jklein
6ade74db80
A two-replica node group was redundancy on paper: nothing kept both pods off one node, and no PodDisruptionBudget kept a drain from taking both at once. Both objects are now rendered per node group, gated on that group's COMPUTED replica count being >= 2. The distribution is a promise, not an effort: maxSkew 1 on kubernetes.io/hostname with whenUnsatisfiable DoNotSchedule, selecting only this group's own pods, so an "mx" and a "full" pod may share a node. The selector comes from the chart (stalwart.groupLabels), never from the operator - a spread constraint whose selector matches no pod is trivially satisfied, distributes nothing and reports nothing. nodeAffinityPolicy: Ignore is set explicitly, and that is the load-bearing detail. Its API default is Honor, under which a narrowing nodeSelector shrinks the set of counted domains until both replicas fit into one: skew 0, both pods co-located, no Pending, no event, and a selector that is correct - so no check can see it. With Ignore the same narrowing gives skew 2 and the second pod is Pending. A render abort was the alternative and would also have forbidden a nodeSelector spanning five nodes, which is legal. A user entry colliding with the chart's own on (topologyKey, whenUnsatisfiable) aborts the render naming both values, because the API rejects the duplicate with a message about a list index and an apply-based path merges the two over patchMergeKey: topologyKey silently, dropping one selector. Minor rather than major, proved with a diff rather than argued: with the shipped defaults (full: 1, mx: 0) and with ci/stalwart/test-values.yaml, helm template renders byte-for-byte what 1.0.1+upv0.16.24 rendered - both diffs empty. New in CI, over every chart on every PR: ci/run-selector-match-test.sh holds every rendered PodDisruptionBudget selector and every topologySpreadConstraints labelSelector against the labels of the pod templates in the same render. Helm itself is the YAML parser (ci/yaml2json), because the runners have neither pyyaml nor yq and a hand-written parser could read nothing and report nothing. podAffinity/podAntiAffinity are out of scope; matchExpressions-only selectors are reported as not checked rather than counted as passing. Two anti-vacuum guards: a chart with an overlay under ci/selector-match/ that renders no selector fails, and a run that checks no selector at all fails. Fixes #96
305 lines
15 KiB
YAML
305 lines
15 KiB
YAML
{{- /*
|
|
One StatefulSet per node group. The group key is the cluster-role name and is
|
|
passed in STALWART_ROLE.
|
|
|
|
A StatefulSet and not a Deployment, although this chart mounts no PVC: the
|
|
node-id lease Stalwart acquires for clustering is keyed on the hostname, and
|
|
only a StatefulSet gives a pod a stable one. STALWART_HOSTNAME is fed from the
|
|
pod name for exactly that reason.
|
|
|
|
NO volumeClaimTemplates. All state is external - messages and metadata in
|
|
Postgres, blobs in S3, the in-memory store in Valkey - so both image volumes
|
|
are emptyDirs. That is also why there is no fsGroup in the pod context.
|
|
*/ -}}
|
|
{{- $root := . -}}
|
|
{{- $ports := fromYaml (include "stalwart.ports" .) -}}
|
|
{{- /* Evaluated here, in a template that is ALWAYS rendered, because this is
|
|
where the "cluster without an in-memory store" render abort lives. A
|
|
guard that only ran from NOTES.txt would never fire under helm template
|
|
or helm lint - NOTES.txt is not rendered by either - and would therefore
|
|
pass every check in CI while being broken (guideline 12.2). The value is
|
|
emitted as a comment below, which is also the address an operator has to
|
|
enter in the WebUI. */ -}}
|
|
{{- $valkeyHost := include "stalwart.valkeyHost" . -}}
|
|
{{- range $group, $spec := fromYaml (include "stalwart.nodes" .) }}
|
|
{{- $name := printf "%s-%s" (include "stalwart.fullname" $root) $group }}
|
|
{{- $replicas := int (include "stalwart.replicas" (dict "scaleDown" $root.Values.scaleDown "replicas" $spec.replicas)) }}
|
|
{{- $settings := fromYaml (include "stalwart.groupSettings" (dict "group" $group "given" $spec)) }}
|
|
---
|
|
{{ if $valkeyHost -}}
|
|
# In-memory store this release expects: {{ $valkeyHost }}:{{ $root.Values.externalServices.valkey.port }}
|
|
# NOT configured by this chart - it is an InMemoryStore object in the database,
|
|
# and this is the address to enter in the WebUI. The cluster coordinator's
|
|
# Default variant reuses the same connection.
|
|
{{ end -}}
|
|
apiVersion: apps/v1
|
|
kind: StatefulSet
|
|
metadata:
|
|
name: {{ $name }}-sfs
|
|
labels:
|
|
app: {{ $name }}-sfs
|
|
{{- include "stalwart.groupLabels" (dict "root" $root "group" $group) | nindent 4 }}
|
|
spec:
|
|
replicas: {{ $replicas }}
|
|
serviceName: {{ $name }}-headless
|
|
selector:
|
|
matchLabels:
|
|
app: {{ $name }}-sfs
|
|
template:
|
|
metadata:
|
|
name: {{ $name }}-sfs
|
|
labels:
|
|
app: {{ $name }}-sfs
|
|
# Scopes every per-port Service to THIS release, so a second stalwart
|
|
# release in the same namespace does not get its pods selected.
|
|
{{- /* These same two labels are also what the group's
|
|
PodDisruptionBudget and the chart's topology spread constraint
|
|
select on, which is why they come from ONE definition: a
|
|
selector matching no pod fails silently in both of those objects
|
|
- see the helper's comment. Kept as a rendered YAML comment above
|
|
rather than folded in here so that the manifest this chart
|
|
produced before Gitea #96 and the one it produces now are
|
|
byte-identical for the default values. */}}
|
|
{{- include "stalwart.groupLabels" (dict "root" $root "group" $group) | nindent 8 }}
|
|
{{- /* One label per port this group may be reached on. The per-port
|
|
Services select on these: NodePorts are cluster-globally
|
|
unique, so "full" and "mx" can only accept SMTP behind the SAME
|
|
Service object, and a label is what lets one selector cover
|
|
both groups while still excluding a group that does not list
|
|
the port. */}}
|
|
{{- range $spec.exposedPorts }}
|
|
{{ include "stalwart.portLabel" . }}: "true"
|
|
{{- end }}
|
|
annotations:
|
|
# Hash of the rendered ConfigMap: a changed config.json changes the pod
|
|
# template and therefore restarts the pods. Without it the change would
|
|
# be invisible to the running container - the file is mounted by
|
|
# subPath, which the kubelet never refreshes, and Stalwart reads it only
|
|
# at start.
|
|
checksum/config: {{ include (print $root.Template.BasePath "/stalwart.configmap.yaml") $root | sha256sum }}
|
|
# Hash of the rendered OnePasswordItems - the op:// PATHS, not their
|
|
# contents. Repointing secrets.* at a different vault item therefore
|
|
# takes effect instead of leaving the old credentials in the running
|
|
# pod. Named separately from checksum/config on purpose: that one
|
|
# promises "the config changed", this one only "the config SOURCE
|
|
# changed". A change inside an item is not covered and cannot be -
|
|
# Helm never sees a secret's content.
|
|
checksum/secret-ref: {{ include (print $root.Template.BasePath "/stalwart.secret.yaml") $root | sha256sum }}
|
|
spec:
|
|
automountServiceAccountToken: false
|
|
securityContext:
|
|
{{- include "stalwart.podSecurityContext" $root.Values.securityContext | nindent 8 }}
|
|
{{- /* SCHEDULING. The first three are pass-through: the chart never reads
|
|
their contents, Kubernetes owns their schema, and an empty block
|
|
renders nothing at all rather than an empty field.
|
|
|
|
MIND THE COMBINATION WITH THE SPREAD CONSTRAINT BELOW. A
|
|
nodeSelector or an affinity that narrows the eligible nodes below
|
|
the group's replica count turns the constraint into a permanent
|
|
Pending - which is the INTENDED, loud outcome (see values.yaml),
|
|
but it reads like a different problem if you have not been told. */}}
|
|
{{- with $settings.affinity }}
|
|
affinity:
|
|
{{- toYaml . | nindent 8 }}
|
|
{{- end }}
|
|
{{- with $settings.nodeSelector }}
|
|
nodeSelector:
|
|
{{- toYaml . | nindent 8 }}
|
|
{{- end }}
|
|
{{- with $settings.tolerations }}
|
|
tolerations:
|
|
{{- toYaml . | nindent 8 }}
|
|
{{- end }}
|
|
{{- /* The chart's own per-node constraint plus any the user appended, or
|
|
nothing while the group is below two effective replicas. The helper
|
|
aborts the render on a collision instead of letting two entries
|
|
with the same (topologyKey, whenUnsatisfiable) be merged silently. */}}
|
|
{{- $spread := include "stalwart.spreadConstraints" (dict "root" $root "group" $group "settings" $settings "replicas" $replicas) }}
|
|
{{- if $spread }}
|
|
topologySpreadConstraints:
|
|
{{- $spread | nindent 8 }}
|
|
{{- end }}
|
|
initContainers:
|
|
# config.json is copied out of the ConfigMap into the emptyDir rather
|
|
# than mounted from it, because STALWART WRITES THIS FILE ITSELF: on
|
|
# completing its setup wizard it "writes config.json ... and restarts
|
|
# into normal operation", and a ConfigMap mount is read-only. Copying
|
|
# keeps /etc/stalwart writable AND keeps checksum/config honest, which
|
|
# a bare emptyDir would not.
|
|
#
|
|
# The copy direction matters: the chart's version wins on every pod
|
|
# start. If an operator has let the server rewrite the file, that
|
|
# rewrite is discarded on the next restart - which is the intended
|
|
# behaviour here (the chart owns the data store setting) but is worth
|
|
# knowing before editing it through the WebUI.
|
|
- name: copy-config
|
|
image: "{{ $root.Values.initImage.repository }}:{{ $root.Values.initImage.tag }}"
|
|
imagePullPolicy: IfNotPresent
|
|
command:
|
|
- /bin/sh
|
|
- -c
|
|
- cp /chart-config/config.json /etc/stalwart/config.json
|
|
volumeMounts:
|
|
- name: config
|
|
mountPath: /chart-config
|
|
readOnly: true
|
|
- name: etc-stalwart
|
|
mountPath: /etc/stalwart
|
|
resources:
|
|
{{- include "stalwart.resources" (dict "limitFactor" 3 "requests" (dict "cpu" "10m" "memory" "16Mi" "ephemeral-storage" "8Mi")) | nindent 12 }}
|
|
securityContext:
|
|
{{- include "stalwart.containerSecurityContext" $root.Values.securityContext | nindent 12 }}
|
|
containers:
|
|
- name: {{ $name }}-sfs
|
|
image: "docker.io/stalwartlabs/stalwart:{{ $root.Chart.AppVersion }}"
|
|
imagePullPolicy: IfNotPresent
|
|
env:
|
|
# The cluster role. A ClusterRole object of exactly this name has
|
|
# to exist in the database - if it does not, the node silently
|
|
# becomes a FULL node (every listener AND every task) while staying
|
|
# Running with a green readiness probe. See values.yaml.
|
|
- name: STALWART_ROLE
|
|
value: {{ $group | quote }}
|
|
# The node identity for the cluster node-id lease. The pod name is
|
|
# stable because this is a StatefulSet; the OS hostname inside the
|
|
# pod would be too, but reading it from the API is what makes the
|
|
# value survive a hostname the image cannot change.
|
|
- name: STALWART_HOSTNAME
|
|
valueFrom:
|
|
fieldRef:
|
|
fieldPath: metadata.name
|
|
{{- if $root.Values.bootstrap.enabled }}
|
|
# RECOVERY MODE IS ON: management HTTP on 8080 only, every mail
|
|
# port closed. Required for a first install because this chart
|
|
# always renders config.json and the bootstrap wizard only appears
|
|
# when that file is ABSENT. Turn it off and upgrade again once the
|
|
# database is provisioned.
|
|
- name: STALWART_RECOVERY_MODE
|
|
value: "1"
|
|
- name: STALWART_RECOVERY_MODE_LOG_LEVEL
|
|
value: {{ $root.Values.bootstrap.logLevel | quote }}
|
|
{{- end }}
|
|
{{- if $root.Values.secrets.dataStoreSecretRef }}
|
|
# Referenced BY NAME from config.json as
|
|
# "authSecret": {"@type":"EnvironmentVariable","variableName":"..."}
|
|
# which is what keeps the password out of the ConfigMap. An empty
|
|
# value counts as "not found" upstream and fails the store open.
|
|
- name: {{ include "stalwart.env.dbPassword" $root }}
|
|
valueFrom:
|
|
secretKeyRef:
|
|
name: {{ include "stalwart.fullname" $root }}-datastore-secret
|
|
key: password
|
|
{{- end }}
|
|
{{- if $root.Values.secrets.recoveryAdminSecretRef }}
|
|
- name: STALWART_RECOVERY_ADMIN
|
|
valueFrom:
|
|
secretKeyRef:
|
|
name: {{ include "stalwart.fullname" $root }}-recovery-admin-secret
|
|
key: recovery-admin
|
|
{{- end }}
|
|
{{- if $root.Values.secrets.s3SecretRef }}
|
|
# Not read by the chart and not by config.json: these exist so the
|
|
# BlobStore object in the DATABASE can reference them instead of
|
|
# storing the credentials there.
|
|
- name: {{ include "stalwart.env.s3AccessKey" $root }}
|
|
valueFrom:
|
|
secretKeyRef:
|
|
name: {{ include "stalwart.fullname" $root }}-s3-secret
|
|
key: access-key
|
|
- name: {{ include "stalwart.env.s3SecretKey" $root }}
|
|
valueFrom:
|
|
secretKeyRef:
|
|
name: {{ include "stalwart.fullname" $root }}-s3-secret
|
|
key: secret-key
|
|
{{- end }}
|
|
{{- range $value := $spec.env }}
|
|
{{- if $value.value }}
|
|
- name: {{ $value.name | quote }}
|
|
# quote is required, not cosmetic: a value like "8080" or "true"
|
|
# renders unquoted as a YAML number/bool and the API server
|
|
# rejects the whole manifest.
|
|
value: {{ tpl $value.value $root | quote }}
|
|
{{- else if $value.valueFrom }}
|
|
{{- if $value.valueFrom.secretKeyRef }}
|
|
- name: {{ $value.name }}
|
|
valueFrom:
|
|
secretKeyRef:
|
|
name: {{ tpl $value.valueFrom.secretKeyRef.name $root | quote }}
|
|
key: {{ tpl $value.valueFrom.secretKeyRef.key $root | quote }}
|
|
{{- else if $value.valueFrom.configMapKeyRef }}
|
|
- name: {{ $value.name }}
|
|
valueFrom:
|
|
configMapKeyRef:
|
|
name: {{ tpl $value.valueFrom.configMapKeyRef.name $root | quote }}
|
|
key: {{ tpl $value.valueFrom.configMapKeyRef.key $root | quote }}
|
|
{{- end }}
|
|
{{- end }}
|
|
{{- end }}
|
|
ports:
|
|
{{- range $portName := $spec.exposedPorts }}
|
|
{{- $port := index $ports $portName }}
|
|
- name: {{ $portName }}
|
|
# containerPort, NOT nodePort. This is the number the application
|
|
# binds and the number the NetworkListener object in the database
|
|
# has to carry; the nodePort only appears in the Service.
|
|
containerPort: {{ int $port.containerPort }}
|
|
protocol: {{ $port.protocol | default "TCP" }}
|
|
{{- end }}
|
|
# Probes, resources and both security contexts go through the
|
|
# null-safe helpers (#99): an erased block means "chart defaults
|
|
# apply" instead of a vanished probe or an unhardened container.
|
|
# Never wrap these in an "if".
|
|
livenessProbe:
|
|
{{- include "stalwart.livenessProbe" $root.Values.livenessProbe | nindent 12 }}
|
|
readinessProbe:
|
|
{{- include "stalwart.readinessProbe" $root.Values.readinessProbe | nindent 12 }}
|
|
startupProbe:
|
|
{{- include "stalwart.startupProbe" $root.Values.startupProbe | nindent 12 }}
|
|
resources:
|
|
{{- include "stalwart.effectiveResources" (dict "group" $group "given" $spec.resources "limitFactor" $root.Values.limitFactor) | nindent 12 }}
|
|
securityContext:
|
|
{{- include "stalwart.containerSecurityContext" $root.Values.securityContext | nindent 12 }}
|
|
volumeMounts:
|
|
- name: etc-stalwart
|
|
mountPath: /etc/stalwart
|
|
- name: var-lib-stalwart
|
|
mountPath: /var/lib/stalwart
|
|
- name: tmp
|
|
mountPath: /tmp
|
|
{{- if $root.Values.certificate.enabled }}
|
|
- name: mail-tls
|
|
mountPath: {{ $root.Values.certificate.mountPath }}
|
|
readOnly: true
|
|
{{- end }}
|
|
volumes:
|
|
- name: config
|
|
configMap:
|
|
name: {{ include "stalwart.fullname" $root }}-configmap
|
|
# Writable because Stalwart rewrites config.json itself after its setup
|
|
# wizard; the init container seeds it from the ConfigMap.
|
|
- name: etc-stalwart
|
|
emptyDir: {}
|
|
# The image's data volume. An emptyDir because the data store, the blob
|
|
# store and the in-memory store are all external.
|
|
#
|
|
# OPEN MEASUREMENT: whether anything under this path has to survive a
|
|
# restart once all three stores are external. If it does, this needs a
|
|
# PVC per group. Not verified at runtime.
|
|
- name: var-lib-stalwart
|
|
emptyDir: {}
|
|
- name: tmp
|
|
emptyDir: {}
|
|
{{- if $root.Values.certificate.enabled }}
|
|
- name: mail-tls
|
|
secret:
|
|
secretName: tls-{{ include "stalwart.fullname" $root }}-mail
|
|
# The pods do not block on the certificate: a Secret that
|
|
# cert-manager has not written yet would otherwise hold every pod
|
|
# in ContainerCreating until the challenge completes. Stalwart
|
|
# falls back to a self-signed certificate on the mail ports in the
|
|
# meantime, which is the better of the two failures.
|
|
optional: true
|
|
{{- end }}
|
|
restartPolicy: Always
|
|
{{- end }}
|