feat(stalwart): spread each node group over nodes and give each one a PDB

A two-replica node group was redundancy on paper: nothing kept both pods off
one node, and no PodDisruptionBudget kept a drain from taking both at once.
Both objects are now rendered per node group, gated on that group's COMPUTED
replica count being >= 2.

The distribution is a promise, not an effort: maxSkew 1 on
kubernetes.io/hostname with whenUnsatisfiable DoNotSchedule, selecting only
this group's own pods, so an "mx" and a "full" pod may share a node. The
selector comes from the chart (stalwart.groupLabels), never from the operator -
a spread constraint whose selector matches no pod is trivially satisfied,
distributes nothing and reports nothing.

nodeAffinityPolicy: Ignore is set explicitly, and that is the load-bearing
detail. Its API default is Honor, under which a narrowing nodeSelector shrinks
the set of counted domains until both replicas fit into one: skew 0, both pods
co-located, no Pending, no event, and a selector that is correct - so no check
can see it. With Ignore the same narrowing gives skew 2 and the second pod is
Pending. A render abort was the alternative and would also have forbidden a
nodeSelector spanning five nodes, which is legal.

A user entry colliding with the chart's own on (topologyKey,
whenUnsatisfiable) aborts the render naming both values, because the API
rejects the duplicate with a message about a list index and an apply-based path
merges the two over patchMergeKey: topologyKey silently, dropping one selector.

Minor rather than major, proved with a diff rather than argued: with the
shipped defaults (full: 1, mx: 0) and with ci/stalwart/test-values.yaml,
helm template renders byte-for-byte what 1.0.1+upv0.16.24 rendered - both
diffs empty.

New in CI, over every chart on every PR: ci/run-selector-match-test.sh holds
every rendered PodDisruptionBudget selector and every topologySpreadConstraints
labelSelector against the labels of the pod templates in the same render. Helm
itself is the YAML parser (ci/yaml2json), because the runners have neither
pyyaml nor yq and a hand-written parser could read nothing and report nothing.
podAffinity/podAntiAffinity are out of scope; matchExpressions-only selectors
are reported as not checked rather than counted as passing. Two anti-vacuum
guards: a chart with an overlay under ci/selector-match/ that renders no
selector fails, and a run that checks no selector at all fails.

Fixes #96

stalwart/templates/NOTES.txt

@@ -217,6 +217,82 @@ WHAT A GREEN POD PROVES HERE, AND WHAT IT DOES NOT
   the node IP, and so every pod on a node shares one bucket. The defaults cost
   6 requests/min/pod and leave room for about 16 pods per node. Tightening them
   produces 429s, and a 429 on liveness is a restart loop.
+{{- $anySpread := false }}
+{{- range $group, $spec := $nodes }}
+{{- $r := int (include "stalwart.replicas" (dict "scaleDown" $.Values.scaleDown "replicas" $spec.replicas)) }}
+{{- $s := fromYaml (include "stalwart.groupSettings" (dict "group" $group "given" $spec)) }}
+{{- if and $s.spreadAcrossNodes (ge $r 2) }}{{ $anySpread = true }}{{ end }}
+{{- end }}
+
+DISTRIBUTION AND DISRUPTION BUDGET — WHAT EACH GROUP ACTUALLY GOT
+  Both objects are conditional, so this is the list rather than something to
+  infer from `kubectl get pdb` and an `-o yaml`.
+{{- range $group, $spec := $nodes }}
+{{- $r := int (include "stalwart.replicas" (dict "scaleDown" $.Values.scaleDown "replicas" $spec.replicas)) }}
+{{- $s := fromYaml (include "stalwart.groupSettings" (dict "group" $group "given" $spec)) }}
+
+  {{ $group }} — {{ $r }} effective replica(s)
+{{- if and $s.spreadAcrossNodes (ge $r 2) }}
+    SPREAD   ENFORCED. maxSkew 1 on kubernetes.io/hostname, whenUnsatisfiable
+             DoNotSchedule, nodeAffinityPolicy Ignore, selecting this group's
+             own pods only — a pod of "{{ $group }}" sharing a node with a pod of
+             another group stays fine. This is a PROMISE, so the price is a Pending pod
+             rather than a co-located one: if fewer nodes are available than
+             this group has replicas, the surplus replica does not schedule and
+             there is NO scheduling event to read. Check with
+               kubectl -n {{ $.Release.Namespace }} get pod -l stalwart.jk-dev.de/node-group={{ $group }} -o wide
+{{- else if $s.spreadAcrossNodes }}
+    SPREAD   NOT RENDERED, because this group has {{ $r }} effective replica(s) and the
+             rule needs two. {{ if eq $r 0 }}With no pod at all there is nothing to
+             distribute{{ else }}With one pod every placement satisfies it, so the object
+             would describe nothing{{ end }}. Raise nodes.{{ $group }}.replicas to 2 to get it —
+             and note that doing so changes the POD TEMPLATE, so {{ if eq $r 0 }}the group
+             starts with the constraint already in place{{ else }}the pod running now is
+             replaced in the same rollout round, not merely joined{{ end }}.
+{{- else }}
+    SPREAD   OFF — nodes.{{ $group }}.spreadAcrossNodes is false. Nothing keeps two
+             pods of this group off the same node, so its replicas are
+             redundancy on paper only unless your own
+             nodes.{{ $group }}.topologySpreadConstraints or affinity does the job.
+{{- end }}
+{{- if and $s.podDisruptionBudget.enabled (ge $r 2) }}
+    PDB      {{ $release }}-{{ $group }}-pdb, minAvailable {{ $s.podDisruptionBudget.minAvailable | int64 }}. A node drain cannot take
+             all {{ $r }} replicas of this group at once.
+{{- else if $s.podDisruptionBudget.enabled }}
+    PDB      NONE, because the effective replica count is {{ $r }} and a budget needs
+             two. Deliberate: {{ if eq $r 0 }}with no pod at all the object would only
+             describe a promise about nothing{{ else }}minAvailable 1 with a single replica
+             forbids every voluntary eviction, so `kubectl drain` would block
+             forever on a pod nothing may move — and the error would name the
+             PDB rather than the replica count{{ end }}.
+{{- else }}
+    PDB      NONE — nodes.{{ $group }}.podDisruptionBudget.enabled is false. A node
+             drain may take every replica of this group at once.
+{{- end }}
+{{- end }}
+{{- if $anySpread }}
+
+  ONE VERSION CAVEAT, AND IT FAILS SILENTLY: nodeAffinityPolicy is a beta field
+  — on by default from Kubernetes 1.26, GA with the feature gate
+  NodeInclusionPolicyInPodTopologySpread in 1.33. An older API server, or one
+  with the gate off, PRUNES IT WITHOUT A WORD, and then a nodeSelector or
+  affinity that narrows this group onto a single node no longer produces a
+  Pending pod but two CO-LOCATED ones, with the constraint reported as
+  satisfied. Compare the live object against what was rendered:
+    kubectl -n {{ .Release.Namespace }} get sts -l stalwart.jk-dev.de/release={{ $release }} \
+      -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.template.spec.topologySpreadConstraints[*].nodeAffinityPolicy}{"\n"}{end}'
+  An empty column where "Ignore" should be is the whole symptom.
+
+  AND A CEILING THAT IS A RULE, NOT A NUMBER: this chart leaves
+  nodeTaintsPolicy at its API default of Ignore, so every node in the cluster
+  counts as a domain — control-plane nodes included, where no Stalwart pod ever
+  runs. They hold the minimum at 0 permanently, so with maxSkew 1 a group can
+  never place more replicas than there are SCHEDULABLE WORKERS. It moves with
+  the cluster: add a worker and it rises, remove one and it falls. Past that
+  point a group needs spreadAcrossNodes: false or an own constraint with
+  ScheduleAnyway. This ceiling comes from nodeTaintsPolicy and existed before
+  nodeAffinityPolicy was set — the two fields are not a package.
+{{- end }}
 {{- if .Values.ingress.enabled }}
 
 WEB UI AND JMAP

stalwart/templates/_helpers.tpl

@@ -354,6 +354,131 @@ accept SMTP behind the SAME Service object.
 {{- printf "stalwart.jk-dev.de/port-%s" . -}}
 {{- end -}}
 
+{{/*
+The labels that identify ONE node group's pods - and the ONLY definition of
+them. Input: dict "root" <the root context> "group" <name>. Output: YAML.
+
+Three places consume this: the pod template's labels, the labelSelector of the
+chart's own topology spread constraint, and the selector of the group's
+PodDisruptionBudget. All three have to agree, and the failure of a disagreement
+is silent in both of the latter two:
+
+  - a spread constraint whose labelSelector matches no pod is TRIVIALLY
+    SATISFIED. It distributes nothing, the rollout is green, and nothing is
+    logged. With whenUnsatisfiable: DoNotSchedule that is worse rather than
+    better, because the object then looks like a guarantee.
+  - a PDB whose selector matches no pod reports ALLOWED DISRUPTIONS equal to
+    its own minAvailable arithmetic over zero pods and never blocks an
+    eviction, so a drain takes every replica at once - the thing the object was
+    added to prevent.
+
+One definition is the precedent set by stalwart.env.dbPassword in this file,
+for exactly the same reason: a mismatch between two copies is not a render
+error. ci/run-selector-match-test.sh is the second half of that - it holds
+every rendered selector in this repository against the labels of the pod
+templates in the same render, because one definition only helps while nobody
+writes a second one.
+*/}}
+{{- define "stalwart.groupLabels" -}}
+stalwart.jk-dev.de/release: {{ include "stalwart.fullname" .root }}
+stalwart.jk-dev.de/node-group: {{ .group }}
+{{- end -}}
+
+{{/*
+The effective scheduling and disruption settings of one node group (#99 shape).
+Input: dict "group" <name> "given" <the group's spec from stalwart.nodes>.
+
+Only the two switches the chart interprets carry defaults. affinity,
+nodeSelector, tolerations and topologySpreadConstraints are pass-through: the
+chart never reads their contents, Kubernetes owns their schema, and a default
+would be a value this chart cannot reason about. They therefore arrive here
+unchanged, including absent.
+*/}}
+{{- define "stalwart.groupSettings" -}}
+{{- $defaults := dict
+      "spreadAcrossNodes" true
+      "podDisruptionBudget" (dict "enabled" true "minAvailable" 1) -}}
+{{- include "stalwart.defaultedDict" (dict "defaults" $defaults "given" .given "path" (printf "nodes.%s" .group)) -}}
+{{- end -}}
+
+{{/*
+The topologySpreadConstraints of one node group: the chart's own entry followed
+by the user's, or nothing at all.
+Input: dict "root" <root context> "group" <name> "settings" <stalwart.groupSettings>
+            "replicas" <the COMPUTED replica count, as an int>.
+
+WHY THE CHART-OWNED ENTRY NEEDS >= 2 REPLICAS. With one pod the constraint is
+satisfied by any placement, so it describes nothing; and the group that ships
+with replicas: 0 ("mx") would carry an object about pods that do not exist. The
+condition is on the COMPUTED count, so scaleDown: true removes it as well. The
+side effect is the one that made this a minor rather than a major change: a
+default install (full: 1, mx: 0) renders byte-for-byte what it rendered before.
+
+WHY nodeAffinityPolicy: Ignore IS SET EXPLICITLY, AND IT IS NOT A STYLE CHOICE.
+The two node-inclusion policies have OPPOSITE defaults, straight from the API
+definition: nodeAffinityPolicy defaults to Honor, nodeTaintsPolicy to Ignore.
+Under Honor, only nodes matching the pod's nodeAffinity/nodeSelector count as
+domains - so a nodeSelector narrowing a 2-replica group onto ONE node leaves one
+eligible domain, both pods land in it, the skew is 0, and the constraint is
+satisfied. Both replicas are co-located, nothing is Pending, no event is
+emitted, and the selector in the object is correct - so the CI selector check
+cannot see it either. With Ignore every node counts, the same narrowing gives
+2,0,0, the skew is 2, and the second pod is Pending: the silent half becomes the
+loud one. A render abort was the alternative and was rejected, because it would
+also have forbidden a nodeSelector spanning five nodes, which is legal and
+works.
+
+Note what this field does NOT add: the ceiling on replicas per group comes from
+nodeTaintsPolicy being Ignore, and it existed before this value did. values.yaml
+argues the two separately for that reason.
+
+WHY A COLLISION ABORTS. Kubernetes keys a spread constraint on
+(topologyKey, whenUnsatisfiable), whose API default is DoNotSchedule - so a
+user entry on kubernetes.io/hostname without an explicit whenUnsatisfiable
+collides with the chart's own. Both ways that collision resolves are worse than
+an abort: `kubectl create` fails validation with a message naming a LIST INDEX
+instead of a value, and an apply-based path merges the two entries over
+patchMergeKey: topologyKey - SILENTLY, dropping one labelSelector and leaving a
+green result. The abort names both values instead, and it only fires while the
+chart's own entry is actually rendered, since without it there is no duplicate.
+*/}}
+{{- define "stalwart.spreadConstraints" -}}
+{{- $root := .root -}}
+{{- $group := .group -}}
+{{- $settings := .settings -}}
+{{- $own := list -}}
+{{- if and $settings.spreadAcrossNodes (ge (int .replicas) 2) -}}
+{{- $own = list (dict
+      "maxSkew" 1
+      "topologyKey" "kubernetes.io/hostname"
+      "whenUnsatisfiable" "DoNotSchedule"
+      "nodeAffinityPolicy" "Ignore"
+      "labelSelector" (dict "matchLabels" (fromYaml (include "stalwart.groupLabels" (dict "root" $root "group" $group))))) -}}
+{{- end -}}
+{{- $given := $settings.topologySpreadConstraints -}}
+{{- if kindIs "invalid" $given -}}
+{{- $given = list -}}
+{{- end -}}
+{{- if not (kindIs "slice" $given) -}}
+{{- fail (printf "nodes.%s.topologySpreadConstraints must be a list of topology spread constraints or null, got %s (%v)" $group (kindOf $given) $given) -}}
+{{- end -}}
+{{- if $own -}}
+{{- range $i, $constraint := $given -}}
+{{- if not (kindIs "map" $constraint) -}}
+{{- fail (printf "nodes.%s.topologySpreadConstraints[%d] must be a mapping, got %s (%v)" $group $i (kindOf $constraint) $constraint) -}}
+{{- end -}}
+{{- $when := $constraint.whenUnsatisfiable | default "DoNotSchedule" -}}
+{{- if and (eq ($constraint.topologyKey | default "") "kubernetes.io/hostname") (eq $when "DoNotSchedule") -}}
+{{- fail (printf "nodes.%s.topologySpreadConstraints[%d] collides with the constraint nodes.%s.spreadAcrossNodes renders: both are keyed on topologyKey kubernetes.io/hostname with whenUnsatisfiable DoNotSchedule (the API default, so an omitted whenUnsatisfiable counts), and Kubernetes keys a spread constraint on exactly that pair. Two entries sharing it are rejected by `kubectl create` with a message naming a list index rather than a value, and on an apply-based path they are MERGED over patchMergeKey: topologyKey - silently, so one labelSelector is dropped and the result is green. Pick one: set nodes.%s.spreadAcrossNodes: false and keep your entry (it then owns the per-node distribution of this group, and the chart's nodeAffinityPolicy: Ignore is yours to repeat), or drop your entry and keep the chart's. Entries on any OTHER topologyKey - topology.kubernetes.io/zone, for instance - are appended and really do apply as an AND" $group $i $group $group) -}}
+{{- end -}}
+{{- end -}}
+{{- end -}}
+{{- $all := concat $own $given -}}
+{{- if $all -}}
+{{- toYaml $all -}}
+{{- end -}}
+{{- end -}}
+
 {{/*
 The address of the in-memory store, and the render abort that goes with it.
 Input: the root context. Returns the host, or "" when none is needed.

stalwart/templates/stalwart.pdb.yaml

@@ -0,0 +1,62 @@
+{{- /*
+ONE PodDisruptionBudget PER NODE GROUP, and that is the whole difference from
+the precedent. keycloak.pdb.yaml (Gitea #83, README guideline 11) set the
+convention this file follows; what it did not have to deal with is a chart whose
+replica count is a property of a GROUP rather than of the release. So the
+condition is evaluated per group, and with full: 2 / mx: 1 exactly one object is
+rendered.
+
+WHY IT EXISTS: with two replicas in a group a node drain is otherwise free to
+take both at once - and a drain is the most common reason a pod moves, because
+it is what a node upgrade does. Without this object the second replica buys
+nothing against the failure it was added for. The chart's topology spread
+constraint and this budget are two halves of one answer: the constraint decides
+that the two pods sit on different nodes, the budget decides that only one of
+those nodes may be emptied at a time.
+
+minAvailable rather than maxUnavailable, as in keycloak: the promise wanted is
+about the service - at least one node of this group keeps answering throughout
+the drain - and not about the size of the disruption.
+
+AND WHY IT IS NOT RENDERED BELOW TWO EFFECTIVE REPLICAS. This is the trap that
+makes a PDB worth a condition rather than an unconditional object, quoted from
+the precedent because it has not changed: minAvailable: 1 with exactly ONE
+replica forbids every voluntary eviction, so `kubectl drain` blocks forever on a
+pod nothing is allowed to move - and the message then names the PDB and not the
+replica count, which sends the reader to the wrong file. The same holds for
+scaleDown: true, where the group has no pods at all and the object would only
+describe a promise about nothing.
+
+The count is therefore the COMPUTED one (stalwart.replicas), not the raw
+nodes.<group>.replicas, which is what makes scaleDown: true remove the budget
+rather than leave it behind. NOTES.txt prints, per group, which of the cases
+applied, so an operator does not have to infer it from `kubectl get pdb`.
+
+THE SELECTOR COMES FROM stalwart.groupLabels, the same definition the pod
+template's labels come from. A PDB whose selector matches no pod does not fail:
+it reports its arithmetic over zero pods, never blocks an eviction, and a drain
+takes the whole group while the object sits there looking like protection.
+ci/run-selector-match-test.sh holds this selector against the rendered pod
+templates on every pull request for exactly that reason.
+*/ -}}
+{{- $root := . -}}
+{{- range $group, $spec := fromYaml (include "stalwart.nodes" .) }}
+{{- $name := printf "%s-%s" (include "stalwart.fullname" $root) $group }}
+{{- $replicas := int (include "stalwart.replicas" (dict "scaleDown" $root.Values.scaleDown "replicas" $spec.replicas)) }}
+{{- $settings := fromYaml (include "stalwart.groupSettings" (dict "group" $group "given" $spec)) }}
+{{- if and $settings.podDisruptionBudget.enabled (ge $replicas 2) }}
+---
+apiVersion: policy/v1
+kind: PodDisruptionBudget
+metadata:
+  name: {{ $name }}-pdb
+  labels:
+    app: {{ $name }}-sfs
+    {{- include "stalwart.groupLabels" (dict "root" $root "group" $group) | nindent 4 }}
+spec:
+  minAvailable: {{ $settings.podDisruptionBudget.minAvailable | int64 }}
+  selector:
+    matchLabels:
+      {{- include "stalwart.groupLabels" (dict "root" $root "group" $group) | nindent 6 }}
+{{- end }}
+{{- end }}

stalwart/templates/stalwart.sfs.yaml

@@ -23,6 +23,8 @@ are emptyDirs. That is also why there is no fsGroup in the pod context.
 {{- $valkeyHost := include "stalwart.valkeyHost" . -}}
 {{- range $group, $spec := fromYaml (include "stalwart.nodes" .) }}
 {{- $name := printf "%s-%s" (include "stalwart.fullname" $root) $group }}
+{{- $replicas := int (include "stalwart.replicas" (dict "scaleDown" $root.Values.scaleDown "replicas" $spec.replicas)) }}
+{{- $settings := fromYaml (include "stalwart.groupSettings" (dict "group" $group "given" $spec)) }}
 ---
 {{ if $valkeyHost -}}
 # In-memory store this release expects: {{ $valkeyHost }}:{{ $root.Values.externalServices.valkey.port }}
@@ -36,10 +38,9 @@ metadata:
   name: {{ $name }}-sfs
   labels:
     app: {{ $name }}-sfs
-    stalwart.jk-dev.de/release: {{ include "stalwart.fullname" $root }}
-    stalwart.jk-dev.de/node-group: {{ $group }}
+    {{- include "stalwart.groupLabels" (dict "root" $root "group" $group) | nindent 4 }}
 spec:
-  replicas: {{ include "stalwart.replicas" (dict "scaleDown" $root.Values.scaleDown "replicas" $spec.replicas) }}
+  replicas: {{ $replicas }}
   serviceName: {{ $name }}-headless
   selector:
     matchLabels:
@@ -51,8 +52,15 @@ spec:
         app: {{ $name }}-sfs
         # Scopes every per-port Service to THIS release, so a second stalwart
         # release in the same namespace does not get its pods selected.
-        stalwart.jk-dev.de/release: {{ include "stalwart.fullname" $root }}
-        stalwart.jk-dev.de/node-group: {{ $group }}
+        {{- /* These same two labels are also what the group's
+               PodDisruptionBudget and the chart's topology spread constraint
+               select on, which is why they come from ONE definition: a
+               selector matching no pod fails silently in both of those objects
+               - see the helper's comment. Kept as a rendered YAML comment above
+               rather than folded in here so that the manifest this chart
+               produced before Gitea #96 and the one it produces now are
+               byte-identical for the default values. */}}
+        {{- include "stalwart.groupLabels" (dict "root" $root "group" $group) | nindent 8 }}
         {{- /* One label per port this group may be reached on. The per-port
                Services select on these: NodePorts are cluster-globally
                unique, so "full" and "mx" can only accept SMTP behind the SAME
@@ -81,6 +89,36 @@ spec:
       automountServiceAccountToken: false
       securityContext:
         {{- include "stalwart.podSecurityContext" $root.Values.securityContext | nindent 8 }}
+      {{- /* SCHEDULING. The first three are pass-through: the chart never reads
+             their contents, Kubernetes owns their schema, and an empty block
+             renders nothing at all rather than an empty field.
+
+             MIND THE COMBINATION WITH THE SPREAD CONSTRAINT BELOW. A
+             nodeSelector or an affinity that narrows the eligible nodes below
+             the group's replica count turns the constraint into a permanent
+             Pending - which is the INTENDED, loud outcome (see values.yaml),
+             but it reads like a different problem if you have not been told. */}}
+      {{- with $settings.affinity }}
+      affinity:
+        {{- toYaml . | nindent 8 }}
+      {{- end }}
+      {{- with $settings.nodeSelector }}
+      nodeSelector:
+        {{- toYaml . | nindent 8 }}
+      {{- end }}
+      {{- with $settings.tolerations }}
+      tolerations:
+        {{- toYaml . | nindent 8 }}
+      {{- end }}
+      {{- /* The chart's own per-node constraint plus any the user appended, or
+             nothing while the group is below two effective replicas. The helper
+             aborts the render on a collision instead of letting two entries
+             with the same (topologyKey, whenUnsatisfiable) be merged silently. */}}
+      {{- $spread := include "stalwart.spreadConstraints" (dict "root" $root "group" $group "settings" $settings "replicas" $replicas) }}
+      {{- if $spread }}
+      topologySpreadConstraints:
+        {{- $spread | nindent 8 }}
+      {{- end }}
       initContainers:
         # config.json is copied out of the ConfigMap into the emptyDir rather
         # than mounted from it, because STALWART WRITES THIS FILE ITSELF: on