feat(stalwart): spread each node group over nodes and give each one a PDB

A two-replica node group was redundancy on paper: nothing kept both pods off
one node, and no PodDisruptionBudget kept a drain from taking both at once.
Both objects are now rendered per node group, gated on that group's COMPUTED
replica count being >= 2.

The distribution is a promise, not an effort: maxSkew 1 on
kubernetes.io/hostname with whenUnsatisfiable DoNotSchedule, selecting only
this group's own pods, so an "mx" and a "full" pod may share a node. The
selector comes from the chart (stalwart.groupLabels), never from the operator -
a spread constraint whose selector matches no pod is trivially satisfied,
distributes nothing and reports nothing.

nodeAffinityPolicy: Ignore is set explicitly, and that is the load-bearing
detail. Its API default is Honor, under which a narrowing nodeSelector shrinks
the set of counted domains until both replicas fit into one: skew 0, both pods
co-located, no Pending, no event, and a selector that is correct - so no check
can see it. With Ignore the same narrowing gives skew 2 and the second pod is
Pending. A render abort was the alternative and would also have forbidden a
nodeSelector spanning five nodes, which is legal.

A user entry colliding with the chart's own on (topologyKey,
whenUnsatisfiable) aborts the render naming both values, because the API
rejects the duplicate with a message about a list index and an apply-based path
merges the two over patchMergeKey: topologyKey silently, dropping one selector.

Minor rather than major, proved with a diff rather than argued: with the
shipped defaults (full: 1, mx: 0) and with ci/stalwart/test-values.yaml,
helm template renders byte-for-byte what 1.0.1+upv0.16.24 rendered - both
diffs empty.

New in CI, over every chart on every PR: ci/run-selector-match-test.sh holds
every rendered PodDisruptionBudget selector and every topologySpreadConstraints
labelSelector against the labels of the pod templates in the same render. Helm
itself is the YAML parser (ci/yaml2json), because the runners have neither
pyyaml nor yq and a hand-written parser could read nothing and report nothing.
podAffinity/podAntiAffinity are out of scope; matchExpressions-only selectors
are reported as not checked rather than counted as passing. Two anti-vacuum
guards: a chart with an overlay under ci/selector-match/ that renders no
selector fails, and a run that checks no selector at all fails.

Fixes #96

stalwart/Chart.yaml

@@ -18,7 +18,14 @@ type: application
 # all three of latest / v0.16 / v0.16.24 resolved to one digest pushed seconds
 # apart, so either floating name would change the running image with no commit
 # in this repository and nothing to report it.
-version: 1.0.1+upv0.16.24
+# MINOR, not major, and the evidence is a diff rather than a judgement: the
+# per-group spread constraint and the per-group PodDisruptionBudget of Gitea #96
+# are both gated on the COMPUTED replica count being >= 2, and the chart ships
+# full: 1 / mx: 0. `helm template` against the previous version therefore renders
+# byte-for-byte identically for the defaults AND for ci/stalwart/test-values.yaml
+# - measured, both empty diffs. New optional values with unchanged defaults is
+# exactly the minor case (README, "Versioning").
+version: 1.1.0+upv0.16.24
 appVersion: "v0.16.24"
 
 maintainers:

stalwart/templates/NOTES.txt

@@ -217,6 +217,82 @@ WHAT A GREEN POD PROVES HERE, AND WHAT IT DOES NOT
   the node IP, and so every pod on a node shares one bucket. The defaults cost
   6 requests/min/pod and leave room for about 16 pods per node. Tightening them
   produces 429s, and a 429 on liveness is a restart loop.
+{{- $anySpread := false }}
+{{- range $group, $spec := $nodes }}
+{{- $r := int (include "stalwart.replicas" (dict "scaleDown" $.Values.scaleDown "replicas" $spec.replicas)) }}
+{{- $s := fromYaml (include "stalwart.groupSettings" (dict "group" $group "given" $spec)) }}
+{{- if and $s.spreadAcrossNodes (ge $r 2) }}{{ $anySpread = true }}{{ end }}
+{{- end }}
+
+DISTRIBUTION AND DISRUPTION BUDGET — WHAT EACH GROUP ACTUALLY GOT
+  Both objects are conditional, so this is the list rather than something to
+  infer from `kubectl get pdb` and an `-o yaml`.
+{{- range $group, $spec := $nodes }}
+{{- $r := int (include "stalwart.replicas" (dict "scaleDown" $.Values.scaleDown "replicas" $spec.replicas)) }}
+{{- $s := fromYaml (include "stalwart.groupSettings" (dict "group" $group "given" $spec)) }}
+
+  {{ $group }} — {{ $r }} effective replica(s)
+{{- if and $s.spreadAcrossNodes (ge $r 2) }}
+    SPREAD   ENFORCED. maxSkew 1 on kubernetes.io/hostname, whenUnsatisfiable
+             DoNotSchedule, nodeAffinityPolicy Ignore, selecting this group's
+             own pods only — a pod of "{{ $group }}" sharing a node with a pod of
+             another group stays fine. This is a PROMISE, so the price is a Pending pod
+             rather than a co-located one: if fewer nodes are available than
+             this group has replicas, the surplus replica does not schedule and
+             there is NO scheduling event to read. Check with
+               kubectl -n {{ $.Release.Namespace }} get pod -l stalwart.jk-dev.de/node-group={{ $group }} -o wide
+{{- else if $s.spreadAcrossNodes }}
+    SPREAD   NOT RENDERED, because this group has {{ $r }} effective replica(s) and the
+             rule needs two. {{ if eq $r 0 }}With no pod at all there is nothing to
+             distribute{{ else }}With one pod every placement satisfies it, so the object
+             would describe nothing{{ end }}. Raise nodes.{{ $group }}.replicas to 2 to get it —
+             and note that doing so changes the POD TEMPLATE, so {{ if eq $r 0 }}the group
+             starts with the constraint already in place{{ else }}the pod running now is
+             replaced in the same rollout round, not merely joined{{ end }}.
+{{- else }}
+    SPREAD   OFF — nodes.{{ $group }}.spreadAcrossNodes is false. Nothing keeps two
+             pods of this group off the same node, so its replicas are
+             redundancy on paper only unless your own
+             nodes.{{ $group }}.topologySpreadConstraints or affinity does the job.
+{{- end }}
+{{- if and $s.podDisruptionBudget.enabled (ge $r 2) }}
+    PDB      {{ $release }}-{{ $group }}-pdb, minAvailable {{ $s.podDisruptionBudget.minAvailable | int64 }}. A node drain cannot take
+             all {{ $r }} replicas of this group at once.
+{{- else if $s.podDisruptionBudget.enabled }}
+    PDB      NONE, because the effective replica count is {{ $r }} and a budget needs
+             two. Deliberate: {{ if eq $r 0 }}with no pod at all the object would only
+             describe a promise about nothing{{ else }}minAvailable 1 with a single replica
+             forbids every voluntary eviction, so `kubectl drain` would block
+             forever on a pod nothing may move — and the error would name the
+             PDB rather than the replica count{{ end }}.
+{{- else }}
+    PDB      NONE — nodes.{{ $group }}.podDisruptionBudget.enabled is false. A node
+             drain may take every replica of this group at once.
+{{- end }}
+{{- end }}
+{{- if $anySpread }}
+
+  ONE VERSION CAVEAT, AND IT FAILS SILENTLY: nodeAffinityPolicy is a beta field
+  — on by default from Kubernetes 1.26, GA with the feature gate
+  NodeInclusionPolicyInPodTopologySpread in 1.33. An older API server, or one
+  with the gate off, PRUNES IT WITHOUT A WORD, and then a nodeSelector or
+  affinity that narrows this group onto a single node no longer produces a
+  Pending pod but two CO-LOCATED ones, with the constraint reported as
+  satisfied. Compare the live object against what was rendered:
+    kubectl -n {{ .Release.Namespace }} get sts -l stalwart.jk-dev.de/release={{ $release }} \
+      -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.template.spec.topologySpreadConstraints[*].nodeAffinityPolicy}{"\n"}{end}'
+  An empty column where "Ignore" should be is the whole symptom.
+
+  AND A CEILING THAT IS A RULE, NOT A NUMBER: this chart leaves
+  nodeTaintsPolicy at its API default of Ignore, so every node in the cluster
+  counts as a domain — control-plane nodes included, where no Stalwart pod ever
+  runs. They hold the minimum at 0 permanently, so with maxSkew 1 a group can
+  never place more replicas than there are SCHEDULABLE WORKERS. It moves with
+  the cluster: add a worker and it rises, remove one and it falls. Past that
+  point a group needs spreadAcrossNodes: false or an own constraint with
+  ScheduleAnyway. This ceiling comes from nodeTaintsPolicy and existed before
+  nodeAffinityPolicy was set — the two fields are not a package.
+{{- end }}
 {{- if .Values.ingress.enabled }}
 
 WEB UI AND JMAP

stalwart/templates/_helpers.tpl

@@ -354,6 +354,131 @@ accept SMTP behind the SAME Service object.
 {{- printf "stalwart.jk-dev.de/port-%s" . -}}
 {{- end -}}
 
+{{/*
+The labels that identify ONE node group's pods - and the ONLY definition of
+them. Input: dict "root" <the root context> "group" <name>. Output: YAML.
+
+Three places consume this: the pod template's labels, the labelSelector of the
+chart's own topology spread constraint, and the selector of the group's
+PodDisruptionBudget. All three have to agree, and the failure of a disagreement
+is silent in both of the latter two:
+
+  - a spread constraint whose labelSelector matches no pod is TRIVIALLY
+    SATISFIED. It distributes nothing, the rollout is green, and nothing is
+    logged. With whenUnsatisfiable: DoNotSchedule that is worse rather than
+    better, because the object then looks like a guarantee.
+  - a PDB whose selector matches no pod reports ALLOWED DISRUPTIONS equal to
+    its own minAvailable arithmetic over zero pods and never blocks an
+    eviction, so a drain takes every replica at once - the thing the object was
+    added to prevent.
+
+One definition is the precedent set by stalwart.env.dbPassword in this file,
+for exactly the same reason: a mismatch between two copies is not a render
+error. ci/run-selector-match-test.sh is the second half of that - it holds
+every rendered selector in this repository against the labels of the pod
+templates in the same render, because one definition only helps while nobody
+writes a second one.
+*/}}
+{{- define "stalwart.groupLabels" -}}
+stalwart.jk-dev.de/release: {{ include "stalwart.fullname" .root }}
+stalwart.jk-dev.de/node-group: {{ .group }}
+{{- end -}}
+
+{{/*
+The effective scheduling and disruption settings of one node group (#99 shape).
+Input: dict "group" <name> "given" <the group's spec from stalwart.nodes>.
+
+Only the two switches the chart interprets carry defaults. affinity,
+nodeSelector, tolerations and topologySpreadConstraints are pass-through: the
+chart never reads their contents, Kubernetes owns their schema, and a default
+would be a value this chart cannot reason about. They therefore arrive here
+unchanged, including absent.
+*/}}
+{{- define "stalwart.groupSettings" -}}
+{{- $defaults := dict
+      "spreadAcrossNodes" true
+      "podDisruptionBudget" (dict "enabled" true "minAvailable" 1) -}}
+{{- include "stalwart.defaultedDict" (dict "defaults" $defaults "given" .given "path" (printf "nodes.%s" .group)) -}}
+{{- end -}}
+
+{{/*
+The topologySpreadConstraints of one node group: the chart's own entry followed
+by the user's, or nothing at all.
+Input: dict "root" <root context> "group" <name> "settings" <stalwart.groupSettings>
+            "replicas" <the COMPUTED replica count, as an int>.
+
+WHY THE CHART-OWNED ENTRY NEEDS >= 2 REPLICAS. With one pod the constraint is
+satisfied by any placement, so it describes nothing; and the group that ships
+with replicas: 0 ("mx") would carry an object about pods that do not exist. The
+condition is on the COMPUTED count, so scaleDown: true removes it as well. The
+side effect is the one that made this a minor rather than a major change: a
+default install (full: 1, mx: 0) renders byte-for-byte what it rendered before.
+
+WHY nodeAffinityPolicy: Ignore IS SET EXPLICITLY, AND IT IS NOT A STYLE CHOICE.
+The two node-inclusion policies have OPPOSITE defaults, straight from the API
+definition: nodeAffinityPolicy defaults to Honor, nodeTaintsPolicy to Ignore.
+Under Honor, only nodes matching the pod's nodeAffinity/nodeSelector count as
+domains - so a nodeSelector narrowing a 2-replica group onto ONE node leaves one
+eligible domain, both pods land in it, the skew is 0, and the constraint is
+satisfied. Both replicas are co-located, nothing is Pending, no event is
+emitted, and the selector in the object is correct - so the CI selector check
+cannot see it either. With Ignore every node counts, the same narrowing gives
+2,0,0, the skew is 2, and the second pod is Pending: the silent half becomes the
+loud one. A render abort was the alternative and was rejected, because it would
+also have forbidden a nodeSelector spanning five nodes, which is legal and
+works.
+
+Note what this field does NOT add: the ceiling on replicas per group comes from
+nodeTaintsPolicy being Ignore, and it existed before this value did. values.yaml
+argues the two separately for that reason.
+
+WHY A COLLISION ABORTS. Kubernetes keys a spread constraint on
+(topologyKey, whenUnsatisfiable), whose API default is DoNotSchedule - so a
+user entry on kubernetes.io/hostname without an explicit whenUnsatisfiable
+collides with the chart's own. Both ways that collision resolves are worse than
+an abort: `kubectl create` fails validation with a message naming a LIST INDEX
+instead of a value, and an apply-based path merges the two entries over
+patchMergeKey: topologyKey - SILENTLY, dropping one labelSelector and leaving a
+green result. The abort names both values instead, and it only fires while the
+chart's own entry is actually rendered, since without it there is no duplicate.
+*/}}
+{{- define "stalwart.spreadConstraints" -}}
+{{- $root := .root -}}
+{{- $group := .group -}}
+{{- $settings := .settings -}}
+{{- $own := list -}}
+{{- if and $settings.spreadAcrossNodes (ge (int .replicas) 2) -}}
+{{- $own = list (dict
+      "maxSkew" 1
+      "topologyKey" "kubernetes.io/hostname"
+      "whenUnsatisfiable" "DoNotSchedule"
+      "nodeAffinityPolicy" "Ignore"
+      "labelSelector" (dict "matchLabels" (fromYaml (include "stalwart.groupLabels" (dict "root" $root "group" $group))))) -}}
+{{- end -}}
+{{- $given := $settings.topologySpreadConstraints -}}
+{{- if kindIs "invalid" $given -}}
+{{- $given = list -}}
+{{- end -}}
+{{- if not (kindIs "slice" $given) -}}
+{{- fail (printf "nodes.%s.topologySpreadConstraints must be a list of topology spread constraints or null, got %s (%v)" $group (kindOf $given) $given) -}}
+{{- end -}}
+{{- if $own -}}
+{{- range $i, $constraint := $given -}}
+{{- if not (kindIs "map" $constraint) -}}
+{{- fail (printf "nodes.%s.topologySpreadConstraints[%d] must be a mapping, got %s (%v)" $group $i (kindOf $constraint) $constraint) -}}
+{{- end -}}
+{{- $when := $constraint.whenUnsatisfiable | default "DoNotSchedule" -}}
+{{- if and (eq ($constraint.topologyKey | default "") "kubernetes.io/hostname") (eq $when "DoNotSchedule") -}}
+{{- fail (printf "nodes.%s.topologySpreadConstraints[%d] collides with the constraint nodes.%s.spreadAcrossNodes renders: both are keyed on topologyKey kubernetes.io/hostname with whenUnsatisfiable DoNotSchedule (the API default, so an omitted whenUnsatisfiable counts), and Kubernetes keys a spread constraint on exactly that pair. Two entries sharing it are rejected by `kubectl create` with a message naming a list index rather than a value, and on an apply-based path they are MERGED over patchMergeKey: topologyKey - silently, so one labelSelector is dropped and the result is green. Pick one: set nodes.%s.spreadAcrossNodes: false and keep your entry (it then owns the per-node distribution of this group, and the chart's nodeAffinityPolicy: Ignore is yours to repeat), or drop your entry and keep the chart's. Entries on any OTHER topologyKey - topology.kubernetes.io/zone, for instance - are appended and really do apply as an AND" $group $i $group $group) -}}
+{{- end -}}
+{{- end -}}
+{{- end -}}
+{{- $all := concat $own $given -}}
+{{- if $all -}}
+{{- toYaml $all -}}
+{{- end -}}
+{{- end -}}
+
 {{/*
 The address of the in-memory store, and the render abort that goes with it.
 Input: the root context. Returns the host, or "" when none is needed.

stalwart/templates/stalwart.pdb.yaml

@@ -0,0 +1,62 @@
+{{- /*
+ONE PodDisruptionBudget PER NODE GROUP, and that is the whole difference from
+the precedent. keycloak.pdb.yaml (Gitea #83, README guideline 11) set the
+convention this file follows; what it did not have to deal with is a chart whose
+replica count is a property of a GROUP rather than of the release. So the
+condition is evaluated per group, and with full: 2 / mx: 1 exactly one object is
+rendered.
+
+WHY IT EXISTS: with two replicas in a group a node drain is otherwise free to
+take both at once - and a drain is the most common reason a pod moves, because
+it is what a node upgrade does. Without this object the second replica buys
+nothing against the failure it was added for. The chart's topology spread
+constraint and this budget are two halves of one answer: the constraint decides
+that the two pods sit on different nodes, the budget decides that only one of
+those nodes may be emptied at a time.
+
+minAvailable rather than maxUnavailable, as in keycloak: the promise wanted is
+about the service - at least one node of this group keeps answering throughout
+the drain - and not about the size of the disruption.
+
+AND WHY IT IS NOT RENDERED BELOW TWO EFFECTIVE REPLICAS. This is the trap that
+makes a PDB worth a condition rather than an unconditional object, quoted from
+the precedent because it has not changed: minAvailable: 1 with exactly ONE
+replica forbids every voluntary eviction, so `kubectl drain` blocks forever on a
+pod nothing is allowed to move - and the message then names the PDB and not the
+replica count, which sends the reader to the wrong file. The same holds for
+scaleDown: true, where the group has no pods at all and the object would only
+describe a promise about nothing.
+
+The count is therefore the COMPUTED one (stalwart.replicas), not the raw
+nodes.<group>.replicas, which is what makes scaleDown: true remove the budget
+rather than leave it behind. NOTES.txt prints, per group, which of the cases
+applied, so an operator does not have to infer it from `kubectl get pdb`.
+
+THE SELECTOR COMES FROM stalwart.groupLabels, the same definition the pod
+template's labels come from. A PDB whose selector matches no pod does not fail:
+it reports its arithmetic over zero pods, never blocks an eviction, and a drain
+takes the whole group while the object sits there looking like protection.
+ci/run-selector-match-test.sh holds this selector against the rendered pod
+templates on every pull request for exactly that reason.
+*/ -}}
+{{- $root := . -}}
+{{- range $group, $spec := fromYaml (include "stalwart.nodes" .) }}
+{{- $name := printf "%s-%s" (include "stalwart.fullname" $root) $group }}
+{{- $replicas := int (include "stalwart.replicas" (dict "scaleDown" $root.Values.scaleDown "replicas" $spec.replicas)) }}
+{{- $settings := fromYaml (include "stalwart.groupSettings" (dict "group" $group "given" $spec)) }}
+{{- if and $settings.podDisruptionBudget.enabled (ge $replicas 2) }}
+---
+apiVersion: policy/v1
+kind: PodDisruptionBudget
+metadata:
+  name: {{ $name }}-pdb
+  labels:
+    app: {{ $name }}-sfs
+    {{- include "stalwart.groupLabels" (dict "root" $root "group" $group) | nindent 4 }}
+spec:
+  minAvailable: {{ $settings.podDisruptionBudget.minAvailable | int64 }}
+  selector:
+    matchLabels:
+      {{- include "stalwart.groupLabels" (dict "root" $root "group" $group) | nindent 6 }}
+{{- end }}
+{{- end }}

stalwart/templates/stalwart.sfs.yaml

@@ -23,6 +23,8 @@ are emptyDirs. That is also why there is no fsGroup in the pod context.
 {{- $valkeyHost := include "stalwart.valkeyHost" . -}}
 {{- range $group, $spec := fromYaml (include "stalwart.nodes" .) }}
 {{- $name := printf "%s-%s" (include "stalwart.fullname" $root) $group }}
+{{- $replicas := int (include "stalwart.replicas" (dict "scaleDown" $root.Values.scaleDown "replicas" $spec.replicas)) }}
+{{- $settings := fromYaml (include "stalwart.groupSettings" (dict "group" $group "given" $spec)) }}
 ---
 {{ if $valkeyHost -}}
 # In-memory store this release expects: {{ $valkeyHost }}:{{ $root.Values.externalServices.valkey.port }}
@@ -36,10 +38,9 @@ metadata:
   name: {{ $name }}-sfs
   labels:
     app: {{ $name }}-sfs
-    stalwart.jk-dev.de/release: {{ include "stalwart.fullname" $root }}
-    stalwart.jk-dev.de/node-group: {{ $group }}
+    {{- include "stalwart.groupLabels" (dict "root" $root "group" $group) | nindent 4 }}
 spec:
-  replicas: {{ include "stalwart.replicas" (dict "scaleDown" $root.Values.scaleDown "replicas" $spec.replicas) }}
+  replicas: {{ $replicas }}
   serviceName: {{ $name }}-headless
   selector:
     matchLabels:
@@ -51,8 +52,15 @@ spec:
         app: {{ $name }}-sfs
         # Scopes every per-port Service to THIS release, so a second stalwart
         # release in the same namespace does not get its pods selected.
-        stalwart.jk-dev.de/release: {{ include "stalwart.fullname" $root }}
-        stalwart.jk-dev.de/node-group: {{ $group }}
+        {{- /* These same two labels are also what the group's
+               PodDisruptionBudget and the chart's topology spread constraint
+               select on, which is why they come from ONE definition: a
+               selector matching no pod fails silently in both of those objects
+               - see the helper's comment. Kept as a rendered YAML comment above
+               rather than folded in here so that the manifest this chart
+               produced before Gitea #96 and the one it produces now are
+               byte-identical for the default values. */}}
+        {{- include "stalwart.groupLabels" (dict "root" $root "group" $group) | nindent 8 }}
         {{- /* One label per port this group may be reached on. The per-port
                Services select on these: NodePorts are cluster-globally
                unique, so "full" and "mx" can only accept SMTP behind the SAME
@@ -81,6 +89,36 @@ spec:
       automountServiceAccountToken: false
       securityContext:
         {{- include "stalwart.podSecurityContext" $root.Values.securityContext | nindent 8 }}
+      {{- /* SCHEDULING. The first three are pass-through: the chart never reads
+             their contents, Kubernetes owns their schema, and an empty block
+             renders nothing at all rather than an empty field.
+
+             MIND THE COMBINATION WITH THE SPREAD CONSTRAINT BELOW. A
+             nodeSelector or an affinity that narrows the eligible nodes below
+             the group's replica count turns the constraint into a permanent
+             Pending - which is the INTENDED, loud outcome (see values.yaml),
+             but it reads like a different problem if you have not been told. */}}
+      {{- with $settings.affinity }}
+      affinity:
+        {{- toYaml . | nindent 8 }}
+      {{- end }}
+      {{- with $settings.nodeSelector }}
+      nodeSelector:
+        {{- toYaml . | nindent 8 }}
+      {{- end }}
+      {{- with $settings.tolerations }}
+      tolerations:
+        {{- toYaml . | nindent 8 }}
+      {{- end }}
+      {{- /* The chart's own per-node constraint plus any the user appended, or
+             nothing while the group is below two effective replicas. The helper
+             aborts the render on a collision instead of letting two entries
+             with the same (topologyKey, whenUnsatisfiable) be merged silently. */}}
+      {{- $spread := include "stalwart.spreadConstraints" (dict "root" $root "group" $group "settings" $settings "replicas" $replicas) }}
+      {{- if $spread }}
+      topologySpreadConstraints:
+        {{- $spread | nindent 8 }}
+      {{- end }}
       initContainers:
         # config.json is copied out of the ConfigMap into the emptyDir rather
         # than mounted from it, because STALWART WRITES THIS FILE ITSELF: on

stalwart/values.schema.json

@@ -191,6 +191,24 @@
             "exposedPorts": {
               "$ref": "#/definitions/exposedPorts"
             },
+            "spreadAcrossNodes": {
+              "$ref": "#/definitions/spreadAcrossNodes"
+            },
+            "podDisruptionBudget": {
+              "$ref": "#/definitions/podDisruptionBudget"
+            },
+            "affinity": {
+              "$ref": "#/definitions/podSpecObject"
+            },
+            "nodeSelector": {
+              "$ref": "#/definitions/podSpecObject"
+            },
+            "tolerations": {
+              "$ref": "#/definitions/podSpecArray"
+            },
+            "topologySpreadConstraints": {
+              "$ref": "#/definitions/topologySpreadConstraints"
+            },
             "resources": {
               "$ref": "#/definitions/resources"
             },
@@ -212,6 +230,24 @@
             "exposedPorts": {
               "$ref": "#/definitions/exposedPorts"
             },
+            "spreadAcrossNodes": {
+              "$ref": "#/definitions/spreadAcrossNodes"
+            },
+            "podDisruptionBudget": {
+              "$ref": "#/definitions/podDisruptionBudget"
+            },
+            "affinity": {
+              "$ref": "#/definitions/podSpecObject"
+            },
+            "nodeSelector": {
+              "$ref": "#/definitions/podSpecObject"
+            },
+            "tolerations": {
+              "$ref": "#/definitions/podSpecArray"
+            },
+            "topologySpreadConstraints": {
+              "$ref": "#/definitions/topologySpreadConstraints"
+            },
             "resources": {
               "$ref": "#/definitions/resources"
             },
@@ -233,6 +269,24 @@
           "exposedPorts": {
             "$ref": "#/definitions/exposedPorts"
           },
+          "spreadAcrossNodes": {
+            "$ref": "#/definitions/spreadAcrossNodes"
+          },
+          "podDisruptionBudget": {
+            "$ref": "#/definitions/podDisruptionBudget"
+          },
+          "affinity": {
+            "$ref": "#/definitions/podSpecObject"
+          },
+          "nodeSelector": {
+            "$ref": "#/definitions/podSpecObject"
+          },
+          "tolerations": {
+            "$ref": "#/definitions/podSpecArray"
+          },
+          "topologySpreadConstraints": {
+            "$ref": "#/definitions/topologySpreadConstraints"
+          },
           "resources": {
             "$ref": "#/definitions/resources"
           },
@@ -410,6 +464,31 @@
         "minLength": 1
       }
     },
+    "spreadAcrossNodes": {
+      "description": "Renders a topologySpreadConstraints entry for THIS group on topologyKey kubernetes.io/hostname, maxSkew 1, whenUnsatisfiable DoNotSchedule, nodeAffinityPolicy Ignore, with a labelSelector the chart fills from its own pod labels - the user never writes it, because a selector that matches no pod is trivially satisfied, distributes nothing and reports nothing. Rendered only while the group's COMPUTED replica count is >= 2, so scaleDown: true removes it as well. DoNotSchedule makes it a promise rather than an effort, and the price is a Pending pod: when two of three workers are gone (while their Node objects still exist), or when a nodeSelector/affinity narrows the eligible nodes below the replica count. nodeAffinityPolicy is beta - on by default from Kubernetes 1.26, GA with the NodeInclusionPolicyInPodTopologySpread gate in 1.33 - and is pruned SILENTLY on anything older, which brings back exactly the co-location it was set against. The replica ceiling (as many per group as there are schedulable workers, because control-plane domains hold the minimum at 0) comes from nodeTaintsPolicy and is NOT a consequence of that field. The whole rule holds only while this chart stays PVC-free - see open measurement 1 on #65.",
+      "type": "boolean"
+    },
+    "podDisruptionBudget": {
+      "description": "One PodDisruptionBudget per node group, rendered only while that group's COMPUTED replica count is >= 2 (so scaleDown: true renders none). The condition is the point, not tidiness: minAvailable 1 with exactly one replica forbids every voluntary eviction, so kubectl drain blocks forever on a pod nothing may move - and the message names the PDB rather than the replica count. minAvailable rather than maxUnavailable, as in keycloak: the promise is about the service. The selector comes from the same definition as the pod template's labels, because a PDB matching no pod never blocks an eviction while looking like protection.",
+      "type": ["object", "null"],
+      "additionalProperties": false,
+      "properties": {
+        "enabled": { "type": "boolean" },
+        "minAvailable": { "type": "integer", "minimum": 1 }
+      }
+    },
+    "podSpecObject": {
+      "description": "Passed verbatim into this group's pod spec (affinity, nodeSelector). Loosely typed on purpose, as with the probes and the security contexts: Kubernetes owns this schema, the chart reads none of the contents, and a stricter copy here would reject valid configurations rather than catch mistakes. Declaring the key is not optional even so, because the level above carries additionalProperties: false. Empty renders no field at all. Mind that narrowing the eligible nodes below a group's replica count while spreadAcrossNodes is on leaves a replica permanently Pending - intended, and loud by design.",
+      "type": ["object", "null"]
+    },
+    "podSpecArray": {
+      "description": "Passed verbatim into this group's pod spec (tolerations). Loosely typed for the same reason as podSpecObject. Note that tolerating a taint does not raise the replica ceiling: with nodeTaintsPolicy at its default of Ignore an untolerated node already counts as an empty domain.",
+      "type": ["array", "null"]
+    },
+    "topologySpreadConstraints": {
+      "description": "Additional constraints for this group, APPENDED after the chart's own and really AND-ed with it, so a second key such as topology.kubernetes.io/zone works as expected. Loosely typed like the other pass-throughs. An entry on kubernetes.io/hostname with whenUnsatisfiable DoNotSchedule - or omitted, which IS DoNotSchedule - aborts the render naming both this value and spreadAcrossNodes: Kubernetes keys a constraint on that pair, kubectl create rejects the duplicate with a message about a list index instead of a value, and an apply-based path merges the two over patchMergeKey: topologyKey silently, dropping one labelSelector for a green result.",
+      "type": ["array", "null"]
+    },
     "externalService": {
       "type": ["object", "null"],
       "additionalProperties": false,

stalwart/values.yaml

@@ -300,6 +300,137 @@ nodes:
     # Every group needs http, including mx: it is the port all three probes
     # use. It is not published for mx.
     exposedPorts: [smtp, submissions, imaps, http]
+
+    # -----------------------------------------------------------------------
+    # Distribute THIS GROUP's pods over distinct nodes (Gitea #96)
+    # -----------------------------------------------------------------------
+    # Renders a topologySpreadConstraints entry on topologyKey
+    # kubernetes.io/hostname with maxSkew: 1 and a labelSelector the CHART
+    # fills in — the two labels it puts on its own pods anyway. You never
+    # write that selector, and that is the point: a spread constraint whose
+    # selector matches no pod is TRIVIALLY SATISFIED. It distributes nothing,
+    # the rollout is green, and nothing anywhere says so.
+    #
+    # ONLY WITHIN THIS GROUP. The selector names the release AND the node
+    # group, so "full" pods are spread against "full" pods and "mx" pods
+    # against "mx" pods. A "full" and an "mx" pod on the same node is
+    # deliberately fine — they are different roles, not redundant copies.
+    #
+    # RENDERED ONLY FROM TWO EFFECTIVE REPLICAS UPWARDS. With one pod any
+    # placement satisfies it, so it would describe nothing; the count used is
+    # the computed one, so scaleDown: true removes it too.
+    #
+    # whenUnsatisfiable IS DoNotSchedule, i.e. THIS IS A PROMISE AND NOT AN
+    # EFFORT — and the price is a pod that stays Pending rather than one that
+    # lands somewhere useless. ScheduleAnyway was the original proposal and was
+    # rejected: with one worker at 97 % requested CPU the scheduler may put both
+    # replicas of a group on the same remaining node, entirely within the rule
+    # and with the redundancy gone. The cases where the promise costs something:
+    #
+    #   - TWO OF THREE WORKERS GONE: the second replica stays Pending instead
+    #     of joining the survivor — but only WHILE THE Node OBJECTS STILL
+    #     EXIST. A NotReady node still counts as an (empty) domain, so the
+    #     third replica would raise the skew to 2. Delete the Node objects and
+    #     the domains disappear with them and the pod schedules. Same
+    #     mechanism, opposite outcome, one step apart.
+    #   - A NARROWING nodeSelector/affinity BELOW: see nodeSelector further
+    #     down. That combination blocks itself, deliberately and loudly.
+    #
+    # A Pending pod with no scheduling event reads like a different problem
+    # than a distribution rule, so NOTES.txt names this rule whenever it is
+    # rendered.
+    #
+    # THE FIELD THAT CAN BE DROPPED WITHOUT A WORD: the constraint carries
+    # nodeAffinityPolicy: Ignore, and that field is BETA — on by default since
+    # Kubernetes 1.26, graduated to GA with the feature gate
+    # NodeInclusionPolicyInPodTopologySpread in 1.33. On an API server older
+    # than 1.26, or one with the gate switched off, the field is pruned
+    # SILENTLY, and the gap it was set against comes straight back (the
+    # nodeSelector case below). Nothing reports it: the object still looks
+    # correct in `kubectl get sts -o yaml`, minus one line. So 1.26 is the
+    # minimum version this chart can make the promise on at all. NOT MEASURED
+    # against an older server — this is the documented pruning behaviour for a
+    # field behind a disabled gate, not something tried here. Check the
+    # rendered StatefulSet against the live one if the server is near that
+    # boundary.
+    #
+    # WHAT IS *NOT* A CONSEQUENCE OF THAT FIELD, because the two
+    # node-inclusion policies are not a package: the ceiling on replicas
+    # described at nodeSelector below comes from nodeTaintsPolicy, which this
+    # chart leaves at its own default. It was there before nodeAffinityPolicy
+    # was set and is not its price.
+    #
+    # ONE PRECONDITION, AND IT IS NOT IN THIS FILE: this works because the
+    # StatefulSet has NO volumeClaimTemplates. Open measurement 1 on #65 asks
+    # whether anything under /var/lib/stalwart has to survive a restart; if
+    # that answer ever becomes yes, a PVC per group appears, and with a
+    # node-local or topology-bound storage class the pod is pinned to wherever
+    # its volume was first provisioned. Distribution is then decided at
+    # provisioning time rather than by this rule, and DoNotSchedule can keep a
+    # returning pod Pending for good. This rule holds only while the chart
+    # stays PVC-free.
+    #
+    # AND THE COST OF TURNING IT ON BY SCALING UP: going from 1 to 2 replicas
+    # changes the POD TEMPLATE (the constraint appears in it), so the existing
+    # pod is replaced in the same rollout round rather than merely joined by a
+    # second one. Expect a restart of the running node, not just an addition.
+    spreadAcrossNodes: true
+
+    # A PodDisruptionBudget for this group, rendered only while the group's
+    # COMPUTED replica count is >= 2 — the spread constraint's companion: that
+    # one decides the two pods sit on different nodes, this one decides only
+    # one of those nodes may be emptied at a time.
+    #
+    # The condition is not tidiness. minAvailable: 1 with exactly one replica
+    # forbids every voluntary eviction, so `kubectl drain` blocks forever on a
+    # pod nothing may move — and the message names the PDB rather than the
+    # replica count, which sends the reader to the wrong file. Precedent and
+    # full argument: keycloak (README, guideline 11).
+    podDisruptionBudget:
+      enabled: true
+      # minAvailable, not maxUnavailable: the promise is about the service —
+      # at least one pod of this group keeps answering through a drain.
+      minAvailable: 1
+
+    # Generic pass-through into this group's pod spec. Rendered verbatim with
+    # toYaml; the chart reads none of their contents, which is why they are
+    # loosely typed in values.schema.json — Kubernetes owns their schema. An
+    # empty value renders no field at all.
+    #
+    # THE ONE COMBINATION THAT BLOCKS ITSELF, and it does so on purpose:
+    # narrowing the eligible nodes below this group's replica count while
+    # spreadAcrossNodes is on leaves a replica permanently Pending. With
+    # nodeAffinityPolicy: Ignore on the chart's constraint, a nodeSelector
+    # down to one node gives 2,0,0 — skew 2 — rather than the one eligible
+    # domain and skew 0 it would give under the API default of Honor. The
+    # second shape is what we do NOT want: both pods co-located, nothing
+    # Pending, no event, and a selector that is perfectly correct, so no check
+    # in CI can see it either. Loud beats silent; that is the whole trade.
+    #
+    # AND THE CEILING, WHICH IS A RULE AND NOT A NUMBER. The chart leaves
+    # nodeTaintsPolicy at its API default of Ignore, so EVERY node counts as a
+    # domain — control-plane nodes included, and no Stalwart pod ever runs
+    # there. Those domains hold the minimum at 0 permanently, so with
+    # maxSkew: 1 a group can never place more replicas than there are
+    # SCHEDULABLE WORKERS. Add a worker and the ceiling rises with it; remove
+    # one and it falls. Do not write today's count down here — it would be
+    # wrong at the next cluster rebuild and nobody would come back for it.
+    # Raising a group past that ceiling needs spreadAcrossNodes: false, or an
+    # own topologySpreadConstraints entry with ScheduleAnyway.
+    affinity: {}
+    nodeSelector: {}
+    tolerations: []
+    # Appended AFTER the chart's own constraint and really AND-ed with it, so
+    # a second key — topology.kubernetes.io/zone, say — works as expected.
+    #
+    # An entry on kubernetes.io/hostname with whenUnsatisfiable DoNotSchedule
+    # (or omitted, which IS DoNotSchedule) ABORTS THE RENDER naming both this
+    # value and spreadAcrossNodes. Kubernetes keys a constraint on exactly
+    # that pair: `kubectl create` rejects the duplicate with a message about a
+    # list index instead of a value, and an apply-based path merges the two
+    # over patchMergeKey: topologyKey SILENTLY — one labelSelector is dropped
+    # and the result is green. A named abort is better than either.
+    topologySpreadConstraints: []
     resources:
       requests:
         cpu: 200m
@@ -324,6 +455,25 @@ nodes:
     # fail-open above), which is the opposite of what this group is for.
     replicas: 0
     exposedPorts: [smtp, http]
+    # Same six keys as nodes.full, same defaults, and argued in full there —
+    # including the three things that are easy to get wrong: that
+    # nodeAffinityPolicy can be pruned silently below Kubernetes 1.26, that the
+    # replica ceiling is "as many as there are schedulable workers" rather than
+    # a number, and that the ceiling comes from nodeTaintsPolicy and not from
+    # the field this chart sets.
+    #
+    # Neither object is rendered for this group while replicas is 0, and the
+    # condition is on the computed count: raising it to 2 (after the "mx"
+    # ClusterRole exists — see above) brings both the spread constraint and the
+    # PodDisruptionBudget with it, independently of what "full" does.
+    spreadAcrossNodes: true
+    podDisruptionBudget:
+      enabled: true
+      minAvailable: 1
+    affinity: {}
+    nodeSelector: {}
+    tolerations: []
+    topologySpreadConstraints: []
     resources:
       requests:
         cpu: 100m