jlab-cluster/helm-charts
3
0 Fork 0
Code Issues 15 Pull Requests 3 Actions Packages Wiki Activity
Page: stalwart
Pages
CI CD Chart Style Guide Charts Forge and Tooling Home Lessons Learned Local Development New Chart Renovate Versioning Workflow apache tika bentopdf bulwark calibre web automated convertx cyberchef freesailarr garage gotenberg homeassistant keycloak matrix synapse minecraft router minecraft server outline paperless ngx patchmon rallly rss bridge samba satisfactory searxng stalwart sure teamspeak server template chart urlaubsverwaltung valkey voucher vault ytdlp web player zipline
Clone
9
stalwart
Justin Klein edited this page 2026-10-14 14:57:00 +00:00
Unescape Escape
Table of Contents
This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

stalwart

Deploys Stalwart — an all-in-one mail server (SMTP, IMAP, POP3, JMAP, ManageSieve, plus CalDAV/CardDAV/WebDAV) — cluster-capable from the start, with roles rather than symmetry: one StatefulSet per entry of the nodes map, each passing its key in STALWART_ROLE.

Read the next paragraph before anything else, because it decides how this page is organised.

Since v0.16 Stalwart keeps its entire configuration inside the data store and manages it over the JMAP API — over 2 400 settings in 117 object types, including every listener, the blob store, the in-memory store, TLS objects, domains and the cluster roles. The only thing left on disk is a small config.json naming the data store: "It only needs to contain the DataStore object." The REST API from v0.15 is gone.

So this chart configures almost nothing, and that is its defining property, not a gap. It renders config.json, the workloads, the Services, the NetworkPolicies, the Ingress and a cert-manager Certificate. Domains, listeners, stores, TLS objects and cluster roles are operator work in the WebUI. That is why First install is eight steps long and why half of this page is Traps.
Image docker.io/stalwartlabs/stalwart (the glibc variant; -alpine is built from the same source and carries the same setcap, so it is not an escape from the capability deviation)
Versions not listed on purpose — read the current pair off stalwart/Chart.yaml, see Chart index
Workloads one StatefulSet per node group, plus one config-seeding init container each
Storage no PVC. Both image volumes are emptyDirs — state lives in Postgres, S3 and Valkey
Routing two mutually exclusive modes for the WebUI and JMAP only, Ingress (default) or Gateway API (httpRoute.enabled) — one object in either mode, pointed at a single node group (ingress.nodeGroup), never per group. SMTP, IMAP and ManageSieve pass through neither. In Gateway mode the CORS configuration the webmail depends on is gone — see Routing
External dependencies Postgres (required), S3 (blob store), Valkey (in-memory store + cluster coordinator)
Secrets up to 3 × OnePasswordItem (data store, recovery admin, S3)
Network one NodePort Service per public port + per-group ClusterIP and headless Services + NetworkPolicy on by default
Subchart optional valkey — this repo's own chart, from the published index, off by default and gated on valkeySubchart.enabled
Install-test ci/stalwart/ — and it proves much less than a green run suggests, see CI

Prerequisites

  • Postgres, reachable from the cluster, with a database and a role for Stalwart. This is the data store and the chart will not render without its host.
  • S3-compatible object storage for the blob store. The filesystem blob store is documented as "available only for single node installations", and RocksDB likewise — two nodes force both stores to be external.
  • A Valkey (or Redis), for the in-memory store. Not optional in a cluster: the coordinator's Default variant "reuses the Redis connection already configured on the in-memory store", so a multi-node install has no coordination path without one. Either point externalServices.valkey.host at an instance or set valkeySubchart.enabled: true to deploy this repo's own valkey chart as a subchart. Redis/Valkey is admissible as the in-memory store only — never as the data or blob store.
  • cert-manager, and for the mail certificate most likely a DNS-01 issuer — see TLS.
  • 1Password operator, plus the items in What the 1Password items must contain.
  • Inbound reachability on port 25 with matching reverse DNS. Nothing in the chart can provide this, and without it a mail server is not operable. See DNS.
  • An HA proxy in front of the cluster that forwards 25, 465 and 993 and can set send-proxy-v2. The whole source-address design rests on it; see Why externalTrafficPolicy must be Cluster.

No full-text search service is needed: the search store has a Default variant that indexes into the data store ("no additional services are required"). No Elasticsearch, no Meilisearch — but the FTS load lands on the same Postgres, which belongs in the sizing.

Installation

The registry URL and its credentials live in 1Password (item jlab-registry-access: username, password, charts-url) and never in a repository or a shell history file.

helm repo add jlab "$CHARTS_URL" --username "$REG_USER" --password "$REG_PASS"
helm repo update

helm install mail jlab/stalwart \
  --namespace mail --create-namespace \
  -f my-values.yaml

A first-install my-values.yaml. Note bootstrap.enabled: true — it is required on a first install, for the reason in step 1:

bootstrap:
  enabled: true          # first install only; set to false in step 7

ingress:
  domain: mail.example.com
  clusterIssuer: letsencrypt-dns

certificate:
  clusterIssuer: letsencrypt-dns
  extraDnsNames:
    - smtp.example.com
    - imap.example.com

dataStore:
  postgres:
    host: postgres.db.svc.cluster.local

externalServices:
  s3:
    host: s3.example.com
  valkey:
    host: mail-valkey.mail.svc.cluster.local

secrets:
  dataStoreSecretRef: op://vault/stalwart-datastore
  recoveryAdminSecretRef: op://vault/stalwart-recovery-admin
  s3SecretRef: op://vault/stalwart-s3

helm install prints the chart's NOTES.txt, which is the per-release version of this page: it names the node groups and their ports, the exact listener numbers to create, the ClusterRole names, the 1Password fields, and a loud banner for as long as recovery mode is on. Read it.

What the 1Password items must contain

One Kubernetes Secret key is created per field of the item, and the templates reference those keys by name — a missing field leaves the container in CreateContainerConfigError. An empty field fails the same way as a missing one: upstream's secret resolver treats an empty environment variable as not found.

Value Item fields Notes
secrets.dataStoreSecretRef password Only the password. The username cannot live here — see why
secrets.recoveryAdminSecretRef recovery-admin The complete value admin:<password>, in one field. Not two fields — see why
secrets.s3SecretRef access-key, secret-key Exposed to the container as STALWART_S3_ACCESS_KEY / STALWART_S3_SECRET_KEY for the BlobStore object to reference

Changing what is inside an item restarts nothing. Helm never sees a secret's content, so there is nothing to hash. Repointing one of the values above at a different vault item does roll the pods — the chart stamps checksum/secret-ref over the rendered OnePasswordItems, which covers the op:// path. After rotating a password inside an item, restart by hand, once per group:

kubectl -n mail rollout restart statefulset/mail-stalwart-full-sfs
kubectl -n mail rollout restart statefulset/mail-stalwart-mx-sfs

The 1Password operator's auto-restart annotation is not set and must not be: verified against operator 1.12.0 it enumerates Deployments only, so on a StatefulSet it is a silent no-op advertising a guarantee it does not provide (§9).

First install: eight steps, none optional

  1. bootstrap.enabled: true, nodes.mx.replicas: 0. The setup wizard everyone reaches for runs in bootstrap mode, and bootstrap mode is entered only when there is no config.json: "The first time Stalwart is started without a config.json, it enters bootstrap mode." This chart always renders one, so the wizard never appears. Against an empty database that would leave a server with no listeners, no domain and no administrator, and nothing to fix it with.

    The documented path for a templated config.json is recovery mode: "template config.json onto the host, start Stalwart once with STALWART_RECOVERY_MODE=1 and a STALWART_RECOVERY_ADMIN supplied from a secrets manager… and then restart Stalwart without the recovery variables." It opens the management endpoint on 8080 and keeps every mail port closed. With a second group running the wizard would be unusable anyway — each request would land on a random pod.

  2. Open the management UI and create the permanent administrator.

    kubectl -n mail port-forward svc/mail-stalwart-full-http-service 8080:8080

    then http://localhost:8080/admin. With recoveryAdminSecretRef set, the credential is the one in the vault item; without it, Stalwart generates a password and prints it to the pod log on every restart.

  3. Create the configuration objects in the WebUI. Blob store (S3), in-memory store (Valkey), coordinator (Default — reuses the in-memory store connection), search store (Default — uses the data store), the TLS object from step 5, the NetworkListener objects, proxyTrustedNetworks, and one ClusterRole per node group.

    The listener ports must match the release exactly, because the chart cannot create listener objects:

    Listener bind published
    smtp 25 node port 30025
    submissions 465 node port 30465
    imaps 993 node port 30993
    http 8080 not published (Ingress + probes)

    And the roles, named exactly full and mx. The example shape:

    {"name":"mx","tasks":{"@type":"DisableAll"},"listeners":{"@type":"EnableAll"}}

    Both fields accept EnableAll, DisableAll, EnableSome and DisableSome. The task types are storeMaintenance, accountMaintenance, metricsCalculate, metricsPush, pushNotifications, searchIndexing, spamClassifierTraining, outboundMta, taskQueueProcessing, taskScheduler. Only spamClassifierTraining may run on at most one node. Note the object encoding: a set is a JSON object mapping each member to true, not an array, and a duration is an integer count of milliseconds.

    Getting a role name wrong here does not fail — read Fail-open before continuing.

  4. Configure HTTP forwarding, before the auto-ban notices the Ingress. Mandatory, not a tip — see The Ingress ban trap.

  5. Point the TLS object at the mounted certificate.

    certificate: /etc/stalwart-tls/tls.crt
privateKey:  /etc/stalwart-tls/tls.key

    cert-manager writes these into tls-<release>-stalwart-mail. The volume is mounted optional, so a pending challenge does not hold the pods in ContainerCreating; until it is issued the mail ports serve a self-signed certificate.

  6. Set send-proxy-v2 on the HA proxy. All-or-nothing together with step 3. Without the header on a trusted network every connection hangs until timeout; with the header from an untrusted network Stalwart reads PROXY TCP4 … as an SMTP command. Test on a spare port with its own frontend and move 25 only once a Received: line shows the real sender address. Background: Why externalTrafficPolicy must be Cluster.

  7. Restart the pods — "a newly added listener binds only on the next start" — and set bootstrap.enabled: false in the same round.

  8. Raise nodes.mx.replicas (only now that the mx role exists) and publish DNS.

DNS a mail server cannot run without

None of this is rendered by the chart; all of it is required.

Record Content
MX the deployment's public name, pointing at the HA proxy
SPF TXT "v=spf1 ..." naming the HA proxy's address as a permitted sender
DKIM TXT at <selector>._domainkey — the key is generated in Stalwart (Domains → DKIM), so this record is published after step 3
DMARC TXT at _dmarc — start at p=none with rua= and tighten only once the reports are clean
Reverse DNS PTR for the HA proxy's public address, matching its forward name

The PTR is the one most often missed. It is not the node's and not the pod's: the HA proxy is what other mail servers see connecting. Many receivers reject outright when it is absent, and nothing in Kubernetes can set it — it belongs to whoever owns the IP range.

Configurable values

Global

Value Default Meaning
scaleDown false true renders every group with replicas: 0 and overrides every nodes.*.replicas. Services, Ingress, Secrets and the Certificate stay, so no certificate is re-issued
limitFactor 3 Chart-wide default limit factor; a group's resources.limitFactor wins

secrets

Value Default Meaning
secrets.dataStoreSecretRef null op:// path; field password. Referenced from config.json by variable name, so the password never enters the ConfigMap
secrets.recoveryAdminSecretRef null op:// path; field recovery-admin holding admin:<password>. null switches STALWART_RECOVERY_ADMIN off — the correct end state once a permanent admin exists
secrets.s3SecretRef null op:// path; fields access-key, secret-key. Exposed as env vars for the BlobStore object to reference

dataStore — the only part of Stalwart the chart configures

Rendered verbatim into config.json as the DataStore object, "@type": "PostgreSql".

Value Default Meaning
dataStore.postgres.host null Required, no default
dataStore.postgres.port 5432
dataStore.postgres.database stalwart
dataStore.postgres.username stalwart Rendered as authUsername. Public by necessity — why
dataStore.postgres.tls.enable true Rendered as useTls. The opposite of upstream's default, deliberately
dataStore.postgres.tls.allowInvalidCerts false Only ever a temporary diagnostic
dataStore.postgres.poolMaxConnections 10
dataStore.postgres.timeoutMs 15000 Integer milliseconds. Named for its unit — why
dataStore.postgres.options null Extra libpq options, passed through; null omits the field

bootstrap

Value Default Meaning
bootstrap.enabled false STALWART_RECOVERY_MODE=1. true for a first install, then false. Leaving it on keeps every mail port closed while all Kubernetes objects look healthy
bootstrap.logLevel info STALWART_RECOVERY_MODE_LOG_LEVEL; no effect once disabled

ports

A map, user-extensible: adding 587, 143, 110, 995 or 4190 is an additive change. Two numbers per entry and they are not interchangeable — see The port duplication.

Entry containerPort nodePort public
smtp 25 30025 true
submissions 465 30465 true
imaps 993 30993 true
http 8080 — false

service

Value Default Meaning
service.externalTrafficPolicy Cluster Local would drop every packet in this topology — why
service.annotations {} Applied to every Service the chart renders

nodes

A map, not a list, so that an override never depends on a position. The key is the cluster-role name and goes into STALWART_ROLE.

Value Default (full) Default (mx) Meaning
nodes.<g>.replicas 1 0 0 allowed; scaleDown wins
nodes.<g>.exposedPorts [smtp, submissions, imaps, http] [smtp, http] Keys of ports; must include http (all probes target it)
nodes.<g>.spreadAcrossNodes true true One pod of this group per node, enforced. Rendered only from two computed replicas upwards — what it costs
nodes.<g>.podDisruptionBudget.enabled true true A PDB per group, rendered only from two computed replicas upwards
nodes.<g>.podDisruptionBudget.minAvailable 1 1 minAvailable, not maxUnavailable: the promise is about the service
nodes.<g>.affinity {} {} Passed verbatim into this group's pod spec. Narrowing it below the replica count blocks the group — why that is the good outcome
nodes.<g>.nodeSelector {} {} As above
nodes.<g>.tolerations [] [] As above. Tolerating a taint does not raise the replica ceiling
nodes.<g>.topologySpreadConstraints [] [] Appended after the chart's own and really AND-ed with it. A collision on kubernetes.io/hostname aborts the render — why
nodes.<g>.resources.requests 200m / 512Mi / 64Mi 100m / 256Mi / 64Mi Limits computed by limitFactor
nodes.<g>.env [] [] Additive; value, secretKeyRef, configMapKeyRef

There is no nodes.<g>.tasks. Which tasks a role runs is a property of the ClusterRole object in the database and the chart cannot write it; a values key that looked like it set them and did not would be worse than its absence.

High availability, per group

spreadAcrossNodes: true renders one topologySpreadConstraints entry for the group — maxSkew: 1 on kubernetes.io/hostname, whenUnsatisfiable: DoNotSchedule, nodeAffinityPolicy: Ignore — with a labelSelector the chart fills from its own pod labels. podDisruptionBudget.enabled: true renders <release>-stalwart-<group>-pdb with the same selector. They are two halves of one answer: the constraint decides that the pods sit on different nodes, the budget decides that only one of those nodes may be emptied at a time.

The distribution is only ever within a group. The selector names the release and the node group, so full pods are spread against full pods and mx pods against mx pods. A full and an mx pod on the same node is deliberately fine — they are different roles, not redundant copies. There is no cross-group condition, and that was decided rather than overlooked.

Both objects are conditional on the computed replica count (so scaleDown: true removes both). With full: 2 and mx: 1 exactly one PDB and one spread constraint exist. NOTES.txt prints the outcome per group — enforced / not rendered / off, and for the PDB its minAvailable or the reason it is absent — so it never has to be inferred from kubectl get pdb.

The selector is the chart's, never yours. stalwart.groupLabels is the single definition behind the pod template's labels, the spread constraint's labelSelector and the PDB's selector, and the reason is that a selector matching no pod fails silently in both objects: the constraint becomes a condition over zero pods and is trivially satisfied, the budget never blocks an eviction while looking like protection. ci/run-selector-match-test.sh holds all three against each other on every pull request — see the selector-match job.

Scaling a group from 1 to 2 replaces the running pod. The constraint lives in the pod template, so the change is a rollout and not merely an addition. Expect a restart of the node that is already up, in the same round.

And one precondition that is not in values.yaml: this works because the StatefulSets have no volumeClaimTemplates. If the open question about /var/lib/stalwart ever turns into a PVC per group, a node-local or topology-bound storage class pins each pod to wherever its volume was first provisioned — distribution is then decided at provisioning time rather than by this rule, and DoNotSchedule can keep a returning pod Pending for good. The rule holds only while this chart stays PVC-free.

networkPolicy

Value Default Meaning
networkPolicy.enabled true Only a boundary where the CNI enforces one
networkPolicy.probeSources [] CIDRs allowed to reach http. Empty means open — why
networkPolicy.allowedNamespaces [] Extra namespaces for http. Not for the ingress controller — use ingressController below; this value is namespace-granular, so naming the controller's namespace here would open http to every pod in it
networkPolicy.ingressController kube-system / app.kubernetes.io/name: rke2-traefik The controller allowed to reach http once probeSources or allowedNamespaces restricts it. The two keys render as one from entry, i.e. an AND — which is the whole reason it is separate from allowedNamespaces. Default measured on the cluster
networkPolicy.egress.dns true UDP+TCP 53 to anywhere
networkPolicy.egress.smtp true TCP 25 to anywhere — outbound delivery
networkPolicy.egress.extraPorts [] Extra outbound TCP ports; add 443 for DNS-01 provider APIs or AI endpoints

Egress is restricted by port, not by destination: a NetworkPolicy matches CIDRs and selectors, never hostnames, so no host value from this chart can appear in it. What it still buys is that a compromised pod cannot open arbitrary outbound ports. Unlike most charts in the repository these are the configured ports and not a guess at the standard ones, because this chart holds them as values (dataStore.postgres.port, externalServices.*.port).

Correction, 2026-10-14 (Gitea #5). This page and the chart's own values.yaml used to advise putting the ingress controller's namespace into allowedNamespaces, with ["ingress-nginx"] as the example. Both halves of that are now wrong. Measured on the live cluster: ingress-nginx is gone — no DaemonSet, no pods — the only IngressClass is traefik and it is the default, and the controller is the DaemonSet rke2-traefik in kube-system. And even with the right name, allowedNamespaces would have opened http to every pod in that namespace. networkPolicy.ingressController replaces the advice and narrows the source to the controller's pods. A policy written against the old controller would have been accepted, would have looked strict, and would have silently blocked all web traffic.

externalServices — Kubernetes only, these do not configure Stalwart

Value Default Used for
externalServices.s3.host "" the NOTES.txt checklist only
externalServices.s3.port 443 the egress rule
externalServices.valkey.host "" the render-abort check and the checklist; empty + valkeySubchart.enabled resolves to <release>-valkey-service
externalServices.valkey.port 6379 the egress rule

Setting a host here does not point Stalwart at it. The blob store and in-memory store are database objects created in the WebUI.

certificate

Value Default Meaning
certificate.enabled true
certificate.clusterIssuer null Falls back to ingress.clusterIssuer; usually a different, DNS-01 issuer
certificate.extraDnsNames [] ingress.domain is always included
certificate.mountPath /etc/stalwart-tls A top-level path so no mount nests inside the config emptyDir

ingress

Value Default Meaning
ingress.enabled true Gates the Ingress only; ingress.domain stays required either way (the mail Certificate reads it). Mutually exclusive with httpRoute.enabled
ingress.clusterIssuer null Required, no default — unless certificate.clusterIssuer is set, which it falls back to. Gateway mode does not free it for that reason: the mail Certificate still needs an issuer
ingress.domain null Required, no default; read three times since the Gateway support — the Ingress host, the mail Certificate's dnsNames, and the HTTPRoute's hostname while httpRoute.hostnames is empty
ingress.className nginx No counterpart in Gateway mode — the class sits on the Gateway
ingress.nodeGroup full Which group the routing object points at, via that group's own http Service. Shared by both modes on purpose: it names the node group the management surface is published from, which is not a property of the Ingress. Must be a key of nodes: and must expose http; both are checked in both templates
ingress.annotations {} Merged with the cert-manager annotation. No counterpart in Gateway mode, and for this chart that is the consequential one: this installation's CORS configuration lives here. See Routing

httpRoute

The Gateway-API alternative to the Ingress, off by default. Added with group 3 of the Gateway-API sweep (Gitea #114); the form is the pilot's and is binding for every chart (style guide §4).

Value Default Meaning
httpRoute.enabled false Renders <release>-stalwart-httproute instead of the Ingress. Mutually exclusive with ingress.enabled — with both true the render aborts naming both values. One route, not one per node group
httpRoute.parentRefs [] The Gateway(s) the route attaches to — Gateway API ParentReference entries (name, and optionally namespace, sectionName, port, group, kind), passed through unchanged. Required while httpRoute.enabled is true: the render aborts on an empty list. The schema requires name on every entry and is otherwise left open, because Kubernetes owns that schema and adds fields to it across releases
httpRoute.hostnames [] Empty means [ingress.domain], so the HTTP hostname has a single source of truth. Note that the server's own hostname lives in the database, not in any chart value: after bootstrap the WebUI is reached at https://<the configured server hostname>/admin, so a second hostname can be reachable without the application agreeing that it is its name

mtaSts — the MTA-STS policy routes

Added with Gitea #123. Stalwart serves the MTA-STS policy itself, per domain and out of its data store; this block is only about routing a sending MTA to it, and it renders in both routing modes.

Value Default Meaning
mtaSts.hosts [] One entry per policy hostname. The list is the switch — there is deliberately no mtaSts.enabled, because "enabled with an empty list" would be a state with no meaning that the chart would then have to catch. The schema rejects mtaSts.enabled outright, so the rule is enforced and not merely documented. Empty renders nothing in either mode and makes no value required
mtaSts.hosts[].host — Required. The policy hostname, i.e. mta-sts.<domain>
mtaSts.hosts[].tlsSecretName — Required in Ingress mode, inert in Gateway mode (TLS sits on the Gateway's listener there). No default on purpose: the plausible one, this release's own tls-<release>-<chart>-ingress, is the wrong certificate — it covers ingress.domain, not this host. Hosts sharing one secret are collected into a single tls: entry, so one certificate with several SANs is requested once instead of once per host
mtaSts.hosts[].clusterIssuer "" Optional, and empty means no annotation — "the certificate comes from somewhere else, do not ask cert-manager for it". The same deliberate omission as the qBittorrent object in freesailarr and two of the four realm ingresses of Gitea #122. Entries sharing a tlsSecretName must agree on it; the render aborts naming both

What is rendered is not the same shape on both sides, and that is forced rather than stylistic:

Mode Objects
Ingress one Ingress per distinct clusterIssuer — <release>-stalwart-mta-sts-<issuer>-ingress, and <release>-stalwart-mta-sts-ingress for the issuer-less group — each with one rule per host and its tls: list grouped by secret
Gateway one HTTPRoute, <release>-stalwart-mta-sts-httproute, carrying every hostname. Nothing differs per host (same path, same backend, no filter), and it attaches to the same Gateway(s) as the WebUI route

"TLS belongs to the platform" is not the same as "there is no mechanism for it." In Gateway mode this chart renders no certificate for these hostnames and both TLS fields are inert — but cert-manager in this cluster runs with enableGatewayAPI: true (measured in its ControllerConfiguration), so it can issue the listener certificates itself, from an annotation on the Gateway rather than on anything here. So the work does not disappear with the Ingress, it moves to the Gateway object — and mind the depth trap below while doing it, because a wildcard listener for the parent domain will not cover mta-sts.sub.<domain>.

The path is not a value. RFC 8461 fixes it at /.well-known/mta-sts.txt, rendered with pathType: Exact (type: Exact in the route) because exactly one file lives there. A values key for a protocol-mandated path is one that can only ever be set wrong — and wrong is invisible from this side, see the traps below. Exact also matters in its own right: a prefix on these hostnames would publish the mail server's whole management surface under them.

The chart's own ingress.annotations are not copied onto these objects. They belong to the WebUI/JMAP host — on this installation they are its CORS configuration — and a policy file served to sending MTAs has no use for them.

Why not fold the names into the mail certificate. The alternative was real: the chart already has certificate.extraDnsNames. It was rejected because that would hang the mail certificate on this list — a bad entry here would then fail the issuance of the certificate STARTTLS on 25 and implicit TLS on 465/993 depend on. A policy route that is a nice-to-have would be able to take SMTP and IMAP with it. Separate secrets keep the blast radius at the entry.

Migration from jk-cluster-config. Where a route for one of these hosts already exists in another repository — and mta-sts.bekemeyer-klein.de does, from the Fleet release jk-cluster-config-fleet-cd-stalwart-mta-sts — the order is remove there first, then add the entry here. The other way round, ingress-nginx's admission webhook rejects the second object on the same host and path. With the default [] this block changes nothing about a running installation.

securityContext and probes

Value Default
securityContext.pod runAsNonRoot: true, runAsUser/runAsGroup: 2000, seccompProfile: RuntimeDefault. No fsGroup — the chart mounts no PVC
securityContext.container allowPrivilegeEscalation: false, readOnlyRootFilesystem: true, runAsNonRoot: true, drop: [ALL], add: [NET_BIND_SERVICE]
livenessProbe GET /healthz/live, period 20s, timeout 5s, failures 3
readinessProbe GET /healthz/ready, period 20s, timeout 5s, failures 3
startupProbe GET /healthz/ready, period 10s, failures 30 (budget 300s)

The intervals are calculated, not chosen — see The /healthz rate limit.

valkeySubchart and valkey (subchart)

The subchart is this repo's own valkey chart, resolved from the published index — not upstream's. So §14 point 2 ("we consume their chart as it comes") does not apply here: there is no third party's defaults to leave alone, because the subchart follows our conventions by being one of ours.

Value Default Meaning
valkeySubchart.enabled false true deploys our valkey chart alongside, reachable at <release>-valkey-service:6379
valkey.* see below Passed through to the subchart — these are its value names

The switch is valkeySubchart.enabled, not valkey.enabled, and that is forced. Everything under valkey: is handed to the subchart as its values, and our valkey chart ships additionalProperties: false with no enabled property — so valkey.enabled is rejected by the subchart:

Error: values don't meet the specifications of the schema(s) in the following chart(s):
valkey:
- at '': additional properties 'enabled' not allowed

And because Helm skips a disabled subchart's schema validation, that error would surface only for whoever first set it to true — the default path renders perfectly clean. The condition therefore lives on a key of this chart's own, which is also the honest shape: enabled was never a valkey value.

The values passed through, with the ones that matter:

Value Default here Meaning
valkey.secrets.authSecretRef null Required once the subchart is on (the subchart's own required() aborts the render). One field, auth-conf, holding the complete line requirepass <password> — not a bare password
valkey.persistence.enabled false Cache, not queue — see below
valkey.valkey.maxmemoryPolicy allkeys-lru The other half of the same pair
valkey.valkey.resources.requests 50m / 128Mi / 32Mi The subchart derives maxmemory from the computed limit
valkey.networkPolicy.allowedNamespaces [] The subchart's own policy; needed only when Stalwart runs in a different namespace, which with the subchart it does not

It is configured as a CACHE, and that is a decision rather than an inherited default. The valkey chart treats persistence and eviction as a pair and warns against mixing them: queue is persistence.enabled: true + noeviction, cache is persistence.enabled: false + allkeys-lru. Stalwart's in-memory store holds rate-limiter counters, auth tokens and sessions, auto-ban entries and the coordination pub/sub — every one of those is reconstructible. Nothing in it is the system of record: messages and metadata are in Postgres, blobs in S3, and the cluster node-id lease is in the data store too, not here. Paying for a PVC and for AOF replay to protect data that is worthless after a restart is the wrong trade.

The cost, named rather than discovered later: allkeys-lru really does discard keys under memory pressure, so the rate limiters and the auto-ban fail open — someone who can drive memory usage can age out their own ban entry. The answer is to size memory for the working set, not to switch to noeviction: with noeviction Stalwart's writes to the store start erroring instead, which takes out authentication rather than just the limits. Treat the limiters as a nuisance filter, not as a security boundary.

scaleDown and checksum/config stop at the subchart boundary, so scaling this release to zero does not stop the Valkey, and a new valkey chart release reaches a cluster only when stalwart is republished.

Never change valkey/ and stalwart/ in the same pull request

The dependency resolves from the published index, not from the working tree, and valkey/ is published only by a push to main that touches it. In one PR carrying both:

  • every helm command — lint, package, render and the install-test — pulls the old valkey .tgz, because the new one does not exist yet;
  • the PR goes green against a version of the subchart that is not the one in the diff, so the change to valkey/ is reviewed while nothing has exercised it;
  • after the merge both publish, and the first install of the new stalwart is the first time the new valkey runs anywhere.

Two consecutive pull requests: change valkey/, merge, let it publish, then raise the version: in stalwart/Chart.yaml. The green check on a combined PR is a statement about the wrong artifact.

Traps

Fail-open: a node whose role is missing becomes a full node

The worst one, and the reason the chart is shaped as it is. From the source, not the documentation.

The bootstrap registry looks a ClusterRole up by name; when it finds none it records a non-fatal error and leaves the role unset. The listener check then reads

if bp.role.as_ref().is_none_or(|r| match &r.listeners { ... })

and is_none_or yields true for every listener when the role is None. The same applies to the tasks, through a different file whose pre-set default has all ten task flags at true.

So a node whose role is not in the database binds every listener and runs every task. The mx pod would serve IMAP and the admin interface, train the spam classifier, and run the outbound MTA a second time. The pod is Running, the log holds one line — Cluster role "mx" not found in registry — and the readiness probe is green. Kubernetes cannot detect it. Worse, the error is logged after the listeners are already bound.

This is not only a first-install concern: it is every typo in a role name and every rename in values.yaml without the matching rename in the database.

Three things contain it, and all three are why the chart looks the way it does:

  • mx ships at replicas: 0;
  • the Services publish only what was asked for, per port;
  • the NetworkPolicy is a ceiling on reachability, independent of what the process actually bound.

Documented as a trap rather than explained away. If you add a node group, create its ClusterRole first.

NET_BIND_SERVICE is needed to exec, not to bind

Both published image variants run

setcap cap_net_bind_service=+ep /usr/local/bin/stalwart

(Dockerfile.build, line 164 for the glibc target and 186 for musl/-alpine). The e in +ep sets the file's effective bit, which makes the binary a capability-dumb binary, and capabilities(7) then applies this at execve time:

When executing a capability-dumb binary, the kernel checks if the process obtained all permitted capabilities that were specified in the file permitted set… If the process did not obtain the full set of file permitted capabilities, then execve(2) fails with the error EPERM.

drop: [ALL] empties the bounding set, so the capability cannot be obtained, so the exec fails outright:

exec /usr/local/bin/stalwart: operation not permitted

The check runs before the program binds anything and does not ask whether the capability is needed. This is why the three obvious escapes all fail for the same reason: high application ports do not help, -alpine carries the identical setcap, and the net.ipv4.ip_unprivileged_port_start sysctl addresses binding rather than exec.

And the second half, which is the part that matters for whoever reads this next: the capability is never actually used. It is spent entirely on satisfying the exec check. Stalwart binds 25/465/993 here, but inside a pod that needs no capability for them anyway — the kubelet sets ip_unprivileged_port_start=0 in the pod network namespace. So a hardening pass grepping for a capability nobody uses will find exactly this one, and removing it stops the chart from starting at all. The day setcap disappears from upstream's Dockerfile.build is the day this deviation can go with it.

Why externalTrafficPolicy must be Cluster

The topology decides it. Three control-plane nodes, two workers; an HA proxy in front distributes 25/465/993 across the control-plane nodes, and no Stalwart pod is ever scheduled on a control-plane node.

externalTrafficPolicy: Local forwards only to pods on the node that received the packet. On a control-plane node there is never one — so under Local every packet is dropped, not some. It has to be Cluster.

The price of Cluster is that kube-proxy masquerades the source address to the receiving node's IP (it must: the reply would otherwise leave through a different node than the one the sender holds its connection with). On a mail server a lost source address is not cosmetic. In order of how fast it hurts:

  1. the auto-ban switches the service off — it bans the masqueraded node IP and with it every sender at once (documented behind Traefik, triggerable from outside);
  2. SPF checks the wrong address and takes DMARC with it;
  3. blocklists report everything clean;
  4. per-IP rate limits collapse onto one address;
  5. greylisting loses its key;
  6. the Received: chain becomes worthless — the only item here that destroys evidence retroactively rather than merely failing.

The HA proxy solves all of it, port 25 included. The objection that foreign mail servers do not send a PROXY header applies only to direct DNAT. Here the HA proxy terminates the inbound connection and opens its own to the node, on which it can prepend a send-proxy-v2 header. The real client address then travels in the header and the masquerading stops mattering.

Which networks are trusted follows from the same topology — and only works because of it. A trusted network makes the PROXY header mandatory: a connection from there without one is misread and hangs until timeout. In Kubernetes that is dangerous, because kubelet probes come from a node IP.

Traffic Source IP Stalwart sees Trusted?
Mail via the HA proxy the control-plane node's IP (after masquerading) yes
Kubelet probes the worker node's IP (the one running the pod) no
Ingress to WebUI/JMAP the ingress controller's pod IP no — different mechanism

So proxyTrustedNetworks contains only the three control-plane node IPs. Because no Stalwart pod ever runs there, probe sources and trusted addresses are disjoint — that is not an operating preference, it is a load-bearing assumption of the design. Anyone who later lets a Stalwart pod onto a control-plane node breaks it.

Which address the masquerading actually inserts is to be measured, not assumed. It appears in the Received: header after the first connection.

The port duplication cannot be avoided

Every listener is a NetworkListener object in the database. The chart cannot create those objects, so ports.*.containerPort is a second copy of a decision really made in the WebUI. If the two disagree, the Service points at a port nothing listens on: connections are refused or time out, every pod stays Ready, and no object reports a problem.

The chart reduces the surprise rather than the duplication, in two ways. The application binds the ordinary 25/465/993, so the numbers an operator types into the WebUI are the ones from every page of Stalwart's documentation and from the image's own EXPOSE list. And nodes.<g>.exposedPorts is deliberately not a copy of the listener configuration but a statement of its own — "this group is reachable here and nowhere else" — which is why a disagreement in either direction is harmless: a role that binds more is unreachable on the extra ports, and a policy that permits more has nothing listening.

An earlier draft had the application bind 30025/30465/30993 so that nothing privileged was bound and the capability could be dropped. That reasoning collapsed against the setcap/execve rule above; since the capability must be granted anyway, high ports bought no hardening and only cost predictability.

And in the NetworkPolicy the pod ports apply, not the node ports — 25/465/993, never 30025/30465/30993. A policy is evaluated at the pod's interface, after kube-proxy has already translated the node port to the target port. Writing 30025 there matches nothing: the packet arrives on 25. The policy then looks strict, is accepted by the API server, and silently blocks everything inbound.

Routing: Ingress or Gateway API

Two mutually exclusive modes for the HTTP surface — WebUI and JMAP — and exactly one of them may be on. ingress.enabled ships true, httpRoute.enabled ships false.

One route, not one per node group. #114 describes this chart as rendering "per node group"; its Services and StatefulSets are per group, the routing object is not and must not be. The route targets exactly ingress.nodeGroup's own http Service and keeps both of the Ingress's guards — the group must be a key of nodes: and must expose http — because the WebUI's setup and management screens are not safe to spread over a group that may not run the management tasks, and a port-label selector would send each request to a random group.

Rule Why
ingress.enabled and httpRoute.enabled cannot both be true — the render aborts naming both value paths two data paths onto one Service under one hostname is a conflict at DNS level already. For Stalwart there is a second edge: the auto-ban sees the proxy's address, so two data paths mean two addresses to get banned and a WebUI that fails for half the requests. A migration runs over a second hostname — and note that the server hostname configured in the database decides where the WebUI sends administrators, which no chart value can change
httpRoute.parentRefs must not be empty — the render aborts a route without a parent attaches to no Gateway. The object is created, it is schema-valid, no controller rejects it, and it routes nothing: no event, no condition, and the only symptom is a hostname that does not answer
ingress.nodeGroup is shared by both modes it names where the management surface is published from, which is not a property of the Ingress

SMTP, IMAP and ManageSieve are not routed in either mode, exactly as before: they keep arriving through the per-port NodePort Services behind the HA proxy. A Gateway could in principle carry them as TLSRoute/TCPRoute listeners; this chart renders neither, and doing so would be a separate decision.

The mail certificate stays in Gateway mode, and must not be cleaned up

This is the only chart in the whole sweep where anything of the TLS path stays with the chart. certificate.enabled keeps rendering <release>-stalwart-mail-certificate in Gateway mode, because that object has nothing to do with the Ingress or the Gateway: it covers STARTTLS on 25 and implicit TLS on 465/993, and those connections pass through no HTTP proxy at all (TLS).

So in Gateway mode:

  • ingress.domain is still required — the certificate's dnsNames read it;
  • ingress.clusterIssuer is still required unless certificate.clusterIssuer is set, which it falls back to;
  • and the Certificate must not be removed as part of a migration. Deleting it drops SMTP and IMAP back to whatever self-signed certificate Stalwart generates for itself: clients see a name mismatch while every Kubernetes object in the release stays green.

What does move to the Gateway is the certificate for the HTTP hostname only, onto its listener.

Gateway mode drops the CORS configuration, and that breaks the webmail

Measured on the live Ingress on 2026-10-12 — operator-supplied through ingress.annotations; this chart sets none of it:

nginx.ingress.kubernetes.io/enable-cors:        true
nginx.ingress.kubernetes.io/cors-allow-origin:  https://<the webmail host>
nginx.ingress.kubernetes.io/cors-allow-methods: GET, POST, OPTIONS
nginx.ingress.kubernetes.io/cors-allow-headers: Authorization, Content-Type

That origin is bulwark, the webmail chart — so this is the one loss in the sweep that breaks a second chart. The CORS configuration is what allows the browser-side webmail to call this server's JMAP endpoint at all. Without it the webmail stops working, with a console error in the browser and nothing whatsoever in this server's logs.

And "no counterpart" would be the wrong thing to write here. Measured by grepping the CRDs this repository pins (ci/gateway-api-crds.sh, gateway-api v1.6.3 standard channel) rather than inferred:

  • HTTPRoute's filters carry a full cors field (GEP-1767) with allowOrigins, allowMethods, allowHeaders, allowCredentials and maxAge. Its documented level, verbatim from that CRD: Support: Extended.
  • responseHeaderModifier is in the same CRD and is also Support: Extended, not Core.

There is a counterpart, then, and this chart still does not render it. Three reasons, in order:

  1. The chart has no CORS values. The live configuration is the operator's — a specific origin, a specific method list — and the chart sets no CORS annotation in Ingress mode either. Rendering a filter means inventing a values surface that would exist in Gateway mode only, and that asymmetry is its own defect and its own piece of work.
  2. Extended support is not support. An implementation that does not support an Extended value reports it in the route's status, and a rejected route is rejected whole — which here would take the WebUI and JMAP down together (the reasoning #119 reached for statusCode). The observed failure is quieter still: Cilium accepts filters.type: CORS and forwards the browser's preflight upstream anyway (cilium/cilium#45272), i.e. Accepted: True with the behaviour absent. A chart cannot know which implementation it landed on.
  3. Half a CORS is worse than none, and the translatable half is the useless one here. A responseHeaderModifier can add the three Access-Control-Allow-* headers to a response. The preflight is a different thing: the browser sends OPTIONS first and the gateway, not the backend, has to answer it — the CORS filter's own contract says a path carrying it must not pass the preflight upstream, which a bare header modifier cannot do. And this installation's header list includes Authorization, which makes every webmail request preflighted: the half that is expressible without the CORS filter is precisely the half that does not help.

So: configure CORS at the Gateway, with that implementation's own support, before you move this host. Then test it from the webmail in a browser — curl sends no preflight, so it will pass while the webmail fails.

MTA-STS: the chart registers a route, it does not create a domain

The limit the mtaSts block has to name rather than suggest away, and it is the same genus as the OnePasswordItem limit this chart already documents: a promise the chart cannot keep.

Read the first paragraph of values.yaml again — since v0.16 the domains live in Stalwart's data store, and the chart never sees them. So whether the server answers for mta-sts.example.com depends on example.com existing in that data store, which no value here can arrange.

An entry without its domain in the data store gives a 404 at a correctly routed endpoint: every Kubernetes object green, the Ingress or route resolving perfectly, nothing wrong on this side at all. And the other half is worse — MTA-STS failures are silent on the sender's side. A sending MTA that finds no policy reports nothing here; it simply delivers in the clear. So the thing the block exists to achieve can be absent with no symptom in this cluster whatsoever.

Two more things the chart does not do, both equally silent: the _mta-sts.<domain> TXT record (without it no sending MTA ever looks for the policy — that is DNS, not Kubernetes), and the policy's content (version/mode/max_age/mx come from the data store, per domain; the running installation serves mode: testing, so sending MTAs enforce nothing).

Verify every entry from outside after adding it — NOTES.txt prints the command per host:

curl https://mta-sts.<domain>/.well-known/mta-sts.txt

MTA-STS on a subdomain: the parent's wildcard does not cover it

The trap that will actually catch somebody. mta-sts.sub.example.com is one label deeper than *.example.com, and a wildcard certificate matches exactly one label (RFC 6125) — so a wildcard for the parent domain does not cover it, and the obvious assumption ("the wildcard already covers that") is false.

A secret per entry solves it in Ingress mode, which is why tlsSecretName is per entry and has no default. The trap is documented anyway because the symptom is a TLS error at the sending MTA: nothing in this cluster logs it, the endpoint looks healthy from here, and mail keeps flowing unencrypted.

In Gateway mode the trap moves to the Gateway's listener, where this chart cannot help at all: a wildcard listener for the parent domain will not terminate TLS for a deeper name, and the route carries no certificate to compensate with.

One Ingress per cluster issuer, and why it cannot be one object

Worth knowing before the next "list of hosts with its own TLS secret" lands in some other chart, because the cause is not specific to this one.

cert-manager reads the two fields at different scopes:

  • cert-manager.io/cluster-issuer is an annotation, i.e. object-scoped. One Ingress carries exactly one, or none — and with it absent, ingress-shim creates nothing for that object, for every tls entry on it rather than for some.
  • secretName is tls-entry-scoped, and the shim creates one Certificate per tls entry.

So an object cannot mix "issued by X" with "comes from somewhere else", and an entry-level optional issuer forces a split on the issuer. Measured in jk-cluster on 2026-10-13, which is where the second half of the rule comes from:

Certificate tls-mta-sts-bekemeyer-klein-de-ingress
  secretName:      tls-mta-sts-bekemeyer-klein-de-ingress
  ownerReferences: Ingress/mta-sts-ingress

ingress-shim names the Certificate after the secret and owns it from the Ingress. Two Ingress objects whose tls entries name the same secret therefore drive the shim at one Certificate name from two owners — so a shared secret must sit in exactly one object (hence the grouping), and a secret may not span two issuer groups (hence the abort).

The objects are named by index all the same — …-mta-sts-ingress, …-mta-sts-ingress-1, … — in the repo's idiom (style guide §4), as homeassistant's routes and keycloak's realm Ingresses do it. The grouping above and the naming are independent decisions. An earlier draft of this chart named them by issuer, arguing that the index rule rested on "a route holds no state or certificate" and that these objects hold one. The reading was right and the rule's stated reason was wrong — it was the reason that got corrected, not the naming of two charts, because keycloak's realm Ingresses own certificates too and are already published. What actually carries the index rule is that the Secret survives a rename, so no certificate is re-issued; the three-step version, and the cluster flag it depends on, is in that style-guide section.

So the name does not say which issuer an object carries. NOTES.txt prints the index → issuer → hosts mapping, and it is the only place that association is visible — read it there rather than inferring it from object names.

And reordering mtaSts.hosts is sharper here than at homeassistant. There an index names a host. Here it names an issuer group derived from the list, so moving one entry can change how many objects exist and which index holds which issuer — renumbering objects that were not themselves moved. Kubernetes recreates them, which takes their Certificate objects along; nothing is re-issued, because the Secrets survive, and that holds only while the cluster leaves --enable-certificate-owner-ref unset. Measured on this cluster: it is unset. It is a property of the cluster, not of this chart.

The Ingress ban trap

The one the HA proxy does not solve. Stalwart sees the ingress controller's pod IP for every JMAP and WebUI request, not the client's, and the auto-ban eventually bans that address — at which point WebUI and JMAP are dead for everyone at once.

For HTTP the remedy is not the PROXY protocol but Stalwart's forwarded handling with X-Forwarded-For evaluation, while discarding forwarding headers the client brought itself. A mandatory step (step 4), because the outage arrives weeks later and looks like a server fault.

Gateway mode does not avoid this and does not reset it: it is one proxy address instead of another — the Gateway's data-plane pods rather than the ingress controller's — and whether the Gateway sends an X-Forwarded-For at all is that implementation's business, not something the route can state. The same applies to networkPolicy.allowedNamespaces: in Gateway mode the namespace to allow is the Gateway's data plane, which usually lives in a namespace of its own, not an ingress controller's.

Second half of the same area: after bootstrap, port 8080 is no longer the admin port. Upstream moves administration to https://<hostname>/admin once setup completes. The Ingress therefore only works while an HTTP NetworkListener on 8080 exists in the database — between finishing setup and creating that listener it resolves to a port that answers nothing.

A missing config.json opens an ephemeral store and looks healthy

If the file is absent — a failed ConfigMap mount, a renamed key — the server does not abort. It sets recovery mode, prints a random temporary admin password, and opens an in-memory EphemeralStore. Probes go green, mail is accepted, and everything is lost on the next restart.

An unparsable or unreadable file does abort. It is specifically the missing-file case that fails open, and the chart's defence is that config.json is always rendered and seeded by an init container, so its absence means something else broke.

The /healthz rate limit sets the probe intervals

Stalwart applies its anonymous HTTP rate limit to /healthz as to everything else, default 100 requests per 60 seconds keyed on the source IP. Kubelet probes arrive from the node's IP, so every Stalwart pod on one node draws from the same bucket. Exceeding it returns 429, a 429 fails the probe, and a failed liveness probe restarts the pod — a self-inflicted restart loop that worsens with pod density.

Intervals Per pod Pods per node before the limit
20/20/10 (shipped) 6 req/min ~16
10/10/5 (the obvious default) 12 req/min ~8

The startup probe gates liveness and readiness, so the three never run simultaneously and 6/min is the worst case in both phases. If a node ever needs more pods than that, raise the limit on the server (the allow-list bypasses the limiter outright for listed addresses) rather than blinding the probes.

This trap is not specific to Stalwart — any application that rate-limits by source IP meets it in Kubernetes, because probes are indistinguishable from each other at the node IP. See Lessons learned.

A tcpSocket readiness probe on the mail port would bypass the bucket, and was rejected: it would be green exactly in the fail-open case (a broken mx node binds 25 too), it proves strictly less than /healthz/ready, and it would have to differ per node group so the fallback default could no longer carry the probe's context.

Empty probeSources means open, and that is the safer default

With networkPolicy.probeSources empty the http rule renders without a from selector, so the port is reachable from anywhere the CNI routes to. Deliberate: the kubelet probes from the worker node's IP, which the chart cannot know, and a guessed CIDR that misses the real addresses blocks every probe — and a pod that never becomes Ready reads as a broken application rather than as a policy. http carries no node port, so "open" means open inside the cluster, not from the internet.

Once probeSources is non-empty the rule is restricted, which also cuts off the ingress controller unless its namespace is in networkPolicy.allowedNamespaces.

The data store username is public and cannot be otherwise

On the DataStore object, authSecret has the type SecretKeyOptional — a tagged sub-object with None, Value, EnvironmentVariable and File variants — while authUsername is a plain String with no secret form at all. So the password is referenced by variable name and never enters the ConfigMap:

"authSecret": {"@type":"EnvironmentVariable","variableName":"STALWART_DB_PASSWORD"}

but the username is rendered into it. An upstream constraint, not a choice. The upside is the reason config.json can be a real ConfigMap with a meaningful checksum/config: Helm can hash the whole file because there is no secret in it.

The recovery admin is one field, not two

STALWART_RECOVERY_ADMIN takes username:password, and the chart reads it as one vault field rather than assembling it. The upstream parser splits on the first : and drops the entry when either half is empty, so a half-filled item would silently mean no recovery admin instead of failing — and this credential is a back door that logs in regardless of directory state.

Remove it when you are done: set secrets.recoveryAdminSecretRef: null once a permanent administrator exists.

Durations are integer milliseconds, not 15s

Stalwart's object encoding is explicit: "A duration is an integer count of milliseconds, and a size is an integer count of bytes. Neither accepts a string." The DataStore reference gives the field as Duration? default: 15000, and the deserializer goes through u64 into Duration::from_millis.

A config.json carrying "15s" is rejected at startup. The chart therefore names the value timeoutMs and types it integer in the schema, so timeoutMs: 15s fails at render with got string, want integer rather than in the server's log.

Two neighbours worth knowing when writing a stalwart-cli apply plan by hand: an ordered list is a JSON object keyed by stringified positions ({"0":…,"1":…}), and a set is a JSON object mapping each member to true — neither is an array.

What a green pod proves here, and what it does not

Liveness asks /healthz/live; readiness and startup ask /healthz/ready. Liveness is on the dependency-free endpoint on purpose, so a Postgres outage cannot restart every pod in the cluster at once (§12.3).

But read from the source: /healthz/ready does not query the database. It returns 200 as soon as the data store handle is not Store::None. So readiness does not hang on the dependency the way the guideline asks, and it cannot — the endpoint upstream offers is shallow. It also does not catch the ephemeral-store case above, since that server has a handle too.

Both endpoints are unauthenticated and carry no network restriction. They do pass the configurable allowedEndpoints expression (default 200, i.e. everything allowed) whose loopback escape hatch does not cover a node IP — so an operator who later writes a restrictive expression breaks every probe.

Distribution is a promise, and the price is a Pending pod

whenUnsatisfiable is DoNotSchedule, not ScheduleAnyway, because the requirement was stated as a guarantee: full pods run on different nodes, mx pods run on different nodes. ScheduleAnyway gives no guarantee at all — with one worker at 97 % requested CPU the scheduler may put both replicas of a group on the one remaining node, entirely within the rule and with the redundancy gone.

The price is a pod that stays Pending rather than one that lands somewhere useless, and a Pending pod with no scheduling event reads like a different problem than a distribution rule. Two cases:

  • Two of three workers gone. The surviving replica keeps running; the second does not join the survivor — but only while the Node objects still exist. A NotReady node still counts as an (empty) domain, so placing another pod would raise the skew to 2. Delete the Node objects and the domains disappear with them, and the pod schedules. Same mechanism, opposite outcome, one step apart — worth knowing before a node is "temporarily" left in the API.
  • A narrowing nodeSelector or affinity. DoNotSchedule plus a node restriction below the group's replica count is a combination that blocks itself. That is intended and loud — see the next trap for why it took an explicit field to make it loud.

Where the 97 %-CPU node actually bites: at the third replica, not the second. With two replicas on three workers the distribution is 1,1,0 → skew 1 → allowed, and the tight CPU produces no Pending yet. Only from three replicas per group upwards must every worker accept one.

The replica ceiling is a rule, not a number: as many replicas per group as there are schedulable workers. The chart leaves nodeTaintsPolicy at its API default of Ignore, so every node counts as a domain — control-plane nodes included, where no Stalwart pod ever runs. Those domains hold the minimum at 0 permanently, so with maxSkew: 1 no domain may hold more than one pod. Add a worker and the ceiling rises with it; remove one and it falls. Past that point a group needs spreadAcrossNodes: false, or an own entry with ScheduleAnyway. Do not write today's count down anywhere — it would be wrong at the next cluster rebuild and nobody would come back for it.

The two node-inclusion policies have opposite defaults

Straight from the API definition, and this asymmetry is the whole reason the chart sets one of them explicitly:

  • nodeAffinityPolicy — "If this value is nil, the behavior is equivalent to the Honor policy", and Honor means only nodes matching the pod's nodeAffinity/nodeSelector are counted as domains.
  • nodeTaintsPolicy — "If this value is nil, the behavior is equivalent to the Ignore policy", i.e. all nodes are counted.

Under the Honor default, a nodeSelector narrowing a 2-replica group onto one node leaves exactly one eligible domain, both pods land in it, and the skew is 0 — the constraint is satisfied. Both replicas co-located, nothing Pending, no event, and the selector in the object is perfectly correct, so the CI selector check cannot see it either. The chart therefore sets nodeAffinityPolicy: Ignore, under which every node counts: the same narrowing gives 2,0,0, skew 2, and the second pod is Pending. The silent half becomes the loud one.

A render abort was the alternative and was rejected, because it would also have forbidden a nodeSelector spanning five nodes, which is legal and works (1,1,0,0,0 → skew 1). The field forbids only the case that is actually broken.

And the two fields are not a package. The replica ceiling above comes from nodeTaintsPolicy ≙ Ignore and existed before any of this; nodeAffinityPolicy: Ignore does not add it, it only closes the silent gap. They are argued separately in values.yaml for exactly that reason.

nodeAffinityPolicy can be dropped without a word

The field is beta — on by default from Kubernetes 1.26, graduated to GA with the feature gate NodeInclusionPolicyInPodTopologySpread in 1.33. On an older API server, or one with the gate switched off, it is pruned silently, and the gap of the previous trap comes straight back: a narrowing nodeSelector again yields co-location with no Pending and no message. The object still looks correct in kubectl get sts -o yaml, minus one line.

So 1.26 is the minimum version on which this chart can make the promise at all. Check it against the live object rather than against the render:

kubectl -n <ns> get sts -l stalwart.jk-dev.de/release=<release>-stalwart \
  -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.template.spec.topologySpreadConstraints[*].nodeAffinityPolicy}{"\n"}{end}'

An empty column where Ignore should be is the whole symptom. Not measured against a pre-1.26 server — this is the documented pruning behaviour for a field behind a disabled gate, not something tried here.

A colliding spread constraint aborts the render

Kubernetes keys a spread constraint on (topologyKey, whenUnsatisfiable), and whenUnsatisfiable defaults to DoNotSchedule — so an entry in nodes.<g>.topologySpreadConstraints on kubernetes.io/hostname without an explicit whenUnsatisfiable collides with the chart's own. The chart aborts the render naming both nodes.<g>.spreadAcrossNodes and nodes.<g>.topologySpreadConstraints, because both ways the collision would otherwise resolve are worse:

  • kubectl create rejects the duplicate with a message naming a list index rather than a value;
  • an apply-based path merges the two entries over patchMergeKey: topologyKey — silently — so one labelSelector is dropped and the result is green.

Entries on any other key (topology.kubernetes.io/zone, for instance) are appended and really do apply as an AND. To own the per-node distribution yourself, set spreadAcrossNodes: false and keep your entry — remembering that nodeAffinityPolicy: Ignore is then yours to repeat. The abort fires only while the chart's own entry is actually rendered, so it cannot trip a single-replica group.

Upgrades and backup

  • A plain app-version bump needs no action here; Renovate opens it and ci/renovate-bump-chart.sh bumps the version — since Gitea #128 at the app's own level, so an app patch is a chart patch and an app minor a chart minor (see Versioning). Chart pages carry no version numbers — Chart.yaml is the source.
  • Read UPGRADING/ for an app major. The v0.15 → v0.16 migration was offline, multi-step and irreversible without a backup; nothing suggests the next major will be gentler. For app major jumps the repo rule applies: first to the last release of the old line, then to the newest major, each in its own PR.
  • The backup is Postgres plus the S3 bucket. No PVC is involved — there is nothing in the pods worth keeping. The configuration lives in Postgres too, which means a Postgres restore also rolls back every listener, domain and role.
  • stalwart-cli snapshot is the configuration backup, and the only one that is readable and diffable. It emits an apply plan; keep one in version control after any WebUI change, or the eight steps above are the recovery procedure.
  • Scaling to zero (scaleDown: true) stops the release without uninstalling it. Mail queued on other servers will be retried for days, so a short window is survivable — but inbound connections are refused meanwhile, and some senders downgrade a repeatedly refused host.

CI

ci/stalwart/ holds test-values.yaml and fixtures.yaml: a throwaway Postgres, the dummy data store Secret, and a stub CRD for cert-manager's Certificate. It runs with bootstrap.enabled: true, which is the only mode that can come up against an empty database.

A green run does not mean a working mail server, and no install-test here could. Every listener lives in the database and is created by a human, so against an empty database there is no SMTP service to test.

What it does prove is the chart's own responsibility, and the first item is why the chart is tested rather than excluded:

  1. the container execs at all — the setcap/execve interaction above would otherwise only be caught by reading a Dockerfile;
  2. readOnlyRootFilesystem: true holds with the two image volumes plus /tmp as emptyDirs, and the init container can write config.json;
  3. the rendered config.json parses as Stalwart reads it — the store opens, so @type, the integer timeout and the authSecret indirection are all accepted, which is the one check that the password really arrives without ever being in the ConfigMap;
  4. the Secret key names match, the node ports can be allocated, the quantities are valid, and the NetworkPolicies do not cut off the probes.

The install-test renders neither the PDB nor the spread constraint. ci/stalwart/test-values.yaml runs full: 1 / mx: 0, so both objects stay below their threshold — nothing about the HA objects is exercised in a cluster, and nothing claims otherwise. What covers them instead is a render-only overlay, ci/selector-match/stalwart.yaml, which raises both groups to two replicas and sets an externalServices.valkey.host (without it the chart's own "cluster with no in-memory store" abort fires). ci/run-selector-match-test.sh then checks that each group's PDB selector and spread labelSelector match that group's own pod labels — and fails if the overlay stops producing a selector, because an overlay that quietly reaches nothing would make the check vacuum-green. See CI/CD.

No Valkey in the fixtures, deliberately. The in-memory store is a database object, so a Valkey in the namespace would be reachable and unused — making the test look as though it covered clustering while covering none of it. The test runs a single node, which needs no in-memory store. The subchart path is covered by a local render only; a known gap.

The Gateway mode is held at render level, by ci/httproute/stalwart.yaml plus ci/run-httproute-match-test.sh, which runs over every chart on every PR (CI/CD). The overlay switches this chart into Gateway mode; 4 assertions since Gitea #123 — parentRefs not empty and the backendRefs entry resolving to that node group's http Service in the same render, for the WebUI route and for the MTA-STS route. The overlay's mtaSts.hosts enters all three shapes of the entry form, so the second route is not green over a single trivial case: an entry with a clusterIssuer, one without, and one host on a subdomain. The port is held by number (ports.http.containerPort), and for this chart that is the check worth having: the NodePort Services publish a different number (ports.<name>.nodePort) than the container binds, so naming the wrong one in a route would be a plausible mistake. The one-route-not-one-per-group property is the other thing the overlay pins down, since it is exactly what a copy would get wrong.

What that does not cover, named so the gaps are not mistaken for coverage:

  • that the hostname answers. No Gateway controller runs in CI and no route reaches Accepted: True.
  • the CORS configuration — the thing this chart actually loses in that mode, and the one that breaks the bulwark webmail. It lives in the operator's ingress.annotations, the chart has no values for it, and the Gateway API's own CORS filter is Extended-support with observed implementation variance. All of that is documented in values.yaml and printed by NOTES.txt, and documentation is not something a test can assert.
  • that SMTP/IMAP still work. They never went through the Ingress and do not go through the route; the mail Certificate that serves them keeps being rendered in Gateway mode, which is this chart's distinguishing feature in the sweep, and no render test looks at whether it was left alone.
  • NOTES.txt itself. helm template does not emit it and no script under ci/ reads it, so its text carries review-only assurance.

The MTA-STS Ingress half is covered by the install-test, not by the render test. The overlay's whole job is to switch the mode off, so it never renders those Ingress objects. ci/stalwart/test-values.yaml therefore carries two mtaSts.hosts entries — one with a clusterIssuer and one without, the second a subdomain — which is the only stage that puts them in front of a real API server: that two objects differing only by the issuer suffix are both accepted, that pathType: Exact on /.well-known/mta-sts.txt is valid, and that a tls: list grouping several hosts under one secret is taken as written. Nothing issues either certificate — cert-manager is not installed in the k3d cluster, so the annotation is inert and the named Secrets never exist, which is harmless because nothing mounts them.

See also

  • Chart style guide — the conventions this chart follows and the two it deviates from
  • Lessons learned — the source-IP mechanism behind NodePort, which recurs in every chart whose application evaluates the client address
  • bulwark — a webmail client for this server: mail, calendar, contacts and files in a browser, over JMAP. A separate release pointed at this one's JMAP endpoint, and the two are only useful together. Two things worth carrying across before installing it: it is a pure client, so nothing in either chart notices the other being away — with this server gone, Bulwark is Ready and green — and its own configuration can be overridden from its volume, silently, which is the first thing its page explains
  • valkey — the chart this one takes as a subchart, and the one to install separately when it should outlive this release
  • Chart index

helm-charts wiki

How we work

How it is built

How charts are written

Charts