{{- $release := include "stalwart.fullname" . -}} {{- $nodes := fromYaml (include "stalwart.nodes" .) -}} {{- $ports := fromYaml (include "stalwart.ports" .) -}} {{- $valkeyHost := include "stalwart.valkeyHost" . -}} {{- $ingressGroup := "" -}} {{- if .Values.ingress.enabled }}{{- $ingressGroup = include "stalwart.ingressNodeGroup" . }}{{- end -}} Stalwart {{ .Chart.AppVersion }} — release {{ .Release.Name }} in namespace {{ .Release.Namespace }}. THIS CHART CONFIGURES ALMOST NOTHING, AND THAT IS NOT A GAP Since v0.16 Stalwart keeps its whole configuration inside the data store and manages it over JMAP. The only file on disk is config.json, which names the data store and nothing else. So this release has given you the workloads, the Services, the NetworkPolicies, the Ingress and a certificate — but no domains, no listeners, no blob store, no in-memory store and no cluster roles. Those are the steps below, and without them this is not yet a mail server. NODE GROUPS IN THIS RELEASE {{- range $group, $spec := $nodes }} {{- $replicas := include "stalwart.replicas" (dict "scaleDown" $.Values.scaleDown "replicas" $spec.replicas) }} {{ printf "%-10s" $group }} replicas {{ $replicas }}, reachable on: {{ $spec.exposedPorts | join ", " }} {{- end }} {{- if .Values.scaleDown }} scaleDown: true — every group is at replicas 0 and nothing is running. The Services, Ingress, Secrets and the Certificate stay in place, so setting it back to false brings the release up without re-issuing the certificate. {{- end }} {{- if .Values.bootstrap.enabled }} !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! RECOVERY MODE IS ON — THIS RELEASE SERVES NO MAIL !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! bootstrap.enabled is true, so every pod runs with STALWART_RECOVERY_MODE=1: the management HTTP endpoint on {{ (index $ports "http").containerPort }} is the ONLY thing listening and every mail port is closed. The Services and the Ingress look exactly as healthy as they do in normal operation, which is why this banner exists. This is correct for a first install (step 1 below). When the database is provisioned, set bootstrap.enabled: false and upgrade again. {{- end }} THE 1PASSWORD ITEMS MUST CARRY THE FIELDS THIS CHART READS One Secret key is created per FIELD of the item, and these templates reference the keys by name — an item missing a field leaves the container in CreateContainerConfigError. An EMPTY field fails the same way as a missing one: Stalwart's secret resolver treats an empty environment variable as "not found". {{- if .Values.secrets.dataStoreSecretRef }} secrets.dataStoreSecretRef → {{ .Values.secrets.dataStoreSecretRef }} field: password (the data store password) The USERNAME is not in this item and cannot be: authUsername is a plain String on the DataStore object with no secret form. It is dataStore.postgres.username = "{{ .Values.dataStore.postgres.username }}" and is rendered into the ConfigMap. {{- else }} secrets.dataStoreSecretRef is NOT SET, so no data store password reaches the pods and config.json references an environment variable that will not exist. Stalwart fails to open the store with Environment variable '{{ include "stalwart.env.dbPassword" . }}' not found Set it unless the database genuinely takes no password. {{- end }} {{- if .Values.secrets.recoveryAdminSecretRef }} secrets.recoveryAdminSecretRef → {{ .Values.secrets.recoveryAdminSecretRef }} field: recovery-admin (the COMPLETE value "admin:") One field, not two. The upstream parser splits on the first ":" and drops the entry when either half is empty, so a half-filled item would silently mean "no recovery admin" instead of failing — and this credential logs in regardless of directory state. REMOVE IT WHEN YOU ARE DONE. It is a back door. Once a permanent administrator exists in the database, set secrets.recoveryAdminSecretRef to null and upgrade; the variable then disappears from the pods. {{- else }} secrets.recoveryAdminSecretRef is NOT SET. That is the right end state once a permanent administrator exists — but during a first install or a recovery start it means Stalwart generates a temporary administrator password and prints it to STDOUT, i.e. into the pod log, freshly on every restart and visible to anyone who can read logs. Catch it with kubectl -n {{ .Release.Namespace }} logs sts/{{ $release }}-{{ $ingressGroup | default "" }}-sfs | grep -A3 bootstrap {{- end }} {{- if .Values.secrets.s3SecretRef }} secrets.s3SecretRef → {{ .Values.secrets.s3SecretRef }} fields: access-key, secret-key Exposed to the containers as {{ include "stalwart.env.s3AccessKey" . }} and {{ include "stalwart.env.s3SecretKey" . }}. The chart does NOT configure the blob store; point the BlobStore object in the WebUI at these names: accessKey: {"@type":"EnvironmentVariable","variableName":"{{ include "stalwart.env.s3AccessKey" . }}"} secretKey: {"@type":"EnvironmentVariable","variableName":"{{ include "stalwart.env.s3SecretKey" . }}"} {{- end }} CHANGING WHAT IS INSIDE AN ITEM RESTARTS NOTHING. Helm never sees a secret's content, so there is nothing to hash. Repointing one of the values above at a different vault item DOES roll the pods (checksum/secret-ref covers the op:// path). After rotating a password inside an item: {{- range $group, $spec := $nodes }} kubectl -n {{ $.Release.Namespace }} rollout restart statefulset/{{ $release }}-{{ $group }}-sfs {{- end }} FIRST INSTALL — EIGHT STEPS, AND NONE OF THEM IS OPTIONAL 1. bootstrap.enabled: true, nodes.mx.replicas: 0. The setup wizard everyone reaches for appears only when config.json is ABSENT, and this chart always renders one — so the wizard never runs here. Recovery mode is the documented path for a templated config.json: it opens the management endpoint and keeps the mail ports shut. With a second group running, the wizard would be unusable anyway: each request would land on a random pod. 2. Open the management UI and create the permanent administrator. kubectl -n {{ .Release.Namespace }} port-forward svc/{{ $release }}-{{ $ingressGroup | default "" }}-http-service {{ (index $ports "http").containerPort }}:{{ (index $ports "http").containerPort }} then http://localhost:{{ (index $ports "http").containerPort }}/admin 3. In the WebUI, create: the blob store (S3), the in-memory store ({{ if $valkeyHost }}{{ $valkeyHost }}:{{ .Values.externalServices.valkey.port }}{{ else }}no address configured — see externalServices.valkey{{ end }}), the coordinator (Default — it reuses the in-memory store connection), the search store (Default — it uses the data store, so no extra service is needed), the TLS object pointing at the certificate from step 5, the NetworkListener objects, and one ClusterRole per node group below. THE LISTENER PORTS MUST MATCH THIS RELEASE EXACTLY. The chart cannot create listener objects, so these numbers are a second copy of a decision made in the WebUI, and a mismatch is silent — the Service points at a port nothing listens on, every pod stays Ready, nothing reports it: {{- range $portName, $port := $ports }} {{ printf "%-12s" $portName }} bind {{ printf "%-5s" (printf "%v" $port.containerPort) }}{{ if $port.public }} (published on node port {{ $port.nodePort }}){{ else }} (not published){{ end }} {{- end }} AND ONE ClusterRole PER GROUP, NAMED EXACTLY: {{- range $group, $spec := $nodes }} {{ $group }} {{- end }} A node whose role name is not in the database does not fail. It becomes a FULL node — every listener and every task — while staying Running with a green readiness probe. The only trace is one log line, 'Cluster role "" not found in registry', and it is printed AFTER the listeners are already bound. 4. Configure HTTP forwarding before the auto-ban notices the Ingress. Stalwart sees the ingress controller's pod IP for every WebUI and JMAP request, not the client's. Sooner or later the auto-ban bans that one address and the WebUI and JMAP go dark for everyone at once. The PROXY protocol is not the fix for HTTP: enable "forwarded" handling with X-Forwarded-For evaluation, and discard forwarding headers the client brought itself. The outage arrives weeks later and looks like a server fault, which is why this is a step and not a tip. 5. Point the TLS object at the mounted certificate. {{- if .Values.certificate.enabled }} certificate: {{ .Values.certificate.mountPath }}/tls.crt privateKey: {{ .Values.certificate.mountPath }}/tls.key cert-manager writes these into tls-{{ $release }}-mail. Watch it with kubectl -n {{ .Release.Namespace }} get certificate {{ $release }}-mail-certificate Until it is issued the mail ports serve a self-signed certificate: the volume is mounted optional so that a pending challenge does not hold every pod in ContainerCreating. {{- else }} certificate.enabled is false, so this chart issues no mail certificate. The Ingress certificate does NOT cover the mail ports — the HA proxy terminates no TLS, so Stalwart has to present one itself for STARTTLS on 25 and implicit TLS on 465/993. Provide it another way or clients will see a name mismatch. {{- end }} 6. Set send-proxy-v2 on the HA proxy. ALL OR NOTHING WITH STEP 3. externalTrafficPolicy is {{ .Values.service.externalTrafficPolicy }}, which it has to be — no Stalwart pod runs on a control-plane node, and the HA proxy delivers to the control-plane nodes, so Local would drop every packet. The cost is that kube-proxy masquerades the source address, and on a mail server a lost source address switches off SPF, takes DMARC with it, makes every blocklist report "clean", collapses per-IP rate limits onto one address, removes greylisting's key, and makes the auto-ban lock out every sender at once. The Received: chain becomes worthless — the one item there that destroys evidence retroactively rather than just failing. The PROXY v2 header restores it, for port 25 as well, because the HA proxy opens its own connection to the node. But a trusted network makes the header MANDATORY: without it a connection from there hangs until timeout, and with it from an untrusted network Stalwart reads "PROXY TCP4 ..." as an SMTP command. Test on a spare port with its own frontend, and move 25 only once a Received: line shows the real sender address. 7. Restart the pods — "a newly added listener binds only on the next start" — and set bootstrap.enabled: false in the same round. 8. Raise nodes.mx.replicas only after the "mx" ClusterRole exists (step 3), and publish DNS (below). DNS A MAIL SERVER CANNOT RUN WITHOUT None of this is rendered by the chart; all of it is required. MX {{ .Values.ingress.domain | default "" }}. → this deployment's public name SPF TXT "v=spf1 ..." naming the HA proxy's address as a sender DKIM TXT at ._domainkey — the key is generated IN Stalwart (Domains → DKIM), so this record is published after step 3 DMARC TXT at _dmarc — start at p=none with rua= and tighten only once the reports are clean REVERSE DNS PTR for the HA PROXY's public address, matching its forward name. Not the node's and not the pod's: the HA proxy is what other mail servers see connecting. Many receivers reject mail outright when it is missing, and nothing in Kubernetes can set it — it belongs to whoever owns the IP range. WHAT A GREEN POD PROVES HERE, AND WHAT IT DOES NOT Liveness asks /healthz/live, readiness and startup ask /healthz/ready. Read off the source at v0.16.24: /healthz/ready returns 200 as soon as a data store HANDLE exists — it does not query the database. So Ready does not mean the database is reachable, and it does not catch a pod that fell back to an ephemeral in-memory store because config.json was missing. A green pod here means the process is answering HTTP. The probe intervals are calculated, not chosen: Stalwart rate-limits anonymous /healthz requests to 100 per 60s PER SOURCE IP, probes arrive from the node IP, and so every pod on a node shares one bucket. The defaults cost 6 requests/min/pod and leave room for about 16 pods per node. Tightening them produces 429s, and a 429 on liveness is a restart loop. {{- if .Values.ingress.enabled }} WEB UI AND JMAP https://{{ .Values.ingress.domain }} → node group "{{ $ingressGroup }}" Certificate requested by cert-manager through the "{{ .Values.ingress.clusterIssuer }}" cluster issuer: kubectl -n {{ .Release.Namespace }} get certificate tls-{{ $release }}-ingress AFTER BOOTSTRAP, PORT {{ (index $ports "http").containerPort }} IS NOT THE ADMIN PORT ANY MORE. Upstream moves administration to https:///admin once setup is complete, so this Ingress only works while an HTTP NetworkListener on {{ (index $ports "http").containerPort }} exists in the database (step 3). Between finishing setup and creating that listener the Ingress resolves to a port that answers nothing. {{- else }} NO INGRESS ingress.enabled is false, so there is no Ingress and no WebUI certificate. ingress.domain STAYS REQUIRED even so — the mail Certificate reads it for its dnsNames, and without it the certificate would carry no name at all. {{- end }} {{- if .Values.valkeySubchart.enabled }} THE VALKEY SUBCHART IS ENABLED Reachable at {{ $valkeyHost }}:{{ .Values.externalServices.valkey.port }} — this repo's own valkey chart, pulled from the published index. Enter that address as the in-memory store in step 3; the coordinator's Default variant reuses the same connection. ITS 1PASSWORD ITEM IS A SEPARATE ONE, AND ITS FIELD IS NOT A PASSWORD: valkey.secrets.authSecretRef points at {{ .Values.valkey.secrets.authSecretRef | default "" }} and that item carries ONE field, "auth-conf", whose value is the complete configuration line requirepass not a bare password. The valkey chart mounts the file and includes it, and an init container refuses to start the pod unless the line is really there — because the alternative failure is an open server that looks healthy. IT IS CONFIGURED AS A CACHE, NOT A QUEUE persistence.enabled false + maxmemoryPolicy allkeys-lru. Everything Stalwart keeps here — rate-limiter counters, auth tokens, sessions, auto-ban entries, the coordination pub/sub — is reconstructible after a restart; nothing in it is the system of record. The cost of allkeys-lru, so it is not a surprise: under memory pressure keys are discarded, so the rate limiters and the auto-ban FAIL OPEN. Size memory for the working set rather than switching to noeviction — with noeviction Stalwart's writes to the store start erroring, which takes out authentication instead of just the limits. scaleDown and checksum/config stop at the subchart boundary, so scaling this release to zero does NOT stop the Valkey, and a new valkey chart release reaches you only when stalwart itself is republished. {{- end }} {{- if not .Values.networkPolicy.enabled }} NETWORKPOLICY IS OFF networkPolicy.enabled is false, so nothing limits reachability except the Services. That removes the second line of defence against the fail-open in step 3: a group that accidentally became a full node is then reachable on every port it bound. {{- else if not .Values.networkPolicy.probeSources }} NETWORKPOLICY: THE HTTP PORT IS OPEN INSIDE THE CLUSTER networkPolicy.probeSources is empty, so the rule for port {{ (index $ports "http").containerPort }} renders with no "from" selector. Deliberate: the kubelet probes from the worker node's IP, which this chart cannot know, and a wrong guess blocks every probe — a pod that never becomes Ready reads as a broken application rather than as a policy. The port carries no nodePort, so this is open inside the cluster, not from outside. Narrow it with networkPolicy.probeSources, and then add the ingress controller's namespace to networkPolicy.allowedNamespaces or the Ingress goes dark. {{- end }}