helm-charts/stalwart/templates/NOTES.txt

{{- $release := include "stalwart.fullname" . -}}
{{- $nodes := fromYaml (include "stalwart.nodes" .) -}}
{{- $ports := fromYaml (include "stalwart.ports" .) -}}
{{- $valkeyHost := include "stalwart.valkeyHost" . -}}
{{- $ingressGroup := "" -}}
{{- if .Values.ingress.enabled }}{{- $ingressGroup = include "stalwart.ingressNodeGroup" . }}{{- end -}}
Stalwart {{ .Chart.AppVersion }} — release {{ .Release.Name }} in namespace {{ .Release.Namespace }}.

THIS CHART CONFIGURES ALMOST NOTHING, AND THAT IS NOT A GAP
  Since v0.16 Stalwart keeps its whole configuration inside the data store and
  manages it over JMAP. The only file on disk is config.json, which names the
  data store and nothing else. So this release has given you the workloads, the
  Services, the NetworkPolicies, the Ingress and a certificate — but no
  domains, no listeners, no blob store, no in-memory store and no cluster
  roles. Those are the steps below, and without them this is not yet a mail
  server.

NODE GROUPS IN THIS RELEASE
{{- range $group, $spec := $nodes }}
{{- $replicas := include "stalwart.replicas" (dict "scaleDown" $.Values.scaleDown "replicas" $spec.replicas) }}
  {{ printf "%-10s" $group }} replicas {{ $replicas }}, reachable on: {{ $spec.exposedPorts | join ", " }}
{{- end }}
{{- if .Values.scaleDown }}

  scaleDown: true — every group is at replicas 0 and nothing is running. The
  Services, Ingress, Secrets and the Certificate stay in place, so setting it
  back to false brings the release up without re-issuing the certificate.
{{- end }}
{{- if .Values.bootstrap.enabled }}

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
RECOVERY MODE IS ON — THIS RELEASE SERVES NO MAIL
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
  bootstrap.enabled is true, so every pod runs with STALWART_RECOVERY_MODE=1:
  the management HTTP endpoint on {{ (index $ports "http").containerPort }} is the ONLY thing listening and
  every mail port is closed. The Services and the Ingress look exactly as
  healthy as they do in normal operation, which is why this banner exists.

  This is correct for a first install (step 1 below). When the database is
  provisioned, set bootstrap.enabled: false and upgrade again.
{{- end }}

THE 1PASSWORD ITEMS MUST CARRY THE FIELDS THIS CHART READS
  One Secret key is created per FIELD of the item, and these templates
  reference the keys by name — an item missing a field leaves the container in
  CreateContainerConfigError. An EMPTY field fails the same way as a missing
  one: Stalwart's secret resolver treats an empty environment variable as "not
  found".
{{- if .Values.secrets.dataStoreSecretRef }}

  secrets.dataStoreSecretRef → {{ .Values.secrets.dataStoreSecretRef }}
    field:  password          (the data store password)
    The USERNAME is not in this item and cannot be: authUsername is a plain
    String on the DataStore object with no secret form. It is
    dataStore.postgres.username = "{{ .Values.dataStore.postgres.username }}" and is rendered into the
    ConfigMap.
{{- else }}

  secrets.dataStoreSecretRef is NOT SET, so no data store password reaches the
  pods and config.json references an environment variable that will not exist.
  Stalwart fails to open the store with
    Environment variable '{{ include "stalwart.env.dbPassword" . }}' not found
  Set it unless the database genuinely takes no password.
{{- end }}
{{- if .Values.secrets.recoveryAdminSecretRef }}

  secrets.recoveryAdminSecretRef → {{ .Values.secrets.recoveryAdminSecretRef }}
    field:  recovery-admin    (the COMPLETE value "admin:<password>")
    One field, not two. The upstream parser splits on the first ":" and drops
    the entry when either half is empty, so a half-filled item would silently
    mean "no recovery admin" instead of failing — and this credential logs in
    regardless of directory state.

    REMOVE IT WHEN YOU ARE DONE. It is a back door. Once a permanent
    administrator exists in the database, set secrets.recoveryAdminSecretRef
    to null and upgrade; the variable then disappears from the pods.
{{- else }}

  secrets.recoveryAdminSecretRef is NOT SET. That is the right end state once a
  permanent administrator exists — but during a first install or a recovery
  start it means Stalwart generates a temporary administrator password and
  prints it to STDOUT, i.e. into the pod log, freshly on every restart and
  visible to anyone who can read logs. Catch it with
    kubectl -n {{ .Release.Namespace }} logs sts/{{ $release }}-{{ $ingressGroup | default "<group>" }}-sfs | grep -A3 bootstrap
{{- end }}
{{- if .Values.secrets.s3SecretRef }}

  secrets.s3SecretRef → {{ .Values.secrets.s3SecretRef }}
    fields: access-key, secret-key
    Exposed to the containers as {{ include "stalwart.env.s3AccessKey" . }} and
    {{ include "stalwart.env.s3SecretKey" . }}. The chart does NOT configure the blob
    store; point the BlobStore object in the WebUI at these names:
      accessKey: {"@type":"EnvironmentVariable","variableName":"{{ include "stalwart.env.s3AccessKey" . }}"}
      secretKey: {"@type":"EnvironmentVariable","variableName":"{{ include "stalwart.env.s3SecretKey" . }}"}
{{- end }}

  CHANGING WHAT IS INSIDE AN ITEM RESTARTS NOTHING. Helm never sees a secret's
  content, so there is nothing to hash. Repointing one of the values above at a
  different vault item DOES roll the pods (checksum/secret-ref covers the
  op:// path). After rotating a password inside an item:
{{- range $group, $spec := $nodes }}
    kubectl -n {{ $.Release.Namespace }} rollout restart statefulset/{{ $release }}-{{ $group }}-sfs
{{- end }}

FIRST INSTALL — EIGHT STEPS, AND NONE OF THEM IS OPTIONAL
  1. bootstrap.enabled: true, nodes.mx.replicas: 0.
     The setup wizard everyone reaches for appears only when config.json is
     ABSENT, and this chart always renders one — so the wizard never runs here.
     Recovery mode is the documented path for a templated config.json: it
     opens the management endpoint and keeps the mail ports shut. With a second
     group running, the wizard would be unusable anyway: each request would
     land on a random pod.

  2. Open the management UI and create the permanent administrator.
     kubectl -n {{ .Release.Namespace }} port-forward svc/{{ $release }}-{{ $ingressGroup | default "<group>" }}-http-service {{ (index $ports "http").containerPort }}:{{ (index $ports "http").containerPort }}
     then http://localhost:{{ (index $ports "http").containerPort }}/admin

  3. In the WebUI, create: the blob store (S3), the in-memory store
     ({{ if $valkeyHost }}{{ $valkeyHost }}:{{ .Values.externalServices.valkey.port }}{{ else }}no address configured — see externalServices.valkey{{ end }}),
     the coordinator (Default — it reuses the in-memory store connection), the
     search store (Default — it uses the data store, so no extra service is
     needed), the TLS object pointing at the certificate from step 5, the
     NetworkListener objects, and one ClusterRole per node group below.

     THE LISTENER PORTS MUST MATCH THIS RELEASE EXACTLY. The chart cannot
     create listener objects, so these numbers are a second copy of a decision
     made in the WebUI, and a mismatch is silent — the Service points at a port
     nothing listens on, every pod stays Ready, nothing reports it:
{{- range $portName, $port := $ports }}
       {{ printf "%-12s" $portName }} bind {{ printf "%-5s" (printf "%v" $port.containerPort) }}{{ if $port.public }}  (published on node port {{ $port.nodePort }}){{ else }}  (not published){{ end }}
{{- end }}

     AND ONE ClusterRole PER GROUP, NAMED EXACTLY:
{{- range $group, $spec := $nodes }}
       {{ $group }}
{{- end }}
     A node whose role name is not in the database does not fail. It becomes a
     FULL node — every listener and every task — while staying Running with a
     green readiness probe. The only trace is one log line,
     'Cluster role "<name>" not found in registry', and it is printed AFTER the
     listeners are already bound.

  4. Configure HTTP forwarding before the auto-ban notices the Ingress.
     Stalwart sees the ingress controller's pod IP for every WebUI and JMAP
     request, not the client's. Sooner or later the auto-ban bans that one
     address and the WebUI and JMAP go dark for everyone at once. The PROXY
     protocol is not the fix for HTTP: enable "forwarded" handling with
     X-Forwarded-For evaluation, and discard forwarding headers the client
     brought itself. The outage arrives weeks later and looks like a server
     fault, which is why this is a step and not a tip.

  5. Point the TLS object at the mounted certificate.
{{- if .Values.certificate.enabled }}
       certificate: {{ .Values.certificate.mountPath }}/tls.crt
       privateKey:  {{ .Values.certificate.mountPath }}/tls.key
     cert-manager writes these into tls-{{ $release }}-mail. Watch it with
       kubectl -n {{ .Release.Namespace }} get certificate {{ $release }}-mail-certificate
     Until it is issued the mail ports serve a self-signed certificate: the
     volume is mounted optional so that a pending challenge does not hold every
     pod in ContainerCreating.
{{- else }}
     certificate.enabled is false, so this chart issues no mail certificate.
     The Ingress certificate does NOT cover the mail ports — the HA proxy
     terminates no TLS, so Stalwart has to present one itself for STARTTLS on
     25 and implicit TLS on 465/993. Provide it another way or clients will see
     a name mismatch.
{{- end }}

  6. Set send-proxy-v2 on the HA proxy. ALL OR NOTHING WITH STEP 3.
     externalTrafficPolicy is {{ .Values.service.externalTrafficPolicy }}, which it has to be — no Stalwart
     pod runs on a control-plane node, and the HA proxy delivers to the
     control-plane nodes, so Local would drop every packet. The cost is that
     kube-proxy masquerades the source address, and on a mail server a lost
     source address switches off SPF, takes DMARC with it, makes every
     blocklist report "clean", collapses per-IP rate limits onto one address,
     removes greylisting's key, and makes the auto-ban lock out every sender at
     once. The Received: chain becomes worthless — the one item there that
     destroys evidence retroactively rather than just failing.

     The PROXY v2 header restores it, for port 25 as well, because the HA proxy
     opens its own connection to the node. But a trusted network makes the
     header MANDATORY: without it a connection from there hangs until timeout,
     and with it from an untrusted network Stalwart reads "PROXY TCP4 ..." as
     an SMTP command. Test on a spare port with its own frontend, and move 25
     only once a Received: line shows the real sender address.

  7. Restart the pods — "a newly added listener binds only on the next start" —
     and set bootstrap.enabled: false in the same round.

  8. Raise nodes.mx.replicas only after the "mx" ClusterRole exists (step 3),
     and publish DNS (below).

DNS A MAIL SERVER CANNOT RUN WITHOUT
  None of this is rendered by the chart; all of it is required.
    MX          {{ .Values.ingress.domain | default "<your mail domain>" }}.            →  this deployment's public name
    SPF         TXT "v=spf1 ..." naming the HA proxy's address as a sender
    DKIM        TXT at <selector>._domainkey — the key is generated IN Stalwart
                (Domains → DKIM), so this record is published after step 3
    DMARC       TXT at _dmarc — start at p=none with rua= and tighten only once
                the reports are clean
    REVERSE DNS PTR for the HA PROXY's public address, matching its forward
                name. Not the node's and not the pod's: the HA proxy is what
                other mail servers see connecting. Many receivers reject mail
                outright when it is missing, and nothing in Kubernetes can set
                it — it belongs to whoever owns the IP range.

WHAT A GREEN POD PROVES HERE, AND WHAT IT DOES NOT
  Liveness asks /healthz/live, readiness and startup ask /healthz/ready.
  Read off the source at v0.16.24: /healthz/ready returns 200 as soon as a data
  store HANDLE exists — it does not query the database. So Ready does not mean
  the database is reachable, and it does not catch a pod that fell back to an
  ephemeral in-memory store because config.json was missing. A green pod here
  means the process is answering HTTP.

  The probe intervals are calculated, not chosen: Stalwart rate-limits
  anonymous /healthz requests to 100 per 60s PER SOURCE IP, probes arrive from
  the node IP, and so every pod on a node shares one bucket. The defaults cost
  6 requests/min/pod and leave room for about 16 pods per node. Tightening them
  produces 429s, and a 429 on liveness is a restart loop.
{{- $anySpread := false }}
{{- range $group, $spec := $nodes }}
{{- $r := int (include "stalwart.replicas" (dict "scaleDown" $.Values.scaleDown "replicas" $spec.replicas)) }}
{{- $s := fromYaml (include "stalwart.groupSettings" (dict "group" $group "given" $spec)) }}
{{- if and $s.spreadAcrossNodes (ge $r 2) }}{{ $anySpread = true }}{{ end }}
{{- end }}

DISTRIBUTION AND DISRUPTION BUDGET — WHAT EACH GROUP ACTUALLY GOT
  Both objects are conditional, so this is the list rather than something to
  infer from `kubectl get pdb` and an `-o yaml`.
{{- range $group, $spec := $nodes }}
{{- $r := int (include "stalwart.replicas" (dict "scaleDown" $.Values.scaleDown "replicas" $spec.replicas)) }}
{{- $s := fromYaml (include "stalwart.groupSettings" (dict "group" $group "given" $spec)) }}

  {{ $group }} — {{ $r }} effective replica(s)
{{- if and $s.spreadAcrossNodes (ge $r 2) }}
    SPREAD   ENFORCED. maxSkew 1 on kubernetes.io/hostname, whenUnsatisfiable
             DoNotSchedule, nodeAffinityPolicy Ignore, selecting this group's
             own pods only — a pod of "{{ $group }}" sharing a node with a pod of
             another group stays fine. This is a PROMISE, so the price is a Pending pod
             rather than a co-located one: if fewer nodes are available than
             this group has replicas, the surplus replica does not schedule and
             there is NO scheduling event to read. Check with
               kubectl -n {{ $.Release.Namespace }} get pod -l stalwart.jk-dev.de/node-group={{ $group }} -o wide
{{- else if $s.spreadAcrossNodes }}
    SPREAD   NOT RENDERED, because this group has {{ $r }} effective replica(s) and the
             rule needs two. {{ if eq $r 0 }}With no pod at all there is nothing to
             distribute{{ else }}With one pod every placement satisfies it, so the object
             would describe nothing{{ end }}. Raise nodes.{{ $group }}.replicas to 2 to get it —
             and note that doing so changes the POD TEMPLATE, so {{ if eq $r 0 }}the group
             starts with the constraint already in place{{ else }}the pod running now is
             replaced in the same rollout round, not merely joined{{ end }}.
{{- else }}
    SPREAD   OFF — nodes.{{ $group }}.spreadAcrossNodes is false. Nothing keeps two
             pods of this group off the same node, so its replicas are
             redundancy on paper only unless your own
             nodes.{{ $group }}.topologySpreadConstraints or affinity does the job.
{{- end }}
{{- if and $s.podDisruptionBudget.enabled (ge $r 2) }}
    PDB      {{ $release }}-{{ $group }}-pdb, minAvailable {{ $s.podDisruptionBudget.minAvailable | int64 }}. A node drain cannot take
             all {{ $r }} replicas of this group at once.
{{- else if $s.podDisruptionBudget.enabled }}
    PDB      NONE, because the effective replica count is {{ $r }} and a budget needs
             two. Deliberate: {{ if eq $r 0 }}with no pod at all the object would only
             describe a promise about nothing{{ else }}minAvailable 1 with a single replica
             forbids every voluntary eviction, so `kubectl drain` would block
             forever on a pod nothing may move — and the error would name the
             PDB rather than the replica count{{ end }}.
{{- else }}
    PDB      NONE — nodes.{{ $group }}.podDisruptionBudget.enabled is false. A node
             drain may take every replica of this group at once.
{{- end }}
{{- end }}
{{- if $anySpread }}

  ONE VERSION CAVEAT, AND IT FAILS SILENTLY: nodeAffinityPolicy is a beta field
  — on by default from Kubernetes 1.26, GA with the feature gate
  NodeInclusionPolicyInPodTopologySpread in 1.33. An older API server, or one
  with the gate off, PRUNES IT WITHOUT A WORD, and then a nodeSelector or
  affinity that narrows this group onto a single node no longer produces a
  Pending pod but two CO-LOCATED ones, with the constraint reported as
  satisfied. Compare the live object against what was rendered:
    kubectl -n {{ .Release.Namespace }} get sts -l stalwart.jk-dev.de/release={{ $release }} \
      -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.template.spec.topologySpreadConstraints[*].nodeAffinityPolicy}{"\n"}{end}'
  An empty column where "Ignore" should be is the whole symptom.

  AND A CEILING THAT IS A RULE, NOT A NUMBER: this chart leaves
  nodeTaintsPolicy at its API default of Ignore, so every node in the cluster
  counts as a domain — control-plane nodes included, where no Stalwart pod ever
  runs. They hold the minimum at 0 permanently, so with maxSkew 1 a group can
  never place more replicas than there are SCHEDULABLE WORKERS. It moves with
  the cluster: add a worker and it rises, remove one and it falls. Past that
  point a group needs spreadAcrossNodes: false or an own constraint with
  ScheduleAnyway. This ceiling comes from nodeTaintsPolicy and existed before
  nodeAffinityPolicy was set — the two fields are not a package.
{{- end }}
{{- if .Values.ingress.enabled }}

WEB UI AND JMAP
  https://{{ .Values.ingress.domain }}  →  node group "{{ $ingressGroup }}"
  Certificate requested by cert-manager through the "{{ .Values.ingress.clusterIssuer }}" cluster issuer:
    kubectl -n {{ .Release.Namespace }} get certificate tls-{{ $release }}-ingress

  AFTER BOOTSTRAP, PORT {{ (index $ports "http").containerPort }} IS NOT THE ADMIN PORT ANY MORE. Upstream moves
  administration to https://<hostname>/admin once setup is complete, so this
  Ingress only works while an HTTP NetworkListener on {{ (index $ports "http").containerPort }} exists in the
  database (step 3). Between finishing setup and creating that listener the
  Ingress resolves to a port that answers nothing.
{{- else }}

NO INGRESS
  ingress.enabled is false, so there is no Ingress and no WebUI certificate.
  ingress.domain STAYS REQUIRED even so — the mail Certificate reads it for its
  dnsNames, and without it the certificate would carry no name at all.
{{- end }}
{{- if .Values.valkeySubchart.enabled }}

THE VALKEY SUBCHART IS ENABLED
  Reachable at {{ $valkeyHost }}:{{ .Values.externalServices.valkey.port }} — this repo's own valkey chart,
  pulled from the published index. Enter that address as the in-memory store in
  step 3; the coordinator's Default variant reuses the same connection.

  ITS 1PASSWORD ITEM IS A SEPARATE ONE, AND ITS FIELD IS NOT A PASSWORD:
    valkey.secrets.authSecretRef points at
      {{ .Values.valkey.secrets.authSecretRef | default "<NOT SET - the subchart's own required() will abort the render>" }}
    and that item carries ONE field, "auth-conf", whose value is the complete
    configuration line

      requirepass <password>

    not a bare password. The valkey chart mounts the file and includes it, and
    an init container refuses to start the pod unless the line is really there —
    because the alternative failure is an open server that looks healthy.

  IT IS CONFIGURED AS A CACHE, NOT A QUEUE
    persistence.enabled false + maxmemoryPolicy allkeys-lru. Everything Stalwart
    keeps here — rate-limiter counters, auth tokens, sessions, auto-ban entries,
    the coordination pub/sub — is reconstructible after a restart; nothing in it
    is the system of record.

    The cost of allkeys-lru, so it is not a surprise: under memory pressure keys
    are discarded, so the rate limiters and the auto-ban FAIL OPEN. Size memory
    for the working set rather than switching to noeviction — with noeviction
    Stalwart's writes to the store start erroring, which takes out
    authentication instead of just the limits.

  scaleDown and checksum/config stop at the subchart boundary, so scaling this
  release to zero does NOT stop the Valkey, and a new valkey chart release
  reaches you only when stalwart itself is republished.
{{- end }}
{{- if not .Values.networkPolicy.enabled }}

NETWORKPOLICY IS OFF
  networkPolicy.enabled is false, so nothing limits reachability except the
  Services. That removes the second line of defence against the fail-open in
  step 3: a group that accidentally became a full node is then reachable on
  every port it bound.
{{- else if not .Values.networkPolicy.probeSources }}

NETWORKPOLICY: THE HTTP PORT IS OPEN INSIDE THE CLUSTER
  networkPolicy.probeSources is empty, so the rule for port {{ (index $ports "http").containerPort }} renders with
  no "from" selector. Deliberate: the kubelet probes from the worker node's IP,
  which this chart cannot know, and a wrong guess blocks every probe — a pod
  that never becomes Ready reads as a broken application rather than as a
  policy. The port carries no nodePort, so this is open inside the cluster, not
  from outside. Narrow it with networkPolicy.probeSources, and then add the
  ingress controller's namespace to networkPolicy.allowedNamespaces or the
  Ingress goes dark.
{{- end }}