From 079639225c060c86de06886e29dc84d0d41bcc35 Mon Sep 17 00:00:00 2001 From: Justin Klein Date: Fri, 2 Oct 2026 19:24:15 +0000 Subject: [PATCH] stalwart: new chart page, index, sidebar and four lessons The chart page carries the full values reference, the eight-step first install (eight, not seven: the bootstrap wizard appears only when config.json is ABSENT, and the chart always renders one, so a first install goes through recovery mode) and a Traps section. Four entries on Lessons-Learned, because none of them is specific to mail: - The source address every NodePort quietly replaces. externalTrafficPolicy Cluster masquerades the client address onto the node IP, and every application that evaluates the client address meets this - bans, rate limits, geo rules, audit logs. Includes the three traps of the PROXY header remedy and the reason the trusted set and the probe set have to be disjoint in Kubernetes. - The rate limit that counted a node instead of a client. A per-IP limit in an application is a per-node limit for probes; the arithmetic that picks the probe intervals, and why a tcpSocket probe is the wrong escape. - The hardening that stopped the container from starting. setcap +ep makes the binary capability-dumb, so drop: [ALL] fails the execve itself. The lesson is that a deviation has to be documented with its mechanism, or the next hardening pass removes it as unused. - The missing file that opened a database nobody meant to use. An absent config.json opens an ephemeral in-memory store with green probes; an unparsable one aborts. The lenient path is the one a broken mount takes. Also on the fail-open: it covers the TASKS as well as the listeners, with all ten enabled by default, and the error is logged after the listeners are already bound. --- Charts.md | 3 +- Lessons-Learned.md | 64 ++++++ _Sidebar.md | 1 + stalwart.md | 486 +++++++++++++++++++++++++++++++++++++++++++++ 4 files changed, 553 insertions(+), 1 deletion(-) create mode 100644 stalwart.md diff --git a/Charts.md b/Charts.md index 61748aa..f50c287 100644 --- a/Charts.md +++ b/Charts.md @@ -1,6 +1,6 @@ # Chart index -27 directories: 26 installable charts plus the `template-chart` scaffold. Each page holds prerequisites, an installation walkthrough, the complete list of configurable values, and the chart's operating traps. +28 directories: 27 installable charts plus the `template-chart` scaffold. Each page holds prerequisites, an installation walkthrough, the complete list of configurable values, and the chart's operating traps. **Versions are not listed here on purpose** — and not on the chart pages either. Renovate moves app versions continuously; a table of numbers in a wiki is wrong within days. Read the current pair off the chart: @@ -29,6 +29,7 @@ grep -E '^(version|appVersion):' /Chart.yaml | [samba](samba) | Samba | StatefulSet | no | needs capabilities — a documented hardening deviation | | [satisfactory](satisfactory) | Satisfactory game server | StatefulSet | no | **excluded from the install-test**; `limitFactor: 2` and `OnRootMismatch` are deliberate | | [searxng](searxng) | SearXNG | Deployment + ConfigMap | yes | `ingress.domain` is read twice, so it stays required with the Ingress off | +| [stalwart](stalwart) | Stalwart mail server (SMTP, IMAP, POP3, JMAP, ManageSieve) | one StatefulSet **per node group** + one NodePort Service per port | yes, WebUI/JMAP only | **the chart configures almost nothing** — since v0.16 Stalwart keeps its whole configuration in the data store, so listeners, stores, TLS objects and cluster roles are operator work and the first install is eight steps. The role lookup **fails open** (a missing `ClusterRole` enables every listener *and* every task, green probe included); `NET_BIND_SERVICE` is a documented deviation needed to `exec`, not to bind; `externalTrafficPolicy` must be `Cluster` and the PROXY v2 header is what restores the source address | | [sure](sure) | Sure (the maintained Maybe Finance fork) | StatefulSet, **two containers** + upload PVC | yes | web and Sidekiq worker share one RWO volume; readiness avoids Rails' `/up`, which proves only that the app booted | | [teamspeak-server](teamspeak-server) | TeamSpeak server | StatefulSet | no | probes address the TCP port by name; the voice service is UDP and unprobeable | | [template-chart](template-chart) | — | scaffold | — | **not installable**; the reference implementation of every convention | diff --git a/Lessons-Learned.md b/Lessons-Learned.md index b36d964..f26efd7 100644 --- a/Lessons-Learned.md +++ b/Lessons-Learned.md @@ -232,6 +232,66 @@ The error was not the table. It was the move from "the product supports X" to "o Generally: **a capability list is about the product; only a run is about the installation.** The same shape as a stale context block — both read as authoritative, and neither is the thing you are actually talking to. +## The source address every NodePort quietly replaces + +Not a mail problem, although [stalwart](stalwart) is where it was worked out. It recurs in **every** chart whose application evaluates the client address — rate limiting, auto-banning, geo rules, audit logs, allow-lists. + +A `Service` of type `NodePort` or `LoadBalancer` with `externalTrafficPolicy: Cluster` makes kube-proxy **masquerade the source address** to the receiving node's IP. It has to: the reply would otherwise leave through a different node than the one the sender holds its connection with. So the application sees one address for every client, forever, and nothing reports it — the connection works, the request is served, and the log line is written with the wrong address in it. + +`Local` preserves the address but forwards only to pods **on the receiving node**. That is not a free choice either, and in the stalwart topology it is not a choice at all: an HA proxy delivers to the three control-plane nodes and no pod is ever scheduled there, so `Local` would drop **every** packet rather than some. + +What the lost address costs, in the order it hurts — from the mail case because it is the most complete: the **auto-ban switches the service off** (it bans the masqueraded node IP and with it every client at once); **SPF checks the wrong address and takes DMARC with it**; **blocklists report everything clean**; **per-IP rate limits collapse onto one address**; **greylisting loses its key**; and the **`Received:` chain becomes worthless** — the only one of those that destroys evidence retroactively instead of merely failing. Read the list as a template: any per-client decision an application makes is being made on one address. + +**The remedy is a proxy that terminates the connection and re-announces the client**, not a Service setting. A reverse or HA proxy accepts the inbound connection and opens its **own** to the node, on which it can prepend a `PROXY v2` header — for raw TCP such as port 25 as well, where no remote peer would ever send one. For HTTP it is `X-Forwarded-For` instead, **plus discarding whatever forwarding headers the client brought itself**, which is the half that gets forgotten and is what turns the mechanism into a spoofing hole. + +Three traps come with it, and they are why this is a page entry rather than a chart note: + +- **A trusted network makes the header mandatory.** A connection from a trusted source *without* one is misread and hangs until timeout; a header from an *untrusted* source is read as protocol data (Stalwart parses `PROXY TCP4 …` as an SMTP command). Enabling trust and enabling the header is one step, not two, and it is tested on a spare port before the real one moves. +- **In Kubernetes the trusted set and the probe set must be disjoint.** Kubelet probes arrive from the node running the pod, with no header. In the stalwart topology that works out only because pods never run on the nodes mail enters through — a load-bearing assumption, not an operating preference, and anyone who later schedules a pod onto one of those nodes breaks it. +- **An ingress controller is a third case again.** The application sees the controller's **pod** IP, which the PROXY protocol does not address; that is the `X-Forwarded-For` path. This failure is the slowest and most confusing: the auto-ban eventually bans the controller and the service dies for everyone at once, weeks after the install, looking like a server fault. + +**Which address the masquerading actually inserts is to be measured, not assumed** — for mail it stands in the first `Received:` header after one connection. + +## The rate limit that counted a node instead of a client + +The same mechanism from the other end, found by reading source rather than by an outage — which is the only reason it is not written in the past tense. + +Stalwart rate-limits anonymous HTTP requests to **100 per 60 seconds, keyed on the source IP**, and that applies to `/healthz` like everything else. Kubelet probes arrive from the **node's** IP, so every pod of that application on one node draws from the **same bucket**. Exceeding it returns 429, a 429 fails a probe, and a failed liveness probe restarts the pod — a restart loop that appears only once a node is dense enough, and reads as an unstable application. + +The arithmetic is short and belongs in the values file rather than in a review: liveness and readiness at `periodSeconds: 10` cost 12 requests/min/pod, so the bucket is gone at about **eight** pods on one node. At `20/20/10` it is 6/min and about **sixteen**. The startup probe *gates* liveness and readiness, so the three never run together and the worst case is the same in both phases — worth knowing, because the intuitive sum is half again too high and would argue for tighter intervals than necessary. + +**Now:** [stalwart](stalwart) ships `20/20/10` with the sum as a comment in `values.yaml` **and** in the probe fallbacks in `_helpers.tpl`, so a later tightening has to pass the reasoning. Generally: **a per-IP limit in an application is a per-node limit for probes**, and it is the first thing to check in anything that rate-limits by address. + +A `tcpSocket` probe bypasses such a bucket entirely and is the obvious escape. It was rejected here for a reason worth reusing: it would have been **green in exactly the failure the chart exists to contain** (a fail-opened node binds the port too), it proves strictly less than an HTTP health endpoint, and it would have had to differ per workload, so the fallback default could no longer carry the probe's context. + +## The hardening that stopped the container from starting + +The plan was the strict one and it was argued well: bind only high ports, so nothing privileged is ever bound, so `capabilities: drop: [ALL]` can stand with **nothing** added — no `NET_BIND_SERVICE`, no sysctl, no documented deviation. Three escape routes from the earlier survey were discarded on that basis. + +It cannot work, and the reason is not in the application at all. The stalwart image runs `setcap cap_net_bind_service=+ep` on its binary. The `e` sets the file's **effective** bit, which makes it a *capability-dumb binary*, and `capabilities(7)` then has the kernel check **at `execve`** that the process obtained the full file permitted set, failing with `EPERM` when the bounding set masked any of it out. `drop: [ALL]` empties the bounding set. So: + +``` +exec /usr/local/bin/stalwart: operation not permitted +``` + +The container never starts. Not unhardened, not degraded — it does not run. + +Two things made this easy to get wrong. The reasoning *"nothing below 1024 is bound, so the capability is not needed"* is **true and irrelevant**: the check happens before the program binds anything and never asks whether the capability would be used. And all three obvious escapes fail for the same single reason — high ports do not help, the `-alpine` variant carries the identical `setcap`, and `ip_unprivileged_port_start` addresses binding rather than exec. + +**Now:** `drop: [ALL]` plus `add: [NET_BIND_SERVICE]` as a documented deviation, in `values.yaml` **and** in the `_helpers.tpl` fallback — an erased `securityContext` falling back to the repo-standard empty `add` would otherwise produce a chart that cannot start, with nothing pointing at the cause. + +And the part that is the actual lesson: **the comment has to say that the capability is never used.** It is spent entirely on the exec check — inside a pod the kubelet sets `ip_unprivileged_port_start=0`, so the low ports need no capability anyway. A deviation documented only with its *name* reads, to the next hardening pass, as a dead grant to clean up; documented with its *mechanism*, it survives. "Needed for binding" would have been both wrong and an invitation. + +Generally: **the image is part of the security context.** A file capability, a fixed uid, an entrypoint that wants to write — these constrain the pod spec regardless of what the application does, and they are read off the Dockerfile that builds the **published** image, not the one at the repository root. Here those were two different files, and only `docker-bake.hcl` said which. + +## The missing file that opened a database nobody meant to use + +If stalwart's `config.json` is **absent** — a renamed ConfigMap key, a failed mount — the server does not abort. It enters recovery mode, prints a temporary admin password, and opens an **in-memory ephemeral store**. Probes go green, mail is accepted, and all of it is gone at the next restart. + +An *unparsable* file does abort. It is specifically the missing-file case that fails open, which is the inversion that makes it dangerous: the careful operator who validates the file's syntax is protected, and the one whose mount silently failed is not. + +**Now:** the chart always renders the file and seeds it with an init container, so its absence means something else is broken, and the trap is written on the chart page. Generally: **check what an application does when its configuration is missing, not only when it is wrong** — the two paths are often written by different people, and "absent" is the one that tends to get the lenient fallback. + ## The recurring shape - **Green is a claim about the assertions that ran, not about the system.** @@ -243,5 +303,9 @@ Generally: **a capability list is about the product; only a run is about the ins - **A document handed to you as context is a copy, not the source.** Correcting the file does not correct the copies already delivered, and a stale copy looks exactly as authoritative as a current one — only the file itself, and its diff against `origin/main`, settle what a rule says. - **Put the warning where the next change will be made.** A risk that only materializes at the next upgrade belongs at the line that upgrade will touch — a note in the pull request that argued the decision is read once, by people who already know. - **A capability list is about the product; only a run is about the installation.** "Supported" on a documentation page and "enabled on the server we call" are different claims, and the first one reads like a promise. +- **The image is part of the security context.** A file capability or a fixed uid constrains the pod spec no matter what the application does — and it is read off the Dockerfile that builds the *published* image, which is not always the one at the repository root. +- **Document a deviation with its mechanism, not its name.** A grant whose comment does not say *why it cannot be removed* reads as dead weight to the next cleanup, and "needed for binding" is worse than nothing when the real reason is `execve`. +- **Inside a cluster, every per-client identity is a per-node identity.** Masqueraded source addresses and probe traffic both collapse onto node IPs, so any application logic keyed on the client address — bans, limits, geo, audit — is being fed one address, silently and correctly-looking. +- **Ask what an application does when its configuration is missing, not only when it is wrong.** The lenient fallback is almost always on the "absent" path, and it is the path a broken mount takes. - **A reference to something that moved is false, not stale.** It still answers the question, which is why nobody re-asks it — so a file leaving the repository is a grep across the whole tree, comments included, not only the lines a tool can resolve. - **An answer that arrived by accident is worth making repeatable.** The all-charts install-test exists because one unplanned run produced a result nobody could have asked for; the fix is a way to ask the question on purpose, not a note to remember the accident. diff --git a/_Sidebar.md b/_Sidebar.md index 74512d6..6612003 100644 --- a/_Sidebar.md +++ b/_Sidebar.md @@ -38,6 +38,7 @@ - [samba](samba) - [satisfactory](satisfactory) - [searxng](searxng) +- [stalwart](stalwart) - [sure](sure) - [teamspeak-server](teamspeak-server) - [template-chart](template-chart) diff --git a/stalwart.md b/stalwart.md new file mode 100644 index 0000000..97d1768 --- /dev/null +++ b/stalwart.md @@ -0,0 +1,486 @@ +# stalwart + +Deploys [Stalwart](https://stalw.art/) — an all-in-one mail server (SMTP, IMAP, POP3, JMAP, ManageSieve, plus CalDAV/CardDAV/WebDAV) — **cluster-capable from the start**, with roles rather than symmetry: one StatefulSet per entry of the `nodes` map, each passing its key in `STALWART_ROLE`. + +**Read the next paragraph before anything else, because it decides how this page is organised.** + +Since v0.16 Stalwart keeps its **entire configuration inside the data store** and manages it over the JMAP API — over 2 400 settings in 117 object types, including every listener, the blob store, the in-memory store, TLS objects, domains and the cluster roles. The only thing left on disk is a small `config.json` naming the data store: *"It only needs to contain the `DataStore` object."* The REST API from v0.15 is gone. + +So **this chart configures almost nothing, and that is its defining property, not a gap.** It renders `config.json`, the workloads, the Services, the NetworkPolicies, the Ingress and a cert-manager `Certificate`. Domains, listeners, stores, TLS objects and cluster roles are operator work in the WebUI. That is why [First install](#first-install-eight-steps-none-optional) is eight steps long and why half of this page is [Traps](#traps). + +| | | +|---|---| +| Image | `docker.io/stalwartlabs/stalwart` (the glibc variant; `-alpine` is built from the same source and carries the same `setcap`, so it is not an escape from the [capability deviation](#net_bind_service-is-needed-to-exec-not-to-bind)) | +| Versions | not listed on purpose — read the current pair off `stalwart/Chart.yaml`, see [Chart index](Charts) | +| Workloads | one StatefulSet **per node group**, plus one config-seeding init container each | +| Storage | **no PVC.** Both image volumes are emptyDirs — state lives in Postgres, S3 and Valkey | +| Ingress | yes, for the WebUI and JMAP only, pointed at a single node group | +| External dependencies | **Postgres** (required), **S3** (blob store), **Valkey** (in-memory store + cluster coordinator) | +| Secrets | up to 3 × `OnePasswordItem` (data store, recovery admin, S3) | +| Network | one NodePort Service **per public port** + per-group ClusterIP and headless Services + **NetworkPolicy on by default** | +| Subchart | optional `valkey` (the Valkey project's own chart), **off by default** | +| Install-test | `ci/stalwart/` — and it proves much less than a green run suggests, see [CI](#ci) | + +## Prerequisites + +- **Postgres**, reachable from the cluster, with a database and a role for Stalwart. This is the data store and the chart will not render without its host. +- **S3-compatible object storage** for the blob store. The filesystem blob store is documented as *"available only for single node installations"*, and RocksDB likewise — two nodes force both stores to be external. +- **A Valkey (or Redis)**, for the in-memory store. Not optional in a cluster: the coordinator's `Default` variant *"reuses the Redis connection already configured on the in-memory store"*, so a multi-node install has no coordination path without one. Either point `externalServices.valkey.host` at an instance (this repo's own [valkey](valkey) chart is the better choice when one application owns it) or set `valkey.enabled: true` for the subchart. **Redis/Valkey is admissible as the in-memory store only — never as the data or blob store.** +- **cert-manager**, and for the mail certificate most likely a **DNS-01** issuer — see [TLS](#tls-cert-manager-not-stalwarts-own-acme). +- **1Password operator**, plus the items in [What the 1Password items must contain](#what-the-1password-items-must-contain). +- **Inbound reachability on port 25 with matching reverse DNS.** Nothing in the chart can provide this, and without it a mail server is not operable. See [DNS](#dns-a-mail-server-cannot-run-without). +- **An HA proxy in front of the cluster** that forwards 25, 465 and 993 and can set `send-proxy-v2`. The whole source-address design rests on it; see [Why `externalTrafficPolicy` must be `Cluster`](#why-externaltrafficpolicy-must-be-cluster). + +No full-text search service is needed: the search store has a `Default` variant that indexes into the data store (*"no additional services are required"*). No Elasticsearch, no Meilisearch — but the FTS load lands on the same Postgres, which belongs in the sizing. + +## Installation + +The registry URL and its credentials live in 1Password (item `jlab-registry-access`: `username`, `password`, `charts-url`) and never in a repository or a shell history file. + +```bash +helm repo add jlab "$CHARTS_URL" --username "$REG_USER" --password "$REG_PASS" +helm repo update + +helm install mail jlab/stalwart \ + --namespace mail --create-namespace \ + -f my-values.yaml +``` + +A first-install `my-values.yaml`. Note `bootstrap.enabled: true` — it is **required** on a first install, for the reason in [step 1](#first-install-eight-steps-none-optional): + +```yaml +bootstrap: + enabled: true # first install only; set to false in step 7 + +ingress: + domain: mail.example.com + clusterIssuer: letsencrypt-dns + +certificate: + clusterIssuer: letsencrypt-dns + extraDnsNames: + - smtp.example.com + - imap.example.com + +dataStore: + postgres: + host: postgres.db.svc.cluster.local + +externalServices: + s3: + host: s3.example.com + valkey: + host: mail-valkey.mail.svc.cluster.local + +secrets: + dataStoreSecretRef: op://vault/stalwart-datastore + recoveryAdminSecretRef: op://vault/stalwart-recovery-admin + s3SecretRef: op://vault/stalwart-s3 +``` + +`helm install` prints the chart's `NOTES.txt`, which is the per-release version of this page: it names the node groups and their ports, the exact listener numbers to create, the `ClusterRole` names, the 1Password fields, and a loud banner for as long as recovery mode is on. Read it. + +### What the 1Password items must contain + +One Kubernetes Secret key is created per **field** of the item, and the templates reference those keys by name — a missing field leaves the container in `CreateContainerConfigError`. An **empty** field fails the same way as a missing one: upstream's secret resolver treats an empty environment variable as *not found*. + +| Value | Item fields | Notes | +|---|---|---| +| `secrets.dataStoreSecretRef` | `password` | **Only the password.** The username cannot live here — see [why](#the-data-store-username-is-public-and-cannot-be-otherwise) | +| `secrets.recoveryAdminSecretRef` | `recovery-admin` | The **complete** value `admin:`, in one field. Not two fields — see [why](#the-recovery-admin-is-one-field-not-two) | +| `secrets.s3SecretRef` | `access-key`, `secret-key` | Exposed to the container as `STALWART_S3_ACCESS_KEY` / `STALWART_S3_SECRET_KEY` for the BlobStore object to reference | + +**Changing what is inside an item restarts nothing.** Helm never sees a secret's content, so there is nothing to hash. Repointing one of the values above at a *different* vault item **does** roll the pods — the chart stamps `checksum/secret-ref` over the rendered `OnePasswordItem`s, which covers the `op://` path. After rotating a password *inside* an item, restart by hand, once per group: + +```bash +kubectl -n mail rollout restart statefulset/mail-stalwart-full-sfs +kubectl -n mail rollout restart statefulset/mail-stalwart-mx-sfs +``` + +The 1Password operator's `auto-restart` annotation is **not** set and must not be: verified against operator 1.12.0 it enumerates Deployments only, so on a StatefulSet it is a silent no-op advertising a guarantee it does not provide ([§9](Chart-Style-Guide#9-config-changes-must-restart-the-workload)). + +## First install: eight steps, none optional + +1. **`bootstrap.enabled: true`, `nodes.mx.replicas: 0`.** + The setup wizard everyone reaches for runs in **bootstrap** mode, and bootstrap mode is entered only when there is **no** `config.json`: *"The first time Stalwart is started without a `config.json`, it enters bootstrap mode."* This chart always renders one, so the wizard never appears. Against an empty database that would leave a server with no listeners, no domain and no administrator, and nothing to fix it with. + + The documented path for a templated `config.json` is **recovery mode**: *"template `config.json` onto the host, start Stalwart once with `STALWART_RECOVERY_MODE=1` and a `STALWART_RECOVERY_ADMIN` supplied from a secrets manager… and then restart Stalwart without the recovery variables."* It opens the management endpoint on 8080 and keeps every mail port closed. With a second group running the wizard would be unusable anyway — each request would land on a random pod. + +2. **Open the management UI and create the permanent administrator.** + + ```bash + kubectl -n mail port-forward svc/mail-stalwart-full-http-service 8080:8080 + ``` + + then `http://localhost:8080/admin`. With `recoveryAdminSecretRef` set, the credential is the one in the vault item; without it, Stalwart generates a password and prints it to the pod log on **every** restart. + +3. **Create the configuration objects in the WebUI.** Blob store (S3), in-memory store (Valkey), coordinator (`Default` — reuses the in-memory store connection), search store (`Default` — uses the data store), the TLS object from step 5, the `NetworkListener` objects, `proxyTrustedNetworks`, and one `ClusterRole` per node group. + + **The listener ports must match the release exactly**, because the chart cannot create listener objects: + + | Listener | bind | published | + |---|---|---| + | `smtp` | 25 | node port 30025 | + | `submissions` | 465 | node port 30465 | + | `imaps` | 993 | node port 30993 | + | `http` | 8080 | not published (Ingress + probes) | + + And the roles, named **exactly** `full` and `mx`. The example shape: + + ```json + {"name":"mx","tasks":{"@type":"DisableAll"},"listeners":{"@type":"EnableAll"}} + ``` + + Both fields accept `EnableAll`, `DisableAll`, `EnableSome` and `DisableSome`. The task types are `storeMaintenance`, `accountMaintenance`, `metricsCalculate`, `metricsPush`, `pushNotifications`, `searchIndexing`, `spamClassifierTraining`, `outboundMta`, `taskQueueProcessing`, `taskScheduler`. Only `spamClassifierTraining` may run on at most one node. Note the object encoding: a **set** is a JSON object mapping each member to `true`, not an array, and a **duration** is an integer count of milliseconds. + + Getting a role name wrong here does not fail — read [Fail-open](#fail-open-a-node-whose-role-is-missing-becomes-a-full-node) before continuing. + +4. **Configure HTTP forwarding, before the auto-ban notices the Ingress.** Mandatory, not a tip — see [The Ingress ban trap](#the-ingress-ban-trap). + +5. **Point the TLS object at the mounted certificate.** + + ``` + certificate: /etc/stalwart-tls/tls.crt + privateKey: /etc/stalwart-tls/tls.key + ``` + + cert-manager writes these into `tls--stalwart-mail`. The volume is mounted `optional`, so a pending challenge does not hold the pods in `ContainerCreating`; until it is issued the mail ports serve a self-signed certificate. + +6. **Set `send-proxy-v2` on the HA proxy. All-or-nothing together with step 3.** Without the header on a trusted network every connection hangs until timeout; with the header from an untrusted network Stalwart reads `PROXY TCP4 …` as an SMTP command. Test on a spare port with its own frontend and move 25 only once a `Received:` line shows the real sender address. Background: [Why `externalTrafficPolicy` must be `Cluster`](#why-externaltrafficpolicy-must-be-cluster). + +7. **Restart the pods** — *"a newly added listener binds only on the next start"* — and set `bootstrap.enabled: false` in the same round. + +8. **Raise `nodes.mx.replicas`** (only now that the `mx` role exists) and **publish DNS**. + +## DNS a mail server cannot run without + +None of this is rendered by the chart; all of it is required. + +| Record | Content | +|---|---| +| **MX** | the deployment's public name, pointing at the HA proxy | +| **SPF** | `TXT "v=spf1 ..."` naming the HA proxy's address as a permitted sender | +| **DKIM** | `TXT` at `._domainkey` — the key is generated **in Stalwart** (Domains → DKIM), so this record is published after step 3 | +| **DMARC** | `TXT` at `_dmarc` — start at `p=none` with `rua=` and tighten only once the reports are clean | +| **Reverse DNS** | `PTR` for the **HA proxy's** public address, matching its forward name | + +The PTR is the one most often missed. It is not the node's and not the pod's: the HA proxy is what other mail servers see connecting. Many receivers reject outright when it is absent, and nothing in Kubernetes can set it — it belongs to whoever owns the IP range. + +## Configurable values + +### Global + +| Value | Default | Meaning | +|---|---|---| +| `scaleDown` | `false` | `true` renders every group with `replicas: 0` and overrides every `nodes.*.replicas`. Services, Ingress, Secrets and the Certificate stay, so no certificate is re-issued | +| `limitFactor` | `3` | Chart-wide default limit factor; a group's `resources.limitFactor` wins | + +### `secrets` + +| Value | Default | Meaning | +|---|---|---| +| `secrets.dataStoreSecretRef` | `null` | `op://` path; field `password`. Referenced from `config.json` by variable name, so the password never enters the ConfigMap | +| `secrets.recoveryAdminSecretRef` | `null` | `op://` path; field `recovery-admin` holding `admin:`. `null` switches `STALWART_RECOVERY_ADMIN` off — the correct end state once a permanent admin exists | +| `secrets.s3SecretRef` | `null` | `op://` path; fields `access-key`, `secret-key`. Exposed as env vars for the BlobStore object to reference | + +### `dataStore` — the only part of Stalwart the chart configures + +Rendered verbatim into `config.json` as the `DataStore` object, `"@type": "PostgreSql"`. + +| Value | Default | Meaning | +|---|---|---| +| `dataStore.postgres.host` | `null` | **Required**, no default | +| `dataStore.postgres.port` | `5432` | | +| `dataStore.postgres.database` | `stalwart` | | +| `dataStore.postgres.username` | `stalwart` | Rendered as `authUsername`. Public by necessity — [why](#the-data-store-username-is-public-and-cannot-be-otherwise) | +| `dataStore.postgres.tls.enable` | `true` | Rendered as `useTls`. **The opposite of upstream's default**, deliberately | +| `dataStore.postgres.tls.allowInvalidCerts` | `false` | Only ever a temporary diagnostic | +| `dataStore.postgres.poolMaxConnections` | `10` | | +| `dataStore.postgres.timeoutMs` | `15000` | **Integer milliseconds.** Named for its unit — [why](#durations-are-integer-milliseconds-not-15s) | +| `dataStore.postgres.options` | `null` | Extra libpq options, passed through; `null` omits the field | + +### `bootstrap` + +| Value | Default | Meaning | +|---|---|---| +| `bootstrap.enabled` | `false` | `STALWART_RECOVERY_MODE=1`. **`true` for a first install**, then `false`. Leaving it on keeps every mail port closed while all Kubernetes objects look healthy | +| `bootstrap.logLevel` | `info` | `STALWART_RECOVERY_MODE_LOG_LEVEL`; no effect once disabled | + +### `ports` + +A map, user-extensible: adding 587, 143, 110, 995 or 4190 is an additive change. **Two numbers per entry and they are not interchangeable** — see [The port duplication](#the-port-duplication-cannot-be-avoided). + +| Entry | `containerPort` | `nodePort` | `public` | +|---|---|---|---| +| `smtp` | `25` | `30025` | `true` | +| `submissions` | `465` | `30465` | `true` | +| `imaps` | `993` | `30993` | `true` | +| `http` | `8080` | — | `false` | + +### `service` + +| Value | Default | Meaning | +|---|---|---| +| `service.externalTrafficPolicy` | `Cluster` | `Local` would drop **every** packet in this topology — [why](#why-externaltrafficpolicy-must-be-cluster) | +| `service.annotations` | `{}` | Applied to every Service the chart renders | + +### `nodes` + +A **map**, not a list, so that an override never depends on a position. The key is the cluster-role name and goes into `STALWART_ROLE`. + +| Value | Default (`full`) | Default (`mx`) | Meaning | +|---|---|---|---| +| `nodes..replicas` | `1` | `0` | `0` allowed; `scaleDown` wins | +| `nodes..exposedPorts` | `[smtp, submissions, imaps, http]` | `[smtp, http]` | Keys of `ports`; **must include `http`** (all probes target it) | +| `nodes..resources.requests` | `200m` / `512Mi` / `64Mi` | `100m` / `256Mi` / `64Mi` | Limits computed by `limitFactor` | +| `nodes..env` | `[]` | `[]` | Additive; `value`, `secretKeyRef`, `configMapKeyRef` | + +**There is no `nodes..tasks`.** Which tasks a role runs is a property of the `ClusterRole` object in the database and the chart cannot write it; a values key that looked like it set them and did not would be worse than its absence. + +### `networkPolicy` + +| Value | Default | Meaning | +|---|---|---| +| `networkPolicy.enabled` | `true` | Only a boundary where the CNI enforces one | +| `networkPolicy.probeSources` | `[]` | CIDRs allowed to reach `http`. **Empty means open** — [why](#empty-probesources-means-open-and-that-is-the-safer-default) | +| `networkPolicy.allowedNamespaces` | `[]` | Extra namespaces for `http`; the ingress controller belongs here once `probeSources` is set | +| `networkPolicy.egress.dns` | `true` | UDP+TCP 53 to anywhere | +| `networkPolicy.egress.smtp` | `true` | TCP 25 to anywhere — outbound delivery | +| `networkPolicy.egress.extraPorts` | `[]` | Extra outbound TCP ports; add `443` for DNS-01 provider APIs or AI endpoints | + +Egress is restricted **by port, not by destination**: a NetworkPolicy matches CIDRs and selectors, never hostnames, so no host value from this chart can appear in it. What it still buys is that a compromised pod cannot open arbitrary outbound ports. + +### `externalServices` — Kubernetes only, these do not configure Stalwart + +| Value | Default | Used for | +|---|---|---| +| `externalServices.s3.host` | `""` | the NOTES.txt checklist only | +| `externalServices.s3.port` | `443` | the egress rule | +| `externalServices.valkey.host` | `""` | the render-abort check and the checklist; empty + `valkey.enabled` resolves to the subchart's Service | +| `externalServices.valkey.port` | `6379` | the egress rule | + +Setting a host here does **not** point Stalwart at it. The blob store and in-memory store are database objects created in the WebUI. + +### `certificate` + +| Value | Default | Meaning | +|---|---|---| +| `certificate.enabled` | `true` | | +| `certificate.clusterIssuer` | `null` | Falls back to `ingress.clusterIssuer`; usually a **different**, DNS-01 issuer | +| `certificate.extraDnsNames` | `[]` | `ingress.domain` is always included | +| `certificate.mountPath` | `/etc/stalwart-tls` | A top-level path so no mount nests inside the config emptyDir | + +### `ingress` + +| Value | Default | Meaning | +|---|---|---| +| `ingress.enabled` | `true` | Gates the Ingress only; **`ingress.domain` stays required either way** (the mail Certificate reads it) | +| `ingress.clusterIssuer` | `null` | **Required**, no default | +| `ingress.domain` | `null` | **Required**, no default; read twice | +| `ingress.className` | `nginx` | | +| `ingress.nodeGroup` | `full` | Which group the Ingress routes to, via that group's own `http` Service | +| `ingress.annotations` | `{}` | Merged with the cert-manager annotation | + +### `securityContext` and probes + +| Value | Default | +|---|---| +| `securityContext.pod` | `runAsNonRoot: true`, `runAsUser`/`runAsGroup: 2000`, `seccompProfile: RuntimeDefault`. **No `fsGroup`** — the chart mounts no PVC | +| `securityContext.container` | `allowPrivilegeEscalation: false`, `readOnlyRootFilesystem: true`, `runAsNonRoot: true`, `drop: [ALL]`, **`add: [NET_BIND_SERVICE]`** | +| `livenessProbe` | `GET /healthz/live`, period **20s**, timeout 5s, failures 3 | +| `readinessProbe` | `GET /healthz/ready`, period **20s**, timeout 5s, failures 3 | +| `startupProbe` | `GET /healthz/ready`, period **10s**, failures 30 (budget 300s) | + +The intervals are **calculated, not chosen** — see [The `/healthz` rate limit](#the-healthz-rate-limit-sets-the-probe-intervals). + +### `valkey` (subchart) + +| Value | Default | Meaning | +|---|---|---| +| `valkey.enabled` | `false` | `true` deploys the Valkey project's own chart alongside | + +Upstream's chart is **consumed as it comes** ([§14](Chart-Style-Guide#14-chart-dependencies)); two of its defaults fall short of our conventions and are documented rather than patched: `auth.enabled` is `false` (unauthenticated access from anything the CNI permits — set `auth.enabled` plus `auth.aclUsers`/`aclConfig`, and prefer `usersExistingSecret` over an inline password), and its NetworkPolicy is its own object with its own semantics that this chart's `networkPolicy` block does not reach. `scaleDown` and `checksum/config` stop at the subchart boundary, and a new Valkey release reaches a cluster only when **this** chart is republished. + +## Traps + +### Fail-open: a node whose role is missing becomes a full node + +The worst one, and the reason the chart is shaped as it is. From the source, not the documentation. + +The bootstrap registry looks a `ClusterRole` up **by name**; when it finds none it records a **non-fatal** error and leaves the role unset. The listener check then reads + +```rust +if bp.role.as_ref().is_none_or(|r| match &r.listeners { ... }) +``` + +and `is_none_or` yields **`true` for every listener** when the role is `None`. The same applies to the **tasks**, through a different file whose pre-set default has all ten task flags at `true`. + +**So a node whose role is not in the database binds every listener and runs every task.** The `mx` pod would serve IMAP and the admin interface, train the spam classifier, and run the outbound MTA a second time. The pod is `Running`, the log holds one line — `Cluster role "mx" not found in registry` — and the readiness probe is **green**. Kubernetes cannot detect it. Worse, the error is logged **after** the listeners are already bound. + +This is not only a first-install concern: it is every typo in a role name and every rename in `values.yaml` without the matching rename in the database. + +Three things contain it, and all three are why the chart looks the way it does: + +- `mx` ships at `replicas: 0`; +- the Services publish only what was asked for, per port; +- the **NetworkPolicy is a ceiling on reachability**, independent of what the process actually bound. + +Documented as a trap rather than explained away. If you add a node group, create its `ClusterRole` **first**. + +### `NET_BIND_SERVICE` is needed to `exec`, not to bind + +Both published image variants run + +``` +setcap cap_net_bind_service=+ep /usr/local/bin/stalwart +``` + +(`Dockerfile.build`, line 164 for the glibc target and 186 for musl/`-alpine`). The `e` in `+ep` sets the file's **effective** bit, which makes the binary a *capability-dumb binary*, and `capabilities(7)` then applies this at `execve` time: + +> When executing a capability-dumb binary, the kernel checks if the process obtained all permitted capabilities that were specified in the file permitted set… If the process did not obtain the full set of file permitted capabilities, then `execve(2)` fails with the error `EPERM`. + +`drop: [ALL]` empties the bounding set, so the capability cannot be obtained, so the exec fails outright: + +``` +exec /usr/local/bin/stalwart: operation not permitted +``` + +The check runs **before** the program binds anything and does not ask whether the capability is needed. This is why the three obvious escapes all fail for the same reason: high application ports do not help, `-alpine` carries the identical `setcap`, and the `net.ipv4.ip_unprivileged_port_start` sysctl addresses binding rather than exec. + +**And the second half, which is the part that matters for whoever reads this next: the capability is never actually used.** It is spent entirely on satisfying the exec check. Stalwart binds 25/465/993 here, but inside a pod that needs no capability for them anyway — the kubelet sets `ip_unprivileged_port_start=0` in the pod network namespace. So a hardening pass grepping for *a capability nobody uses* will find exactly this one, and removing it stops the chart from starting at all. The day `setcap` disappears from upstream's `Dockerfile.build` is the day this deviation can go with it. + +### Why `externalTrafficPolicy` must be `Cluster` + +The topology decides it. Three control-plane nodes, two workers; an HA proxy in front distributes 25/465/993 across the **control-plane** nodes, and **no Stalwart pod is ever scheduled on a control-plane node**. + +`externalTrafficPolicy: Local` forwards only to pods on the node that received the packet. On a control-plane node there is never one — so under `Local` **every** packet is dropped, not some. It has to be `Cluster`. + +The price of `Cluster` is that kube-proxy **masquerades the source address** to the receiving node's IP (it must: the reply would otherwise leave through a different node than the one the sender holds its connection with). On a mail server a lost source address is not cosmetic. In order of how fast it hurts: + +1. the **auto-ban switches the service off** — it bans the masqueraded node IP and with it every sender at once (documented behind Traefik, triggerable from outside); +2. **SPF checks the wrong address and takes DMARC with it**; +3. **blocklists report everything clean**; +4. **per-IP rate limits collapse onto one address**; +5. **greylisting loses its key**; +6. the **`Received:` chain becomes worthless** — the only item here that destroys evidence retroactively rather than merely failing. + +**The HA proxy solves all of it, port 25 included.** The objection that foreign mail servers do not send a PROXY header applies only to direct DNAT. Here the HA proxy terminates the inbound connection and opens its **own** to the node, on which it can prepend a `send-proxy-v2` header. The real client address then travels in the header and the masquerading stops mattering. + +**Which networks are trusted follows from the same topology — and only works because of it.** A trusted network makes the PROXY header **mandatory**: a connection from there without one is misread and hangs until timeout. In Kubernetes that is dangerous, because kubelet probes come from a node IP. + +| Traffic | Source IP Stalwart sees | Trusted? | +|---|---|---| +| Mail via the HA proxy | the **control-plane node's** IP (after masquerading) | yes | +| Kubelet probes | the **worker node's** IP (the one running the pod) | no | +| Ingress to WebUI/JMAP | the ingress controller's **pod** IP | no — different mechanism | + +So `proxyTrustedNetworks` contains **only the three control-plane node IPs**. Because no Stalwart pod ever runs there, probe sources and trusted addresses are **disjoint** — that is not an operating preference, it is a load-bearing assumption of the design. Anyone who later lets a Stalwart pod onto a control-plane node breaks it. + +**Which address the masquerading actually inserts is to be measured, not assumed.** It appears in the `Received:` header after the first connection. + +### The port duplication cannot be avoided + +Every listener is a `NetworkListener` object in the database. The chart cannot create those objects, so `ports.*.containerPort` is a **second copy** of a decision really made in the WebUI. If the two disagree, the Service points at a port nothing listens on: connections are refused or time out, every pod stays `Ready`, and no object reports a problem. + +The chart reduces the surprise rather than the duplication, in two ways. The application binds the **ordinary** 25/465/993, so the numbers an operator types into the WebUI are the ones from every page of Stalwart's documentation and from the image's own `EXPOSE` list. And `nodes..exposedPorts` is deliberately **not** a copy of the listener configuration but a statement of its own — *"this group is reachable here and nowhere else"* — which is why a disagreement in either direction is harmless: a role that binds more is unreachable on the extra ports, and a policy that permits more has nothing listening. + +An earlier draft had the application bind 30025/30465/30993 so that nothing privileged was bound and the capability could be dropped. That reasoning collapsed against the `setcap`/`execve` rule above; since the capability must be granted anyway, high ports bought no hardening and only cost predictability. + +**And in the NetworkPolicy the pod ports apply, not the node ports** — 25/465/993, never 30025/30465/30993. A policy is evaluated at the pod's interface, after kube-proxy has already translated the node port to the target port. Writing 30025 there matches nothing: the packet arrives on 25. The policy then looks strict, is accepted by the API server, and silently blocks everything inbound. + +### The Ingress ban trap + +The one the HA proxy does **not** solve. Stalwart sees the ingress controller's **pod IP** for every JMAP and WebUI request, not the client's, and the auto-ban eventually bans that address — at which point WebUI and JMAP are dead for everyone at once. + +For HTTP the remedy is not the PROXY protocol but Stalwart's `forwarded` handling with `X-Forwarded-For` evaluation, **while discarding forwarding headers the client brought itself**. A mandatory step (step 4), because the outage arrives weeks later and looks like a server fault. + +Second half of the same area: **after bootstrap, port 8080 is no longer the admin port.** Upstream moves administration to `https:///admin` once setup completes. The Ingress therefore only works while an HTTP `NetworkListener` on 8080 exists in the database — between finishing setup and creating that listener it resolves to a port that answers nothing. + +### A missing `config.json` opens an ephemeral store and looks healthy + +If the file is **absent** — a failed ConfigMap mount, a renamed key — the server does **not** abort. It sets recovery mode, prints a random temporary admin password, and opens an **in-memory `EphemeralStore`**. Probes go green, mail is accepted, and everything is lost on the next restart. + +An unparsable or unreadable file *does* abort. It is specifically the missing-file case that fails open, and the chart's defence is that `config.json` is always rendered and seeded by an init container, so its absence means something else broke. + +### The `/healthz` rate limit sets the probe intervals + +Stalwart applies its anonymous HTTP rate limit to `/healthz` as to everything else, default **100 requests per 60 seconds keyed on the source IP**. Kubelet probes arrive from the **node's** IP, so every Stalwart pod on one node draws from the **same bucket**. Exceeding it returns 429, a 429 fails the probe, and a failed liveness probe restarts the pod — a self-inflicted restart loop that worsens with pod density. + +| Intervals | Per pod | Pods per node before the limit | +|---|---|---| +| **20/20/10 (shipped)** | 6 req/min | ~16 | +| 10/10/5 (the obvious default) | 12 req/min | ~8 | + +The startup probe **gates** liveness and readiness, so the three never run simultaneously and 6/min is the worst case in both phases. If a node ever needs more pods than that, raise the limit on the server (the allow-list bypasses the limiter outright for listed addresses) rather than blinding the probes. + +**This trap is not specific to Stalwart** — any application that rate-limits by source IP meets it in Kubernetes, because probes are indistinguishable from each other at the node IP. See [Lessons learned](Lessons-Learned). + +A `tcpSocket` readiness probe on the mail port would bypass the bucket, and was rejected: it would be **green exactly in the fail-open case** (a broken `mx` node binds 25 too), it proves strictly less than `/healthz/ready`, and it would have to differ per node group so the fallback default could no longer carry the probe's context. + +### Empty `probeSources` means open, and that is the safer default + +With `networkPolicy.probeSources` empty the `http` rule renders **without** a `from` selector, so the port is reachable from anywhere the CNI routes to. Deliberate: the kubelet probes from the worker node's IP, which the chart cannot know, and a guessed CIDR that misses the real addresses blocks **every** probe — and a pod that never becomes Ready reads as a broken application rather than as a policy. `http` carries no node port, so "open" means open inside the cluster, not from the internet. + +Once `probeSources` is non-empty the rule is restricted, which **also cuts off the ingress controller** unless its namespace is in `networkPolicy.allowedNamespaces`. + +### The data store username is public and cannot be otherwise + +On the `DataStore` object, `authSecret` has the type `SecretKeyOptional` — a tagged sub-object with `None`, `Value`, `EnvironmentVariable` and `File` variants — while `authUsername` is a plain `String` with no secret form at all. So the password is referenced by variable name and never enters the ConfigMap: + +```json +"authSecret": {"@type":"EnvironmentVariable","variableName":"STALWART_DB_PASSWORD"} +``` + +but the username is rendered into it. An upstream constraint, not a choice. The upside is the reason `config.json` can be a real ConfigMap with a meaningful `checksum/config`: Helm can hash the whole file because there is no secret in it. + +### The recovery admin is one field, not two + +`STALWART_RECOVERY_ADMIN` takes `username:password`, and the chart reads it as **one** vault field rather than assembling it. The upstream parser splits on the first `:` and **drops the entry when either half is empty**, so a half-filled item would silently mean *no recovery admin* instead of failing — and this credential is a back door that logs in regardless of directory state. + +Remove it when you are done: set `secrets.recoveryAdminSecretRef: null` once a permanent administrator exists. + +### Durations are integer milliseconds, not `15s` + +Stalwart's object encoding is explicit: *"A duration is an integer count of milliseconds, and a size is an integer count of bytes. **Neither accepts a string.**"* The `DataStore` reference gives the field as `Duration? default: 15000`, and the deserializer goes through `u64` into `Duration::from_millis`. + +A `config.json` carrying `"15s"` is rejected at startup. The chart therefore names the value `timeoutMs` and types it `integer` in the schema, so `timeoutMs: 15s` fails at render with `got string, want integer` rather than in the server's log. + +Two neighbours worth knowing when writing a `stalwart-cli apply` plan by hand: an ordered **list** is a JSON object keyed by stringified positions (`{"0":…,"1":…}`), and a **set** is a JSON object mapping each member to `true` — neither is an array. + +### What a green pod proves here, and what it does not + +Liveness asks `/healthz/live`; readiness and startup ask `/healthz/ready`. Liveness is on the dependency-free endpoint on purpose, so a Postgres outage cannot restart every pod in the cluster at once ([§12.3](Chart-Style-Guide#12-values-that-must-not-lie)). + +But read from the source: **`/healthz/ready` does not query the database.** It returns 200 as soon as the data store *handle* is not `Store::None`. So readiness does not hang on the dependency the way the guideline asks, and it cannot — the endpoint upstream offers is shallow. It also does not catch the ephemeral-store case above, since that server has a handle too. + +Both endpoints are unauthenticated and carry no network restriction. They do pass the configurable `allowedEndpoints` expression (default `200`, i.e. everything allowed) whose loopback escape hatch does **not** cover a node IP — so an operator who later writes a restrictive expression breaks every probe. + +## Upgrades and backup + +- **A plain app-version bump** is a chart patch and needs no action here; Renovate opens it and `ci/renovate-bump-chart.sh` bumps the version. Chart pages carry no version numbers — `Chart.yaml` is the source. +- **Read `UPGRADING/` for an app major.** The v0.15 → v0.16 migration was offline, multi-step and irreversible without a backup; nothing suggests the next major will be gentler. For app major jumps the repo rule applies: first to the last release of the old line, then to the newest major, each in its own PR. +- **The backup is Postgres plus the S3 bucket.** No PVC is involved — there is nothing in the pods worth keeping. The configuration lives in Postgres too, which means a Postgres restore also rolls back every listener, domain and role. +- **`stalwart-cli snapshot` is the configuration backup**, and the only one that is readable and diffable. It emits an `apply` plan; keep one in version control after any WebUI change, or the eight steps above are the recovery procedure. +- **Scaling to zero** (`scaleDown: true`) stops the release without uninstalling it. Mail queued on other servers will be retried for days, so a short window is survivable — but inbound connections are refused meanwhile, and some senders downgrade a repeatedly refused host. + +## CI + +`ci/stalwart/` holds `test-values.yaml` and `fixtures.yaml`: a throwaway Postgres, the dummy data store Secret, and a stub CRD for cert-manager's `Certificate`. It runs with `bootstrap.enabled: true`, which is the only mode that can come up against an empty database. + +**A green run does not mean a working mail server, and no install-test here could.** Every listener lives in the database and is created by a human, so against an empty database there is no SMTP service to test. + +What it does prove is the chart's own responsibility, and the first item is why the chart is tested rather than excluded: + +1. **the container execs at all** — the `setcap`/`execve` interaction above would otherwise only be caught by reading a Dockerfile; +2. `readOnlyRootFilesystem: true` holds with the two image volumes plus `/tmp` as emptyDirs, and the init container can write `config.json`; +3. the rendered `config.json` parses **as Stalwart reads it** — the store opens, so `@type`, the integer timeout and the `authSecret` indirection are all accepted, which is the one check that the password really arrives without ever being in the ConfigMap; +4. the Secret key names match, the node ports can be allocated, the quantities are valid, and the NetworkPolicies do not cut off the probes. + +**No Valkey in the fixtures, deliberately.** The in-memory store is a database object, so a Valkey in the namespace would be reachable and unused — making the test look as though it covered clustering while covering none of it. The test runs a single node, which needs no in-memory store. The subchart path is covered by a local render only; a known gap. + +## See also + +- [Chart style guide](Chart-Style-Guide) — the conventions this chart follows and the two it deviates from +- [Lessons learned](Lessons-Learned) — the source-IP mechanism behind NodePort, which recurs in every chart whose application evaluates the client address +- [valkey](valkey) — this repo's own single-instance chart, usually the better in-memory store than the subchart +- [Chart index](Charts)