feat(stalwart): new chart for the Stalwart mail server #77

Merged

jklein merged 2 commits from feat/stalwart-chart into main 2026-10-02 22:59:18 +02:00

Conversation 2 Commits 2 Files Changed 19

+3166 -4

jlab-bot commented 2026-10-02 21:16:28 +02:00

Member

Copy link

Copy source

Implements the approved plan on #65, including the two decisions from the blocker comment (NET_BIND_SERVICE as a documented deviation, and the container binding the ordinary 25/465/993 behind NodePorts 30025/30465/30993).

Part of #65, not Fixes: every runtime measurement is still open (list at the bottom), and this workspace has no k3d and no Docker daemon, so none of it could be closed here.

What the chart does, and why it is so small

Since v0.16 Stalwart keeps its entire configuration — every listener, the blob store, the in-memory store, TLS objects, domains and the cluster roles — inside the data store, managed over JMAP. The only file on disk is a config.json that names the data store and nothing else ("It only needs to contain the DataStore object"). So the chart renders that ConfigMap, one StatefulSet per entry of the nodes map, the Services, the NetworkPolicies, the Ingress and a cert-manager Certificate. Everything else is operator work in the WebUI, which is why NOTES.txt carries the full eight-step first install.

Services are per port, not per group: a NodePort is cluster-unique, so full and mx can only accept SMTP behind the same Service, selected through a per-port pod label. The plan did not resolve how http could be both port-selected and Ingress-group-scoped; it cannot, so http gets a per-group ClusterIP Service and the Ingress names one of them. That is the only place I filled in an under-specification rather than following the text literally.

The four traps, documented in values.yaml rather than smoothed over

• The role lookup fails open. A node whose ClusterRole is missing from the database enables every listener and every task (the plan knew only about the listeners; network.rs:341-363 does the same for tasks, defaulting all ten to true), while staying Running with a green readiness probe. log_errors() runs after Listeners::parse, so the one log line arrives when the listeners are already bound. Containment: mx at replicas: 0, per-port Services, and a NetworkPolicy that caps reachability independently of what was bound.
• NET_BIND_SERVICE is needed to exec, not to bind. The comment says both halves, as instructed — the execve rule and that the capability is never used — because the second half is what stops the next hardening pass from deleting it. It is in the _helpers.tpl fallback too, so an erased securityContext cannot silently produce a chart that will not start.
• externalTrafficPolicy: Cluster, with the PROXY v2 header as its mandatory companion.
• /healthz is rate-limited to 100 req/60s per source IP, and kubelet probes come from the node IP, so every pod on a node shares one bucket. See the probe decision below.

The probe decision you asked me to justify

HTTP kept, intervals widened to 20/20/10. The arithmetic is a comment in values.yaml and in the _helpers.tpl fallback:

	per pod	pods per node before 100/60s
20/20/10 (shipped)	6 req/min	~16
10/10/5 (the obvious default)	12 req/min	~8

The startup probe gates liveness and readiness, so the three never run together and 6/min is the worst case in both phases — which also corrects the "~18/min, ~5 pods" figure in the decision comment.

A tcpSocket readiness probe on the mail port was considered and rejected on three grounds, none of them convenience:

1. It would be green in exactly the failure this chart exists to contain — a fail-opened mx node binds 25, so a TCP connect succeeds on a broken node as readily as on a healthy one.
2. It proves strictly less: a TCP accept says a socket is bound; /healthz/ready additionally says a data store handle exists.
3. It would have to differ per node group (mx has no 465/993), so the fallback default could no longer carry the probe's context — the failure guideline 12.5 exists for.

Honest limit on the kept option: /healthz/ready does not query the database (it returns 200 iff the data store handle is not Store::None). So readiness does not hang on the dependency the way guideline 12.3 asks, and it cannot — the endpoint upstream offers is shallow. Named in values.yaml and NOTES.txt rather than papered over.

Field names verified at source, not derived

As asked, against crates/registry/src/schema/structs.rs at v0.16.24 rather than inferred:

• PostgreSqlStore carries #[serde(rename = "timeout")] pub timeout: Option<Duration> — so the JSON key is timeout, while the values key in this chart is timeoutMs to make the unit unmissable.
• The Duration deserializer is <u64>::deserialize(...).map(Duration::from_millis) — integer milliseconds; a string cannot parse. The schema types it integer, so timeoutMs: 15s is rejected at render with got string, want integer, and the plan's key timeout is rejected with not allowed.
• SecretKeyOptional is #[serde(tag = "@type")] over None | Value | EnvironmentVariable | File, with SecretKeyEnvironmentVariable { variableName }. Correction 3 stands.

CI fixtures: fixtures, not an exclusion — and why

ci/stalwart/test-values.yaml and fixtures.yaml: throwaway Postgres, the one dummy Secret, a stub CRD for Certificate, and bootstrap.enabled: true (recovery mode — the only mode that can come up against an empty database, since the wizard needs config.json to be absent).

It cannot prove a working mail server, and no install-test here could. I chose fixtures anyway because of what it does prove, and the first item decided it: the container execs at all. The whole blocker on this issue was a container that could not start; excluding the chart would give up precisely the signal that mattered most. It also covers readOnlyRootFilesystem against the two emptyDirs, the init container's copy, that the rendered config.json parses as Stalwart reads it (the store opens, so @type, the integer timeout and the authSecret indirection are all accepted — the one check that the password really arrives without ever being in the ConfigMap), the Secret key names, NodePort allocation and the NetworkPolicy objects.

No Valkey in the fixtures, deliberately: the in-memory store is a database object, so a Valkey in the namespace would be reachable and unused — making the test look like it covered clustering while covering none of it. The test runs a single node, which needs no in-memory store. The subchart path is covered by local render only; named as a known gap.

The one thing I could not pre-verify: whether /healthz is served on the recovery listener. The source shows the handler in the general HTTP path with no auth and no network restriction, but recovery mode binds a single listener and skips most config parsing, and there is no cluster here to try it on. If the install-test goes red with the pods never becoming Ready, that is the reason, and the fix is an exec probe or an entry in ci/excluded-charts.txt. Flagging it rather than letting it surprise you.

A bug the testing found

certificate.clusterIssuer | default (required "..." ingress.clusterIssuer) — the obvious one-liner — is wrong: Go evaluates pipeline arguments eagerly, so required() fires whenever ingress.clusterIssuer is null even when certificate.clusterIssuer is set and the fallback is never needed. Measured with ingress.enabled=false; replaced by an explicit if with the reason in a comment.

Local verification (no runtime)

• ci/helm-dependencies.sh stalwart → fetches valkey 0.12.0; charts/*.tgz and Chart.lock correctly gitignored
• helm lint --strict stalwart → 1 chart(s) linted, 0 chart(s) failed (the [INFO] Missing required value lines are the required() messages on chart defaults, exactly as for outline and valkey)
• helm package → stalwart-1.0.0+upv0.16.24.tgz
• helm template against chart defaults and against ci/stalwart/test-values.yaml
• the rendered config.json parsed with json.loads and its field types checked: timeout and port integers, useTls and allowInvalidCerts booleans, authSecret the tagged object
• ci/run-quantity-render-test.sh → failed: 0, stalwart exercised with both nodes.full.resources and nodes.mx.resources
• NOTES.txt rendered through the scratch-ConfigMap trick (helm template does not render it), across the ingress-on, ingress-off, recovery-mode and scaleDown paths
• null-safety and fail paths: securityContext=null still renders NET_BIND_SERVICE (4 occurrences); nodes.mx.resources=null falls back to mx's own lighter figures, not the generic ones; a cluster with no in-memory store, an unknown exposedPorts name, a group without http, and an unknown ingress.nodeGroup each abort with a message naming the value; the schema rejects timeout, a string timeoutMs, and a list-shaped nodes

Not run: trivy (not installed here), and the install-test itself.

Still open — every runtime measurement, nothing closed

1. Do /healthz/live and /healthz/ready answer a plain kubelet probe, and is /healthz served on the recovery listener? (also gates the install-test above)
2. Does readOnlyRootFilesystem: true hold with /etc/stalwart, /var/lib/stalwart and /tmp as emptyDirs, and must anything under /var/lib/stalwart survive a restart once all three stores are external? If yes, this needs a PVC per group.
3. Does Stalwart rewrite config.json itself when the file exists and points at an empty Postgres? The init-container-into-emptyDir shape assumes it may.
4. Which address the SNAT masquerading actually inserts — it determines the contents of proxyTrustedNetworks, and it is readable off the first Received: header.
5. Does this cluster's --service-node-port-range include 30025 to 30993? (the default 30000-32767 does)
6. The NET_BIND_SERVICE conclusion itself: the setcap lines and the docker-bake.hcl mapping I read myself, but the capabilities(7) rule is applied from the specification and not measured against a running runtime. The install-test is what would close it.
7. The anonymous rate limit was read from the registry defaults in source; the 100/60s figure is not measured against a running server.
8. The Valkey subchart path — render only, never installed.

Implements the approved plan on [#65](https://git.jk-dev.de/jlab-cluster/helm-charts/issues/65), including the two decisions from the blocker comment (`NET_BIND_SERVICE` as a documented deviation, and the container binding the ordinary 25/465/993 behind NodePorts 30025/30465/30993). `Part of #65`, not `Fixes`: **every runtime measurement is still open** (list at the bottom), and this workspace has no k3d and no Docker daemon, so none of it could be closed here. ## What the chart does, and why it is so small Since v0.16 Stalwart keeps its entire configuration — every listener, the blob store, the in-memory store, TLS objects, domains and the cluster roles — **inside the data store**, managed over JMAP. The only file on disk is a `config.json` that names the data store and nothing else ("It only needs to contain the DataStore object"). So the chart renders that ConfigMap, one StatefulSet per entry of the `nodes` map, the Services, the NetworkPolicies, the Ingress and a cert-manager `Certificate`. Everything else is operator work in the WebUI, which is why `NOTES.txt` carries the full eight-step first install. Services are **per port, not per group**: a NodePort is cluster-unique, so `full` and `mx` can only accept SMTP behind the *same* Service, selected through a per-port pod label. The plan did not resolve how `http` could be both port-selected and Ingress-group-scoped; it cannot, so `http` gets a per-group ClusterIP Service and the Ingress names one of them. That is the only place I filled in an under-specification rather than following the text literally. ## The four traps, documented in `values.yaml` rather than smoothed over - **The role lookup fails open.** A node whose `ClusterRole` is missing from the database enables every listener **and every task** (the plan knew only about the listeners; `network.rs:341-363` does the same for tasks, defaulting all ten to `true`), while staying `Running` with a green readiness probe. `log_errors()` runs *after* `Listeners::parse`, so the one log line arrives when the listeners are already bound. Containment: `mx` at `replicas: 0`, per-port Services, and a NetworkPolicy that caps reachability independently of what was bound. - **`NET_BIND_SERVICE` is needed to `exec`, not to bind.** The comment says both halves, as instructed — the `execve` rule *and* that the capability is never used — because the second half is what stops the next hardening pass from deleting it. It is in the `_helpers.tpl` fallback too, so an erased `securityContext` cannot silently produce a chart that will not start. - **`externalTrafficPolicy: Cluster`**, with the PROXY v2 header as its mandatory companion. - **`/healthz` is rate-limited to 100 req/60s per source IP**, and kubelet probes come from the node IP, so every pod on a node shares one bucket. See the probe decision below. ## The probe decision you asked me to justify **HTTP kept, intervals widened to 20/20/10.** The arithmetic is a comment in `values.yaml` and in the `_helpers.tpl` fallback: | | per pod | pods per node before 100/60s | |---|---|---| | 20/20/10 (shipped) | 6 req/min | ~16 | | 10/10/5 (the obvious default) | 12 req/min | ~8 | The startup probe **gates** liveness and readiness, so the three never run together and 6/min is the worst case in both phases — which also corrects the "~18/min, ~5 pods" figure in the decision comment. A `tcpSocket` readiness probe on the mail port was considered and **rejected** on three grounds, none of them convenience: 1. It would be **green in exactly the failure this chart exists to contain** — a fail-opened `mx` node binds 25, so a TCP connect succeeds on a broken node as readily as on a healthy one. 2. It proves strictly less: a TCP accept says a socket is bound; `/healthz/ready` additionally says a data store handle exists. 3. It would have to differ per node group (`mx` has no 465/993), so the fallback default could no longer carry the probe's context — the failure guideline 12.5 exists for. Honest limit on the kept option: **`/healthz/ready` does not query the database** (it returns 200 iff the data store handle is not `Store::None`). So readiness does not hang on the dependency the way guideline 12.3 asks, and it cannot — the endpoint upstream offers is shallow. Named in `values.yaml` and `NOTES.txt` rather than papered over. ## Field names verified at source, not derived As asked, against `crates/registry/src/schema/structs.rs` at `v0.16.24` rather than inferred: - `PostgreSqlStore` carries `#[serde(rename = "timeout")] pub timeout: Option<Duration>` — so the **JSON key is `timeout`**, while the values key in this chart is `timeoutMs` to make the unit unmissable. - The `Duration` deserializer is `<u64>::deserialize(...).map(Duration::from_millis)` — integer milliseconds; a string cannot parse. The schema types it `integer`, so `timeoutMs: 15s` is rejected at render with `got string, want integer`, and the plan's key `timeout` is rejected with `not allowed`. - `SecretKeyOptional` is `#[serde(tag = "@type")]` over `None | Value | EnvironmentVariable | File`, with `SecretKeyEnvironmentVariable { variableName }`. Correction 3 stands. ## CI fixtures: fixtures, not an exclusion — and why `ci/stalwart/test-values.yaml` and `fixtures.yaml`: throwaway Postgres, the one dummy Secret, a stub CRD for `Certificate`, and `bootstrap.enabled: true` (recovery mode — the only mode that can come up against an empty database, since the wizard needs `config.json` to be absent). It **cannot** prove a working mail server, and no install-test here could. I chose fixtures anyway because of what it *does* prove, and the first item decided it: **the container execs at all.** The whole blocker on this issue was a container that could not start; excluding the chart would give up precisely the signal that mattered most. It also covers `readOnlyRootFilesystem` against the two emptyDirs, the init container's copy, that the rendered `config.json` parses *as Stalwart reads it* (the store opens, so `@type`, the integer timeout and the `authSecret` indirection are all accepted — the one check that the password really arrives without ever being in the ConfigMap), the Secret key names, NodePort allocation and the NetworkPolicy objects. No Valkey in the fixtures, deliberately: the in-memory store is a database object, so a Valkey in the namespace would be reachable and unused — making the test look like it covered clustering while covering none of it. The test runs a single node, which needs no in-memory store. The subchart path is covered by local render only; **named as a known gap.** **The one thing I could not pre-verify:** whether `/healthz` is served on the **recovery** listener. The source shows the handler in the general HTTP path with no auth and no network restriction, but recovery mode binds a single listener and skips most config parsing, and there is no cluster here to try it on. If the install-test goes red with the pods never becoming Ready, that is the reason, and the fix is an exec probe or an entry in `ci/excluded-charts.txt`. Flagging it rather than letting it surprise you. ## A bug the testing found `certificate.clusterIssuer | default (required "..." ingress.clusterIssuer)` — the obvious one-liner — is **wrong**: Go evaluates pipeline arguments eagerly, so `required()` fires whenever `ingress.clusterIssuer` is null even when `certificate.clusterIssuer` is set and the fallback is never needed. Measured with `ingress.enabled=false`; replaced by an explicit `if` with the reason in a comment. ## Local verification (no runtime) - `ci/helm-dependencies.sh stalwart` → fetches `valkey` 0.12.0; `charts/*.tgz` and `Chart.lock` correctly gitignored - `helm lint --strict stalwart` → `1 chart(s) linted, 0 chart(s) failed` (the `[INFO] Missing required value` lines are the `required()` messages on chart defaults, exactly as for `outline` and `valkey`) - `helm package` → `stalwart-1.0.0+upv0.16.24.tgz` - `helm template` against chart defaults **and** against `ci/stalwart/test-values.yaml` - the rendered `config.json` parsed with `json.loads` and its field **types** checked: `timeout` and `port` integers, `useTls` and `allowInvalidCerts` booleans, `authSecret` the tagged object - `ci/run-quantity-render-test.sh` → `failed: 0`, stalwart exercised with both `nodes.full.resources` and `nodes.mx.resources` - `NOTES.txt` rendered through the scratch-ConfigMap trick (`helm template` does not render it), across the ingress-on, ingress-off, recovery-mode and `scaleDown` paths - null-safety and fail paths: `securityContext=null` still renders `NET_BIND_SERVICE` (4 occurrences); `nodes.mx.resources=null` falls back to **mx's own** lighter figures, not the generic ones; a cluster with no in-memory store, an unknown `exposedPorts` name, a group without `http`, and an unknown `ingress.nodeGroup` each abort with a message naming the value; the schema rejects `timeout`, a string `timeoutMs`, and a list-shaped `nodes` Not run: `trivy` (not installed here), and the install-test itself. ## Still open — every runtime measurement, nothing closed 1. Do `/healthz/live` and `/healthz/ready` answer a plain kubelet probe, and **is `/healthz` served on the recovery listener**? (also gates the install-test above) 2. Does `readOnlyRootFilesystem: true` hold with `/etc/stalwart`, `/var/lib/stalwart` and `/tmp` as emptyDirs, and must anything under `/var/lib/stalwart` survive a restart once all three stores are external? If yes, this needs a PVC per group. 3. Does Stalwart rewrite `config.json` itself when the file exists and points at an empty Postgres? The init-container-into-emptyDir shape assumes it may. 4. Which address the SNAT masquerading actually inserts — it determines the contents of `proxyTrustedNetworks`, and it is readable off the first `Received:` header. 5. Does this cluster's `--service-node-port-range` include 30025 to 30993? (the default 30000-32767 does) 6. The `NET_BIND_SERVICE` conclusion itself: the `setcap` lines and the `docker-bake.hcl` mapping I read myself, but the `capabilities(7)` rule is applied from the specification and **not** measured against a running runtime. The install-test is what would close it. 7. The anonymous rate limit was read from the registry defaults in source; the 100/60s figure is **not** measured against a running server. 8. The Valkey subchart path — render only, never installed.

jlab-bot added 1 commit 2026-10-02 21:16:29 +02:00

feat(stalwart): new chart for the Stalwart mail server ... 26be67f4cd

Cluster-capable from the start, with roles rather than symmetry: one
StatefulSet per entry of the "nodes" map, the key passed in STALWART_ROLE.

Since v0.16 Stalwart keeps its whole configuration inside the data store and
manages it over JMAP, so the chart's scope is deliberately narrow: it renders
config.json (exactly the DataStore object), the workloads, the Services, the
NetworkPolicies, the Ingress and a cert-manager Certificate. Listeners, blob
store, in-memory store, TLS object and cluster roles are operator work in the
WebUI, which is why NOTES.txt carries an eight-step first install.

config.json stays secret-free: authSecret uses the EnvironmentVariable variant
(verified against the serde definition of PostgreSqlStore at v0.16.24), so the
password arrives through secretKeyRef and checksum/config stays meaningful.
The username cannot follow it - authUsername is a plain String with no secret
form - so it is a public value and says so.

Services are per PORT, not per group: a NodePort is cluster-unique, so "full"
and "mx" can only accept SMTP behind one Service, selected through a per-port
pod label. The http port gets a per-group ClusterIP Service instead, so the
Ingress reaches exactly ingress.nodeGroup.

Four traps, documented rather than explained away:

- The role lookup FAILS OPEN. A node whose ClusterRole is missing from the
  database enables every listener and every task while staying Running with a
  green probe, and the log line arrives after the listeners are bound. Hence
  mx at replicas 0, per-port Services, and a NetworkPolicy that is a ceiling
  independent of what was bound.
- NET_BIND_SERVICE is a documented deviation needed to EXEC, not to bind. The
  image sets setcap cap_net_bind_service=+ep, which makes the kernel fail
  execve when the bounding set is empty. The values comment says both why it
  is there and that it is never used, because the second half is what stops
  the next hardening pass from removing it.
- externalTrafficPolicy must be Cluster - no pod runs on a control-plane node
  while the HA proxy delivers to them, so Local would drop every packet - and
  the source address that costs is restored only by the HA proxy's PROXY v2
  header.
- /healthz is rate-limited to 100 requests/60s per source IP, which is a node
  IP for kubelet probes, so all pods on a node share one bucket. The probe
  intervals are calculated (6 req/min/pod, ~16 pods per node) and the sum is a
  comment in values.yaml.

A first install needs bootstrap.enabled: true: the setup wizard appears only
when config.json is ABSENT, and this chart always renders one, so recovery
mode is upstream's documented path for a templated config.json.

Second chart in the repo with a dependency - the Valkey project's own chart,
off by default behind condition:, because Stalwart's cluster coordinator rides
on the in-memory store connection. The render aborts when more than one pod
runs with no in-memory store address anywhere.

Also wired up: publish-stalwart.yml through the shared block in
workflow-util/pipelines, the change-app-version.yml choice list, a Renovate
customManagers block anchored on the v prefix (which excludes the moving
latest / v0.16 / -alpine tags), CI fixtures, and the README and CLAUDE.md
lines that claimed ytdlp-web-player was the only chart with a dependency.

Verified locally: ci/helm-dependencies.sh, helm lint --strict (0 failed),
helm package, helm template against chart defaults and against
ci/stalwart/test-values.yaml, the rendered config.json parsed as JSON with the
field types the serde definition requires, ci/run-quantity-render-test.sh
(0 failed, both nodes.*.resources paths exercised), NOTES.txt rendered through
a scratch ConfigMap, and the null-safety and fail paths (securityContext=null
keeps NET_BIND_SERVICE; a cluster without an in-memory store, an unknown port
name, a group without http and an unknown ingress.nodeGroup all abort with a
named message; the schema rejects timeout, a string timeoutMs and a list-shaped
nodes). NO runtime verification was possible - there is no k3d or Docker daemon
in this workspace.

Part of #65

jklein added 1 commit 2026-10-02 22:40:30 +02:00

feat(stalwart): take our own published valkey chart as the subchart ... 2768d0ae05

Replaces the dependency on upstream's valkey chart with this repo's own,
resolved from the published index:

  version: "1.0.0+up9.1.2.alpine"
  repository: "https://git.jk-dev.de/api/packages/jlab-cluster/helm"

The condition is "valkeySubchart.enabled", NOT "valkey.enabled", and that is
forced rather than stylistic. Everything under "valkey:" is handed to the
subchart as its values and validated against ITS schema, and our valkey chart
ships additionalProperties: false with no "enabled" property - so
"valkey.enabled" is rejected by the subchart:

  valkey: - at '': additional properties 'enabled' not allowed

Measured. And because Helm skips a DISABLED subchart's schema validation, that
error would have surfaced only for whoever first set it to true: every render,
lint and install-test in CI would have stayed green. Adding "enabled" to
valkey/values.schema.json was rejected as the repair - that chart does not read
the key, and it would have needed its own PR and a publish first.

The host helper was also wrong for our chart and is fixed: our Service is
"<release>-valkey-service", not upstream's bare "<release>-valkey".

The values block is rewritten for our chart's schema, with the pair chosen
rather than inherited: CACHE (persistence off + allkeys-lru), because
everything Stalwart keeps in the in-memory store - rate limiters, auth tokens,
sessions, auto-ban entries, coordination pub/sub - is reconstructible, and the
node-id lease lives in the data store. The cost is named where it is set: with
allkeys-lru the rate limiters and the auto-ban FAIL OPEN under memory pressure,
so the answer is to size memory, not to switch to noeviction (which would make
writes error and take out authentication instead). secrets.authSecretRef wants
an item whose single field "auth-conf" holds the complete "requirepass <pw>"
line, not a bare password.

Chart.yaml's comments are corrected where they became false: nothing is
"consumed as it comes" any more and there is no shortfall of someone else's to
document. The Renovate reasoning stays and is now backed by measurement.

New trap, in Chart.yaml and in three places in the wiki: NEVER change valkey/
and a chart depending on it in the same PR. The dependency resolves from the
published index, so a combined PR lints, packages, renders AND install-tests
against the OLD subchart - green on an artifact that is not the one in the
diff. Two consecutive PRs.

Two things measured, not assumed, because either could have broken the update
path with no symptom but "no updates ever":
- helm dependency update AND build resolve the exact build-metadata version
  from the URL alone with an EMPTY HELM_REPOSITORY_CONFIG - a fresh runner's
  state - so the anonymous index needs no credential and
  ci/helm-dependencies.sh is untouched;
- Renovate 44.132.2 dry run with the dependency pinned low: manager helmv3
  matched the file, took registryUrls from Chart.yaml and offered
  newVersion "1.0.0+up9.1.2.alpine", versioning "helm". The self-hosted index
  and the "+up..." suffix are both handled.

README: the claim "no registry address appears in this repository" is narrowed
to what still holds - the publish target and its credentials come from
1Password and are not in the repo. The read path is now in Chart.yaml by
necessity, carries no credential, and the host was always in CLAUDE.md. Done
here rather than later so the paperless PR finds it already corrected.

Re-verified after the change: helm lint --strict (0 failed), helm package (the
subchart is embedded in the .tgz), helm template with the subchart OFF and ON
(ON was previously impossible), against CI test-values, the rendered
config.json, ci/run-quantity-render-test.sh (0 failed), NOTES.txt through the
scratch-ConfigMap trick, and the fail paths including the reworded abort. The
rendered subchart Service name matches what the helper emits, the cache pairing
really renders (maxmemory-policy allkeys-lru, appendonly no, save "", emptyDir)
and no PVC appears.

Part of #65

jlab-bot commented 2026-10-02 22:52:07 +02:00

Author

Member

Copy link

Copy source

Update: subchart switched to our own published chart, and CI is now fully green

Pushed 2768d0a. Two things to record, because the description above was written when both were open.

The subchart is now our own valkey, from the published index

Per the decision on #65. One finding worth the reviewer’s attention, because the obvious form does not work:

The condition had to become valkeySubchart.enabled, not valkey.enabled. Everything under valkey: is handed to the subchart as its values and validated against its schema, and our valkey chart ships additionalProperties: false with no enabled property:

Error: values don’t meet the specifications of the schema(s) in the following chart(s):
valkey:
- at bracket-empty: additional properties ‘enabled’ not allowed

What makes it worth flagging rather than just fixing: Helm skips schema validation for a disabled subchart, so with the conventional key every render, lint and install-test in this PR would have stayed green, and the error would have waited for the first operator who set it to true. Adding enabled to valkey/values.schema.json was rejected as the repair — that chart does not read the key, and it would have needed its own PR and a publish first (see the PR-ordering rule below).

Also corrected: the host helper was reproducing upstream’s naming. Our Service is <release>-valkey-service, not a bare <release>-valkey.

The values block is rewritten for our chart’s schema, and the persistence/eviction pair is chosen, not inherited: CACHE (persistence.enabled: false + allkeys-lru), because everything Stalwart keeps there — rate limiters, auth tokens, sessions, auto-ban entries, coordination pub/sub — is reconstructible, and the cluster node-id lease lives in the data store. Its cost is named where it is set: allkeys-lru discards keys under memory pressure, so the rate limiters and the auto-ban fail open. The answer is to size memory, not to switch to noeviction — that would make Stalwart’s writes to the store error and take out authentication instead of just the limits.

Both requested proofs hold

Question	Result
Does helm resolve the build-metadata version 1.0.0+up9.1.2.alpine from our index?	Yes, update and build, and with an empty HELM_REPOSITORY_CONFIG — a fresh runner’s state. Helm reports “Getting updates for unmanaged Helm repositories”: no helm repo add, no credential, ci/helm-dependencies.sh untouched
Does Renovate’s helmv3 manager follow a self-hosted Gitea index?	Yes. Renovate 44.132.2, dry run with the dependency pinned low: manager helmv3 matched the file, took registryUrls from Chart.yaml, and offered newVersion: "1.0.0+up9.1.2.alpine", versioning: "helm", release timestamp read from our index

The second was load-bearing rather than nice-to-have: without it the published-version form would have had no update path at all and the comment naming Renovate as the safety net would have been false.

CI is green, and it closes four of the eight open measurements

Run 419: all six checks pass, including Install-test stalwart and Trivy config stalwart (the latter needed no new ignore entries despite the added capability). The previous run 415 hung on the runner, not on the chart.

ci-stalwart-full-sfs-0   1/1   Running   0   21s
OK: chart ‘stalwart’ installed and all workloads are Ready.

What that measures, replacing inference in the list above:

1. The container execs. drop: [ALL] + add: [NET_BIND_SERVICE] against the image’s setcap cap_net_bind_service=+ep really does start — the capabilities(7) conclusion was applied from the specification and is now observed.
2. /healthz is served on the recovery listener, and answers a plain kubelet probe. This was the one thing the description flagged as unverifiable and as the likely cause if the test went red. Resolved positively: readiness went green in 21s, well inside the 300s startup budget.
3. readOnlyRootFilesystem: true holds with /etc/stalwart, /var/lib/stalwart and /tmp as emptyDirs, and the init container’s copy of config.json into the writable config volume works.
4. The authSecret EnvironmentVariable indirection resolves. Store construction reads the variable and a missing or empty one is a hard error upstream, so the password really does arrive without ever being in the ConfigMap.
5. nodes.mx.replicas: 0 behaves (0 new pods have been updated), so the explicit-zero path of the replicas helper is exercised.
6. The anonymous index resolution works on a real runner, not only locally.

Two honest limits on point 4, so it is not over-read: readiness proves the store was constructed and the secret indirection resolved — it does not prove a successful query against Postgres, because /healthz/ready only checks that a data-store handle exists and deadpool builds its pool lazily. And the test still proves nothing about mail: every listener lives in the database.

Still open (four, down from eight)

1. Does anything under /var/lib/stalwart need to survive a restart once all three stores are external? If yes, a PVC per group.
2. Does Stalwart rewrite config.json itself when the file exists and points at an empty Postgres?
3. Which address the SNAT masquerading inserts — determines proxyTrustedNetworks, readable off the first Received: header.
4. Does this cluster’s --service-node-port-range include 30025–30993? (the default does)

Plus two now only partially closed: the /healthz rate limit is still read from source rather than measured against a running server, and the Valkey subchart path is render-only — the install-test deliberately runs with it off, since the in-memory store is a database object and an unused Valkey would make the test look as if it covered clustering.

New rule, in the repo and in three wiki places

Never change valkey/ and a chart that depends on it in the same pull request.

The dependency resolves from the published index, not the working tree, so a combined PR lints, packages, renders and install-tests against the old subchart — green on an artifact that is not the one in the diff. Two consecutive PRs. It is written at the pinned version in Chart.yaml, on stalwart’s wiki page, and — because that is where the next change will actually be made — on valkey’s own page.

README

The claim “no registry address appears in this repository — not even the owner path segment” became false, since the read path is now in Chart.yaml by necessity. Narrowed in this PR (not a later one, so #78 finds it corrected) to what still holds: the publish target and its credentials come from 1Password and appear nowhere in the repo. The read path carries no credential, the index is anonymously readable, and the host was always in CLAUDE.md.

## Update: subchart switched to our own published chart, and CI is now fully green Pushed `2768d0a`. Two things to record, because the description above was written when both were open. ### The subchart is now our own `valkey`, from the published index Per the decision on #65. One finding worth the reviewer’s attention, because the obvious form does not work: **The condition had to become `valkeySubchart.enabled`, not `valkey.enabled`.** Everything under `valkey:` is handed to the subchart as its values and validated against *its* schema, and our valkey chart ships `additionalProperties: false` with no `enabled` property: ``` Error: values don’t meet the specifications of the schema(s) in the following chart(s): valkey: - at bracket-empty: additional properties ‘enabled’ not allowed ``` What makes it worth flagging rather than just fixing: **Helm skips schema validation for a *disabled* subchart**, so with the conventional key every render, lint and install-test in this PR would have stayed green, and the error would have waited for the first operator who set it to `true`. Adding `enabled` to `valkey/values.schema.json` was rejected as the repair — that chart does not read the key, and it would have needed its own PR and a publish first (see the PR-ordering rule below). Also corrected: the host helper was reproducing upstream’s naming. Our Service is `<release>-valkey-service`, not a bare `<release>-valkey`. The values block is rewritten for our chart’s schema, and the persistence/eviction pair is **chosen, not inherited**: CACHE (`persistence.enabled: false` + `allkeys-lru`), because everything Stalwart keeps there — rate limiters, auth tokens, sessions, auto-ban entries, coordination pub/sub — is reconstructible, and the cluster node-id lease lives in the data store. **Its cost is named where it is set:** `allkeys-lru` discards keys under memory pressure, so the rate limiters and the auto-ban *fail open*. The answer is to size memory, not to switch to `noeviction` — that would make Stalwart’s writes to the store error and take out authentication instead of just the limits. ### Both requested proofs hold | Question | Result | |---|---| | Does `helm` resolve the build-metadata version `1.0.0+up9.1.2.alpine` from our index? | **Yes**, `update` *and* `build`, and with an **empty `HELM_REPOSITORY_CONFIG`** — a fresh runner’s state. Helm reports *“Getting updates for unmanaged Helm repositories”*: no `helm repo add`, no credential, `ci/helm-dependencies.sh` untouched | | Does Renovate’s `helmv3` manager follow a self-hosted Gitea index? | **Yes.** Renovate 44.132.2, dry run with the dependency pinned low: manager `helmv3` matched the file, took `registryUrls` from `Chart.yaml`, and offered `newVersion: "1.0.0+up9.1.2.alpine"`, `versioning: "helm"`, release timestamp read from our index | The second was load-bearing rather than nice-to-have: without it the published-version form would have had **no update path at all** and the comment naming Renovate as the safety net would have been false. ### CI is green, and it closes four of the eight open measurements Run 419: **all six checks pass**, including `Install-test stalwart` and `Trivy config stalwart` (the latter needed no new ignore entries despite the added capability). The previous run 415 hung on the runner, not on the chart. ``` ci-stalwart-full-sfs-0 1/1 Running 0 21s OK: chart ‘stalwart’ installed and all workloads are Ready. ``` What that measures, replacing inference in the list above: 1. **The container execs.** `drop: [ALL]` + `add: [NET_BIND_SERVICE]` against the image’s `setcap cap_net_bind_service=+ep` really does start — the `capabilities(7)` conclusion was applied from the specification and is now observed. 2. **`/healthz` *is* served on the recovery listener, and answers a plain kubelet probe.** This was the one thing the description flagged as unverifiable and as the likely cause if the test went red. Resolved positively: readiness went green in 21s, well inside the 300s startup budget. 3. **`readOnlyRootFilesystem: true` holds** with `/etc/stalwart`, `/var/lib/stalwart` and `/tmp` as emptyDirs, and the init container’s copy of `config.json` into the writable config volume works. 4. **The `authSecret` `EnvironmentVariable` indirection resolves.** Store construction reads the variable and a missing or empty one is a hard error upstream, so the password really does arrive without ever being in the ConfigMap. 5. `nodes.mx.replicas: 0` behaves (`0 new pods have been updated`), so the explicit-zero path of the replicas helper is exercised. 6. The anonymous index resolution works **on a real runner**, not only locally. **Two honest limits on point 4, so it is not over-read:** readiness proves the store was *constructed* and the secret indirection resolved — it does **not** prove a successful query against Postgres, because `/healthz/ready` only checks that a data-store handle exists and deadpool builds its pool lazily. And the test still proves nothing about mail: every listener lives in the database. ### Still open (four, down from eight) 1. Does anything under `/var/lib/stalwart` need to survive a restart once all three stores are external? If yes, a PVC per group. 2. Does Stalwart rewrite `config.json` itself when the file exists and points at an empty Postgres? 3. Which address the SNAT masquerading inserts — determines `proxyTrustedNetworks`, readable off the first `Received:` header. 4. Does this cluster’s `--service-node-port-range` include 30025–30993? (the default does) Plus two now only partially closed: the `/healthz` rate limit is still read from source rather than measured against a running server, and the Valkey subchart path is render-only — the install-test deliberately runs with it off, since the in-memory store is a database object and an unused Valkey would make the test look as if it covered clustering. ### New rule, in the repo and in three wiki places > **Never change `valkey/` and a chart that depends on it in the same pull request.** The dependency resolves from the **published index**, not the working tree, so a combined PR lints, packages, renders **and install-tests** against the *old* subchart — green on an artifact that is not the one in the diff. Two consecutive PRs. It is written at the pinned version in `Chart.yaml`, on `stalwart`’s wiki page, and — because that is where the next change will actually be made — on **`valkey`’s own** page. ### README The claim *“no registry address appears in this repository — not even the owner path segment”* became false, since the read path is now in `Chart.yaml` by necessity. Narrowed in this PR (not a later one, so #78 finds it corrected) to what still holds: **the publish target and its credentials come from 1Password and appear nowhere in the repo.** The read path carries no credential, the index is anonymously readable, and the host was always in `CLAUDE.md`.

jlab-bot referenced this pull request 2026-10-02 22:54:22 +02:00

paperless-ngx: Valkey optional als Abhaengigkeit, Passwort aus dem Redis-Item #78

jklein approved these changes 2026-10-02 22:59:12 +02:00

jklein merged commit 07c75e80c1 into main 2026-10-02 22:59:18 +02:00

jklein deleted branch feat/stalwart-chart 2026-10-02 22:59:18 +02:00

jklein referenced this issue from a commit 2026-10-02 23:29:34 +02:00

feat(paperless-ngx): optional valkey subchart, password from the redis item

jlab-bot referenced this pull request 2026-10-02 23:30:13 +02:00

feat(paperless-ngx): optional valkey subchart, password from the redis item #80

jlab-bot referenced this pull request 2026-10-04 21:26:33 +02:00

Neuer Chart: stalwart (Mailserver) - erster Plan und offene Entscheidungen #65

Sign in to join this conversation.

Reviewers

No Reviewers

jklein

Labels

Clear labels

inhouse-chart

Inhouse-Chart aus der JLab-Git-Paketregistry

renovate

Von Renovate angelegt

No labels

Milestone

No items

No Milestone

Assignees

Clear assignees

jklein (Justin Klein) jlab-bot (jlab-git-bot) jlab-renovate-bot (jlab-renovate-bot)

No Assignees

2 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: jlab-cluster/helm-charts#77