jlab-cluster/helm-charts
3
0 Fork 0
Code Issues 15 Pull Requests 3 Actions Packages Wiki Activity
Page: Lessons Learned
Pages
CI CD Chart Style Guide Charts Forge and Tooling Home Lessons Learned Local Development New Chart Renovate Versioning Workflow apache tika bentopdf bulwark calibre web automated convertx cyberchef freesailarr garage gotenberg homeassistant keycloak matrix synapse minecraft router minecraft server outline paperless ngx patchmon rallly rss bridge samba satisfactory searxng stalwart sure teamspeak server template chart urlaubsverwaltung valkey voucher vault ytdlp web player zipline
Clone
39
Lessons Learned
Justin Klein edited this page 2026-10-15 17:51:52 +00:00
Unescape Escape
Table of Contents
This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

Lessons learned

Every rule in the style guide and every check in CI was bought. This page is the receipt.

They share a shape: the broken state looked exactly like the working one. That is why the fixes are structural — a rule, a check, a rename — rather than "be careful next time".

The config that changed nothing

A ConfigMap was updated. Nothing restarted, because a changed ConfigMap does not change the pod template — and subPath mounts, the usual form here, are never refreshed by the kubelet at all. minecraft-router kept routing to Services that no longer existed. Every bundle was green, the pod was Ready, and the routing table in memory was from the previous week.

Now: every pod template consuming a ConfigMap carries checksum/config, and adding it to an existing chart is a major bump. See §9.

The readiness probe that could not fail

urlaubsverwaltung probed /actuator/health/liveness and /actuator/health/readiness for two years. The split was textbook. Then the database went down and the pod stayed 1/1 Ready through the whole outage, serving errors.

Spring Boot's readiness group contains readinessState alone. The paths were right; the group behind one of them was empty of anything external.

Now: whatever aggregates behind a probe path — Spring health groups, a framework-assembled /healthz — is read off the running endpoint, not off the path name. The chart names the members explicitly, so the composition lives in the template rather than in an upstream default. See §12.3.

It recurred, and the second time it was caught before shipping — by reading the check list rather than the path. keycloak has three separate native endpoints, which looks like the problem being solved for you: /health/live without a database check, /health/ready with one. Except that upstream's table of available checks has a "Requires Metrics" column, and the database check is the single row that says yes — so with metrics-enabled=false, the chart's own default in its first draft, /health/ready would have answered 200 with the database gone. The dependency was not missing from a health group this time; it was gated behind an option named for observability, which is a place nobody looks when reviewing a probe. The chart now defaults metrics on and argues for it as a readiness setting. So the rule generalises past health groups: find the list of checks the endpoint actually performs, and find out what each one is conditional on. "This path includes the database" is a claim about configuration, not about the path.

The value that meant its own opposite

satisfactory shipped game.seasonalEvents, passed straight into the image's DISABLESEASONALEVENTS. The default true switched the events off — in every install, for the chart's entire life — while the schema described the key as "true means events are enabled".

The tempting fix is to invert the template. That would silently change behaviour under existing releases. The loud fix is a rename: the old key then fails schema validation instead of quietly meaning the opposite.

Two mechanics come with it: Go's not treats every non-empty string as true, so not "false" is false; and a value readable as neither boolean belongs in a fail, not in an implicit "off". See §12.6.

The quantity that was silently unusable

Resource quantities were formatted with printf "%v", and the float64 the fallback helpers produce flips to exponential notation from about 1e6. The bug had a loud half and a silent half.

Loud: the reformatted quantity carries e+06, the helper rejects it, the render exits non-zero. Silent: the product of a fractional limitFactor and a bare number is reformatted after the suffix check, so the render succeeds and the manifest carries memory: "1.4999985e+06" — rejected by nothing until the API server sees it at apply time.

Now: the quantity render test runs two cases over every chart on every PR, and the second one searches the rendered text rather than the exit code. It covers all charts because the helper is copied per chart, so the regression arrives by copy — in a PR where the copied bug is in no diff anyone reads.

The override that deleted what it was supposed to configure

An override written without children is YAML null, and null erases the block: securityContext: null renders an unhardened container that is healthy and silent; a probe simply disappears; resources loses its requests and its computed limits.

Wrapping the block in {{- if }} is the same bug with the evidence removed — a guard does not render a visible null, it removes the probe entirely.

Now: probes, security contexts and resources go through the null-safe helper, whose semantics are the opposite — an empty block means "chart defaults apply", while an explicitly set value still wins, false and 0 included. See §12.2.

The half-protected chart, and the template no pipeline renders

The fix above was applied per block, and satisfactory reached 7.0.0 with game null-safe and its two neighbours not. That is worse than none of them being safe: an operator who watches --set game=null fall back cleanly concludes the chart handles erased blocks, and --set server=null then aborts the render. The quieter half was the single erased key — a nulled server.gamePort rendered a Service with port: and nodePort: empty and a StatefulSet with an empty containerPort. Nothing rejects that until the API server does, at apply time.

The second finding is the one that generalises. server was read in four files, and the fourth was templates/NOTES.txt — which helm template does not render. A fix confined to the manifests would have passed every local render, passed lint, passed the pipeline, and still aborted on the first real helm install. For a chart excluded from the install-test, as this one is, no stage renders the notes at all: it is template code the whole pipeline is blind to. helm install --dry-run=client is not the workaround — it still wants a reachable API server.

Now: every chart-owned block a template dereferences goes through the fallback helper, not just the three kinds named above. A key deliberately left un-defaulted is marked as the exception it is — in the schema, in values.yaml and on the chart page — with a pointer to the issue where the trade is argued; satisfactory.storage is exactly that, and it was left standing rather than overturned in passing while the bug next to it was fixed. And when a values change touches a chart's notes, the notes are rendered explicitly, without a cluster: wrap the verbatim NOTES.txt body in a {{- define }} in a throwaway copy of the chart and include it into a scratch ConfigMap — same engine, same root context, and helm template renders that.

The third lesson is about the fix, not the bug: a bugfix that also settles an open design question is no longer a bugfix. The tempting move here was to default storage along with everything else — it is one line, it removes a visible inconsistency, and the argument for it is good. It also would have been the only change in the release that altered rendered output, against a documented decision someone else had reasoned out. It went to its own issue instead, and the patch kept the property that made it cheap to review: byte-identical output for every valid input.

The strict schema that had to meet reality first

A schema draft demanded a boolean for tikaGotenberg.enabled. All three live paperless releases write it as the string "true". Rolling that out would have broken them at upgrade time.

Now: every schema is validated against the real values — the cluster bundle and helm get values — not only against chart defaults, which are valid by construction and prove nothing. The result per chart is a list of rejected keys: the cluster side's work order before the version is pulled up.

The green tick for a chart nothing was attempted on

Excluded charts used to get an install-test job that ran the script, printed SKIP and exited 0. The forge knows only the exit code, so the check list showed Install-test satisfactory ✓ — indistinguishable from a chart that really came up in a cluster.

Now: excluded charts are removed before the matrix is built, so no job exists. And because an absence says nothing on its own, the excluded charts of a PR are written into the job summary with their documented justification — "why" is the question a review actually asks. See CI/CD.

The check a broken cluster passed

kubectl get nodes exits 0 on an empty list. A readiness check written that way went green against a cluster whose node had never registered; the failure surfaced five minutes later, somewhere unrelated.

kubectl wait --for=condition=Ready nodes --all is not the shorter way to write it — with zero nodes the condition is vacuously satisfied.

Now: assert the thing you mean — at least one node, and Ready. Generalized: a check that a broken system passes is worse than no check, because it also buys confidence.

A generous environment is no friend

The k3d cluster name was hard-coded as install-test. On GitHub each matrix leg gets its own VM with its own Docker daemon, so two legs never saw each other's cluster and the bug could not show. On Gitea all legs share one daemon: one leg logs Re-using existing network, the other dies on The container name /k3d-install-test-server-0 is already in use, and the first leg's rollback then deletes the shared network out from under the second. Two red legs, and neither message names the cause.

The bug was wrong on GitHub too. For two years the roomier environment carried it instead of showing it.

Now: one cluster per matrix leg, named after the chart, with a length check. And the general form: when you want to know whether something is correct, a generous environment is no friend.

The publish that uploaded nothing and went green

curl exits 0 on HTTP 401, 403 and 409. Unlike helm push, a failed upload therefore left the job green with nothing published.

Now: --fail-with-body — non-zero exit, and the response body still printed so the log names the cause. Two neighbours of the same step: credentials go in via --config - rather than --user so they stay out of argv, and the file name is built rather than globbed, because --upload-file takes one file and a glob matching two silently turns the second into curl's URL argument.

The secrets that were loaded and then empty

After the bump to load-secrets-action v5, every ${{ env.* }} reference in later steps resolved to an empty string — while the loading step still logged Populating variable and stayed green. v5 no longer exports the values as environment variables for later steps the way v4 did.

Now: values are read via steps.<id>.outputs.*, mapped into the consuming step's own env:, quoted, and guarded with :? — because an empty value used to collapse into a missing argument rather than an error.

The update rule that was valid but matched nothing

A Renovate rule that is syntactically fine and matches nothing produces no PRs — and no PRs is also what "everything is current" looks like. The mediathekarr digest rule sharpens it: while the pin is current, a matching rule and a broken one produce the identical empty result.

Now: a new rule is verified with two dry-run fixtures — the real value (expect: no update) and a deliberately stale one (expect: exactly one update back to the pinned value). Anchored patterns are documented as what they are: reordering or renaming the keys they match fails nothing, it silently stops matching.

The comparison that read a hash as a number

Renovate's regex versioning parses major, minor, patch, build and revision with Number.parseInt. A commit hash captured as build therefore becomes whatever digits lead it: 9fea41204 is 9, 777ba8fa4 is 777, a hash starting with a letter is NaN. Renovate offered an image pushed earlier the same day as an update to a newer one — a downgrade that looks exactly like the routine date bump a reviewer waves through, and automerge then carries it to main.

Now: a capture group is a claim about how the value is compared. Parts of a tag that carry no ordering stay outside every group. prerelease is not the escape hatch — it sorts correctly but marks every tag unstable, handing the chart to ignoreUnstable.

The hook that was skipped without a word

postUpgradeTasks only runs when the command is allow-listed in the instance config. A missing entry is skipped silently, and every chart PR then stays red on the version check with nothing pointing at the cause.

Now: it is verified rather than assumed — the proof is a Renovate PR whose diff carries a version: change, because nothing but ci/renovate-bump-chart.sh writes that line. The allow-list is not visible from this repo, so if it is ever dropped the symptom here is red chart PRs with no explanation.

The annotation that promised a restart it cannot perform

operator.1password.io/auto-restart enumerates exactly one workload type in the operator's source: DeploymentList. On a StatefulSet it is a silent no-op — verified against operator 1.12.0, with the generalized restart code present since 1.11.0 but the PR that would add StatefulSets still open.

Setting it there is worse than not setting it: it advertises a guarantee that does not exist. The affected charts document the manual kubectl rollout restart instead.

The environment variable that was lost on upgrade, quietly

An env entry repeating a name the chart already sets behaves differently depending on when it arrives. At install both reach the pod and the user's wins. On an existing release Kubernetes merges env by name, the pair collapses to the chart's own value — and helm upgrade still reports success.

Measured: helm get manifest shows the variable twice while the live Deployment shows it once, carrying the chart's value.

Now: a chart whose NOTES.txt tells operators to override one of its own variables says that the override only takes at install time, and that a green upgrade proves nothing — the value belongs read back off the Deployment.

The indexer that answered every search with nothing

MediathekArr fetches its ruleset once at start and never retries. Started before gluetun's tunnel was up, it answered every Newznab search with 0 results for over eleven minutes — container up, Ready, nothing in the log. The failure case looked exactly like the healthy one.

Kubernetes cannot order two regular containers, and an init container waiting for gluetun would deadlock, since gluetun only starts once the init containers are done.

Now: a start gate in front of the image entrypoint polls gluetun's health server and exits when the timeout passes instead of starting anyway — a crash-looping container is visible, a silently empty indexer is not.

A guard that never fires still looks like a guard

Five OnePasswordItem templates rendered itemPath: {{ .Values.secrets.<x>Ref }} without a required(). Unset, the CR is created with an empty itemPath: helm install succeeds, the object exists, and nothing reports a problem. The operator has nothing to resolve, the Kubernetes Secret never appears, and the failure arrives minutes later in the pod — CreateContainerConfigError, or a mount that fails with couldn't find key ... so the container is never created at all. Four of the five were not optional; the fifth, searxng's valkey reference, was correctly optional and gated with {{- if }}, which is the other half of the rule.

The second finding came out of fixing the first, and is the more general one. The plan was to back each guard with a required array in values.schema.json. Measured on Helm 3.22.0 with a probe chart, that array could never fire: these keys ship with null as their default, so the key is present at validation time, and JSON Schema's required tests presence, not null-ness. Only a user null written over a non-null default is coalesced away — a null default is not. And both ways of making the array fire (dropping the default, or narrowing ["string", "null"] to "string") turn helm lint --strict red on the chart's own defaults, which the pipeline runs.

Now: a value with no valid default is enforced by required() in the template, with a message naming the values path, and described as required in the schema — never by a required array, which would be a safeguard that cannot fire, dressed as one that can. An optional reference says so in values.yaml and in the schema description, and its template renders nothing when the value is empty. See §12.

The temporary directory Ruby refused

sure mounts /tmp as an emptyDir so readOnlyRootFilesystem: true can hold. Kubernetes creates an emptyDir mode 0777 without the sticky bit, and Ruby's Dir.tmpdir skips exactly that shape (world_writable? && !sticky?). With the image's root filesystem read-only there was no candidate left at all, so both containers died during boot on ArgumentError: could not find a temporary directory — raised from MiniMagick while Rails was still loading, long before anything asked for a temp file.

The emptyDir root is owned by root, so its mode cannot be changed from a non-root container. What can be done needs no privilege at all: create a subdirectory in it (allowed, it is 0777) and chmod 700 what you just created.

Now: a tiny init container — the app's own image, same non-root uid, no added capability — creates ruby and home directories inside the tmp emptyDirs, and the containers run with TMPDIR and HOME pointing there. HOME is part of the same lesson: /home/rails is read-only, and Bundler answers that by building a throwaway home per invocation, worker probe included.

The probe that followed a redirect into the wrong scheme

Readiness for sure's web container was an httpGet on / — the path that provably reaches the database. The pod never became Ready, with http: server gave HTTP response to HTTPS client, while the application was answering correctly.

The kubelet's HTTP prober follows redirects to the same host. Rails, told by RAILS_ASSUME_SSL=true to treat the request as secure, answered 302 with https://<pod-ip>:3000/registration/new — and Puma does not speak TLS. The redirect that proved the probe was doing its job (that target is the branch User.count decides) was also what broke it.

Now: the probe runs curl without -L and compares the status code itself, accepting 2xx–3xx exactly as the kubelet would. Generally: an httpGet probe against an app that redirects to itself is a probe on the redirect target, and a scheme or port the app does not serve turns a correct answer into a failure.

The pod evicted for a cache nobody sized

ephemeral-storage: 10Mi is the scaffold's placeholder, and it was copied into a Rails chart unchanged. Every start ended in Pod ephemeral local storage usage exceeds the total limit of containers 60Mi after about two minutes — an eviction, not a crash, so the container logs showed a healthy boot and nothing else.

Two things count against that limit and are easy to forget: emptyDirs and the container's stdout log. This image also rebuilds its Bootsnap cache under /rails/tmp on every pod start, because the emptyDir mounted there hides the copy baked into the image.

Now: the chart requests 256Mi per app container, and says in values.yaml that this is headroom rather than a measurement. The general form: ephemeral-storage is not boilerplate — a chart that mounts emptyDirs has to budget for what is written into them.

The CLI that reports an error and exits 0

Not an outage — a probe that was checked before it shipped, and the check is the entry. The obvious exec probe for valkey is valkey-cli ping. Against a password-protected server, an unauthenticated ping answers NOAUTH Authentication required. — and valkey-cli exits 0 while printing it. In valkey 9.1.2 (src/valkey-cli.c) an error reply becomes a non-zero exit in exactly one branch, guarded by config.set_errcode, and the -e flag is the only thing that sets it.

So the natural probe would have been Ready for a server that rejected every single command — including, for instance, one started without the password at all.

Now: the chart's probes authenticate (through VALKEYCLI_AUTH, never an -a argument that would land in the process list), pass -e, and assert the answer (grep -qx PONG). Generally: an exec probe wrapping a CLI is only an assertion once that CLI's exit code has been read off its source or measured. "The command ran" and "the server is healthy" are different statements, and a client that prints errors to stdout with status 0 turns the first into the second for free. The same question applies to psql, curl without --fail, and every *-cli ping in this repo.

The password that would have been optional

Same chart, the other half of the same review. valkey takes its password as a finished configuration line out of a 1Password field (requirepass <password>), mounted as a file and included — which is what keeps it out of the process list and out of every rendered manifest. The cost of that design is that the vault field is now syntax: requirepas, a stray quote or an empty value yields a Valkey without a password. No render error, no crash, a green pod — and the official image compiles protected-mode off, so "no password" means "reachable from the whole pod network".

Now: an init container greps the mounted file for a requirepass <value> line and refuses to let the pod start without it, naming the 1Password field and never printing its contents. Generally: when a secret's format becomes load-bearing, something has to validate the format — a value that is merely present satisfies Kubernetes, and the failure it produces can be a working server with its protection off.

The green history from another machine

The move to Gitea changed the execution environment. The install-test matrix follows the charts a pull request changes, so a chart nobody had touched since the move still carried its last green tick — earned on a GitHub runner, months earlier — and it was read as "this chart comes up". It said nothing about the current runner.

It surfaced by accident. A pull request correcting a comment in 21 charts made every one of them "changed" and thereby became the first all-charts install-test after the move: 19 of 20 matrix charts came up in 12m41s, and rss-bridge did not. Its last green run was from 2026-08-29 and a GitHub runner (Gitea #51).

Now: the install-test job is one reusable workflow with two callers, and the second runs it over every chart on demand (Gitea #52) — after a runner change, a K3S_IMAGE or kubectl change, a fixture change, a forge move. Every leg records date, runner label, K3S_IMAGE and kubectl version next to its outcome, because a green run nobody notes is unknown again next time. See CI/CD. Generalized: a check whose scope follows the diff never re-checks what the diff does not touch — and what it quietly stops covering is the environment itself.

The instruction block no fetch could refresh

The rules an agent works under arrive as a project-instructions block at the head of its context — the binding copy of CLAUDE.md. It reads as the current file, and carries nothing that would suggest otherwise. It is a snapshot of the spawning session's working directory, taken when that session started.

On 2026-09-27 an agent reported a "contradiction" in CLAUDE.md — the remote naming — that two pull requests had already fixed, and quoted the stale text as the rule in force. The obvious explanation was a stale checkout, and that was half of it: in the shared checkout, main..HEAD showed 105 differing files while the difference against origin/main was a single line. The sharper measurement came from the next agent the same day, after that working directory had been deliberately fast-forwarded and the result checked ("Already up to date"): its block was still from before Gitea #47, while its own worktree was identical to origin/main — git rev-list --left-right --count origin/main...HEAD returned 0 0. A current checkout and stale instructions, at the same moment. Nothing broke either time, and the reason belongs to the lesson: both agents had checked their work against origin/main instead of against what they had been handed.

Now: worktrees are cut from a freshly fetched origin/main, diffs are read against origin/main and never against the local main, and an agent checks a rule against the file in its own worktree rather than against its context block. The spawning session's directory has to be current before it starts, because a fetch afterwards refreshes nothing that was already handed out. See Forge and tooling. Generally: a document delivered as context is a copy with an age, and the copy shows no sign of it — so what a rule says has to be settled against the source, not against the copy.

The probe that would have passed on a redirect, and meant something else next week

Not an outage — the third entry on this page about reading a path off the running service rather than off its name, and the first where the wrong probe would have been green rather than red.

/ is the obvious probe target for convertx, and the kubelet would have accepted it: httpGet counts 3xx as success. But / in ConvertX is never a 200. It answers 302 to /setup while the instance has no users, and 302 to /login from the moment the first account exists. So the probe would have passed on day one by proving that the setup form was reachable, and on day two by proving something entirely different — with no event, no restart and no diff to mark the switch. A redirect target that is a function of application state is not a health check; it is a health check per state, and only one of them was ever considered.

Now: all three probes use /healthcheck, a static 200 the application exempts from authentication, and the chart's values.yaml carries the reason next to the probes so the "inconsistency" is not tidied away later. Generally: a 3xx makes a probe pass, so a redirecting path hides the question instead of answering it — and the answer has to hold for every state the application can be in, not just the one it is in while you are testing. Compare the sure entry above: there a redirect turned a correct answer into a failure, here it would have turned an irrelevant answer into a success.

The entrypoint we skipped, and the obligation that created

convertx's image carries no USER and starts as root; its /entrypoint.sh exists solely to drop to uid 1000 with gosu. Rather than relax the hardening baseline for it, the chart overrides command with the image's own CMD and lets runAsUser and fsGroup do that work. Read at v0.19.0 the entrypoint does nothing else — no migration, no config generation — and with PUID unset its first branch is already a plain exec "$@". The bypass costs nothing today.

That is exactly the problem. The cost is not in the version you verified; it is in the next one. If upstream ever moves real work into that entrypoint — a data migration is the obvious candidate — the chart skips it with no error, no log line and no failed probe: the application simply starts against an unmigrated volume. Nothing in an app-version bump would surface it, because the bump changes one line in Chart.yaml and the entrypoint is not in the diff anyone reads.

Now: the obligation is written at the command: line in the StatefulSet, not only in the chart page and not only in the pull request — re-read the upstream entrypoint.sh at every app-version update, and if it has grown beyond uid/gid handling, stop. Generally: overriding a container's entrypoint forks it silently, and a fork with no diff is a fork nobody reviews. The mitigation for a decision whose risk is entirely in the future is to leave the instruction where the future change will be made, not where the decision was argued.

The file that moved, and the comments that kept vouching for it

Moving publish-helm-chart.yml out to workflow-util/pipelines left three references behind in this repository, and the sharpest of them was a comment in install-test.yml that cited the departed file as documenting a trap "at length".

That comment still reads as a complete justification. Nothing about it looks broken: it is grammatical, it is specific, and it points at a long explanation — which is now in a different repository, where nobody following the comment will look. The README had the same shape in its "runs in five places" list, naming a file that no longer existed here, and CLAUDE.md still called the workflow ours.

A moved file does not make its inbound references stale. It makes them false — and a false reference is worse than a missing one, because it answers the question instead of raising it.

Now: before a file leaves the repository, grep for its name across the whole tree, comments included, not only the uses: and include lines that a tool would find. All three cases here came out of that one grep, and none of them was in the diff the move itself produced.

The capability list that described a product, not our server

The plan for the Sonar analysis carried a table: 5 shell scripts under ci/ → Shell analyser, 30 workflow files → GitHub Actions analyser, charts → Kubernetes/Helm, schemas → JSON, everything → Secrets. It was taken from SonarQube's "supported languages" page, and it was used to argue that the addition was worth having — the files with the actual logic get analysed.

The first run loaded two quality profiles, json and yaml. The sensors that did work were IaC Kubernetes and the secrets detection. Shell and GitHub Actions never ran, because analysers are server-side plugins and this instance does not have them.

The error was not the table. It was the move from "the product supports X" to "our pipeline will analyse X" — a documentation page describes the newest release of someone else's software, and says nothing about the installation we actually call. The gap was invisible because both halves were true: the product does support Shell, and our scanner did run.

Now: the table in README and on this page carries a third column, "Runs?", filled in from a measured run and headed as measured rather than quoted. The two "no" rows are the valuable ones — they say plainly that the five ci/ scripts and the thirty workflow files are not analysed at all, so nobody mistakes the addition for a check it is not.

Generally: a capability list is about the product; only a run is about the installation. The same shape as a stale context block — both read as authoritative, and neither is the thing you are actually talking to.

The source address every NodePort quietly replaces

Not a mail problem, although stalwart is where it was worked out. It recurs in every chart whose application evaluates the client address — rate limiting, auto-banning, geo rules, audit logs, allow-lists.

A Service of type NodePort or LoadBalancer with externalTrafficPolicy: Cluster makes kube-proxy masquerade the source address to the receiving node's IP. It has to: the reply would otherwise leave through a different node than the one the sender holds its connection with. So the application sees one address for every client, forever, and nothing reports it — the connection works, the request is served, and the log line is written with the wrong address in it.

Local preserves the address but forwards only to pods on the receiving node. That is not a free choice either, and in the stalwart topology it is not a choice at all: an HA proxy delivers to the three control-plane nodes and no pod is ever scheduled there, so Local would drop every packet rather than some.

What the lost address costs, in the order it hurts — from the mail case because it is the most complete: the auto-ban switches the service off (it bans the masqueraded node IP and with it every client at once); SPF checks the wrong address and takes DMARC with it; blocklists report everything clean; per-IP rate limits collapse onto one address; greylisting loses its key; and the Received: chain becomes worthless — the only one of those that destroys evidence retroactively instead of merely failing. Read the list as a template: any per-client decision an application makes is being made on one address.

The remedy is a proxy that terminates the connection and re-announces the client, not a Service setting. A reverse or HA proxy accepts the inbound connection and opens its own to the node, on which it can prepend a PROXY v2 header — for raw TCP such as port 25 as well, where no remote peer would ever send one. For HTTP it is X-Forwarded-For instead, plus discarding whatever forwarding headers the client brought itself, which is the half that gets forgotten and is what turns the mechanism into a spoofing hole.

Three traps come with it, and they are why this is a page entry rather than a chart note:

  • A trusted network makes the header mandatory. A connection from a trusted source without one is misread and hangs until timeout; a header from an untrusted source is read as protocol data (Stalwart parses PROXY TCP4 … as an SMTP command). Enabling trust and enabling the header is one step, not two, and it is tested on a spare port before the real one moves.
  • In Kubernetes the trusted set and the probe set must be disjoint. Kubelet probes arrive from the node running the pod, with no header. In the stalwart topology that works out only because pods never run on the nodes mail enters through — a load-bearing assumption, not an operating preference, and anyone who later schedules a pod onto one of those nodes breaks it.
  • An ingress controller is a third case again. The application sees the controller's pod IP, which the PROXY protocol does not address; that is the X-Forwarded-For path. This failure is the slowest and most confusing: the auto-ban eventually bans the controller and the service dies for everyone at once, weeks after the install, looking like a server fault.

Which address the masquerading actually inserts is to be measured, not assumed — for mail it stands in the first Received: header after one connection.

The rate limit that counted a node instead of a client

The same mechanism from the other end, found by reading source rather than by an outage — which is the only reason it is not written in the past tense.

Stalwart rate-limits anonymous HTTP requests to 100 per 60 seconds, keyed on the source IP, and that applies to /healthz like everything else. Kubelet probes arrive from the node's IP, so every pod of that application on one node draws from the same bucket. Exceeding it returns 429, a 429 fails a probe, and a failed liveness probe restarts the pod — a restart loop that appears only once a node is dense enough, and reads as an unstable application.

The arithmetic is short and belongs in the values file rather than in a review: liveness and readiness at periodSeconds: 10 cost 12 requests/min/pod, so the bucket is gone at about eight pods on one node. At 20/20/10 it is 6/min and about sixteen. The startup probe gates liveness and readiness, so the three never run together and the worst case is the same in both phases — worth knowing, because the intuitive sum is half again too high and would argue for tighter intervals than necessary.

Now: stalwart ships 20/20/10 with the sum as a comment in values.yaml and in the probe fallbacks in _helpers.tpl, so a later tightening has to pass the reasoning. Generally: a per-IP limit in an application is a per-node limit for probes, and it is the first thing to check in anything that rate-limits by address.

A tcpSocket probe bypasses such a bucket entirely and is the obvious escape. It was rejected here for a reason worth reusing: it would have been green in exactly the failure the chart exists to contain (a fail-opened node binds the port too), it proves strictly less than an HTTP health endpoint, and it would have had to differ per workload, so the fallback default could no longer carry the probe's context.

The hardening that stopped the container from starting

The plan was the strict one and it was argued well: bind only high ports, so nothing privileged is ever bound, so capabilities: drop: [ALL] can stand with nothing added — no NET_BIND_SERVICE, no sysctl, no documented deviation. Three escape routes from the earlier survey were discarded on that basis.

It cannot work, and the reason is not in the application at all. The stalwart image runs setcap cap_net_bind_service=+ep on its binary. The e sets the file's effective bit, which makes it a capability-dumb binary, and capabilities(7) then has the kernel check at execve that the process obtained the full file permitted set, failing with EPERM when the bounding set masked any of it out. drop: [ALL] empties the bounding set. So:

exec /usr/local/bin/stalwart: operation not permitted

The container never starts. Not unhardened, not degraded — it does not run.

Two things made this easy to get wrong. The reasoning "nothing below 1024 is bound, so the capability is not needed" is true and irrelevant: the check happens before the program binds anything and never asks whether the capability would be used. And all three obvious escapes fail for the same single reason — high ports do not help, the -alpine variant carries the identical setcap, and ip_unprivileged_port_start addresses binding rather than exec.

Now: drop: [ALL] plus add: [NET_BIND_SERVICE] as a documented deviation, in values.yaml and in the _helpers.tpl fallback — an erased securityContext falling back to the repo-standard empty add would otherwise produce a chart that cannot start, with nothing pointing at the cause.

And the part that is the actual lesson: the comment has to say that the capability is never used. It is spent entirely on the exec check — inside a pod the kubelet sets ip_unprivileged_port_start=0, so the low ports need no capability anyway. A deviation documented only with its name reads, to the next hardening pass, as a dead grant to clean up; documented with its mechanism, it survives. "Needed for binding" would have been both wrong and an invitation.

Generally: the image is part of the security context. A file capability, a fixed uid, an entrypoint that wants to write — these constrain the pod spec regardless of what the application does, and they are read off the Dockerfile that builds the published image, not the one at the repository root. Here those were two different files, and only docker-bake.hcl said which.

The missing file that opened a database nobody meant to use

If stalwart's config.json is absent — a renamed ConfigMap key, a failed mount — the server does not abort. It enters recovery mode, prints a temporary admin password, and opens an in-memory ephemeral store. Probes go green, mail is accepted, and all of it is gone at the next restart.

An unparsable file does abort. It is specifically the missing-file case that fails open, which is the inversion that makes it dangerous: the careful operator who validates the file's syntax is protected, and the one whose mount silently failed is not.

Now: the chart always renders the file and seeds it with an init container, so its absence means something else is broken, and the trap is written on the chart page. Generally: check what an application does when its configuration is missing, not only when it is wrong — the two paths are often written by different people, and "absent" is the one that tends to get the lenient fallback.

The green check that tested the wrong artifact

Caught by reasoning before it cost anything, which is why it reads as a rule rather than an incident — and it is the one hazard that appears only once a repo's own chart becomes somebody's subchart (stalwart → valkey, Gitea #65).

A Helm dependency resolves from the published index, not from the working tree, and a chart here is published only by a push to main that touches it. Put both changes in one pull request and every helm command in that PR — lint, package, render and the install-test — pulls the old subchart, because the new one does not exist yet. The PR goes green against an artifact that is not the one in the diff, the subchart's change is reviewed while nothing exercised it, and the first real use of the new subchart is somebody's first install after the merge.

Now: two consecutive PRs — change the subchart, merge, let it publish, then raise the version: in the dependent chart. The warning sits in three places, and the choice of places is half the lesson: stalwart/Chart.yaml at the pinned version, the dependent chart's page, and — because that is where the next change will actually be made — the subchart's own page, which someone editing valkey/ is far likelier to be reading than the page of a chart they may not know depends on them.

Generally: a dependency on your own artifact tests the published copy, not the one you are editing. Any build that resolves from a registry rather than from the checkout has this property, and the symptom is always a green check on the wrong input.

The condition key the subchart rejected

Two findings from the same change, both of which could have shipped silently.

The one that was caught. The convention for a Helm dependency is condition: <name>.enabled — and it is a convention, not a rule: everything under that key is handed to the subchart as its values and validated against its schema. Our charts ship additionalProperties: false, so valkey.enabled is rejected by the subchart itself:

Error: values don't meet the specifications of the schema(s) in the following chart(s):
valkey:
- at '': additional properties 'enabled' not allowed

What earns this an entry is when it appears: Helm skips schema validation for a disabled subchart, so the default path renders perfectly clean. Every render, lint and install-test passes, and the error waits for the first operator who switches the dependency on. The repo's previous dependency — upstream's oauth2-proxy — never showed it, because that chart's schema is permissive, so the convention looked safe.

Now: the condition is a key of the parent's own, valkeySubchart.enabled, which is also the honest shape, since enabled was never a value of the subchart. The tempting repair — adding an enabled property to the subchart's schema — was rejected for the reason this page keeps repeating: that chart does not read the key, and a values key that does nothing is worse than its absence. It would also have needed its own PR and a publish first, per the entry above.

And the two things measured rather than assumed, because this dependency points at our own self-hosted index and the version carries build metadata (1.0.0+up9.1.2.alpine) — either of which could have broken the update path with no symptom but "no updates, ever":

  • helm dependency update and build resolve the exact build-metadata version from the URL alone with an empty HELM_REPOSITORY_CONFIG — the state a fresh runner is in, so no helm repo add and no credential; the index is anonymously readable.
  • Renovate's built-in helmv3 manager does follow a self-hosted Gitea index: a dry run (Renovate 44.132.2) with the dependency pinned low returned newVersion: "1.0.0+up9.1.2.alpine", versioning: "helm", with the timestamp read from our index.

The second was load-bearing for the decision, not a nice-to-have: without it the published-version form would have had no update path at all, and the comment naming Renovate as the safety net would have been false. "A capability list is about the product; only a run is about the installation" applies to update automation too.

The guard that could not fire, and the switch that defaulted to ON

The second chart to take valkey as a subchart — paperless-ngx, Gitea #78 — produced three findings, and all three were found by trying the thing rather than by reading about it.

A parent cannot default a subchart's value, and the attempt fails in the most misleading way available. The plan said valkey.secrets.authSecretRef should default to secrets.redisSecretRef, because the auth-conf field lives in that same 1Password item. That is not implementable: a values.yaml is never templated, and values are merged statically before any template runs, so there is no mechanism — import-values runs child→parent, and the subchart does not read global. What makes it an entry rather than a footnote is the failure mode. The expression lands verbatim in the subchart's itemPath, where YAML reads {{ as a flow mapping:

Error: YAML parse error on paperless-ngx/charts/valkey/templates/valkey.secret.yaml:
error converting YAML to JSON: yaml: invalid map key:
map[interface {}]interface {}{".Values.secrets.redisSecretRef":interface {}(nil)}

A parse error, in somebody else's template, naming neither the value that caused it nor the file it was written in. The plan was wrong and was corrected on the issue before anything was built; the operator now types the path twice.

A fail() in the parent could not have reported it, because it never runs. The first draft put a guard in the parent's _helpers.tpl and called it from a rendered template — the right shape, per the entry above about validation that is skipped. It still fired never: Helm renders a subchart's templates before the parent's, so the subchart's own required() always wins, and its message names secrets.authSecretRef — the path inside the subchart, which is not the path anyone has to edit. The rule moved to the parent's values.schema.json as an if/then, because schema validation runs before any rendering at all. The general form: a rule about the relationship between two charts' values has exactly one place it can live, and it is not a template.

And the one nobody was looking for: erasing the dependency switch turns the subchart ON. valkeySubchart: null does not mean "the default applies". A null override deletes the key, Helm then cannot resolve the condition: path, and an unresolvable condition counts as enabled — so the block that reads like "off" produces a Valkey with no password configured, reported only by the subchart's own error. Everywhere else in these charts an erased block falls back to the chart defaults (§12, issue #99); the dependency switch is the one block where that is backwards, so the schema now requires it. An empty {} is fine, because Helm merges that with the default — which is precisely why the dangerous case is easy to miss: the two spellings look equivalent and behave oppositely.

The snapshot that was not a restore point

The plan for garage (Gitea #81) rested on a backup story that read as complete: the data lives on an encrypted Longhorn volume, Longhorn snapshots it and backs the snapshot up off-site, and the backup target only ever sees ciphertext because the backup engine reads below the decryption layer. All of that is true. It is also only true of the blocks.

A file-system snapshot of a live embedded database is not a restore point. Taken in the middle of a write it captures a half-updated file, and Garage says so about its own metadata database in as many words — which is why it recommends its own snapshot mechanism "in all cases", not as an extra for the careful. So the half of the store that can be rebuilt from elsewhere was protected, and the index that makes the blocks addressable was not. Nothing reports that asymmetry: the snapshot exists, the backup job is green, the volume is restorable, and the restored database may simply not open.

Two things made it worse in that specific chart, and both generalise:

  • The durability setting was off by default. metadata_fsync ships disabled, and a replicated block layer cannot help with a write that was never handed to the kernel — it is in no replica and in no snapshot. A storage layer with three replicas reads like it covers this. It does not.
  • Replication replicates corruption. At replication_factor = 1 the metadata has no peer to be repaired from, and the volume layer copies a corrupted file as faithfully as a good one. Redundancy against device failure is not redundancy against file corruption, and the two are easy to file under the same heading.

Now: a chart that puts an embedded database on a volume answers three questions before it is called backed up — does the engine have a consistent-copy mechanism of its own, is it switched on, and does its output live on a different volume than the database? The last one is the cheap part and the easy one to skip: a recovery point stored on the volume whose corruption it exists to repair is not a recovery point. garage keeps metadata and data on one PVC (so a volume snapshot catches index and blocks atomically) and the database snapshots on a second one.

And know the window, because "snapshots are on" is not a retention policy: Garage keeps exactly two, the first after half the interval, so the oldest recovery point is between one and two and a half intervals old and never older. A corruption found the next working day is older than everything that still exists.

A permission check that assumes it owns the filesystem

garage offers rpc_secret_file, which is the shape this repo prefers for a secret — a mounted file instead of an environment variable, as valkey argues at length. The plan specified it. It cannot work, and the reason is a collision of two correct-looking checks:

  • Garage refuses to read a secret file whose mode carries a group or other bit: metadata.mode() & 0o077 != 0, with an error that says world-readable while the mask also tests the group.
  • The kubelet ORs the group-read bit onto every file of a secret volume as soon as fsGroup is set (Chmod(info.Mode() | 0440) for a read-only volume). Without fsGroup the file stays owned by root — and a non-root container then cannot read it at all.

Kubernetes can set the group of such a file; it can never set its owner. So for a hardened, non-root pod there is no combination of defaultMode and fsGroup that satisfies both sides: one of the two always refuses. The application's check is not wrong — it is written for a host where the operator owns chmod 0600 and the file's uid — it is simply unsatisfiable on this platform.

What makes this an entry rather than a quirk of one chart is how it presents and what the tempting repairs cost. It presents as a pod that will not start, with a message about a file mode nobody can change from inside Kubernetes, which sends the reader looking for the wrong knob. And the two repairs that suggest themselves are both worse than the environment variable: allow_world_readable_secrets = true switches the check off for all of that application's secret files, including ones nobody has configured yet, and an init container that copies the file with mode 0600 needs a second container image when the main one is FROM scratch — plus an emptyDir with a secret in it.

Now: before specifying *_secret_file for an application, check what its permission test actually masks. A 0o077 test (or any test that includes the group bits) means the file form is unavailable to a non-root pod, and the honest route is secretKeyRef with the price written at the value: the secret is in the process environment, readable by anyone who can exec into the pod. Write the mechanism next to it, not the conclusion — "we use the env var here" invites the next round to tidy it back into the file form, and then the pod stops starting.

The application that rewrites its own installation at start

The plan for keycloak (Gitea #83) specified the three things this repo specifies for every chart: the stock upstream image, probes on the application's own health endpoints, and readOnlyRootFilesystem: true. Each is right. Together they cannot start a pod, and nothing in the plan could have shown that, because the conflict is not between any two of them — it is between all three and a property of the image.

Keycloak splits its options into build options and runtime options. db, health-enabled and metrics-enabled are build options; the published image is augmented with their defaults; and supplying a different value at runtime makes the server re-augment itself at start, rewriting four files inside its own installation directory. Upstream describes that start without euphemism — it "creates an image that is mutable" — which is exactly what a read-only root filesystem forbids. And the escape that suggests itself is not one: "then don't configure the database at runtime" leaves the probes, and health-enabled is a build option too. The hardening baseline and the probes were in conflict through a third thing neither mentions.

The repair has a trap of its own, and it is the part worth remembering, because the obvious version of it looks finished and fails: mounting an empty volume over the directory the augmentation writes does not work either. The runtime's entry point loads a serialized application file from that directory before it dispatches any command — the build command included — so an empty mount breaks the very command meant to fill it. What works is seeding: one init container copies the image's own output into the volume, a second runs the build into the now-seeded volume, and the serving container consumes it with --optimized. Three containers, all three with a read-only root filesystem, no documented deviation anywhere.

Now: before pairing a stock image with readOnlyRootFilesystem: true, ask what the application does at start and not only at run — specifically whether any setting is baked into the image rather than read, and what happens when the running configuration disagrees with the baked one. Where that produces a write, find out where it writes before designing around it: inside the install tree is a different problem from a data or temp directory, and a volume mounted over an install directory hides whatever was in it. Two further rules fell out of this one and both are cheap: build-time options belong in exactly one place in a chart, because two places mean a disagreement that the application answers by rebuilding — and where a values key could reintroduce one, the render should fail naming it rather than let it reach a pod. And when a pre-build step is the answer, prefer two hardened containers over one writable one: a single init container with readOnlyRootFilesystem: false would have been shorter and would have put a documented hardening deviation into an identity provider, plus a Trivy suppression that also silences the serving container.

The upgrade procedure the automerge did not know about

keycloak supports a rolling update only within a minor line. A minor jump needs every node shut down first, the configuration and database backed up, and the cluster brought back up on the new version. Our Renovate automerges every update type, majors included, after a three-day age gate.

Put those two sentences next to each other and the failure writes itself: a 26.8 → 26.9 app-version PR merges itself, publishes the chart, and gets rolled out pod by pod — the one procedure upstream excludes for that step. Nothing reports it. Both pods are Ready, both are green, and the symptom surfaces whenever two incompatible versions happen to need each other inside one Infinispan cluster.

What makes this an entry rather than a chart note is the shape, which has nothing to do with Keycloak: an upgrade constraint that lives in the application's documentation is invisible to the machinery that performs upgrades. Renovate knows versions and update types. It does not know that this particular minor step is a maintenance window. Any clustered application whose versions must not be mixed has the same gap — and the gap is widest exactly where the automation is best, because a smooth pipeline is what turns "somebody will notice" into "nobody has to look".

The decision taken here was not to switch automerge off for that image (Gitea #83): the global policy stands, the risk is accepted knowingly, and main still requires a review. That is defensible, and it moves the whole weight onto documentation — so the procedure went into the chart's NOTES.txt and onto two wiki pages, in each case next to the thing somebody will actually be looking at.

Now: when a chart's application restricts how it may be upgraded, write the restriction where the upgrade is performed — the chart's NOTES.txt, the chart page, and the Renovate page if the automation could trip it — and say explicitly that the automation does not know about it. If upstream ships a tool that decides the question, name the tool, its exact invocation and its exit codes, because an exit code is the only form of that knowledge a pipeline can use. And where upstream's own documentation contradicts itself about the constraint, record the contradiction instead of picking the convenient half: the honest instruction is then "run the check every time", not "patches are safe".

The rescue instruction that ran through the fault it was repairing

The approved plan for bulwark (Gitea #89) accepted a real risk knowingly — sign-in is OIDC-only, so a broken identity provider locks everybody out — and it did the right thing by writing down the way back. The way back was: helm upgrade with OAUTH_ONLY=false, "the only door that does not depend on Keycloak".

Reading the source showed that door can be locked exactly when it is needed.

The application resolves every configuration value as admin override > environment variable > default, where the override is a file on the volume written by its own admin dashboard. The plan knew this — it had documented it, correctly, for the server URL, because that is the one case upstream's documentation mentions. What it had not noticed is that the rule is not about the server URL: it is the behaviour of every key, oauthOnly included, and the dashboard's OAuth setup writes exactly that key when somebody uses it. So on any installation where the dashboard had been used, the documented repair would have changed nothing, reported success, and left no trace of why.

The actual repair is one layer further in: open the dashboard (its own password, not the provider's) and remove the override, which is what hands control back to the chart. Same door, different handle — and the chart now sets an admin password because that is the door, rather than for the reason the issue originally gave.

Now: when a plan documents a recovery path, check it against the same mechanism the fault runs through. A recovery step is the one instruction nobody rehearses, so it is the one that must be derived rather than assumed — and a precedence rule that upstream documents for a single value should be read as a property of the system until something proves it is a special case. Where the recovery depends on a component staying reachable, say which component that is and what keeps it alive, because the next hardening pass will otherwise switch it off as unused.

The read-only switch whose reach exceeded its name

The same chart gained a value to close that precedence trap: a flag that refuses all writes to the application's configuration directory, making the chart authoritative. Its price was understood as "the settings policy freezes too" — true, and already a reason for defaulting it off.

Measuring found two more, and the second one is the reason this is an entry:

  • the function that removes an override is guarded by the same check, so turning the flag on does not clear an existing override — it freezes it in place, which converts a recoverable lockout into a permanent one and removes the repair in the same move;
  • the admin password is persisted through the same guarded write, so on a fresh install the flag prevents the password file from ever being created. The dashboard then never comes up and its login endpoint answers HTTP 500 — while the pod is Ready and serving traffic the whole time. The flag intended to protect the configuration would have removed the only way back into it.

None of that is visible in a render, and none of it would have failed a pipeline.

Now: when a switch says "read-only", find every call site of the check it sets, not just the one the documentation names. A write barrier in an application tends to be implemented once and applied broadly, so its blast radius is whatever shares that helper — typically including bootstrap writes that happen exactly once, on a path nobody tests because it only exists on a first install. And where a flag is only safe in a particular order relative to the first start, the order is the documentation: say which state must already exist on disk before it is switched on, and say that the chart cannot verify it.

The probe method you do not get to choose

The same plan specified HEAD /api/health for all three probes, reasoning — correctly — that the application offers a HEAD handler and that it is the cheapest call the endpoint supports.

A Kubernetes probe cannot send HEAD. The API type is HTTPGetAction; it has no method field, and the kubelet always issues a GET. There is no value anywhere in a pod spec that changes it. The only way to issue a HEAD is an exec probe calling an HTTP client — and in a minimal image that often means there is no client to call: this one ships neither curl nor wget, with npm and npx deliberately removed, so it would have meant starting a language runtime every few seconds against an endpoint whose GET is already cheap.

The outcome was harmless because GET is marginally more capable here than HEAD. That is luck, not design: had the application's GET been the expensive variant — a handler that touches a database, or one that only answers 200 when fully warmed — the plan's "cheapest form" would have silently become "the heaviest form", at the probe interval.

Now: when choosing a health endpoint, check that the platform can make the call you are designing around before writing it into a plan. HTTP probes are GET, period; a method, a body or a non-trivial header set is an exec probe, and an exec probe is a promise about what binaries the image contains. And when the achievable form differs from the planned one, say which it is at the value — otherwise the next reader sees a comment describing HEAD next to a spec that performs GET and has to re-derive which one is true.

The secret file the container could not read

A mode of 0400 on a mounted secret looks like the careful choice. In a pod it can lock out the very process that needs the file: files in a secret volume are owned by root, and what gives a non-root container access is fsGroup, which only supplies the group. With no group read bit, the application's own uid is denied.

What makes it worth an entry is the failure mode rather than the mechanics. The kubelet ORs a group-read bit onto secret volumes once fsGroup is set, so 0400 is in practice corrected to 0440 — meaning the mistake usually works, for a reason nowhere in the chart. And where it does not work, the application in question answered EACCES exactly as it answered "file absent": a feature switched itself off, a credential went out empty, and nothing logged a permission error. (The opposite trap is already on this page under A permission check that assumes it owns the filesystem — garage rejects a group-readable secret file. Two applications, opposite requirements, same mount.)

Now: write the mode you actually need rather than the strictest one, and write down why the group bit is there — a mode that only works because the platform silently widens it is a mode the next hardening pass will narrow again. And when a chart hands a secret to an application as a file, find out what that application does when the read fails: if EACCES and ENOENT collapse into the same branch, a permission problem will present as a disabled feature, not as an error.

The mount that would have arrived after the build it was for

keycloak got a generic extraVolumes / extraVolumeMounts pair (Gitea #88), and the plan justified the generic shape — rather than a theme-specific key — with the cases that follow: extra root CAs and SPI extensions. The naive implementation renders the mounts on the serving container, which is where every chart in this repo mounts things. For a theme that is right. For an extension it is wrong, and wrong in the quietest available way.

This chart runs start --optimized, so the Quarkus augmentation happens before the serving container exists, in a build init container. Upstream on adding a provider JAR: "After that if you are using --optimized, you must run the build command in order to update the server's provider registry with the implementations from the JAR file." A JAR that is mounted only on the serving container is not there when the registry is written. Nothing fails: the build succeeds, the pod becomes Ready, the file is present at runtime, and the extension simply does not exist. No error, no warning, nothing to grep for.

So the mounts are rendered on the build init container and on the serving container, from one list — two lists would be two chances to put an extension in the wrong one, and the wrong one is silent. The price is named at the value: an extra volume is then present at build time too, which is harmless for a theme and worth knowing for a Secret.

What generalizes is not the Keycloak detail but the question the plan did not ask: where does the thing that consumes this file run? Any chart that pre-builds, pre-compiles, pre-indexes or augments in an init container has a second consumer of its volumes, and the two consumers want different things — the runtime one wants the file while serving, the build one wants it while building. A values key whose documentation names only one of them will be used for both.

Now: before adding a pass-through volume pair to a chart with a pre-build step, enumerate every container that could legitimately need the file, and render the mounts at each of them. Write the reason at the value, not only in the pull request: an occurrence that looks redundant is exactly what the next round removes while tidying up, and this one cannot be noticed by any test that checks whether the pod is Ready. And where a pass-through value only places a file, say so at the value — mounting a CA does not add it to a truststore, and mounting a theme does not select it — because the next reader's expectation is set by the use case the key was introduced for.

The restart that the green run could not explain

The install-test for keycloak in Gitea PR #92 passed and reported, in the final kubectl get pods, one restart on one of the two pods. The three runs of PR #87 — the chart's own introduction — had shown 0 three times. The PR added volume mounts to the build init container, so "probably a flake" was not available: a counter moved in the same round as a change that touches container startup.

The first explanation was plausible and wrong. That run was slower everywhere — the unchanged postgres fixture needed 51.5s instead of 27.4s — so "a loaded runner trips a first install that serializes two replicas on a migration lock" wrote itself. Two more runs disposed of it: the slowest run of the whole series (191s) restarted nothing, and the two that restarted sat in the middle of the range. Five runs in, the tally was three restarts out of five against a baseline of none out of three, which reads like a regression.

What settled it was the control case on the same machine: the volume pair commented out of the fixture, which reproduces the pre-change pod spec exactly — the pod template hash went back to the one PR #87 had measured, so the comparison was byte-identical rather than approximate. Two control runs: one clean, one with the same single restart. The restart happens without the change. It is pre-existing, intermittent behaviour of that chart's first install, and the three clean runs two days earlier were a fast afternoon rather than a property of the chart.

The second finding is the one that cost the detour, and it is not about Keycloak: the harness dumps container logs only when a run fails. diagnose() hangs off the EXIT trap behind RC -ne 0, so every one of the seven runs printed a restart count with nothing to explain it. A green run that reports a restart and cannot say what it was is a measurement that invites exactly the wrong conclusion in either direction.

Now: when a counter moves in the same round as a change, do not argue from runs made earlier or elsewhere — reproduce the old state on the current machine and measure it there. For a chart that means disabling the new values in the fixture and checking that the pod template hash returns to the previous one, which turns "roughly the same spec" into a fact. Budget for more than one run per side: a 1-in-3 behaviour is invisible in a single run and looks like a regression in two. And where a green path reports a number nobody can explain, say so in the file that carries the number and name the harness change that would answer it — the next person to see that restart should not have to re-derive this.

The two defaults that pointed in opposite directions

Both halves of stalwart's distribution rule are node-inclusion policies on the same topologySpreadConstraints entry, so they read as a pair. They are not, and the API says so in two lines that sit a few fields apart:

  • nodeAffinityPolicy — "If this value is nil, the behavior is equivalent to the Honor policy"
  • nodeTaintsPolicy — "If this value is nil, the behavior is equivalent to the Ignore policy"

A review of the plan had written down, as the price of whenUnsatisfiable: DoNotSchedule, that a nodeSelector or tolerations narrowing the eligible nodes would leave a replica permanently Pending. For tolerations that is right. For nodeSelector it is wrong in the dangerous direction: under the Honor default only nodes matching the pod's affinity count as domains, so narrowing a 2-replica group onto one node leaves exactly one eligible domain, both pods land in it, the skew is 0 and the constraint is satisfied. No Pending, no event, and — the part that closes every remaining escape — the labelSelector in the object is perfectly correct, so the CI selector check cannot see it either. The promise is gone and every instrument reads normal.

Two things were wrong at once, and the second is the more useful one. The first: a documented symptom that does not occur is worse than no documentation, because somebody will wait for it. The second: the review that found the error had itself been written as an authoritative-sounding table, and the correction came only from reading the API definition for both fields rather than for the one under discussion.

The fix is one field, nodeAffinityPolicy: Ignore, which makes every node count: the same narrowing becomes 2,0,0, skew 2, Pending. A render abort was the obvious alternative and is the wrong one — it would also have forbidden a nodeSelector spanning five nodes, which is legal and works. And the field must be documented apart from the replica ceiling it sits next to: the ceiling (as many replicas per group as there are schedulable workers) comes from nodeTaintsPolicy ≙ Ignore and existed before any of this, so presenting the two as one block invites the next reader to pay a price to a decision that did not cause it.

Now: when a feature has two switches that look like siblings, read the default of each one separately — and when a protection rests on a field, check whether the field can be absent. nodeAffinityPolicy is beta: on by default from Kubernetes 1.26, GA with NodeInclusionPolicyInPodTopologySpread in 1.33, and pruned without a word on anything older, which restores the exact gap it was set against. That sentence belongs at the value with the minimum version, never assumed as known. See §11.

The duplicate that the API rejected and the apply path merged

Two topologySpreadConstraints entries with the same topologyKey and the same whenUnsatisfiable are a duplicate — and whenUnsatisfiable defaults to DoNotSchedule, so a user entry on kubernetes.io/hostname that simply omits the field collides with a chart-rendered one. What happens next depends on how the manifest reaches the cluster, and the two answers differ in kind:

  • kubectl create fails validation with a message naming a list index, not a value. The reader is sent to a rendered manifest to count entries instead of to the two values that produced them.
  • an apply-based path merges the two over patchMergeKey: topologyKey — silently. One labelSelector is dropped, one constraint survives, and the result is green.

The second is the one that matters here, because a dropped labelSelector is exactly the trivially-satisfied case: a condition over zero pods, distributing nothing, under an object that reads like a guarantee.

Now: where a chart renders an entry into a list the user can also append to, the chart aborts the render on a collision and names both values — the switch that produced its own entry and the list the user wrote. The abort fires only while the chart's entry is actually rendered, so it cannot trip a configuration that has no duplicate. The generalisation beyond this field: a list merged by a key is a namespace, and two writers into one namespace need a referee at render time — the API's own answer is either an index or a silent merge, and neither names a value anybody set.

The stub CRD that would have proved nothing

The Gateway-API pilot (cyberchef, Gitea #114) needed an HTTPRoute to be installed somewhere, and this repo already had a pattern for a CRD the CI cluster does not carry: ci/common/ holds a schema-less stub for OnePasswordItem (x-kubernetes-preserve-unknown-fields). Reaching for it a second time was the obvious move and the wrong one.

The stub is right for OnePasswordItem. Nobody is testing the 1Password operator's API there — only that the chart's CR applies at all, so the absence of a schema costs nothing. For an HTTPRoute the schema is the whole exercise. Against a schema-less stub the API server cheerfully accepts a malformed parentRefs, a port written as a string, a path.type that does not exist; the install goes green, and the only thing demonstrated is "something was created" — which is the class of signal this repository removes rather than adds.

Now: the real CRDs (standard channel, pinned, Renovate-tracked, applied --server-side) are fetched by ci/gateway-api-crds.sh for any chart whose render contains an HTTPRoute. Two further findings came out of doing it properly, both measured rather than assumed:

  • The schema still cannot catch the two failures that matter. It accepts an empty parentRefs list — a route attached to no Gateway, created, valid, rejected by nobody, routing nothing — and it accepts a backendRefs entry naming a Service that was never created, because that reference is resolved by name at runtime. Real CRDs prove an object is well-formed; they say nothing about whether it points somewhere. That half needed a render test of its own (httproute-match), and the generalisation is worth keeping: validating against a real schema and validating against the rest of the render are two different checks, and the first one is the one that looks sufficient.
  • Client-side kubectl apply was 93 % of the way to a failure nobody would have attributed correctly. It stores the whole applied object in the last-applied-configuration annotation, annotations total at most 262144 bytes, and that one CRD is 244792 bytes as JSON — fitting today, one upstream release away from metadata.annotations: Too long on a Renovate PR that touches nothing related. Hence --server-side, which writes no such annotation. A limit one release away is not headroom, and the apply mode is part of a fixture's definition rather than a detail of how it happens to be installed.

And the part that stayed honest: none of this proves that traffic flows. No Gateway controller runs in k3d, so no route is ever reconciled and no status is ever written. What the pipeline claims is exactly what it checks — schema-valid object, Ready pod, a route pointing at a Service and a port the same render creates.

The switch that stopped meaning what it was read for

Group 1 of the Gateway-API sweep (Gitea #114) touched eight charts that all looked alike: one Ingress object, no behaviour-bearing nginx annotation, nothing to decide. sure was one of them, and it carried this:

- name: RAILS_ASSUME_SSL
  value: "{{ .Values.ingress.enabled }}"
- name: RAILS_FORCE_SSL
  value: "{{ .Values.ingress.enabled }}"

Which was correct, and documented, and the right call when it was written: with the Ingress terminating TLS, Rails must trust the forwarded scheme; with nothing in front, both flags have to be off or a plain-HTTP port-forward cannot log in. One value, two readings — "is there an Ingress" and "is there a TLS-terminating proxy in front" — and for as long as an Ingress was the only way to publish, they were the same question.

The Gateway-API mode separates them, and it does so by construction: the two modes are mutually exclusive, so httpRoute.enabled: true forces ingress.enabled: false while TLS is still terminated in front of the app, now at the Gateway's listener. Rendered unchanged, a Gateway-mode release would have come up with both flags false. Rails would then not trust X-Forwarded-Proto, would hand out session cookies without the Secure attribute over a connection that is in fact TLS, would send no HSTS, and would build its absolute URLs — mail links, OIDC redirects — as http://.

Nothing about that fails. The pod becomes Ready, the UI works, the route is Accepted, every pipeline stage is green. The only way to notice is to read a Set-Cookie header.

Now: the question is asked once, in a helper (sure.publishedOverTls), as ingress.enabled OR httpRoute.enabled, and both flags read the answer. With httpRoute.enabled at its default of false that expression is ingress.enabled — which is why the change stayed minor and the default render is byte-identical; the behaviour differs only for a configuration that did not exist before.

Three things worth carrying forward, and the first is the one that nearly got missed:

  • A chart's second routing mode is not a template change, it is a change to what ingress.enabled means — and the readers that are affected are the ones outside the Ingress template, where nobody reviewing a routing PR is looking. The grep that finds them is .Values.ingress.enabled across the chart, minus the Ingress template and NOTES.txt. Still open on that count when group 1 landed: freesailarr and ytdlp-web-player, both of which read it in their helpers. Group 2 confirmed the prediction on freesailarr, and the symptom was a different one; group 3 confirmed it on ytdlp-web-player, and the symptom was worse than either — both below. The screen is closed at three finds, which was every chart that had the pattern.
  • "Sets no behaviour-bearing annotation" is not the same question as "needs none." Group 1's selection criterion screened on annotations a chart sets, measured and correct — and garage passed it with annotations: {} while its own values comment tells the operator to set proxy-body-size: "0", because ingress-nginx truncates S3 uploads at 1 MB. proxy-body-size has no Gateway-API counterpart at all, so garage is a sixth chart that loses an upload limit in Gateway mode where the issue had counted five. A screen over what a chart declares misses what a chart documents.
  • A claim about a value's only reader expires when a second template is added. garage's Ingress template stated, under the guideline that exists for exactly this, that nothing outside it read ingress.domain — true when written, false the moment an HTTPRoute read it as its hostname. The same sentence existed in its values.schema.json. A "checked in both directions" note is a measurement with a date on it, and the work that invalidates it is the work that has to correct it. calibre-web-automated carried the identical claim and needed the identical correction in group 2 — the pattern repeats because the sentence is copied.

ingress.enabled meant "is anything published", and that broke a sub-path, not a cookie

Where: freesailarr, group 2 of the Gateway-API sweep (Gitea #114) — the chart the note above had flagged as still open.

What it was: the chart's helper for "which apps are served under the shared host, and at which path" was gated on ingress.enabled. That list has three consumers, and only two of them are routing objects: the Ingresses, NOTES.txt, and the <APP>__SERVER__URLBASE environment variables the chart sets for radarr, sonarr and prowlarr so that those apps know their own base path.

In Gateway mode ingress.enabled is false by construction, so the list would have come out empty while the HTTPRoute served the apps under /radarr, /sonarr and /prowlarr. Each app would then have answered at its sub-path with a page whose assets and API calls are linked at the host root: the page loads, the browser fetches nothing that exists, and the pod is Ready with zero restarts. The two render guards that live in the same helper — seerr and mediathekarr can only be served at /, and no two apps may claim the same path — would have gone unenforced in that mode for the same reason, so a configuration the chart is supposed to refuse would have rendered quietly.

Measured before and after, which is what distinguishes this from an argument: helm template on origin/main with ingress.enabled: false emits zero URLBASE variables; the fixed chart in Gateway mode emits three, with the right paths.

Now: one helper, freesailarr.published = ingress.enabled OR httpRoute.enabled, and the list asks that instead. Same shape as sure.publishedOverTls, same reason it stayed minor: with httpRoute.enabled at its default the expression is ingress.enabled.

The generalisation, which is the point of recording it twice: the question a chart asks ingress.enabled is almost never "does an Ingress object exist". In sure it was "is TLS terminated in front of me". Here it was "is this app published at all, and under which prefix". Both are properties of the release, not of the object — and both answers flip when the object changes while the property does not. Before adding a second routing mode, read every ingress.enabled outside the Ingress template and ask what the value is standing in for; where it stands in for something, the fix is one helper and never a new value.

The gap the chart never had, and the operator had already closed

Where: freesailarr, Gitea #118.

What it was: qBittorrent has no base-path setting, so the chart serves it under a prefix and strips that prefix. A request for the bare prefix — /qbittorrent, no trailing slash — is stripped to /, qBittorrent answers it 200 with its login page, and the browser's document URL keeps the prefixless form. Every relative link in that page then resolves against the host root, which on this chart's shared host is seerr. The login form comes up unstyled and does nothing while a different application answers its assets and XHRs. Nothing fails: not a probe, not a render, not the schema, not the install. The symptom points at qBittorrent, and the fault is in the URL.

What made it a lesson rather than a bug. The chart had never shipped a fix for this, in either routing mode — the pickaxe searches for configuration-snippet and for return 301 over freesailarr/ both return zero commits, and values.yaml did not mention the trap at all. And yet curl -D - https://freesailarr.jk-dev.de/qbittorrent answered 301 to the slashed form, and had done for as long as anyone had used it. The redirect was real, and it came from somewhere else: an nginx.ingress.kubernetes.io/configuration-snippet on the live Ingress objects, supplied through the chart's global ingress.annotations by Fleet, from the separate jk-cluster-config repository. An annotation of risk level Critical, on a controller deliberately started with allow-snippet-annotations=true, in a repository the chart knows nothing about.

So three things were true at once, and each one made the others harder to see:

  • the chart was incomplete — it documented neither the trap nor the remedy;
  • the running system was correct — the redirect was served;
  • the Gateway-API migration therefore looked like it would break something the chart had never provided. The behaviour would have disappeared on migration, and the diff that removed it would have contained nothing: the annotation channel simply does not exist in Gateway mode.

Now: the Gateway mode renders the redirect itself, as an Exact-match rule with a RequestRedirect filter (301 — statusCode defaults to 302, and only 301/302 are Core in that API, so an Extended value an implementation declines fails the whole route and takes the neighbouring apps with it). The Ingress mode is unchanged on purpose: its only mechanism is the snippet annotation, which is a silent no-op on any cluster that keeps allow-snippet-annotations at its post-1.9 default of off — so the chart writes the snippet, and its price, into values.yaml instead of shipping a no-op. The operator's snippet is not redundant and must not be removed: freesailarr.jk-dev.de still runs in Ingress mode, where it is the only thing performing the redirect. ci/run-httproute-match-test.sh now asserts, per HTTPRoute, that a prefix-stripping rewrite is accompanied by a redirect for the bare prefix — see CI/CD — so the gap cannot reappear in the chart that has it or in the next chart to strip a prefix.

The method half, which generalises further than the bug

The investigation produced a correct search and a wrong conclusion, and the gap between them is the part worth keeping.

"The chart has never contained this redirect" was measured properly: the right paths, the right pickaxe searches, zero hits, no ambiguity. What it does not support — and what was briefly inferred from it anyway — is "therefore the redirect is not happening". A chart is one input to a running system. The others are the cluster repository's values, the controller's own flags, anything an admission webhook injects, and whatever a previous operator added by hand.

So: a search over the chart answers what the chart does. What the running system does is a separate measurement — at the live object (kubectl get ingress -o yaml, and read the annotations, including the ones no chart template could have written) or in the cluster repository that supplies the values. The objectset.rio.cattle.io/id annotation named Fleet and the bundle in one line; the question had simply not been asked.

Where a chart finding is meant to carry a claim about behaviour, that second measurement is mandatory. Not as diligence, but because the two can disagree in both directions and both directions are expensive: a chart that is missing something the cluster supplies (this case — a migration silently drops it), and a chart that supplies something the cluster overrides (where the chart's documentation is wrong about the system and nobody notices until the override moves).

The recurring shape

  • Green is a claim about the assertions that ran, not about the system.
  • A failure that looks like success is more expensive than a loud one — which is why several of these fixes deliberately turn a silent state into a crash.
  • The environment can hide a bug for years. Portability work is also archaeology.
  • When a rename and a silent behaviour change are both available, take the rename. Schema validation failing loudly beats a value that quietly means the opposite.
  • A safeguard is only a safeguard if it can fail. Both the missing required() and the required array that replaced it read as protection; neither could ever have fired.
  • Write down what was measured, and say when something was not. Half of this page exists because someone recorded a number instead of an impression.
  • A document handed to you as context is a copy, not the source. Correcting the file does not correct the copies already delivered, and a stale copy looks exactly as authoritative as a current one — only the file itself, and its diff against origin/main, settle what a rule says.
  • Put the warning where the next change will be made. A risk that only materializes at the next upgrade belongs at the line that upgrade will touch — a note in the pull request that argued the decision is read once, by people who already know.
  • A capability list is about the product; only a run is about the installation. "Supported" on a documentation page and "enabled on the server we call" are different claims, and the first one reads like a promise.
  • The image is part of the security context. A file capability or a fixed uid constrains the pod spec no matter what the application does — and it is read off the Dockerfile that builds the published image, which is not always the one at the repository root.
  • Document a deviation with its mechanism, not its name. A grant whose comment does not say why it cannot be removed reads as dead weight to the next cleanup, and "needed for binding" is worse than nothing when the real reason is execve.
  • Inside a cluster, every per-client identity is a per-node identity. Masqueraded source addresses and probe traffic both collapse onto node IPs, so any application logic keyed on the client address — bans, limits, geo, audit — is being fed one address, silently and correctly-looking.
  • Ask what an application does when its configuration is missing, not only when it is wrong. The lenient fallback is almost always on the "absent" path, and it is the path a broken mount takes.
  • A snapshot of a live database is a backup of a file, not of a database. Replication, encryption and an off-site copy all operate below that distinction and none of them notices it.
  • A safety check written for a host can be unsatisfiable in a pod. Before adopting an application's preferred mechanism, check what it verifies — the platform may be unable to produce the state it demands, and the symptom will name the wrong knob.
  • Read the flags of the version you are shipping, not the ones the documentation pages describe. Two decisions in the garage plan were argued from constraints that an existing --single-node had already removed; the plan was not wrong about the old way, it was wrong about which way existed.
  • A dependency on your own artifact tests the published copy, not the one you are editing. Two changes in one PR means a green check on the wrong input — and nothing in the output says which copy was used.
  • A convention is not a rule, and the difference shows up in somebody else's validator. condition: <name>.enabled is fine until the subchart has a strict schema; the error then belongs to the subchart and appears only once the feature is switched on.
  • Validation that is skipped in the default path is not validation. A check that only runs for the non-default branch passes CI forever and fails for the first person who deviates.
  • A reference to something that moved is false, not stale. It still answers the question, which is why nobody re-asks it — so a file leaving the repository is a grep across the whole tree, comments included, not only the lines a tool can resolve.
  • A rule about two components' configuration cannot live in either one's templates. Whatever renders first reports the error, and it reports it in its own vocabulary — so the check belongs where validation happens before rendering does.
  • "Erased" and "empty" are not the same value, and they can behave oppositely. A null override removes a key; an empty object is merged with the default. Wherever a missing key is read as enabled, erasing a switch turns a feature on.
  • An answer that arrived by accident is worth making repeatable. The all-charts install-test exists because one unplanned run produced a result nobody could have asked for; the fix is a way to ask the question on purpose, not a note to remember the accident.
  • Three correct requirements can still be an impossible pod. Conflicts do not only live between pairs of settings — the stock image, the probes and a read-only root filesystem each read as right, and what they collided through was a property of the image none of them names. So the question to ask of a combination is what the application does at start, not whether each half is defensible.
  • A constraint that lives in documentation is invisible to the machinery. Automation knows versions, update types and exit codes. It does not know that a particular step is a maintenance window — and the better the pipeline, the less likely anybody is standing in the path to notice. If a tool can decide it, automate on its exit code; if nothing can, write the obligation where the work is done.
  • Check the recovery path against the mechanism the fault runs through. A repair nobody rehearses is the one most likely to be assumed — and a precedence rule documented for one value is usually a property of the whole system, not a special case.
  • A write barrier reaches as far as the helper that implements it. "Read-only configuration" also catches the bootstrap writes that happen once, on a first install, on the path no test walks — including the one that creates the credential for the door you kept open on purpose.
  • Check that the platform can make the call before designing around it. An HTTP probe is a GET; anything else is an exec probe, and an exec probe is a claim about which binaries the image still contains.
  • The strictest file mode is not the correct one. A mounted secret is owned by root, fsGroup supplies only the group, and an application that treats EACCES like "absent" turns a permission mistake into a feature that quietly switched itself off.
  • A counter that moved alongside a change is not evidence until the control case ran on the same machine. Earlier green runs are a different environment, and an intermittent behaviour needs several runs per side before either answer is honest.
  • Ask where the consumer of a file runs, not only where the application runs. A chart with a pre-build step has two consumers of its volumes at two different times, and a mount that reaches only the serving one fails without a symptom.
  • Two switches that look like siblings can have opposite defaults. Read the default of each one on its own, and keep the consequences of each in its own paragraph — a price documented next to the wrong decision is paid by whoever reads that decision later.
  • A protection that rests on a field can be lost by the field being absent. Beta fields are pruned without a word on an older API server, so the question is not only "is the value right" but "does this version keep it at all" — with the minimum version written at the value.
  • A documented symptom that does not occur is worse than no documentation, because somebody will wait for it instead of looking.
  • A list merged by a key is a namespace, and two writers into one namespace need a referee at render time. The API's own answers are an error naming a list index and a silent merge that drops one half; neither names a value anybody set.
  • A fixture is defined by what it validates, not by what it stands in for. The same trick — a schema-less stub CRD — is correct for one API and worthless for the next, and the question that separates them is whether the schema is the thing under test. Reusing a pattern because the shape matches is how a green run comes to mean "something was created".
  • When the fix needs a privilege, ask whether two cheap steps can replace one privileged one. Splitting a pre-build into a seeding container plus a build container kept the hardening baseline intact across the whole pod, where the shorter single-container version would have bought a documented deviation and a suppression that silences the container that actually serves traffic.
  • A values flag can be correct and still be the wrong question. When one value answers two questions that happen to coincide, the day they stop coinciding is a day nothing fails: the flag still renders, the pod still starts, and the second question is now answered wrong. Adding a second mode to anything is the moment to ask which of its readers were reading it for the other reason.
  • A search over the chart answers what the chart does; what the running system does is a second measurement. The chart is one input among several — the cluster repository's values, the controller's flags, an admission webhook, a previous operator's hand edit. Where a finding about a chart is meant to carry a claim about behaviour, read the live object or the repository that supplies its values; a correct search over the wrong scope produces a confident wrong answer.
  • An operator can close a chart's gap invisibly, and the chart still looks correct. Where the closure lives in another repository, the chart is incomplete, the system works, and the change that removes the closure contains no trace of it — which is why "this has always worked" is evidence about a deployment and not about a chart.
  • The size of a version number is a claim about an API, not about a default. Upstream's semver answers "will your integration still compile", and a flipped security default fits inside a patch without breaking that promise. So the update class a pipeline computes decides who looks at the change — and the class with no documentation duty is the one where a changed default arrives unread.
  • A "nothing else reads this" note is a measurement, not a property. It is true on the day it is written, it is what later decisions rest on, and the change that falsifies it is the change obliged to rewrite it — in every copy, the schema description included.

Four of six charts leaned on an annotation the chart never set

Where: group 3 of the Gateway-API sweep (Gitea #114) — keycloak, matrix-synapse, stalwart, bulwark; and not homeassistant or ytdlp-web-player.

What it is: the generalisation of the entry above, and the reason that one is recorded as a method and not as a qBittorrent bug. At #118 the discovery was singular and late: one chart, one annotation, found after shipping, while migrating a host. Group 3 asked the same question before writing any Gateway branch, with read-only kubectl against the live cluster — and found it was not singular at all.

Chart Set on the live Ingress by the operator, never by the chart
keycloak proxy-buffer-size: 16k, proxy-buffers-number: 4 — on both of its Ingress objects
matrix-synapse proxy-body-size: 50m, the nginx.org/client-max-body-size twin for the other nginx controller, and a configuration-snippet
stalwart enable-cors: true plus cors-allow-origin/-methods/-headers
bulwark proxy-body-size: 25m

None of it is findable from inside the repository. A git grep over those six chart directories for proxy-buffer, app-root, more_set_headers, enable-cors, cors-allow, proxy-body-size and configuration-snippet hits documentation only — bulwark/values.yaml recommending proxy-body-size: "25m", ytdlp-web-player/values.yaml recommending proxy-buffer-size: "16k". No template sets any of them. Every one of the four would have vanished at the mode switch, in a diff containing no trace of it, because the annotation channel does not exist in Gateway mode.

So the measurement at the live object is part of the procedure, not a flourish. Before writing a chart's Gateway branch: read its live routing objects. Reading the chart is not a substitute, and neither is reading the issue that planned the sweep — #114's own measured table listed proxy-body-size for five charts and did not include matrix-synapse, which has it.

And three qualifications, because the measurement is only the first step

Read the directive before calling an annotation load-bearing. matrix-synapse's snippet looks exactly like what Synapse's x_forwarded setting depends on — client IP for rate limiting, and federation — and it is the entry in the table that would frighten a reviewer most. It is not load-bearing at all: more_set_headers is the headers-more module's output-header directive ("Replaces (if any) or adds (if not any) the specified output headers"), the request-side directive being more_set_input_headers. Those three headers go on the response to the client; Synapse never sees them, and what it reads with x_forwarded: true is the request's X-Forwarded-For, which the proxy supplies itself. The chart's notes therefore say in as many words not to re-create it at the Gateway and not to read its absence as the cause of a rate-limiting problem. An annotation that looks alarming and does nothing costs as much review time as one that matters — and had this gone the other way round, the sweep would have shipped a scary paragraph about a non-problem.

"No counterpart" is a claim to check against the pinned CRDs, and once it was false. stalwart's CORS is the loss that breaks a second chart of the same group — the allow-origin names bulwark, the webmail, so the symptom is a browser console error with nothing in the mail server's log. The convenient conclusion would have been "no counterpart, document it". Grepping the CRD set this repository actually pins says otherwise: HTTPRoute's filters carry a full cors field (GEP-1767) with allowOrigins, allowMethods, allowHeaders, allowCredentials and maxAge, and responseHeaderModifier sits beside it — both Support: Extended, neither Core.

The chart still renders neither, and the three reasons are the transferable part:

  1. No values to render it from. The origin and the method list are the operator's, and the chart sets no CORS annotation in Ingress mode either — so a filter would exist in Gateway mode only. That asymmetry is its own defect and its own piece of work, not something to slip into a sweep.
  2. Extended support is not support. A rejected route is rejected whole, which here would take the WebUI and JMAP down together. And the observed failure is quieter than rejection: Cilium accepts filters.type: CORS and forwards the browser's preflight upstream anyway — Accepted: True, behaviour absent. A chart cannot know which implementation it landed on.
  3. Half a feature is worse than none, and the expressible half was the useless one. A responseHeaderModifier can add the Access-Control-Allow-* headers; the preflight must be answered by the gateway rather than the backend, which a header filter cannot do. And because the allowed headers include Authorization, every request from the webmail is preflighted — so the half that needs no CORS filter is precisely the half that does not help. A rendered half would have looked configured and failed on the first real call.

A guard can matter more in the new mode, and that is a reason to look at what it protects. ytdlp-web-player was the last open chart on the ingress.enabled-derived screen, and the find was not an environment variable but a render guard: the one refusing to publish the application and its bundled oauth2-proxy on the same host. In Ingress mode ignoring that guard fails loudly — ingress-nginx's admission webhook rejects the second Ingress and the Fleet bundle lands in ErrApplied. In Gateway mode the webhook never fires: it compares Ingress objects with Ingress objects, and an HTTPRoute is a different kind. Two valid objects then serve one hostname over two data paths that are not equivalent — the proxy's Ingress through authentication, the chart's route straight to the application — and which one a client reaches depends on what DNS resolves to. So a guard whose Ingress-mode job was to pre-empt a noisy webhook becomes, in Gateway mode, the only thing between a values file and an authentication bypass that comes and goes with a DNS record. When screening for a derived value, ask what it protects, not only what it computes.

And a shape read off an issue's summary table is a hypothesis. Three of #114's four descriptions of the group-3 shapes were wrong — bulwark has one Ingress rather than two and has no rewrite-target at all; matrix-synapse renders one object with two host rules rather than a loop over apps; stalwart's routing object is not per node group. Two of the three had already been copied into README.md and into the header comment of ci/run-httproute-match-test.sh, where they read as settled fact and would have been copied again. The bulwark one has a mechanism worth naming: the table was built by grepping for the string rewrite-target, and every occurrence of that string in bulwark is an explanation of why the annotation does not help it. A grep counts refutations as occurrences.

The security default that a patch bump switched off

Where: freesailarr, in the round that collapsed eight simultaneous Renovate PRs into one. prowlarr 2.5.2 → 2.6.5 and radarr 6.3.0 → 6.4.4 — and sonarr, where it had already happened and nobody had noticed.

What it is: the *arr applications used to hard-code the private networks whose X-Forwarded-* headers they trust. Read at the tags this chart pins rather than from the release notes alone — src/NzbDrone.Host/Startup.cs at prowlarr v2.5.2.5491:

options.KnownNetworks.Add(new IPNetwork(IPAddress.Parse("10.0.0.0"), 8));
options.KnownNetworks.Add(new IPNetwork(IPAddress.Parse("172.16.0.0"), 12));
options.KnownNetworks.Add(new IPNetwork(IPAddress.Parse("192.168.0.0"), 16));
options.KnownNetworks.Add(new IPNetwork(IPAddress.Parse("fc00::"), 7));
options.KnownNetworks.Add(new IPNetwork(IPAddress.Parse("fe80::"), 10));

prowlarr 2.6.3 and radarr 6.4.3 removed all five and replaced them with a TrustedNetworks setting whose own default is string.Empty; ForwardedHeadersConfigurator.Configure returns before registering a single KnownNetwork while it is empty. Upstream's reason is sound — trusting X-Forwarded-For from any private network is a spoofing vector — and the consequence behind an in-cluster ingress controller is still a behaviour change: the client address the app logs and displays is the controller's pod IP, the scheme reads as http behind TLS termination, and under "Disabled for Local Addresses" a proxied request stops counting as local and is asked to authenticate.

The part that is a lesson rather than an upstream note: for sonarr this had already shipped, inside a patch. 4.0.19 — the version the chart was already running — has no hard-coded networks and the same empty-default TrustedNetworks. It arrived as what the convention calls a plain app-version bump: the class ci/renovate-bump-chart.sh turns into a chart patch automatically, and the one class the wiki obligation explicitly exempts. So nothing documented it and nothing reviewed it. The two remaining apps were caught only because their jumps crossed a minor boundary, which the convention does treat as possibly behaviour-changing — the safety net that worked was the version number's size, which is a statement about upstream's API compatibility and not about its defaults.

Three transferable pieces:

  1. A one-line tag diff is not a small change; it is an unread one. What carried this was a diff shaped - tag: 4.0.19 / + tag: 4.0.20 — nothing in it can show a default being inverted. Where a bump is classified automatically, the classification is also what decides whether anybody looks.
  2. Read the source at both tags when a default is in question. The release note says "New: Add Trusted Networks setting", which reads as purely additive; the removal of the five networks is visible only in the diff of Startup.cs. A new setting and a changed default are the same commit here, and only one of them is in the headline.
  3. A guessed tag's 404 looks exactly like "not affected". The first check of sonarr used an invented tag, v4.0.20.2912: the file 404'd and the grep came back empty — the identical result to a version that never had the code. The real tags are v4.0.20.3014 and v4.0.19.3011, and against those the answer inverted. Where absence is the finding, confirm that the thing you are reading exists.

What the chart does about it, and what it deliberately does not. A <app>.trustedNetworks value was added for all three apps — sonarr included, because repairing two and knowingly leaving the third is the same silent pattern — rendered as <APP>__SERVER__TRUSTEDNETWORKS and only when non-empty, since an empty variable is indistinguishable from an unset one to the app but looks handled to a reader of the manifest. Its default is empty, so the chart does not restore the old behaviour: it follows upstream's intent and makes the state visible, with a NOTES.txt warning that fires in exactly the combination that costs something — the app published through an Ingress or an HTTPRoute, and the value empty. AllowedHosts, added by the same upstream commits, is deliberately not exposed: its default is string.Empty too and the app then accepts any host, so there is nothing to repair. If anyone ever sets it, the trap to know is that the app appends only localhost, 127.0.0.1, [::1] and its own hostname, so neither the pod IP nor the Service DNS name is covered — and the kubelet probes the pod by IP, which an incomplete list answers with HTTP 400.

Decided since, in Gitea #128 — and the decision does less than this incident asks for, on purpose. The convention now ties the chart jump to the app jump: app major → chart major, app minor → chart minor, patch and digest → chart patch, as a maximum against the chart's own classification rather than a replacement. The level comes from Renovate's own update type, passed into ci/renovate-bump-chart.sh, because an appVersion comparison would have been blind to the very update that started this — prowlarr's 2.5.2 → 2.6.5 was an image tag in freesailarr/values.yaml, where appVersion never moves. Non-semver app versions (CalVer, date tags, four-part and suffixed tags — seven charts) stay at a patch for a minor, since a month rolling over says nothing about risk.

What was explicitly rejected, and therefore what stays broken. The variant that would have caught this incident at its origin — a chart minor or major obliges reading the upstream release notes, with the result stated in the PR even when it is "nothing found" — was turned down. So:

This lesson's own mechanism survives the fix. The classification stops understating the jump; it creates no duty to look behind it. A patch-sized app update that inverts a default can still reach main with nobody reading a release note, exactly as it did here. The repair is that the number is now honest, not that somebody is now watching.

What the honest number buys is not cosmetic, and it is worth knowing precisely because it is indirect: anything that watches Helm release versions — Fleet's fleet.yaml in cluster-config included — sees only our chart version and nothing inside the package, so a minor is a value other gates can act on where a patch is invisible to them. Whether cluster-config actually distinguishes update types is not verifiable from this repository, and was explicitly left out of scope: it is the mechanism that would make the number act, not a claim that it does.

And the smaller, transferable piece of that decision: a classification that is not defined is decided by whoever is in a hurry. "Plain" sat in two convention texts for two years without a definition, and its effect was to tell the reviewer there was nothing to check. Replacing it with something mechanical — here, a field Renovate had already computed — costs one argument and removes the judgement entirely.

The claim four pull requests refused to make, and the measurement that finally reached it — one layer further out than wanted

The Gateway-API sweep (Gitea #114) ran as a pilot plus three groups, and every one of those pull requests says in as many words that "it routes" is not demonstrable. No Gateway controller runs in CI, no HTTPRoute ever reaches Accepted: True there, and what the pipeline proves is a schema-valid object against the real CRDs plus the render assertions. That discipline was right, and it was also uncomfortable: 22 charts gained a routing mode that nothing had ever exercised end to end.

On 2026-10-13 freesailarr.jk-dev.de was switched to Gateway mode, and the claim became measurable. What came back, read-only:

  • No freesailarr Ingress object exists, in any namespace, while the host answers.
  • GET /radarr → 302, server: Kestrel, location: …/radarr/login?returnUrl=%2Fradarr. The app knows its own base path, so the freesailarr.published helper is working in production — without it RADARR__SERVER__URLBASE would be empty in this mode and the UI broken behind a Ready pod.
  • GET /qbittorrent → 301 to the slashed form, 17-byte body, no content-type, no HSTS. The redirect the Ingress mode cannot render is being served.
  • GET /qbittorrent?foo=bar → location: …/qbittorrent/?foo=bar. The query string survives.

The lesson is not "it works" — it is where the measurement stopped

Two limits are real, and a third one that was first written here turned out to be a mistake of my own — which is the more instructive half.

  1. The implementation cannot be named. httproutes and gatewayclasses are RBAC-forbidden to the read-only identity. That the 301 comes from the chart's rule rather than from the operator's nginx snippet is inferred — from the snippet's carrier being gone (ingress.annotations are dropped in this mode) and from the response signature changing with it (before: nginx, text/html, 162 bytes, HSTS). A good inference is still an inference, and it was written down as one.
  2. Every HTTP measurement passed through a front proxy. The hosts resolve to a single address that is not the ingress/gateway load balancer. So the response signature is evidence about what the proxy returned, one hop removed from what the Gateway did. This limit applies to the HTTP measurements and not to anything read from the API server.

The third "limit" was wrong, and it was wrong in the direction of understating the evidence. It read: there is no freesailarr namespace in the cluster and no pods of it visible, so where the backend runs is not determinable. The first clause is literally true and the conclusion does not follow. The release runs in namespace jellyfin — freesailarr-freesailarr-sfs-0, 10/10 Running, 0 restarts, Helm annotations naming release freesailarr in namespace jellyfin. All ten containers are readable, and so is the evidence that mattered most: RADARR__SERVER__URLBASE=/radarr in the running pod, in a namespace with no Ingress object at all.

How the error was produced: by searching over namespace names — for freesailarr, and for anything containing "sail" or "arr". None matched, because the release lives beside its neighbouring service rather than in a namespace named after its chart.

So: a release does not have to live in a namespace named after its chart, and a query over namespace names does not answer "is this running". kubectl get pods -A | grep <release>, or the Helm release annotations, answers it. This is the same genus as the #118 finding one level over: a search that answers a different question than the one asked, and returns an empty result that reads like a fact. The rewrite-target grep that counted bulwark's refutations as occurrences is the same mistake in the other direction — a non-empty result that reads like a fact.

The generalisable point, now with a sharper edge: a measurement that reaches the system is worth far more than a render, and it still has a boundary — but check whether the boundary is the infrastructure's or your own query's. Two of the three limits here were the system's; the third was mine, and it was the one that made the evidence look weaker than it was. The honest form is to state what was observed, what was inferred, and which layer the observation came from — and to treat an empty result as a question about the query before accepting it as a finding.

And the query-string answer is a data point, not a promise

The specification is unchanged: HTTPRequestRedirectFilter has scheme, hostname, path, port and statusCode, and no query field. So the chart still cannot promise the query survives a redirect, and the documentation still says so. What changed is that for this implementation, on this day, it demonstrably does. Both halves belong in the page — a measured behaviour that the specification does not guarantee is exactly the kind of thing that gets remembered as a guarantee, and then relied on by a cluster running something else.

A screen that finds nothing is still a result worth writing down

The sweep's ingress.enabled-derived-value screen found three real defects — sure's TLS flags, freesailarr's sub-path variables, ytdlp-web-player's authentication bypass — each of them a value computed from a switch that Gateway mode flips by construction, wrong while nothing fails.

Run over rss-bridge, the last chart, it came back clean: outside the Ingress template the only readers of an ingress value are NOTES.txt, which knows its own mode, and a TLS-grouping helper called only from behind the switch. The application takes no base URL and no hostname in its environment, so there was nothing to compute and no helper to add.

That result was recorded in the README and on the chart's page anyway, and the reason is practical rather than tidy: an unrecorded clean screen is indistinguishable from a screen nobody ran. The next person to wonder whether this chart has the defect its three siblings had would otherwise have to redo the work — and the cheaper path, deciding it "probably doesn't", is exactly how the ytdlp-web-player bypass survived two earlier rounds of the same sweep. The value of a screen is that it was run; that fact is only available if the negative is written down too.

The red check that was kept red so it could say something — and then said it

rss-bridge's install-test had failed on the Gitea runner since the move: php-fpm as uid 33 unable to read /app/index.php, the startup probe 403ing through its whole 150 s budget, the pod never Ready. Everything else about the chart was healthy — the same chart and the same image ran in production with Ready=True and 0 restarts, the runner pulled the same image digest Docker Hub serves, and /app/index.php is mode 0644 in a 0755 directory, i.e. world-readable. 23 of 24 charts came up on that same runner. The defect was localised to the environment and was not reproducible anywhere else.

An exclusion in ci/excluded-charts.txt was proposed and rejected. The argument: an exclusion makes the pipeline honestly green and simultaneously destroys the only instrument pointed at the problem. A skipped test says nothing — not while the fault persists, and especially not once it is gone.

So the chart stayed in the matrix and the check stayed red, explicitly as the status indicator for the issue.

On 2026-10-13 it went green. Pod 1/1 Ready in 22 s, 0 restarts, no permission error in the job. And the run that produced it happened to carry the strongest possible control: the PR's rendered output for the fixture values was byte-identical to origin/main, proven as part of justifying a Minor version bump. So the pod spec that succeeded was, line for line, the pod spec that had been failing. The chart was ruled out not by reasoning but by a diff.

What generalises

  • An excluded test is not a quiet test, it is a blind one. The cost of an exclusion is usually discussed as "we lose coverage while the bug exists". The larger cost is that you also lose the moment the bug stops existing — and nobody schedules a day to go and re-check a skipped test.
  • Keep a known failure attached to the thing it is about. Red-and-explained is a worse pipeline and a better instrument than green-and-skipped. That trade is only correct when the explanation is written down where the red will be seen, which is why the chart's page and the issue both carried it.
  • A green that arrives on an unchanged input is worth more than a green that arrives with a fix, because it localises the cause by elimination. Here the byte-identical render was produced for an unrelated reason — version-bump justification — and turned out to be the control case. Cheap evidence collected as a matter of routine is what makes an accident interpretable.
  • A reproduced colour is still not a repair. It went green on every run of that branch after the first, across several commits, so it is not a fluke — and nobody has identified what changed on the runner. A green test does not explain itself, and the diagnostic programme that would (docker info on the runner for userns and the dind configuration, the same image in a local k3d without remapping) remains unexecuted. The issue was closed on that round anyway, which makes the unexplained part easy to lose: a fresh red is a finding rather than a surprise, and with the tracker closed that warning has to live in the files somebody will be reading. Close an issue on a cause, not on a colour — which was the standing instruction on this one before the green arrived, written precisely so that the green could not be mistaken for the answer.

The precise message that hid the structural one

Gitea #122 added keycloak.realmHosts, a list rendering one routing object per entry in both modes. Its per-entry checks live in one helper so that neither mode is missing them, and the Ingress template calls that helper whenever ingress.enabled is true. Which looked right, and produced this: with ingress.enabled and httpRoute.enabled both true — the combination the chart has aborted over since the Gateway work, with a message about two data paths onto one hostname and Keycloak's absolute redirect URLs — the error a user got was

keycloak.realmHosts[0] (host a.example.com) has no tlsSecretName, and in Ingress mode it is required.

Both statements are true. The second one is useless: the values file's defect is not a missing secret name, it is that both modes are on, and "in Ingress mode" is a claim about a configuration that is not coherent in the first place. Someone would have added the tlsSecretName, re-run, and met the real message one round later — or, worse, concluded the realm feature was the problem.

The mechanism is ordering, and a chart does not get to rely on it. Helm renders templates in an order a chart cannot pin, so "the structural guard runs first" is not something the Ingress template could assume about the HTTPRoute template. The fix is not ordering but scope: the Ingress template computes the realm list only while ingress.enabled is true and httpRoute.enabled is false, so in the incoherent combination it validates nothing and the only fail left is the one that names it.

The transferable rule: a specific validation must not be reachable in a state that a structural validation already rejects. Fine-grained checks are the ones that fire first by accident, because they sit closer to the data — and the more useful the message, the more convincing the wrong diagnosis. When adding a per-entry check to a chart that already aborts on a mode conflict, gate the new check on the conflict being absent, and verify it by rendering the conflicting combination rather than by reading the templates. It was found that way here: the combination was in the test matrix for the new value, and the output named the wrong thing.

A second, smaller piece from the same round, for anyone adding validation to a chart whose schema is strict: with additionalProperties: false and required/minLength declared, the schema rejects a missing host or realm before any template runs, so the template's own fail for those is unreachable through an ordinary helm template. Keep it — it is the second guard the repo's required() convention already asks for — but check it is not dead by rendering once with --skip-schema-validation. A guard nobody has ever seen fire can carry a typo in its own condition for years.

The two fields cert-manager reads at different scopes

Gitea #123, and the reason it is here rather than only on the stalwart page: the cause is not specific to that chart, and the next "list of hosts, each with its own TLS secret" will meet it unchanged. Gitea #122 is already one.

The approved plan asked for three things that read as independent and are not jointly satisfiable:

  1. one Ingress object, one rule per host, tls: grouped by secret;
  2. a per-entry clusterIssuer, optional, where empty means no annotation — so a secret owned by another release is not fought over;
  3. a fixture carrying one entry with an issuer and one without.

They collide because cert-manager reads the two fields at different scopes, and nothing in either field's name says so:

  • cert-manager.io/cluster-issuer is an annotation — object-scoped. An Ingress carries exactly one, or none. With it absent, ingress-shim creates nothing for that object: for every tls entry on it, not for some.
  • secretName is tls-entry-scoped, and the shim creates one Certificate per entry.

So an object cannot mix "issued by X" with "comes from elsewhere". Point 2 forces a split on the issuer, and point 1 as written cannot hold.

And a second rule came out of measuring rather than reasoning — ingress-shim names the Certificate after the secret and owns it from the Ingress:

Certificate tls-mta-sts-bekemeyer-klein-de-ingress
  secretName:      tls-mta-sts-bekemeyer-klein-de-ingress
  ownerReferences: Ingress/mta-sts-ingress

Two Ingress objects whose tls entries name the same secret therefore drive the shim at one Certificate name from two owners. So a shared secret must sit in exactly one object, and a secret must not span two issuer groups.

Now: group by issuer for the objects, by secret for the tls list, and abort when one secret appears under two issuers. Three consequences worth carrying forward:

  1. The failure this avoids is not a render error. Every shape above produces valid YAML that a cluster accepts. What goes wrong is one Certificate with two claimants, or a certificate silently requested for a secret another release owns — visible in cert-manager's log and nowhere in the release.

  2. "One object" is a shape, not a requirement. Where an approved plan names an object count, check it against the scope at which the controller reads each field before implementing it. Writing the contradiction down beat resolving it quietly in either direction.

  3. A rule can be right while its written reason is wrong — and the reason is what the next person reads. The index-naming rule said "acceptable only because a route holds no state and no certificate". Reading that literally, this round named its objects by issuer instead, since an Ingress that owns a cert-manager Certificate fails that precondition. The parallel realm-host round read the same rule and named by index, citing it as binding — and added keycloak to it, whose realm Ingresses also own certificates. Two charts, one shape, opposite answers, and a rule extended onto cases its own stated precondition excluded.

    It was the reason that was broken, not the rule. The maintainer settled it on index — keycloak was already merged and published, so unifying the other way would have meant renaming a shipped chart for a sentence — and the justification was replaced with the one that holds: the Secret survives a rename, so only the Certificate object is recreated and nothing is re-issued. Measured for this cluster (--enable-certificate-owner-ref is not set, so cert-manager's default applies and it owner-references the Certificate but not the Secret); documented behaviour for the consequence; not applied — no rename was performed. And the promise rests on a cluster flag this repository does not control: turn that flag on and a reorder costs a fresh ACME request per renamed object.

    Three transferable pieces. A justification carried in a rule is load-bearing — it is what someone reaches for when the rule meets a case its author did not have, so a reason that is merely true of the first instance will mislead at the second. Parallel rounds are a conformance test nobody wrote: this surfaced only because two agents implemented the same shape from the same sentence at the same time and disagreed; sequentially, the second would simply have followed the first and the broken reason would have survived. And where a rule is corrected rather than the code, say which way round it went, because the next reader will otherwise re-derive the original objection — which is exactly what happened here, correctly, and cost a round.

And a second occurrence of the entry above it, which is what turns one chart's story into a rule. stalwart's per-entry tlsSecretName check had exactly the defect the realm-list round describes: with both routing modes on and a tlsSecretName missing, the render reported "in Ingress mode every entry needs one" — true, and the wrong diagnosis, because that configuration is not coherent to begin with. Found by rendering the conflicting combination rather than by reading the template, and fixed the same way: the Ingress-mode block is gated on ingress.enabled and not httpRoute.enabled, so in that state it validates nothing and the only fail left is the structural one. Two charts met it in the same round, from two agents, neither having seen the other's code — so expect it in the next chart that gains a per-entry check beside a mode switch, and put the conflicting combination in the test matrix for the new value.

The value the operator quoted correctly, and the manifest where it stood bare

A chart rendered a cron schedule straight into its CronJob:

schedule: {{ .Values.smbConnection.cronJob }}

The default, "0 0,12 * * *", rendered cleanly — because it begins with a digit. The commonest notation of all, */15 * * * *, did not:

Error: YAML parse error on paperless-ngx/templates/paperless-consume.cronjob.yaml:
error converting YAML to JSON: yaml: line 6: did not find expected alphabetic or numeric character

The part that makes this a genus and not a typo: quoting the value does not help. Helm hands the template the string's content, so the quotes an operator writes in their values file are gone by the time the scalar is emitted, and the star stands bare in the manifest again. The operator does everything right and gets an error that names the template — a file they did not write, on a line their value does not appear on. @daily fails identically (@ is a reserved YAML indicator), and so would any value beginning with &, {, [, % or !.

The fix is one word, | quote. Four things generalise, and the first two are why this is worth a page.

The quiet half is the dangerous one

The same missing | quote has a second failure mode that does not abort anything. A value whose content is a legal plain scalar but not a string renders perfectly and lands in the manifest as the wrong type:

Value in values.yaml In the manifest What happens
"*/15 * * * *" */15 * * * * render aborts — loud, and misdirected
"true" true a YAML bool; the API server rejects the object at apply time
"0755" 0755 an integer; same

Both of the latter are schema-valid strings on the way in. helm template exits 0, helm lint is happy, the diff looks right, and the failure arrives at kubectl apply with cannot unmarshal bool into Go struct field EnvVar.value of type string. So "does the render succeed" is not the test — the output has to be read back and type-checked. Kubernetes types env[].value, every label value and every annotation value as a string, so there is no chart for which the opposite is legitimate: that is what makes it checkable without per-chart expectations.

A defect lives where no fixture renders

The schedule survived in a shipped chart for months for one reason: the object carrying it was rendered by nothing. The CronJob sits behind smbConnection.enabled, which is false in the chart defaults and false in the install-test fixture — deliberately, since the CI cluster has no SMB CSI driver. Neither lint, nor the render tests, nor the install-test ever produced that template's output.

A test over what the fixtures already render would therefore have been just as blind, and the test that was added is the overlay as much as the script: ci/yaml-scalar/<chart>.yaml switches such objects on and feeds them hostile values, and the run fails when an overlay stops reaching its object or when nothing was asserted at all. The transferable question for any new check: is there a values combination under which this template produces no output? If yes, that combination is where the next defect will sit, and an anti-vacuum guard is the only thing keeping the green tick honest.

A convention that 23 charts follow and 7 do not is not yet a convention

template-chart carried the rule all along, in a comment ending: "Every chart in this repo quotes here." It was false when it was written — seven charts rendered their env block bare. The comment was not a lie, it was a generalisation from the file it sits in, and nothing ever compared it with the other 29 charts.

Two consequences. Write the exceptions, or write a test — a count in prose decays silently, and this one had decayed into the opposite of the truth. And when a sweep conforms one chart, say which ones remain: the README now names the six charts that still render it bare, and that bringing each one over is a major bump for that chart, instead of repeating the claim.

The issue can be wrong about the defect and right about the pain

The companion issue reported that the chart had no way to set an arbitrary PAPERLESS_* variable. It had one — the standard additive env block, in the schema and rendering after the fixed entries. The report had read helm show values of a deployed, older release and listed only the top-level keys; the block sits one level in.

The reported pain was real anyway, because the block could not be trusted with an arbitrary value. So the right move was neither "works as designed" nor building what was asked for (an extraEnv, which would have been a second name beside env) — it was fixing what was actually broken about the existing one and naming the deviation from the issue's title in the pull request. Check a "the chart cannot do X" report against the chart on main before designing X; and where the finding differs, say so instead of quietly delivering something else.

The net that existed, for a different failure than the one it was built for

The scope of Gitea #5 — a NetworkPolicy in 29 charts at once, on by default — rested on one claim: that the install-test enforces policies, so a wrong one goes red in CI rather than silently in production. The workflow creates its k3d cluster without --disable-network-policy, so k3s brings its policy controller, and the plan concluded that a policy blocking the probe path makes the pod never Ready and the install-test red.

That was read off a flag, so it was measured instead. Two experiments on the PR, both reverted afterwards and both kept in the history:

Experiment Result
cyberchef's probe port restricted to 192.0.2.11/32 — an RFC 5737 documentation address that cannot be the k3d node's install-test green, 1m9s
outline's DNS egress rule removed, so postgres and redis do not resolve and /_health cannot pass install-test red, 5m59s

Both results were needed, because one alone is ambiguous. Green on the first could have meant no enforcement at all — in which case there was no net and the scope had to be renegotiated. Red on the second rules that out: policies are enforced. So the two together say something more precise than either: k3s exempts the kubelet's probe traffic from policy, while enforcing everything else.

The lesson is not "the net was missing". It is that the net was real and covered a different failure than the one it was signed off for — and that the difference only became visible because the claim was tested in both directions. Three things followed from it, and none of them would have been found by reading documentation:

  • The convention's default never restricts a probe port, so the trap CI cannot see is not reachable from the rollout itself — only from an operator narrowing probeSources deliberately, which is why that value carries a warning in values.yaml and a required sentence in NOTES.txt.
  • A wrong ingress port — the NodePort mistake — is therefore caught by the render assertion or not at all: probes are exempt, and no ingress controller runs in CI either. That raised the render test from a convenience to the load-bearing check.
  • And it says nothing about production, whose CNI is Calico. There probe traffic is subject to policy, which is precisely why the default has to stay open.

The transferable part: when a check is the justification for a large change, test it in both directions before relying on it. A green result on a deliberately-broken input does not mean "the check is broken" — it may mean the check is sound and the claim about it was wrong, and those need different fixes.

A comment that could not describe its own syntax

The convention for #5 says reasoning belongs in Go template comments rather than # comments, because a # comment is part of the rendered object and ships to the cluster. The header of the reference implementation said so — and spelled the delimiters out literally to make the point.

The render then failed with error converting YAML to JSON: yaml: did not find expected node content, pointing at the template. The Go template parser had ended the comment at the first occurrence of the closing delimiter inside the prose, and emitted the rest of the sentence as manifest content.

It is a small thing with a sharp edge, because the error names neither the sentence nor the cause, and the obvious reading of it — "my YAML is malformed" — is about the wrong half of the file. Describe the delimiters, never quote them, inside the construct they close. The same applies to any templating language whose comment has no escape for its own terminator.

Three values that reached no rule, and would have read as controls

The #5 rollout settled on five value names so every chart's policy looks the same: enabled, allowedNamespaces, probeSources, ingressController, egress.extraPorts. Uniformity is worth something — an operator who has read one chart's block has read them all.

Then the verification harness reported, for samba and satisfactory, that setting allowedNamespaces and probeSources changed nothing in the render. Both charts publish every port as a NodePort, so every inbound rule is open to any source by design; there was no rule for a source list to attach to. teamspeak-server had the same problem with probeSources alone.

Uniformity lost, and it should have. A value that silently does nothing is worse than a value that is absent, because it reads as a control: someone narrowing a policy would have set it, seen no error, and believed the port was restricted. Those charts now ship only the values that reach a rule — samba and satisfactory carry enabled alone — and each says in values.yaml why the others are missing.

The transferable part, and it generalises past this issue: a convention about value names is not a convention about which values exist. Check that each one changes the render in the chart you are adding it to; a harness that renders both branches finds it in seconds, and a review almost never does.

The subchart whose comments the parent could not fix

The #5 render test fails a chart whose rendered NetworkPolicy contains a # comment. valkey and stalwart both failed it on the first run — correctly, and both were fixed.

paperless-ngx then kept failing, for a policy that was not in its diff. Its valkey dependency is resolved from the published Helm index, not from the copy in this repository, so the subchart it bundles is whatever version is on the registry — and that one still had the old comments. Nothing in the parent chart could change them, and nothing would until valkey's fix was merged and published, which cannot happen in the same pull request (style guide §14: never change a chart and a chart that depends on it in the same PR).

So the assertion now skips documents whose # Source: path contains /charts/. Not out of leniency: holding the parent responsible would have made the test unfixable within the PR and would fail a chart for a file its own diff does not contain. The subchart's own PR is where it is caught, because the job runs over every chart including valkey.

The transferable part: a check that runs over a rendered tree is running over code from more than one release. Before making one a gate, ask which failures the author of this diff can actually fix — and scope it to those. The rest belongs to whoever owns the other artifact.

The rule that named the Service port, and the sentence that stopped anyone checking

An egress rule names the pod port of its destination, never the Service port, because the destination NAT happens before the policy is evaluated. And the symptom of getting it wrong is a timeout, not connection refused.

paperless-ngx 14.0.0 allowed outbound 80 for Tika and Gotenberg — the port those two charts' Services listen on — and said why, in the template, in values.yaml and on its wiki page:

an EGRESS rule names the port on the DESTINATION as the client dials it, and the client dials the Service.

kube-proxy rewrites a ClusterIP's address and port before the packet reaches the pod's network interface, which is where the policy is evaluated. The policy sees podIP:targetPort. Both Services forward 80 to 9998 and 3000, so the rule matched nothing, and the release cut Paperless off from both services on 2026-10-15: mail import, EML, HTML and Office documents stopped, plain PDFs and images kept working, every pod stayed Ready, and the error was httpx.ConnectTimeout naming the Tika URL. Nothing mentioned a NetworkPolicy.

The error type is the diagnostic, and it is the cheapest thing to carry away: a missing listener refuses, a dropped packet runs into the timeout. A timeout to a service that demonstrably exists is a policy question. Nothing in Kubernetes will say so, and the application's own error — correctly — blames the application's own dependency.

Why only this one rule broke, and why that made it so quiet: 5432, 6379, 993 and 443 all have Service port and pod port the same number, so the wrong mental model produces the right rule everywhere else. Checked across the repository: 58 Services in the cluster map a different port, and the charts' egress rules named exactly these two. Three other charts name port: 80 in a policy and are correct — their destination is the internet, where there is no DNAT and 80 is the real port.

How the wrong sentence got in, which is the part worth the entry

The information was already written down, in this repository, two days earlier and by the same session. The style guide's mechanics for #5 say:

A policy applies to traffic that came through a Service… a Service is only destination NAT and the policy is evaluated at the pod's interface, so a ClusterIP exempts nothing.

That is the correct mechanism. It was used to argue the ingress direction (why a podSelector can still match the Traefik pod) and never carried over to the egress direction, where the same destination NAT decides which port to name. The premise was present, the conclusion was missing, and the sentence written in its place was fluent, specific and plausible — which is precisely why no review stopped on it. It survived a pull request, a render test, an install-test and three documents.

So: a confident explanation in a comment is not evidence, and it costs more than no comment at all — it reads as the work of checking, so the checking does not happen. Where a rule rests on a mechanism, the comment should name that mechanism in a form a measurement could contradict ("the policy sees podIP:targetPort"), not in a form that merely sounds like a reason ("the client dials the Service"). And when a mechanism is established for one direction, ask what it implies for the other in the same sitting — that is the moment the knowledge is actually present.

What now catches it, and the limit of that

The #5 assertions hold a policy's ports against the pod spec it selects — right for ingress, meaningless for egress, where the destination is in another chart's render or in no render at all. Within one chart this mistake is not catchable, and that is not a defect of the check.

ci/run-networkpolicy-test.sh now also renders apache-tika and gotenberg alongside paperless-ngx and requires the egress rule to contain their containerPorts, resolved Service port -> targetPort -> containerPort so the output prints the DNAT chain rather than two bare numbers. It is checked with the feature gate on and off; the pairs are an explicit table, because no render can know which Service another chart dials (the address is a value or a vault field); and a row naming a chart that no longer exists fails rather than being skipped.

The limit, named so it is not mistaken for coverage: this works only because both destinations are charts of this repository, and paperless-ngx is the only place where that is true. For a database, a cache, an OIDC issuer or anything else outside the repo, nothing can check the number — the only defence there is knowing that the DNAT, and not the direction, decides it.

And one decision that went the other way on purpose

The obvious-looking fix was a second value, networkPolicy.egress.tikaGotenberg. It was rejected. A policy switch that can drift from the feature switch re-arms the identical outage: whoever enables Tika and forgets the policy value lands in the same timeout with the same unhelpful error. The rule stays gated on the feature itself. A guard that can be out of step with the thing it guards is not a guard — and the gap between two values is invisible in exactly the way the first bug was.

helm-charts wiki

How we work

How it is built

How charts are written

Charts