valkey: chart page, index row, sidebar entry, two lessons

The new valkey chart (helm-charts #53, issue #46): one instance per
application instead of the shared cluster-redis, whose separation rested
on a single ACL file in a single 1Password field.

The page carries what an operator cannot read off the templates: that the
1Password field auth-conf holds a finished "requirepass <password>" line
rather than a password, that uid 999 / gid 1000 are the image's own user
and why fsGroup has to match them, that maxmemory is derived from the
computed memory limit and a value above it aborts the render, and that
persistence and the eviction policy are a pair rather than two switches.
Also what the install-test does not cover: one variant per chart, and
fsGroup itself is unproven on k3d's node-directory volumes.

Two Lessons-Learned entries, both from this chart's review rather than
from an outage, because both generalize: a CLI that prints an error and
exits 0 (valkey-cli without -e, so the natural ping probe cannot fail),
and a secret whose FORMAT is load-bearing - a present-but-malformed field
yields a running server with its protection off.
jklein committed 2026-09-27 11:08:43 +00:00
1 parent 311293e164
commit d91927d2d0
4 files changed
+284 -1

No files matched your search

+2 -1
@@ -1,6 +1,6 @@
# Chart index
25 directories: 24 installable charts plus the `template-chart` scaffold. Each page holds prerequisites, an installation walkthrough, the complete list of configurable values, and the chart's operating traps.
26 directories: 25 installable charts plus the `template-chart` scaffold. Each page holds prerequisites, an installation walkthrough, the complete list of configurable values, and the chart's operating traps.
**Versions are not listed here on purpose** — and not on the chart pages either. Renovate moves app versions continuously; a table of numbers in a wiki is wrong within days. Read the current pair off the chart:
@@ -32,6 +32,7 @@ grep -E '^(version|appVersion):' <chart>/Chart.yaml
| [teamspeak-server](teamspeak-server) | TeamSpeak server | StatefulSet | no | probes address the TCP port by name; the voice service is UDP and unprobeable |
| [template-chart](template-chart) | — | scaffold | — | **not installable**; the reference implementation of every convention |
| [urlaubsverwaltung](urlaubsverwaltung) | Urlaubsverwaltung | Deployment | yes | readiness names its health-group members explicitly after a silent outage |
| [valkey](valkey) | Valkey | StatefulSet + validating init container | no | one instance per application instead of the shared `cluster-redis`; the password is a finished config line from the vault, `maxmemory` is derived from the memory limit, and the first chart with a NetworkPolicy on by default |
| [voucher-vault](voucher-vault) | VoucherVault | StatefulSet | yes | DB and OIDC secrets |
| [ytdlp-web-player](ytdlp-web-player) | ytdlp_web_player | Deployment | yes | **the only chart with a dependency**: an optional oauth2-proxy subchart, off by default |
| [zipline](zipline) | Zipline | StatefulSet | yes | liveness deliberately avoids the database — the positive example of [§12.3](Chart-Style-Guide#12-values-that-must-not-lie) |
+14
@@ -164,6 +164,20 @@ Two things count against that limit and are easy to forget: **emptyDirs** and th
**Now:** the chart requests `256Mi` per app container, and says in `values.yaml` that this is headroom rather than a measurement. The general form: **`ephemeral-storage` is not boilerplate** — a chart that mounts emptyDirs has to budget for what is written into them.
## The CLI that reports an error and exits 0
Not an outage — a probe that was **checked before** it shipped, and the check is the entry. The obvious exec probe for [valkey](valkey) is `valkey-cli ping`. Against a password-protected server, an unauthenticated `ping` answers `NOAUTH Authentication required.` — and `valkey-cli` **exits 0** while printing it. In `valkey` 9.1.2 (`src/valkey-cli.c`) an error reply becomes a non-zero exit in exactly one branch, guarded by `config.set_errcode`, and the `-e` flag is the only thing that sets it.
So the natural probe would have been Ready for a server that rejected every single command — including, for instance, one started without the password at all.
**Now:** the chart's probes authenticate (through `VALKEYCLI_AUTH`, never an `-a` argument that would land in the process list), pass `-e`, **and** assert the answer (`grep -qx PONG`). Generally: **an exec probe wrapping a CLI is only an assertion once that CLI's exit code has been read off its source or measured.** "The command ran" and "the server is healthy" are different statements, and a client that prints errors to stdout with status 0 turns the first into the second for free. The same question applies to `psql`, `curl` without `--fail`, and every `*-cli ping` in this repo.
## The password that would have been optional
Same chart, the other half of the same review. [valkey](valkey) takes its password as a **finished configuration line** out of a 1Password field (`requirepass <password>`), mounted as a file and `include`d — which is what keeps it out of the process list and out of every rendered manifest. The cost of that design is that the vault field is now syntax: `requirepas`, a stray quote or an empty value yields a Valkey **without a password**. No render error, no crash, a green pod — and the official image compiles `protected-mode` **off**, so "no password" means "reachable from the whole pod network".
**Now:** an init container greps the mounted file for a `requirepass <value>` line and refuses to let the pod start without it, naming the 1Password field and never printing its contents. Generally: **when a secret's *format* becomes load-bearing, something has to validate the format** — a value that is merely *present* satisfies Kubernetes, and the failure it produces can be a working server with its protection off.
## The recurring shape
- **Green is a claim about the assertions that ran, not about the system.**
+1
@@ -41,6 +41,7 @@
- [teamspeak-server](teamspeak-server)
- [template-chart](template-chart)
- [urlaubsverwaltung](urlaubsverwaltung)
- [valkey](valkey)
- [voucher-vault](voucher-vault)
- [ytdlp-web-player](ytdlp-web-player)
- [zipline](zipline)
+267
@@ -0,0 +1,267 @@
# valkey
Deploys one [Valkey](https://valkey.io/) instance for **one** application — a cache or a queue backend — as a single-replica StatefulSet. It is the replacement for the shared `cluster-redis`, whose separation rested on a single ACL file in a single 1Password field: every change there touched six applications at once, needed a restart to take effect, and produced a month of Paperless standstill when one application's keys were not covered by the pattern (`cluster-config#102`). With one instance per application the ACL question disappears rather than being solved — each instance has exactly one user, and that user may do `~*`.
| | |
|---|---|
| Image | `docker.io/valkey/valkey` (the official image, **not** `bitnami/valkey` — Bitnami publishes only `latest` since August 2025) |
| Versions | not listed on purpose — read the current pair off `valkey/Chart.yaml`, see [Chart index](Charts) |
| Workloads | StatefulSet (1 replica) + one validating init container |
| Storage | one RWO PVC when `persistence.enabled` (default), otherwise an emptyDir |
| Ingress | **no** — and no value to add one |
| External dependencies | none |
| Secrets | 1 × `OnePasswordItem` |
| Network | ClusterIP Service on 6379, plus a **NetworkPolicy on by default** |
| Install-test | `ci/valkey/` (fixtures + test values, the **persistent** variant) |
## Prerequisites
- **1Password operator** in the cluster, plus one 1Password item carrying exactly one field — see [the field below](#what-the-1password-item-must-contain). The field holds a finished configuration line, not a password, and the exact spelling matters.
- **A storage class** matching `persistence.storageClass` (default `longhorn-encrypted-ns`) when persistence is on.
- **Nothing else.** Valkey depends on no database, no other service and no network egress.
## Installation
The registry URL and its credentials live in 1Password (item `jlab-registry-access`: `username`, `password`, `charts-url`) and never in a repository or a shell history file.
```bash
helm repo add jlab "$CHARTS_URL" --username "$REG_USER" --password "$REG_PASS"
helm repo update
helm install sure-valkey jlab/valkey \
--namespace sure --create-namespace \
-f my-values.yaml
```
Minimal `my-values.yaml` — one required value, because everything else has a default that is safe for a queue:
```yaml
secrets:
authSecretRef: op://vault/sure-valkey
```
For a pure cache, two values instead of one (they belong together, see [Persistence and eviction are a pair](#persistence-and-eviction-are-a-pair-not-two-switches)):
```yaml
secrets:
authSecretRef: op://vault/searxng-valkey
persistence:
enabled: false
valkey:
maxmemoryPolicy: allkeys-lru
```
`helm install` prints the chart's `NOTES.txt`, which names the shape the release ended up in (queue / cache / the mistaken pair), the computed `maxmemory`, and the connection string to put in the *consuming* application's vault item. Read it — it is the per-release version of this page.
### Naming
One instance per application, so the release name says which application it belongs to: `sure-valkey`, `outline-valkey`, `oneuptime-valkey`. Objects are then `sure-valkey-sfs`, `sure-valkey-service` and so on. The instance normally lives in the **consumer's** namespace, which is also what makes the default NetworkPolicy sufficient without a single extra value.
### What the 1Password item must contain
The chart renders one `OnePasswordItem`; the operator turns it into a Kubernetes Secret whose **keys are the field names of the 1Password item**.
| Values key | Kubernetes Secret | Field | Content |
|---|---|---|---|
| `secrets.authSecretRef` | `<release>-valkey-secret` | **`auth-conf`** | a complete Valkey configuration line: `requirepass <the password>` — **not** the bare password |
Two consequences, and both are the point of the design:
- **The password never appears anywhere but the vault and the pod's own tmpfs.** The file is mounted and `include`d by the configuration, so there is no `--requirepass` in the process list, no environment variable, and nothing Helm renders. The same applies to the probes, which read the password out of that same file.
- **A typo in the field would be an open server.** `requirepas`, a stray quote, or an empty field means Valkey starts *without* a password — no render error, no crash, a green pod, and (because the official image compiles `protected-mode` **off**) reachable from the whole pod network. An init container therefore checks the mounted file before the server starts and refuses to let the pod run without a `requirepass <value>` line. If the pod hangs in `Init:Error` or `Init:CrashLoopBackOff`, that is the first thing to read:
```bash
kubectl -n <namespace> logs <release>-valkey-sfs-0 -c <release>-valkey-check-auth
```
It names the field and never prints its contents.
A wrongly **named** field fails differently and earlier: the volume cannot be projected and the pod stays in `ContainerCreating` with `couldn't find key auth-conf` in its events.
The connection string for the consuming application belongs in **that application's** item, not here:
```
redis://:<password>@<release>-valkey-service.<namespace>.svc.cluster.local:6379/0
```
`redis://` is correct — Valkey speaks the same protocol and every client library still calls the scheme that.
## Configurable values
### Global
| Key | Type | Default | Description |
|---|---|---|---|
| `scaleDown` | bool | `false` | Renders the StatefulSet with `replicas: 0` without uninstalling. Takes precedence over `valkey.replicas`. PVC, Secret, Service and NetworkPolicy stay. |
### `secrets`
| Key | Type | Default | Description |
|---|---|---|---|
| `secrets.authSecretRef` | string | `null` | **Required.** `op://` path of the item carrying the `auth-conf` field. |
### `valkey`
| Key | Type | Default | Description |
|---|---|---|---|
| `valkey.replicas` | int | `1` | `0` or `1` — the schema rejects anything higher. See [One instance, on purpose](#one-instance-on-purpose). |
| `valkey.maxmemoryFactor` | number | `0.75` | Fraction of the **computed memory limit** that Valkey may fill with data. Ignored when `valkey.maxmemory` is set. See [maxmemory is derived, not configured](#maxmemory-is-derived-not-configured). |
| `valkey.maxmemory` | string | `null` | Explicit `maxmemory` as a Kubernetes quantity (`512Mi`). **A value above the computed memory limit aborts the render**, and so does `0`. |
| `valkey.maxmemoryPolicy` | string | `noeviction` | Valkey's `maxmemory-policy`, as an enum of the eight real policies. `noeviction` for queues, `allkeys-lru` for caches. |
| `valkey.extraConfig` | string | `""` | Extra configuration lines, appended verbatim (rendered through `tpl`). **Cannot switch the password off** — the auth `include` is rendered after it and the last directive wins. It *can* override `maxmemory`, which then escapes the limit check; use `valkey.maxmemory`. |
| `valkey.securityContext.pod` | object | non-root **999**, `runAsGroup`/`fsGroup` **1000**, `fsGroupChangePolicy: Always`, `seccompProfile: RuntimeDefault` | Passed verbatim into the pod spec. The uid is the image's own `valkey` user — see [Why uid 999 and gid 1000](#why-uid-999-and-gid-1000). The seccomp profile sits here so it also covers the init container. |
| `valkey.securityContext.container` | object | `readOnlyRootFilesystem: true`, all caps dropped, no privilege escalation, non-root | Applied to **both** containers. `/data` is the only writable path and needs no emptyDir beyond it. |
| `valkey.livenessProbe` | object | `exec /bin/sh /etc/valkey/ping.sh`, period 10s, timeout 5s, 3 failures | An **authenticated** `valkey-cli ping`. See [What a green pod proves](#what-a-green-pod-proves-and-why--e-is-in-the-command). |
| `valkey.readinessProbe` | object | same command, period 10s, timeout 5s, 3 failures | |
| `valkey.startupProbe` | object | same command, period 5s, 30 failures (= 150s) | Gates liveness and readiness. The budget covers replaying a large append-only file. |
| `valkey.env` | list | `[]` | Extra environment variables. Rendered through `tpl`; `value`, `secretKeyRef` and `configMapKeyRef` forms. **Additive, not overriding** — see [Environment variables](Chart-Style-Guide#10-environment-variables). Note that the password deliberately does **not** travel this way: the image entrypoint appends `$VALKEY_EXTRA_FLAGS` to the server command line, which would put it in the process list. |
| `valkey.resources.limitFactor` | number | `3` | Memory and ephemeral-storage limits are computed as factor × request. **This is also what sizes `maxmemory`.** |
| `valkey.resources.requests` | object | `cpu: 0.1`, `memory: 256Mi`, `ephemeral-storage: 64Mi` | |
| `valkey.resources.limits` | object | unset | An explicit limit always wins. A CPU limit is only rendered when set here. |
### `persistence`
| Key | Type | Default | Description |
|---|---|---|---|
| `persistence.enabled` | bool | `true` | `true` = RWO PVC, `false` = emptyDir **and** both persistence mechanisms switched off (`appendonly no`, `save ""`), so nothing is written to ephemeral storage. |
| `persistence.appendonly` | bool | `true` | AOF instead of RDB-only. For a queue the question is not "is there a backup" but "how much is lost when the pod dies": a snapshot loses every job since the last one, silently. Has no effect while `persistence.enabled` is `false`. |
| `persistence.size` | string | `1Gi` | Small on purpose: on encrypted Longhorn volumes space is never returned to the pool once written, and a PVC can grow but never shrink. |
| `persistence.storageClass` | string | `longhorn-encrypted-ns` | `null` uses the cluster default — which is **not** encrypted, so leaving the default is a decision about data at rest. A cache regularly holds the same payloads as the application's database. |
### `networkPolicy`
| Key | Type | Default | Description |
|---|---|---|---|
| `networkPolicy.enabled` | bool | `true` | Ingress to 6379 from the release's own namespace. Egress unrestricted (this instance opens no outbound connection). |
| `networkPolicy.allowedNamespaces` | list | `[]` | Additional namespaces, matched on the `kubernetes.io/metadata.name` label Kubernetes sets itself — so no labelling work, but also no typo protection: a name that does not exist matches nothing and is silently ineffective. |
There is **no `ingress` section**. A Valkey whose only authentication is one password does not belong behind a public hostname, so the values that would allow it are absent rather than switched off.
## Operating notes
### Persistence and eviction are a pair, not two switches
The only two combinations that make sense:
| Shape | `persistence.enabled` | `valkey.maxmemoryPolicy` | For |
|---|---|---|---|
| **Queue** (chart default) | `true` | `noeviction` | Sidekiq, Celery — entries are work that must not vanish |
| **Cache** | `false` | `allkeys-lru` | recomputable data only |
Mixing them is what the defaults exist to prevent:
- A **queue with `allkeys-lru`** throws jobs away under memory pressure while every application stays green. That was one of the hypotheses in `cluster-config#102` and it is far cheaper to prevent here than to diagnose there.
- **No persistence with `noeviction`** is almost never intended: the data is gone at every restart, so nothing is being protected — but a full instance starts *refusing writes* instead of dropping cold keys. The rendered `NOTES.txt` prints a warning for exactly this pair.
### `maxmemory` is derived, not configured
`maxmemory` comes from the container's **computed** memory limit (`limitFactor` × the memory request, or an explicit `limits.memory`), multiplied by `maxmemoryFactor` (`0.75`). With the chart defaults: request `256Mi` → limit `768Mi` → `maxmemory 603979776` (576 MiB).
Two properties worth knowing before changing it:
- **The number is read back out of the rendered resources block**, not calculated a second time, so it cannot drift from what the container actually gets.
- **A configured `valkey.maxmemory` above that limit aborts the render**, with both byte counts in the message. It is not clamped and not warned about: above the limit, the kubelet's OOM kill decides what the eviction policy was configured to decide, and that arrives as a restarted pod with no explanation. `valkey.maxmemory: 0` aborts too — in Valkey `0` means *unlimited*, i.e. the exact opposite of what the value is for.
The quarter that is not handed to `maxmemory` is not slack: client output buffers, the AOF buffer, and above all the copy-on-write pages of the `fork()` that writes a snapshot or rewrites the AOF all live **outside** `maxmemory`.
### Why uid 999 and gid 1000
Not the repo's usual `20000`/`1000`, and not interchangeable. The official image creates its own user (`9.1/alpine/Dockerfile`):
```
addgroup -S -g 1000 valkey; \
adduser -S -G valkey -u 999 valkey
```
— gid 1000 because Alpine already uses gid 999. The image sets **no `USER`** and starts as root; its entrypoint then steps down by itself, and on the alpine flavour it does so with `setpriv` (the Debian flavour uses gosu):
```sh
if [ "$1" = 'valkey-server' ]; then
if [ "$(id -u)" = '0' ]; then
find . \! -user valkey -exec chown valkey '{}' +
exec setpriv --reuid=valkey --regid=valkey --clear-groups -- "$0" "$@"
```
Running the pod directly as uid 999 — which `runAsNonRoot` requires — **skips that branch**, including its `chown` of `/data`. So `fsGroup: 1000` is not decoration: it is the only thing that makes the volume writable. Change one of the three (`runAsUser`, `runAsGroup`, `fsGroup`) and the instance either refuses to start or cannot write its append-only file.
The server is started through the image's entrypoint with `args: [/etc/valkey/valkey.conf]`, not with `command:` — the entrypoint prepends `valkey-server` for anything ending in `.conf`, and replacing it would lose `tini` as PID 1.
### What a green pod proves, and why `-e` is in the command
All three probes run the same script, which authenticates with the password from the mounted file and requires the answer `PONG`:
```sh
VALKEYCLI_AUTH="$pass" valkey-cli -e -p 6379 ping | grep -qx PONG
```
Both halves matter, and the reason is the most expensive bug class in this repository:
- **`valkey-cli ping` exits 0 on an error reply** unless `-e` is given. In `valkey` 9.1.2 (`src/valkey-cli.c`) an error reply becomes a non-zero exit in exactly one branch, guarded by `config.set_errcode`, and `-e` is the only thing that sets it. A probe written as a bare `valkey-cli ping` therefore prints `NOAUTH Authentication required.` **and reports success** — an assertion that cannot fail.
- **`grep -qx PONG` asserts the answer**, not that the command returned. Either half alone would catch a `NOAUTH`; `-e` additionally sends the server's own message (`NOAUTH…`, `WRONGPASS…`) to stderr, where `kubectl describe` shows it in the probe's failure output.
- **The password goes through `VALKEYCLI_AUTH`**, never `-a <password>` — an argument is readable in the process list, and `valkey-cli -a` says so itself.
So `Ready` means: the server accepted a connection, accepted that password, and answered. It does **not** mean the dataset is intact — a restarted instance whose AOF was lost answers `PONG` on an empty database — and it says nothing about whether the consuming application's connection URL is correct.
To check by hand without putting the password in your shell history:
```bash
kubectl -n <namespace> exec <release>-valkey-sfs-0 -- /bin/sh /etc/valkey/ping.sh && echo OK
```
### One instance, on purpose
`valkey.replicas` is capped at 1 by the schema. There is no cluster mode and no replication: two pods would fight over the same ReadWriteOnce volume, and clients behind the Service would silently talk to two different datasets. Scaling this chart means *another release for another application*, not another replica.
### Rotating the password does not restart the pod
The pod template carries `checksum/config` over the configuration ConfigMap and `checksum/secret-ref` over the rendered `OnePasswordItem` — so a changed `maxmemory` or eviction policy rolls the pod, and so does pointing `secrets.authSecretRef` at a **different** item. A new password **inside the same item** cannot be covered: Helm never sees a Secret the 1Password operator fills in ([style guide §9](Chart-Style-Guide#9-config-changes-must-restart-the-workload)). After such a rotation:
```bash
kubectl -n <namespace> rollout restart statefulset/<release>-valkey-sfs
```
`operator.1password.io/auto-restart` is deliberately **not** set: the operator restarts Deployments only, so on a StatefulSet it is a silent no-op that would advertise a guarantee that does not exist.
One side effect, written down as **unverified** rather than promised: because the probes read the password out of the same mounted file, the kubelet's refresh of that file (roughly a minute) should make the liveness probe fail against the still-old server and restart the container by itself. Do the rollout restart rather than waiting for it — and mind that the rotation is only complete when the *consuming* application's connection URL has been changed too.
### The NetworkPolicy is only a boundary where the CNI enforces one
The API server accepts a `NetworkPolicy` whatever the cluster does with it. Where the CNI does not enforce policies, everything looks configured and nothing is filtered — which is worse than no policy, because it invites the assumption that the password is the second line of defence rather than the only one. Verify from an unrelated namespace instead of reading it off the manifest:
```bash
kubectl -n <other-namespace> run t --rm -it --image=docker.io/valkey/valkey:<appVersion> \
-- valkey-cli -h <release>-valkey-service.<namespace> ping
```
That must hang or be refused. `NOAUTH Authentication required.` means the policy is not being enforced.
### The tag moves, and that is accepted
`appVersion` is `9.1.2-alpine`, and **no** tag of this image is immutable: upstream rebuilds the whole matrix periodically, including the fully numbered `-alpine3.24` variants (`9.1-alpine` and `9.0.6-alpine3.24` were pushed minutes apart on 2026-09-21, two different release lines). So a rebuild changes the running image with no commit in the repository and nothing reports it.
The `-alpine` tag was chosen over `-alpine3.24` precisely because both move equally, while only `-alpine` keeps a Renovate rule that cannot freeze: docker versioning splits at the first dash and offers only candidates with the same suffix, and upstream ships exactly one Alpine base per release — so a base change arrives as `9.1.3-alpine3.25`, which a rule bound to `alpine3.24` would never see. The remedy for the moving tag, if it ever matters, is `pinDigests` for this one image; see [Renovate](Renovate).
## Upgrades and backup
- **A Valkey upgrade is a restart, and a restart is a data question**, not a version question. With `persistence.enabled: false` the dataset is gone; with AOF it is replayed. Plan the upgrade of a *queue* instance for a moment when losing in-flight work would be tolerable anyway.
- **Raise `valkey.startupProbe.failureThreshold` before an instance's AOF grows large**, not after the crash loop has started: the budget is 150s by default, and replaying a multi-GB file takes longer.
- **`helm uninstall` does not delete the PVC.** The data survives a reinstall under the same release name; a release recreated under a *different* name starts empty while the old volume keeps occupying (never-reclaimed, on encrypted Longhorn) space.
- **Migrating an application off `cluster-redis` has a fixed order**: roll out the instance → repoint the `connection-url` in the application's vault item → restart its pods → verify → **last** remove the ACL user from `users.acl`, so a fallback stays possible. Outline is nearly free (three keys); the three Paperless instances only after 3.2.0, so two causes do not mix; OneUptime last, because moving it means draining its queues or accepting the loss.
## CI
`ci/valkey/` holds `test-values.yaml` and `fixtures.yaml` (the one Secret the operator would materialize). The install-test runs the **persistent** variant — the risky path: a provisioned PVC, the pod's `fsGroup` against it, and an append-only file created on a volume the container did not create.
Two limits of that coverage, named so they are not mistaken for tested:
- **The machinery runs one variant per chart**, so the non-persistent path is covered by a local render only.
- **`fsGroup` itself is not proven.** k3d's `local-path` volumes are node-directory backed and created world-writable, so the container can write whether or not the kubelet applied `fsGroup`. The test proves writability, not the ownership mechanism. (A hostPath PV fixture would be worse: the kubelet does not apply `fsGroup` to those at all.)
## See also
- [Chart style guide](Chart-Style-Guide) — the rules this chart implements
- [Lessons learned](Lessons-Learned) — including the always-green assertion this chart's probes were checked against
- [Versioning](Versioning) — what a bump of `1.0.0+up9.1.2.alpine` means
- [Chart index](Charts)