feat(paperless-ngx): optional valkey subchart, password from the redis item #80

Merged

jklein merged 1 commits from feat/paperless-valkey-subchart into main 2026-10-03 00:06:59 +02:00

Conversation 2 Commits 1 Files Changed 10

+521 -57

jlab-bot commented 2026-10-02 23:30:13 +02:00

Member

Copy link

Copy source

paperless-ngx can now bring its own Valkey, as an optional subchart of this repo's own valkey chart resolved from the published index — off by default, gated on valkeySubchart.enabled. Additive, default behaviour unchanged for every existing release → minor, 12.1.3+up3.2.1 → 12.2.0+up3.2.1.

PAPERLESS_REDIS is the Celery broker, so the application cannot run without a Redis at all. Before this, every installation needed a second Helm release whose address lived in a 1Password item and in neither chart.

The form follows #77 (stalwart), which is the reference implementation and was not re-invented here.

The plan had one error, and it was corrected before anything was built

The issue asked for valkey.secrets.authSecretRef to default to .Values.secrets.redisSecretRef while also forbidding changes to valkey/. That is not implementable: a values.yaml is never templated and a parent cannot compute a subchart's value. Measured — the expression lands verbatim in the subchart's itemPath and dies as a YAML parse error in somebody else's template, naming neither the value nor the cause. Recorded as a comment on #78, and the maintainer picked option A: the path is typed twice, and forgetting it fails loudly.

Two findings from actually trying it

1. A fail() in the parent cannot fire. The first draft had the guard in _helpers.tpl, called from a rendered template — the shape #77 said to use. It still never ran: Helm renders a subchart's templates before the parent's, so the subchart's own required() always wins and reports secrets.authSecretRef, which is not the path anyone edits. Rather than ship a check that cannot fire, the rule moved to values.schema.json as an if/then, because schema validation runs before any rendering:

- at '': 'allOf' failed
  - at '/valkey/secrets/authSecretRef': got null, want string

2. Erasing the switch turns the subchart ON. valkeySubchart: null deletes the key, Helm then cannot resolve the condition: path, and an unresolvable condition counts as enabled — so the block that reads "off" produced a Valkey with no password. Everywhere else here an erased block means "the defaults apply" (#99); this is the one block where that is backwards, so the schema requires it. valkeySubchart: {} stays harmless (Helm merges it with the default).

Queue, not cache — argued, not inherited

Read off the paperless-ngx 3.2.1 source, not from memory: one PAPERLESS_REDIS url serves five consumers.

Consumer	Setting	Character
Celery broker	CELERY_BROKER_URL	queue
Celery result backend	CELERY_RESULT_BACKEND, expires 3600	Redis in 3.x, the database before
Channel layer	channels_redis.pubsub, expiry 15s	pub/sub, not stored keys
Django cache default	RedisCache	recomputable
Cachalot read cache	off by default, TTL 3600	recomputable

allkeys-lru would not be a trade here but actively wrong: LRU evicts typelessly across the whole keyspace, and a backlog of waiting consume tasks is exactly what it reads as cold. A dropped task is a document silently never processed.

The conflict is named, not smoothed over: two of the five consumers are caches, so a full instance refuses writes — new tasks included — instead of dropping cold keys. The answer is memory sizing, never a policy change. All three cache-like consumers are TTL-bounded, which is what keeps that price small.

The three traps, documented

1. The password is in the item twice (connection url + requirepass line) — a rotation touches both fields.
2. With the subchart on, redis-connection-url must be repointed at <release>-valkey-service:6379, or the bundled Valkey runs unused and nothing says so.
3. A OnePasswordItem materializes every field of its item, so both Secrets carry all three keys. Untidy, harmless, written down so nobody "tidies it up".

That valkey/ must not share a PR with a dependent chart is mentioned in values.yaml only — it is already at three wiki places.

The install-test now installs this path for the first time

#77 covered the subchart by a render only, deliberately. Here the subchart is genuinely used, so ci/paperless-ngx/ switches it on and drops the throwaway redis Deployment; the url names ci-valkey-service. Both fixture Secrets carry all three fields, because in a real cluster both CRs name the same item.

Not verified locally, and not claimed: this workspace has no k3d and no Docker daemon, so the install-test runs for the first time in this PR's pipeline. Result to follow as a comment.

Measured here

• ci/helm-dependencies.sh paperless-ngx with an empty HELM_REPOSITORY_CONFIG: resolves 1.0.0+up9.1.2.alpine from the URL alone.
• helm lint --strict: 0 failed. helm package: paperless-ngx-12.2.0+up3.2.1.tgz.
• helm template: subchart off → 0 valkey references; on → renders, maxmemory-policy noeviction, appendonly yes, PVC on local-path.
• ci/run-quantity-render-test.sh paperless-ngx: ok, both cases exercised.
• NOTES.txt rendered in all branches (on, off, and divergent items).
• Renovate's helmv3 reach and the index resolution were not re-measured — proven in #77 and explicitly out of scope.

Wiki: paperless-ngx, valkey, Charts, Chart-Style-Guide §14, Lessons-Learned — commit 43aef80.

Fixes #78

`paperless-ngx` can now bring its own Valkey, as an optional subchart of **this repo's own `valkey` chart** resolved from the published index — off by default, gated on `valkeySubchart.enabled`. Additive, default behaviour unchanged for every existing release → **minor**, `12.1.3+up3.2.1` → `12.2.0+up3.2.1`. `PAPERLESS_REDIS` is the Celery broker, so the application cannot run without a Redis at all. Before this, every installation needed a second Helm release whose address lived in a 1Password item and in neither chart. The form follows #77 (`stalwart`), which is the reference implementation and was not re-invented here. ## The plan had one error, and it was corrected before anything was built The issue asked for `valkey.secrets.authSecretRef` to **default** to `.Values.secrets.redisSecretRef` while also forbidding changes to `valkey/`. That is not implementable: a `values.yaml` is never templated and a parent cannot compute a subchart's value. Measured — the expression lands verbatim in the subchart's `itemPath` and dies as a YAML parse error in somebody else's template, naming neither the value nor the cause. Recorded as a comment on #78, and the maintainer picked **option A**: the path is typed twice, and forgetting it fails loudly. ## Two findings from actually trying it **1. A `fail()` in the parent cannot fire.** The first draft had the guard in `_helpers.tpl`, called from a rendered template — the shape #77 said to use. It still never ran: **Helm renders a subchart's templates before the parent's**, so the subchart's own `required()` always wins and reports `secrets.authSecretRef`, which is not the path anyone edits. Rather than ship a check that cannot fire, the rule moved to `values.schema.json` as an `if`/`then`, because schema validation runs **before any rendering**: ``` - at '': 'allOf' failed - at '/valkey/secrets/authSecretRef': got null, want string ``` **2. Erasing the switch turns the subchart ON.** `valkeySubchart: null` **deletes** the key, Helm then cannot resolve the `condition:` path, and an unresolvable condition counts as *enabled* — so the block that reads "off" produced a Valkey with no password. Everywhere else here an erased block means "the defaults apply" (#99); this is the one block where that is backwards, so the schema requires it. `valkeySubchart: {}` stays harmless (Helm merges it with the default). ## Queue, not cache — argued, not inherited Read off the paperless-ngx 3.2.1 source, not from memory: one `PAPERLESS_REDIS` url serves **five** consumers. | Consumer | Setting | Character | |---|---|---| | Celery broker | `CELERY_BROKER_URL` | **queue** | | Celery result backend | `CELERY_RESULT_BACKEND`, expires 3600 | **Redis in 3.x**, the database before | | Channel layer | `channels_redis.pubsub`, expiry 15s | pub/sub, not stored keys | | Django cache `default` | `RedisCache` | recomputable | | Cachalot read cache | off by default, TTL 3600 | recomputable | `allkeys-lru` would not be a trade here but **actively wrong**: LRU evicts typelessly across the whole keyspace, and a backlog of waiting consume tasks is exactly what it reads as cold. A dropped task is a document silently never processed. **The conflict is named, not smoothed over:** two of the five consumers are caches, so a full instance **refuses writes** — new tasks included — instead of dropping cold keys. The answer is memory sizing, never a policy change. All three cache-like consumers are TTL-bounded, which is what keeps that price small. ## The three traps, documented 1. The password is in the item **twice** (connection url + `requirepass` line) — a rotation touches both fields. 2. With the subchart on, `redis-connection-url` **must** be repointed at `<release>-valkey-service:6379`, or the bundled Valkey runs unused and nothing says so. 3. A `OnePasswordItem` materializes **every** field of its item, so both Secrets carry all three keys. Untidy, harmless, written down so nobody "tidies it up". That `valkey/` must not share a PR with a dependent chart is mentioned in `values.yaml` only — it is already at three wiki places. ## The install-test now installs this path for the first time #77 covered the subchart by a **render** only, deliberately. Here the subchart is genuinely used, so `ci/paperless-ngx/` switches it **on** and drops the throwaway redis Deployment; the url names `ci-valkey-service`. Both fixture Secrets carry all three fields, because in a real cluster both CRs name the same item. **Not verified locally, and not claimed:** this workspace has no `k3d` and **no Docker daemon**, so the install-test runs for the first time in this PR's pipeline. Result to follow as a comment. ## Measured here - `ci/helm-dependencies.sh paperless-ngx` with an **empty** `HELM_REPOSITORY_CONFIG`: resolves `1.0.0+up9.1.2.alpine` from the URL alone. - `helm lint --strict`: 0 failed. `helm package`: `paperless-ngx-12.2.0+up3.2.1.tgz`. - `helm template`: subchart **off** → 0 valkey references; **on** → renders, `maxmemory-policy noeviction`, `appendonly yes`, PVC on `local-path`. - `ci/run-quantity-render-test.sh paperless-ngx`: ok, both cases exercised. - `NOTES.txt` rendered in all branches (on, off, and divergent items). - Renovate's `helmv3` reach and the index resolution were **not** re-measured — proven in #77 and explicitly out of scope. Wiki: `paperless-ngx`, `valkey`, `Charts`, `Chart-Style-Guide` §14, `Lessons-Learned` — commit `43aef80`. Fixes #78

jlab-bot added 1 commit 2026-10-02 23:30:13 +02:00

feat(paperless-ngx): optional valkey subchart, password from the redis item ... a9ca5302de

paperless-ngx cannot run without a Redis - PAPERLESS_REDIS is the Celery
broker - so every installation needed a second Helm release whose address
lived in a 1Password item and in neither chart. This adds our own published
valkey chart as an optional dependency, off by default and gated on
valkeySubchart.enabled (not valkey.enabled, which the subchart's strict
schema rejects and which Helm would only have complained about for whoever
first switched it on). Additive, default behaviour unchanged: minor,
12.1.3+up3.2.1 -> 12.2.0+up3.2.1.

Configured as a QUEUE (PVC + noeviction, the subchart's defaults), unlike
stalwart's cache. Read off the 3.2.1 source rather than assumed: one
PAPERLESS_REDIS url serves five consumers - the Celery broker, the result
backend (Redis in 3.x, the database before), the channels pub/sub layer and
two Django caches. allkeys-lru would not be a trade here but actively wrong,
because LRU evicts typeless across the whole keyspace and a backlog of
waiting consume tasks is exactly what it reads as cold; a dropped task is a
document silently never processed. The price, named in values.yaml rather
than smoothed over: two of the five consumers are caches, so a full instance
refuses writes - new tasks included - and the answer is memory sizing, not a
policy change. All three cache-like consumers are TTL-bounded, which is what
keeps that price small.

The password comes from the SAME item as the Redis connection, in a third
field auth-conf holding the complete line "requirepass <password>". It cannot
DEFAULT to secrets.redisSecretRef, and that is a property of Helm, not an
omission: a values.yaml is never templated and a parent cannot compute a
subchart's value. Measured - a template expression there lands verbatim in
the subchart's itemPath and fails as a YAML parse error naming neither the
value nor the cause.

Forgetting it is therefore caught in values.schema.json, by a conditional
rule rather than a fail() in a template. Also measured, and the reason:
Helm renders a subchart's templates BEFORE the parent's, so a guard in this
chart loses the race to the subchart's own required(), which reports
"secrets.authSecretRef" and cannot know the path belongs to the redis item.
A check that cannot fire is the mistake found while testing #77, so the
first draft of that guard was removed rather than shipped. The schema runs
before any rendering and names /valkey/secrets/authSecretRef.

One trap found while testing and closed here: "valkeySubchart: null" DELETES
the key, Helm then cannot resolve the condition path and treats the
dependency as ENABLED - so erasing the switch turned the subchart on with no
password set. Unlike every other block in this chart (issue #99), that one
is rejected by the schema instead of defaulting.

The install-test now runs with the subchart ON and the throwaway redis
Deployment removed: the url in the fixtures names ci-valkey-service, so the
bundled Valkey is the broker the release actually uses. #65 covered this
path by a render only - deliberately, since an unused Valkey would have made
that test look like it covered clustering. This is the first real
installation of the valkey-subchart path.

Documented, not glossed: the password now sits in that item twice and a
rotation has to touch both fields; with the subchart on the vault url is a
mandatory step, not a hint; and a OnePasswordItem materializes every field
of its item, so both Secrets carry all three keys - untidy, harmless, and
written down so nobody "tidies it up".

Fixes #78

jlab-bot commented 2026-10-02 23:37:04 +02:00

Author

Member

Copy link

Copy source

Pipeline: alles grün, und der Install-Test hat den Subchart-Pfad wirklich installiert

Lauf 427, alle sechs Kontexte success:

Check	Ergebnis
Detect changed charts	success
Build paperless-ngx (lint --strict, package, Versions-Bump)	success
Resource quantities (alle Charts)	success
Trivy config paperless-ngx	success
Install-test paperless-ngx	success
Validate Helm Charts / Install-test	success

Was der Install-Test belegt

secret/ci-valkey-secret created
Downloading valkey from repo https://git.jk-dev.de/api/packages/jlab-cluster/helm
...
Waiting for statefulset.apps/ci-paperless-ngx-sfs...
partitioned roll out complete: 1 new pods have been updated...
Waiting for statefulset.apps/ci-valkey-sfs...
partitioned roll out complete: 1 new pods have been updated...
--- final state ---
NAME                        READY   STATUS      RESTARTS   AGE
ci-paperless-ngx-sfs-0      1/1     Running     0          2m11s
ci-valkey-sfs-0             1/1     Running     0          2m11s
node-setup-nncb4            0/1     Completed   0          2m48s
postgres-7457d4fb7b-lm9fx   1/1     Running     0          2m48s
OK: chart 'paperless-ngx' installed and all workloads are Ready.

Das ist die erste echte Installation des Valkey-Subchart-Pfades in diesem Repository — #77 hatte ihn nur gerendert. Belegt ist damit:

• Die Abhängigkeit löst sich auf einem echten Runner ohne Zugangsdaten aus unserem Index auf.
• Die Objekte des Subcharts werden innerhalb eines Elternteil-Releases akzeptiert, und die Namen sind tatsächlich <release>-valkey-<kind> (ci-valkey-sfs, ci-valkey-service, ci-valkey-secret).
• ci-valkey-sfs-0 wird 1/1 Running — das heißt der Init-Container hat eine requirepass -Zeile in der gemounteten Datei gefunden (sonst verweigert er den Start), und die Readiness-Probe hat sich authentifiziert und PONG bekommen (sonst würde der Pod nie Ready).
• Der PVC-Pfad des Subcharts funktioniert unter einem fremden Release-Namen.

Was er nicht belegt, und ich behaupte es auch nicht

• Dass paperless den mitgelieferten Valkey tatsächlich benutzt hat. Alle drei Probes von paperless fragen nur / auf Port 8000, und die Diagnose-Ausgabe läuft nur im Fehlerfall — es liegt also kein Log vor, das eine erfolgreiche Celery-Verbindung zeigt. Was der Lauf zeigt, ist dass nichts an der Verdrahtung den Start verhindert hat. Ein verarbeitetes Dokument wäre der ehrliche Nachweis, und der war schon vorher außerhalb des Umfangs.
• Die fehlschlagende Richtung der Schema-Regel (valkeySubchart.enabled an, authSecretRef leer) ist nur lokal gerendert, nicht in CI — die Maschinerie fährt eine Konfiguration pro Chart.

## Pipeline: alles grün, und der Install-Test hat den Subchart-Pfad wirklich installiert Lauf 427, alle sechs Kontexte `success`: | Check | Ergebnis | |---|---| | Detect changed charts | success | | Build paperless-ngx (`lint --strict`, package, Versions-Bump) | success | | Resource quantities (alle Charts) | success | | Trivy config paperless-ngx | success | | Install-test paperless-ngx | **success** | | Validate Helm Charts / Install-test | success | ### Was der Install-Test belegt ``` secret/ci-valkey-secret created Downloading valkey from repo https://git.jk-dev.de/api/packages/jlab-cluster/helm ... Waiting for statefulset.apps/ci-paperless-ngx-sfs... partitioned roll out complete: 1 new pods have been updated... Waiting for statefulset.apps/ci-valkey-sfs... partitioned roll out complete: 1 new pods have been updated... --- final state --- NAME READY STATUS RESTARTS AGE ci-paperless-ngx-sfs-0 1/1 Running 0 2m11s ci-valkey-sfs-0 1/1 Running 0 2m11s node-setup-nncb4 0/1 Completed 0 2m48s postgres-7457d4fb7b-lm9fx 1/1 Running 0 2m48s OK: chart 'paperless-ngx' installed and all workloads are Ready. ``` Das ist die **erste echte Installation des Valkey-Subchart-Pfades** in diesem Repository — #77 hatte ihn nur gerendert. Belegt ist damit: - Die Abhängigkeit löst sich **auf einem echten Runner ohne Zugangsdaten** aus unserem Index auf. - Die Objekte des Subcharts werden innerhalb eines Elternteil-Releases akzeptiert, und die Namen sind tatsächlich `<release>-valkey-<kind>` (`ci-valkey-sfs`, `ci-valkey-service`, `ci-valkey-secret`). - `ci-valkey-sfs-0` wird `1/1 Running` — das heißt der Init-Container **hat** eine `requirepass `-Zeile in der gemounteten Datei gefunden (sonst verweigert er den Start), und die Readiness-Probe hat sich **authentifiziert** und `PONG` bekommen (sonst würde der Pod nie Ready). - Der PVC-Pfad des Subcharts funktioniert unter einem fremden Release-Namen. ### Was er nicht belegt, und ich behaupte es auch nicht - **Dass paperless den mitgelieferten Valkey tatsächlich benutzt hat.** Alle drei Probes von paperless fragen nur `/` auf Port 8000, und die Diagnose-Ausgabe läuft nur im Fehlerfall — es liegt also kein Log vor, das eine erfolgreiche Celery-Verbindung zeigt. Was der Lauf zeigt, ist dass nichts an der Verdrahtung den Start verhindert hat. Ein verarbeitetes Dokument wäre der ehrliche Nachweis, und der war schon vorher außerhalb des Umfangs. - **Die fehlschlagende Richtung der Schema-Regel** (`valkeySubchart.enabled` an, `authSecretRef` leer) ist nur lokal gerendert, nicht in CI — die Maschinerie fährt eine Konfiguration pro Chart.

jklein approved these changes 2026-10-03 00:06:43 +02:00

jklein merged commit a9ca5302de into main 2026-10-03 00:06:59 +02:00

jklein deleted branch feat/paperless-valkey-subchart 2026-10-03 00:07:00 +02:00

jlab-bot referenced this pull request 2026-10-04 21:26:33 +02:00

Neuer Chart: stalwart (Mailserver) - erster Plan und offene Entscheidungen #65

Sign in to join this conversation.

Reviewers

No Reviewers

jklein

Labels

Clear labels

inhouse-chart

Inhouse-Chart aus der JLab-Git-Paketregistry

renovate

Von Renovate angelegt

No labels

Milestone

No items

No Milestone

Assignees

Clear assignees

jklein (Justin Klein) jlab-bot (jlab-git-bot) jlab-renovate-bot (jlab-renovate-bot)

No Assignees

2 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: jlab-cluster/helm-charts#80