feat(paperless-ngx): optional valkey subchart, password from the redis item
Validate Helm Charts / Detect changed charts (pull_request) Successful in 4s
Validate Helm Charts / Resource quantities (pull_request) Successful in 12s
Validate Helm Charts / Build paperless-ngx (pull_request) Successful in 6s
Validate Helm Charts / Trivy config paperless-ngx (pull_request) Successful in 22s
Install-test / Install-test paperless-ngx (pull_request) Successful in 3m31s
Validate Helm Charts / Install-test (pull_request) Successful in 3m31s
Publish Helm Chart with patched Application Version / publish-helm-chart (push) Successful in 10s
Publish Paperless-NGX Helm Chart / Publish Paperless-NGX Helm Chart (push) Successful in 10s
SonarQube Analysis / SonarQube Analysis (push) Successful in 33s
Validate Helm Charts / Detect changed charts (pull_request) Successful in 4s
Validate Helm Charts / Resource quantities (pull_request) Successful in 12s
Validate Helm Charts / Build paperless-ngx (pull_request) Successful in 6s
Validate Helm Charts / Trivy config paperless-ngx (pull_request) Successful in 22s
Install-test / Install-test paperless-ngx (pull_request) Successful in 3m31s
Validate Helm Charts / Install-test (pull_request) Successful in 3m31s
Publish Helm Chart with patched Application Version / publish-helm-chart (push) Successful in 10s
Publish Paperless-NGX Helm Chart / Publish Paperless-NGX Helm Chart (push) Successful in 10s
SonarQube Analysis / SonarQube Analysis (push) Successful in 33s
paperless-ngx cannot run without a Redis - PAPERLESS_REDIS is the Celery broker - so every installation needed a second Helm release whose address lived in a 1Password item and in neither chart. This adds our own published valkey chart as an optional dependency, off by default and gated on valkeySubchart.enabled (not valkey.enabled, which the subchart's strict schema rejects and which Helm would only have complained about for whoever first switched it on). Additive, default behaviour unchanged: minor, 12.1.3+up3.2.1 -> 12.2.0+up3.2.1. Configured as a QUEUE (PVC + noeviction, the subchart's defaults), unlike stalwart's cache. Read off the 3.2.1 source rather than assumed: one PAPERLESS_REDIS url serves five consumers - the Celery broker, the result backend (Redis in 3.x, the database before), the channels pub/sub layer and two Django caches. allkeys-lru would not be a trade here but actively wrong, because LRU evicts typeless across the whole keyspace and a backlog of waiting consume tasks is exactly what it reads as cold; a dropped task is a document silently never processed. The price, named in values.yaml rather than smoothed over: two of the five consumers are caches, so a full instance refuses writes - new tasks included - and the answer is memory sizing, not a policy change. All three cache-like consumers are TTL-bounded, which is what keeps that price small. The password comes from the SAME item as the Redis connection, in a third field auth-conf holding the complete line "requirepass <password>". It cannot DEFAULT to secrets.redisSecretRef, and that is a property of Helm, not an omission: a values.yaml is never templated and a parent cannot compute a subchart's value. Measured - a template expression there lands verbatim in the subchart's itemPath and fails as a YAML parse error naming neither the value nor the cause. Forgetting it is therefore caught in values.schema.json, by a conditional rule rather than a fail() in a template. Also measured, and the reason: Helm renders a subchart's templates BEFORE the parent's, so a guard in this chart loses the race to the subchart's own required(), which reports "secrets.authSecretRef" and cannot know the path belongs to the redis item. A check that cannot fire is the mistake found while testing #77, so the first draft of that guard was removed rather than shipped. The schema runs before any rendering and names /valkey/secrets/authSecretRef. One trap found while testing and closed here: "valkeySubchart: null" DELETES the key, Helm then cannot resolve the condition path and treats the dependency as ENABLED - so erasing the switch turned the subchart on with no password set. Unlike every other block in this chart (issue #99), that one is rejected by the schema instead of defaulting. The install-test now runs with the subchart ON and the throwaway redis Deployment removed: the url in the fixtures names ci-valkey-service, so the bundled Valkey is the broker the release actually uses. #65 covered this path by a render only - deliberately, since an unused Valkey would have made that test look like it covered clustering. This is the first real installation of the valkey-subchart path. Documented, not glossed: the password now sits in that item twice and a rotation has to touch both fields; with the subchart on the vault url is a mandatory step, not a hint; and a OnePasswordItem materializes every field of its item, so both Secrets carry all three keys - untidy, harmless, and written down so nobody "tidies it up". Fixes #78
This commit was merged in pull request #80.
This commit is contained in:
10 files changed
+521
-57
No files matched your search
@@ -25,7 +25,7 @@ Every chart has a wiki page of its own — `https://git.jk-dev.de/jlab-cluster/h
|
||||
| `minecraft-router` | itzg/mc-router | Deployment + routing ConfigMap + NodePort Service (TCP only, no ingress); `mappings` is a required value |
|
||||
| `minecraft-server` | itzg/minecraft-server | StatefulSet + PVC + Service + init-script ConfigMap (no ingress); service type is a value (`ClusterIP` by default, routed via `minecraft-router`). **Only the newest Java line is maintained** (chart 5.0.0, #158): the directory carries one line, older ones stay published as frozen artifacts (java8/java17 at `4.0.0`, java21 at `4.1.0`) and receive no further chart changes — publishing once per line out of one directory is what made the lines drift. To run an older JVM with the current chart, set `image.tagOverride`; the chart then checks that `minecraft.version` fits the Java line of the **effective** tag and fails the render with both values named. Two cases pass unchecked on purpose: a tag without a `java<N>` part and a `minecraft.version` that is not a plain numeric `x[.y[.z]]`. `image.tagOverride` is deliberately outside Renovate's reach — `renovate.json5` tracks `appVersion` only. Since chart 2.0.0 the tag is an **immutable dated upstream release** (`appVersion: <release>-java8` / `-java17` / `-java21` / `-java25`, e.g. `2026.8.2-java8`) instead of the continuously rebuilt rolling line tag. Pilot for the strict `values.schema.json` (#68), which is now binding for every chart |
|
||||
| `outline` | Outline | StatefulSet, separate DB/Redis secrets |
|
||||
| `paperless-ngx` | Paperless-ngx | Largest chart: consume CronJob, SMB-connector PV/PVC, pre/post-consumption scripts ConfigMap |
|
||||
| `paperless-ngx` | Paperless-ngx | Largest chart: consume CronJob, SMB-connector PV/PVC, pre/post-consumption scripts ConfigMap. Optional `valkey` subchart — **this repo's own chart, from the published index**, off by default and gated on `valkeySubchart.enabled` (not `valkey.enabled`, which the subchart's strict schema rejects). Configured as a **queue** (PVC + `noeviction`), unlike `stalwart`'s cache: `PAPERLESS_REDIS` is the Celery broker, and `allkeys-lru` evicts across the whole keyspace, where a backlog of waiting consume tasks is exactly what LRU reads as cold. The password comes from the **same 1Password item** as the Redis connection, in a third field `auth-conf` holding the complete line `requirepass <password>` — so the password sits in that item **twice** and a rotation has to touch both fields. Two more traps: with the subchart on, `redis-connection-url` in the vault **must** be repointed at `<release>-valkey-service:6379` or the bundled Valkey runs unused, and `valkeySubchart:` must not be erased — a null override deletes the key, Helm reads an unresolvable `condition:` as *enabled*, so erasing the switch turns the subchart **on** (rejected by `values.schema.json`). Never change `valkey/` and this chart in the same PR, see "Chart dependencies" |
|
||||
| `patchmon` | PatchMon | One Deployment (the 2.x server serves frontend and `/api/*` from one process) plus an optional guacd sidecar; OIDC-only sign-in |
|
||||
| `rallly` | Rallly | Deployment, DB/OIDC/SMTP secrets |
|
||||
| `rss-bridge` | RSS-Bridge | Deployment + whitelist ConfigMap; `ingress.hosts` is a required list of host + optional `tlsSecretName`, so several hosts can keep their own existing certificates (documented deviation from the one-TLS-secret rule, see its `values.yaml`) |
|
||||
@@ -39,7 +39,7 @@ Every chart has a wiki page of its own — `https://git.jk-dev.de/jlab-cluster/h
|
||||
| `urlaubsverwaltung` | Urlaubsverwaltung | Deployment, DB + OIDC secrets |
|
||||
| `valkey` | Valkey | One instance per application (cache or queue backend), replacing the shared `cluster-redis` whose single ACL file coupled six applications together. StatefulSet with one replica + ClusterIP Service, no cluster mode, no Ingress and no value to add one. The password arrives as a **finished configuration line** (`requirepass <secret>`) in the 1Password field `auth-conf`, is mounted as a file and `include`d by the config — never in argv, never in the environment; an init container refuses to start the pod unless that line is really there, because the alternative failure is an open server that looks healthy. `maxmemory` is **derived from the computed memory limit** and a configured value above it aborts the render. Persistence and eviction are a pair, not two switches (queue = PVC + `noeviction`, the default; cache = emptyDir + `allkeys-lru`). First chart carrying a **NetworkPolicy** (on by default, Gitea #5), and the first whose probes had to be proven capable of failing: `valkey-cli` exits 0 on an error reply unless `-e` is given |
|
||||
| `voucher-vault` | VoucherVault | StatefulSet, DB + OIDC secrets |
|
||||
| `ytdlp-web-player` | ytdlp_web_player | Stateless Deployment, video cache and tool cache as emptyDirs (no PVC). **The first chart with a dependency** (`stalwart` is the second): an optional `oauth2-proxy` subchart, off by default, wired in front of the app because the application ships no authentication at all — see "Chart dependencies" below |
|
||||
| `ytdlp-web-player` | ytdlp_web_player | Stateless Deployment, video cache and tool cache as emptyDirs (no PVC). **The first chart with a dependency** (`stalwart` and `paperless-ngx` followed): an optional `oauth2-proxy` subchart, off by default, wired in front of the app because the application ships no authentication at all — see "Chart dependencies" below |
|
||||
| `zipline` | Zipline | StatefulSet, DB secret |
|
||||
|
||||
## Versioning
|
||||
@@ -65,7 +65,7 @@ Every chart has a wiki page of its own — `https://git.jk-dev.de/jlab-cluster/h
|
||||
|
||||
Note: the registry **accepts a re-upload of an already published version** (HTTP 201) and replaces the stored artifact, so a republish of an existing version is still silent — see #160. That is also what makes a manual `workflow_dispatch` of any `publish-<chart>.yml` a harmless way to prove the shared block works.
|
||||
- `change-app-version.yml` — manually dispatched app-version bump; the chart must be listed in its `chart-name` choice list.
|
||||
- `ci/helm-dependencies.sh <chart-dir>` — not a workflow but the step they all share: fetches a chart's declared dependencies into `charts/`, and returns silently for the charts that declare none (all but one). It runs in **five** places, because every helm command this repo builds on exits 1 while a declared dependency is missing: before lint and package in `validate-charts.yml`, before package in the shared publish block (which is why that block names this script as a prerequisite of its callers — see above), and inside `ci/run-install-test.sh`, `ci/run-quantity-render-test.sh` and `ci/run-trivy-config.sh`. See "Chart dependencies" in the style guidelines for what that costs.
|
||||
- `ci/helm-dependencies.sh <chart-dir>` — not a workflow but the step they all share: fetches a chart's declared dependencies into `charts/`, and returns silently for the charts that declare none (all but three: `ytdlp-web-player`, `stalwart`, `paperless-ngx`). It runs in **five** places, because every helm command this repo builds on exits 1 while a declared dependency is missing: before lint and package in `validate-charts.yml`, before package in the shared publish block (which is why that block names this script as a prerequisite of its callers — see above), and inside `ci/run-install-test.sh`, `ci/run-quantity-render-test.sh` and `ci/run-trivy-config.sh`. See "Chart dependencies" in the style guidelines for what that costs.
|
||||
- `validate-charts.yml` — PR pipeline: detects which charts a PR touches and, for exactly those, runs `helm lint --strict`, `helm package`, and a chart-version bump check (`template-chart` is exempt from the version check). PRs without chart changes skip the build matrix. After the build stage, an **install-test** job (the same changed-charts matrix, minus the charts listed in `ci/excluded-charts.txt`) installs each changed chart into an ephemeral [k3d](https://k3d.io) cluster on the runner and requires every workload to become Ready — see below. Both matrices follow the changed chart directories, so a PR that only changes the harness under `ci/` exercises nothing by itself; the Trivy scan has a fixed-sample fallback for that case, the install-test has none — bump the affected chart's patch version to pull it into the matrix, or run the all-charts install-test after the merge. The **resource-quantity render test** is the exception to that rule: it runs over every chart on every PR — see below.
|
||||
- `install-test.yml` — reusable workflow (`workflow_call`) holding the install-test **job** itself: the runner-label expression, the pinned `kubectl` with its skew check, `envsubst`, the containerised-job detection, the per-leg k3d cluster and the `freesailarr` VPN credentials. Its one input is a JSON array of chart names; its callers decide *which* charts are tested and whether the job runs at all, never *how*. Two callers: `validate-charts.yml` and `install-test-all-charts.yml`. **It is not copied for the second caller** — a second copy is a second chance to test differently, unnoticed, the same argument that gives `ci/excluded-charts.txt` exactly one parser.
|
||||
|
||||
@@ -440,7 +440,9 @@ Reference implementation: `template-chart/values.schema.json` and `template-char
|
||||
|
||||
Until `ytdlp-web-player` (#214) this section read "charts have no dependencies (`charts/` directories are empty)". That rule is gone, replaced by a bar rather than a ban — a dependency is **admissible, not free**, and the reason it stopped being a ban is worth keeping: the chart needed an oauth2-proxy in the data path in front of an application that ships no authentication at all, and proxy and app are only useful wired to each other.
|
||||
|
||||
Two charts carry one, and comparing them is the quickest way to see where the bar sits. `ytdlp-web-player` takes **upstream's** `oauth2-proxy` because the app has no authentication and the two are useless apart; there, point 2 above applies in full. `stalwart` (Gitea #65) takes **one of our own published charts**, `valkey`, because Stalwart's **cluster coordinator rides on the in-memory store connection**, so a multi-node install does not work without one. Both are `condition:`-gated and off by default — for `stalwart` the subchart is the path for an installation that has no Valkey yet, and one that already has points `externalServices.valkey.host` at it so the dependency renders nothing.
|
||||
Three charts carry one, and comparing them is the quickest way to see where the bar sits. `ytdlp-web-player` takes **upstream's** `oauth2-proxy` because the app has no authentication and the two are useless apart; there, point 2 above applies in full. `stalwart` (Gitea #65) takes **one of our own published charts**, `valkey`, because Stalwart's **cluster coordinator rides on the in-memory store connection**, so a multi-node install does not work without one. `paperless-ngx` (Gitea #78) takes the same one, because `PAPERLESS_REDIS` is the Celery broker and the application therefore cannot run without a Redis at all — before the subchart, every installation needed a second Helm release whose address lived in a 1Password item and in neither chart. All three are `condition:`-gated and off by default: the subchart is the path for an installation that has no Valkey yet, and one that already has points its connection value at that instead, so the dependency renders nothing.
|
||||
|
||||
**The same subchart, opposite configuration — and that comparison is the useful part.** `stalwart` runs it as a **cache** (`persistence.enabled: false` + `allkeys-lru`): its in-memory store holds rate-limiter counters, tokens and pub/sub, all reconstructible, and the cost it accepts is that the limiters fail open under memory pressure. `paperless-ngx` runs it as a **queue** (PVC + `noeviction`, the subchart's defaults): the broker's contents are work, and a dropped consume task is a document that is silently never processed. `allkeys-lru` would not be a trade there but actively wrong, because LRU evicts typeless across the whole keyspace and a backlog of waiting tasks is exactly what it reads as cold. The price `paperless-ngx` accepts instead: the same instance also serves the Django cache and the Celery result backend (both Redis in 3.x), so a full instance **refuses writes** — new tasks included — rather than dropping cold keys, and the answer is memory sizing, not a policy change. The valkey chart treats the two values as a **pair** and warns against mixing them; these two charts are the worked examples of picking either half deliberately.
|
||||
|
||||
**A dependency on one of our own charts is cheaper in one way and more dangerous in another.** Point 2 stops applying: there are no third-party defaults to leave unpatched, because the subchart follows these guidelines by being one of ours. In exchange it adds a hazard an external dependency does not have, and it is sharp enough to be a rule:
|
||||
|
||||
|
||||
Reference in new issue
Block a user