feat(paperless-ngx): optional valkey subchart, password from the redis item
Validate Helm Charts / Detect changed charts (pull_request) Successful in 4s
Validate Helm Charts / Resource quantities (pull_request) Successful in 12s
Validate Helm Charts / Build paperless-ngx (pull_request) Successful in 6s
Validate Helm Charts / Trivy config paperless-ngx (pull_request) Successful in 22s
Install-test / Install-test paperless-ngx (pull_request) Successful in 3m31s
Validate Helm Charts / Install-test (pull_request) Successful in 3m31s
Publish Helm Chart with patched Application Version / publish-helm-chart (push) Successful in 10s
Publish Paperless-NGX Helm Chart / Publish Paperless-NGX Helm Chart (push) Successful in 10s
SonarQube Analysis / SonarQube Analysis (push) Successful in 33s
Validate Helm Charts / Detect changed charts (pull_request) Successful in 4s
Validate Helm Charts / Resource quantities (pull_request) Successful in 12s
Validate Helm Charts / Build paperless-ngx (pull_request) Successful in 6s
Validate Helm Charts / Trivy config paperless-ngx (pull_request) Successful in 22s
Install-test / Install-test paperless-ngx (pull_request) Successful in 3m31s
Validate Helm Charts / Install-test (pull_request) Successful in 3m31s
Publish Helm Chart with patched Application Version / publish-helm-chart (push) Successful in 10s
Publish Paperless-NGX Helm Chart / Publish Paperless-NGX Helm Chart (push) Successful in 10s
SonarQube Analysis / SonarQube Analysis (push) Successful in 33s
paperless-ngx cannot run without a Redis - PAPERLESS_REDIS is the Celery broker - so every installation needed a second Helm release whose address lived in a 1Password item and in neither chart. This adds our own published valkey chart as an optional dependency, off by default and gated on valkeySubchart.enabled (not valkey.enabled, which the subchart's strict schema rejects and which Helm would only have complained about for whoever first switched it on). Additive, default behaviour unchanged: minor, 12.1.3+up3.2.1 -> 12.2.0+up3.2.1. Configured as a QUEUE (PVC + noeviction, the subchart's defaults), unlike stalwart's cache. Read off the 3.2.1 source rather than assumed: one PAPERLESS_REDIS url serves five consumers - the Celery broker, the result backend (Redis in 3.x, the database before), the channels pub/sub layer and two Django caches. allkeys-lru would not be a trade here but actively wrong, because LRU evicts typeless across the whole keyspace and a backlog of waiting consume tasks is exactly what it reads as cold; a dropped task is a document silently never processed. The price, named in values.yaml rather than smoothed over: two of the five consumers are caches, so a full instance refuses writes - new tasks included - and the answer is memory sizing, not a policy change. All three cache-like consumers are TTL-bounded, which is what keeps that price small. The password comes from the SAME item as the Redis connection, in a third field auth-conf holding the complete line "requirepass <password>". It cannot DEFAULT to secrets.redisSecretRef, and that is a property of Helm, not an omission: a values.yaml is never templated and a parent cannot compute a subchart's value. Measured - a template expression there lands verbatim in the subchart's itemPath and fails as a YAML parse error naming neither the value nor the cause. Forgetting it is therefore caught in values.schema.json, by a conditional rule rather than a fail() in a template. Also measured, and the reason: Helm renders a subchart's templates BEFORE the parent's, so a guard in this chart loses the race to the subchart's own required(), which reports "secrets.authSecretRef" and cannot know the path belongs to the redis item. A check that cannot fire is the mistake found while testing #77, so the first draft of that guard was removed rather than shipped. The schema runs before any rendering and names /valkey/secrets/authSecretRef. One trap found while testing and closed here: "valkeySubchart: null" DELETES the key, Helm then cannot resolve the condition path and treats the dependency as ENABLED - so erasing the switch turned the subchart on with no password set. Unlike every other block in this chart (issue #99), that one is rejected by the schema instead of defaulting. The install-test now runs with the subchart ON and the throwaway redis Deployment removed: the url in the fixtures names ci-valkey-service, so the bundled Valkey is the broker the release actually uses. #65 covered this path by a render only - deliberately, since an unused Valkey would have made that test look like it covered clustering. This is the first real installation of the valkey-subchart path. Documented, not glossed: the password now sits in that item twice and a rotation has to touch both fields; with the subchart on the vault url is a mandatory step, not a hint; and a OnePasswordItem materializes every field of its item, so both Secrets carry all three keys - untidy, harmless, and written down so nobody "tidies it up". Fixes #78
This commit was merged in pull request #80.
This commit is contained in:
10 files changed
+521
-57
No files matched your search
@@ -3,6 +3,9 @@
|
||||
{{- $barcodes := fromYaml (include "paperless-ngx.barcodes" .Values.paperless.files.barcodes) -}}
|
||||
{{- $pod := printf "%s-%s-sfs-0" .Release.Name .Chart.Name -}}
|
||||
{{- $scripts := .Values.paperless.scripts | default dict -}}
|
||||
{{- $valkeySub := (.Values.valkeySubchart | default dict).enabled -}}
|
||||
{{- $valkeyAuth := ((.Values.valkey | default dict).secrets | default dict).authSecretRef -}}
|
||||
{{- $valkeySvc := printf "%s-valkey-service" .Release.Name -}}
|
||||
{{- $pre := default (list) $scripts.preConsumptionScripts -}}
|
||||
{{- $post := default (list) $scripts.postConsumptionScripts -}}
|
||||
Paperless-ngx {{ .Chart.AppVersion }} — release {{ .Release.Name }} in namespace {{ .Release.Namespace }}.
|
||||
@@ -70,6 +73,12 @@ WHAT THE 1PASSWORD ITEMS MUST CONTAIN
|
||||
key-prefix PAPERLESS_REDIS_PREFIX — set this per release when
|
||||
several releases share one Redis, or two instances
|
||||
will consume each other's task queue
|
||||
auth-conf the COMPLETE line "requirepass PASSWORD", not a
|
||||
bare password. Read by the bundled Valkey only, so
|
||||
it is {{ if $valkeySub }}REQUIRED for this release — see the Valkey
|
||||
section below{{ else }}unused while valkeySubchart.enabled is
|
||||
false, and the item needs it only once you switch
|
||||
the bundled Valkey on{{ end }}.
|
||||
|
||||
secrets.paperlessSecretRef → {{ .Values.secrets.paperlessSecretRef }}
|
||||
Secret {{ .Release.Name }}-{{ .Chart.Name }}-secret, fields:
|
||||
@@ -117,9 +126,9 @@ CHANGING A SECRET'S CONTENT RESTARTS NOTHING
|
||||
fields points the release at whatever that new endpoint contains, empty or
|
||||
not, and the next start migrates it.
|
||||
|
||||
POSTGRES AND REDIS ARE NOT PART OF THIS CHART
|
||||
Both have to exist and be reachable before the first start. They fail in
|
||||
opposite ways, and only one of them is loud:
|
||||
POSTGRES IS NOT PART OF THIS CHART{{ if not $valkeySub }}, AND NEITHER IS REDIS{{ end }}
|
||||
{{ if $valkeySub }}The database{{ else }}Both{{ end }} ha{{ if $valkeySub }}s{{ else }}ve{{ end }} to exist and be reachable before the first start.{{ if not $valkeySub }} They fail in
|
||||
opposite ways, and only one of them is loud:{{ end }}
|
||||
|
||||
- Without the database paperless does not come up at all. All three probes
|
||||
ask for "/" on port 8000, so you get no useful error: the pod never turns
|
||||
@@ -129,7 +138,72 @@ POSTGRES AND REDIS ARE NOT PART OF THIS CHART
|
||||
broker: consumption, OCR, the scheduled tasks and the index updates simply
|
||||
never run, and documents dropped into the consume directory stay there.
|
||||
A wrong redis-connection-url costs you the whole processing side and
|
||||
nothing turns red.
|
||||
nothing turns red.{{ if $valkeySub }} This release brings its own Valkey (below), so
|
||||
the address is in your hands — which is exactly why the url is the one
|
||||
thing to check first.{{ end }}
|
||||
{{ if $valkeySub }}
|
||||
THIS RELEASE BRINGS ITS OWN VALKEY — ONE MANDATORY STEP IS IN THE VAULT
|
||||
valkeySubchart.enabled is true, so the release contains a Valkey of its own:
|
||||
StatefulSet {{ .Release.Name }}-valkey-sfs, Service {{ $valkeySvc }}, its own
|
||||
NetworkPolicy, and a PersistentVolumeClaim (persistence is on — this is a
|
||||
queue, see below).
|
||||
|
||||
1. THE CONNECTION URL MUST POINT AT IT. This is a step, not a hint: the url
|
||||
lives in the vault and the chart cannot write it. In the item
|
||||
"{{ .Values.secrets.redisSecretRef }}", field redis-connection-url:
|
||||
|
||||
redis://:PASSWORD@{{ $valkeySvc }}:6379
|
||||
|
||||
Pointing elsewhere is not an error anywhere: paperless talks to that other
|
||||
Redis, the Valkey next to it runs along unused, and nothing reports it.
|
||||
|
||||
2. THE PASSWORD IS IN THE ITEM TWICE — once inside redis-connection-url and
|
||||
once in the "requirepass PASSWORD" line of auth-conf. Both are the same
|
||||
password, and a rotation has to touch BOTH fields. Change only one and
|
||||
paperless can no longer authenticate against its own Valkey. After any
|
||||
such change, restart both workloads by hand (Helm cannot see inside a
|
||||
1Password item, so nothing rolls automatically):
|
||||
|
||||
kubectl -n {{ .Release.Namespace }} rollout restart statefulset/{{ .Release.Name }}-{{ .Chart.Name }}-sfs
|
||||
kubectl -n {{ .Release.Namespace }} rollout restart statefulset/{{ .Release.Name }}-valkey-sfs
|
||||
|
||||
3. AUTH-CONF IS A CONFIGURATION LINE, NOT A PASSWORD. The Valkey mounts the
|
||||
field as a file and includes it, so the password never reaches a process
|
||||
list or an environment variable. An init container refuses to start the
|
||||
pod unless a line beginning with "requirepass " is really there — without
|
||||
that check a typo would leave a Valkey running with NO password and
|
||||
looking perfectly healthy.
|
||||
{{- if and $valkeyAuth (ne ($valkeyAuth | toString) ($.Values.secrets.redisSecretRef | toString)) }}
|
||||
|
||||
NOTE: valkey.secrets.authSecretRef points at a DIFFERENT item than
|
||||
secrets.redisSecretRef:
|
||||
valkey.secrets.authSecretRef → {{ $valkeyAuth }}
|
||||
secrets.redisSecretRef → {{ .Values.secrets.redisSecretRef }}
|
||||
That is supported and may well be deliberate — the bundled Valkey is then fed
|
||||
from its own item. It is only worth knowing that you now keep TWO items in
|
||||
step: the password in auth-conf over there and the one inside
|
||||
redis-connection-url here have to stay identical.
|
||||
{{- end }}
|
||||
|
||||
BOTH ITEMS' FIELDS LAND IN BOTH SECRETS. A OnePasswordItem materializes every
|
||||
field of the item it names, so with one shared item the Valkey's Secret also
|
||||
carries redis-connection-url and key-prefix, and this chart's redis Secret
|
||||
also carries auth-conf. Untidy and harmless — nothing mounts a field it does
|
||||
not reference. It is mentioned so it does not get "cleaned up".
|
||||
|
||||
IT IS CONFIGURED AS A QUEUE, and that is a decision: persistence on, eviction
|
||||
policy "noeviction". PAPERLESS_REDIS is the Celery broker, so its contents are
|
||||
work that must not vanish — "allkeys-lru" would evict across the whole
|
||||
keyspace and a backlog of waiting tasks is exactly what LRU reads as cold.
|
||||
The price: the same instance also serves the Django cache, so when it fills,
|
||||
writes are REFUSED rather than old keys dropped — new tasks included. The
|
||||
answer is to give it more memory (valkey.valkey.resources.requests.memory),
|
||||
never to switch the policy. values.yaml argues this in full.
|
||||
|
||||
SCALEDOWN DOES NOT REACH IT. scaleDown: true scales paperless to zero and
|
||||
leaves the Valkey running, which is the right way round for a queue: the
|
||||
backlog stays where it is.
|
||||
{{- end }}
|
||||
|
||||
THE FIRST START MIGRATES, AND THE STARTUP PROBE IS THE CLOCK
|
||||
Every start runs the database migrations before the server answers, and the
|
||||
|
||||
Reference in new issue
Block a user