sure: new chart for the we-promise Sure app (1.0.0+up0.7.4) #36

Merged

jklein merged 3 commits from feat/sure-chart into main 2026-09-26 23:08:56 +02:00

Conversation 1 Commits 3 Files Changed 21

+1692 -3

jlab-bot commented 2026-09-22 21:54:48 +02:00

Member

Copy link

Copy source

Fixes #33

New chart sure for Sure, the maintained Maybe Finance fork, at 1.0.0+up0.7.4. All four decisions from the issue are implemented as approved: one StatefulSet with two containers, appVersion: 0.7.4, AI options only through the additive env block, Postgres and Redis external.

What is in it

• One StatefulSet, two containers (-web Puma, -worker Sidekiq) sharing one RWO PVC at /rails/storage, because both processes use the same Active Storage files and an RWO volume cannot be attached to two pods.
• Only the web container migrates. The image entrypoint runs ./bin/rails db:prepare solely when the command is ./bin/rails server. The worker is therefore started through args:, not command: — command: would replace the ENTRYPOINT and with it the jemalloc preload upstream names as one of the three reasons its memory figures hold.
• One small init container, the app's own image running as the same non-root uid with no added capability, which prepares the two private directories TMPDIR and HOME point at. Why that is not decoration is the first of the three findings below.
• Three OnePasswordItem CRs (app / database / redis), all wired via secretKeyRef. No operator.1password.io/auto-restart (silent no-op on a StatefulSet); instead three checksum/secret-ref annotations plus the manual kubectl rollout restart in NOTES and wiki.
• Probes chosen for what they prove. Liveness/startup use /up, which is Rails' boot check and touches no database. Readiness asks for /: without a session cookie the request runs authenticate_user! → self_hosted_first_login? → User.count, i.e. an SQL query. The worker has no HTTP at all; all three of its probes run bundle exec sidekiqmon processes | grep -q "^${HOSTNAME}:", which proves a live Sidekiq heartbeat in Redis — a hung worker drops out of the process set within ~60s while its PID still exists, and a plain process check could never fail at all because Sidekiq is PID 1 there.
• ingress.domain stays required with ingress.enabled: false (§12.1): besides the Ingress host it feeds APP_DOMAIN, WEBAUTHN_RP_ID and WEBAUTHN_ALLOWED_ORIGINS. ingress.enabled additionally decides RAILS_ASSUME_SSL/RAILS_FORCE_SSL.
• Strict values.schema.json, null-safe fallbacks reproducing this chart's own defaults, replicas + scaleDown, additive env per container through tpl, resources with limitFactor.
• Wiring: publish-sure.yml, sure in the change-app-version.yml choice list, ci/sure/ fixtures (throwaway Postgres 16 + Redis, dummy secrets, INSTALL_TIMEOUT=900), a Renovate manager on versioning: semver, the README chart row, and one added name in the repo-wide AVD-KSV-0020 statement.

Resource requests come from upstream's own sizing note for 0.7.4 (docs/hosting/docker.md): ~352 MB steady state, "at least 1 GB" with AI enabled, and the sample-data generator measured at ~680 MB in the worker. 512Mi request × limitFactor: 3 = 1536Mi limit covers all three without reserving the large number.

What was red, and why it is green now

The install-test failed twice before it passed, and neither cause was visible in any render. Both are in the chart with the measured evidence next to them, and all three lessons are on the wiki's Lessons learned page, because none of them is specific to this chart.

Run 1 — both containers crash-looped during boot (e92286b):

system temporary path is world-writable: /tmp
/tmp is world-writable: /tmp
. is not writable: /rails
ArgumentError: could not find a temporary directory

Ruby's Dir.tmpdir skips a candidate that is world-writable without a sticky bit, and a Kubernetes emptyDir is created mode 0777 with no sticky bit — so with a read-only root filesystem there was no candidate left at all. The error comes from MiniMagick, which Active Storage loads while Rails boots; nothing had asked for a temp file yet. The emptyDir root is root-owned, so its mode cannot be changed from a non-root container — but creating a subdirectory in a 0777 directory and chmod 700ing it needs no privilege at all. Hence the init container, and TMPDIR=/tmp/ruby.

Run 2 — the pod booted, and still never became Ready (ee62607). Two independent causes:

1. Readiness probe failed: Get "https://10.42.0.22:3000/registration/new": http: server gave HTTP response to HTTPS client. The kubelet's HTTP prober follows redirects to the same host, and with RAILS_ASSUME_SSL=true Rails builds its redirect as https://<pod-ip>:3000/..., which Puma does not speak. The probe now runs curl without -L and compares the status code itself, accepting 2xx–3xx exactly as the kubelet would. The same log line is also the proof the probe works as intended: the redirect target /registration/new is the branch User.count decides, so the path demonstrably reaches Postgres.
2. Pod ephemeral local storage usage exceeds the total limit of containers 60Mi — an eviction roughly two minutes into every start, with a perfectly healthy container log. emptyDirs and the stdout log both count against that limit, and this image rebuilds its Bootsnap cache under /rails/tmp on every start because the emptyDir hides the copy baked into the image. ephemeral-storage is now requested at 256Mi per app container (limit 768Mi); values.yaml says in as many words that this is headroom, not a measurement.

Also fixed in the same round, from the same logs: /home/rails is read-only, so Bundler built a throwaway home per invocation (including every worker probe, every 30s) — HOME now points at a directory the init container creates. And the CI Postgres fixture got -d sure on its readiness probe, which stops the server logging a missing database that nothing asked for.

Run 3 — green in 2m37s: OK: chart 'sure' installed and all workloads are Ready. The pod, first start and schema creation included, was Ready about 80 seconds after helm install.

Rebase note

Rebased onto current main after #35 landed. The sure Renovate block was written for the new reality: automerge is global including majors, so the comment explains that the filtering has to happen at the lookup (semver marks -alpha.N/-rc.N/-hotfix.N unstable and rejects stable/latest/nightly/sha-… outright) rather than at the merge. Two counts that my change makes wrong were pulled up by one: 24 thin publish hulls (README, wiki CI-CD) and 24 charts (renovate.json5 comment, wiki chart index).

Validated

In the pipeline (the only place with a Docker daemon):

• install-test green, which is the runtime proof: both containers Ready in a k3d cluster against throwaway Postgres 16 and Redis
• lint/package, the quantity render test over all charts, and the Trivy config scan all green

Locally:

• helm lint --strict sure, also with -f ci/sure/test-values.yaml
• helm template with only the required values, with the CI values, with scaleDown: true (renders replicas: 0), and with ingress.enabled=false — the last renders no Ingress but still requires the domain, and aborts with A Domain/Host must be provided (ingress.domain) when it is unset, which is the §12.1 guard doing its job
• helm package sure → sure-1.0.0+up0.7.4.tgz; ci/run-quantity-render-test.sh over all charts: 23 exercised, 0 failed
• NOTES.txt rendered out of a throwaway copy of the chart, since helm template does not render it
• Upstream facts re-checked against the registry and the 0.7.4 source: stable and 0.7.4 are the same digest sha256:391b2c52…, 0.7.4 is the newest non-prerelease tag, USER 1000:1000, EXPOSE 3000, the entrypoint's migration condition, config/storage.yml, config/database.yml, config/environments/production.rb, the WebAuthn and Active Record encryption initializers, Sidekiq 8.0.5's identity format

Still unverified, in those words

• Behaviour under failure. Nobody has taken Postgres or Redis away from a running instance and watched the probes react. That readiness reaches the database is measured; that it drops out of the Service on an outage is inferred.
• The steady-state ephemeral-storage figure. 256Mi is headroom chosen after an eviction at 60Mi, not a measurement of what the pod actually writes.
• The cost of a worker probe. bundle exec sidekiqmon starts a Ruby process every 30s; it passes well within its 15s timeout in CI, but the CPU cost was not measured.
• Whether the emptyDir over /rails/tmp costs noticeable boot time by hiding the image's precompiled Bootsnap cache.
• Whether the emptyDirs cover every path the two processes write under real use — imports, exports and AI features were never exercised.

Wiki

Page sure added (outline page as the template: same sections, same table shapes, full values reference), a row in Charts, an entry in _Sidebar, the two stale publish-hull counts in CI-CD corrected, and three new entries on Lessons-Learned for the findings above. Pushed — the wiki has no PR flow.

Fixes #33 New chart `sure` for [Sure](https://github.com/we-promise/sure), the maintained Maybe Finance fork, at `1.0.0+up0.7.4`. All four decisions from the issue are implemented as approved: one StatefulSet with two containers, `appVersion: 0.7.4`, AI options only through the additive `env` block, Postgres and Redis external. ## What is in it - **One StatefulSet, two containers** (`-web` Puma, `-worker` Sidekiq) sharing one RWO PVC at `/rails/storage`, because both processes use the same Active Storage files and an RWO volume cannot be attached to two pods. - **Only the web container migrates.** The image entrypoint runs `./bin/rails db:prepare` solely when the command is `./bin/rails server`. The worker is therefore started through **`args:`**, not `command:` — `command:` would replace the ENTRYPOINT and with it the jemalloc preload upstream names as one of the three reasons its memory figures hold. - **One small init container**, the app's own image running as the same non-root uid with no added capability, which prepares the two private directories `TMPDIR` and `HOME` point at. Why that is not decoration is the first of the three findings below. - **Three `OnePasswordItem` CRs** (app / database / redis), all wired via `secretKeyRef`. No `operator.1password.io/auto-restart` (silent no-op on a StatefulSet); instead three `checksum/secret-ref` annotations plus the manual `kubectl rollout restart` in NOTES and wiki. - **Probes chosen for what they prove.** Liveness/startup use `/up`, which is Rails' boot check and touches no database. Readiness asks for `/`: without a session cookie the request runs `authenticate_user!` → `self_hosted_first_login?` → `User.count`, i.e. an SQL query. The worker has no HTTP at all; all three of its probes run `bundle exec sidekiqmon processes | grep -q "^${HOSTNAME}:"`, which proves a live Sidekiq heartbeat in Redis — a hung worker drops out of the process set within ~60s while its PID still exists, and a plain process check could never fail at all because Sidekiq is PID 1 there. - **`ingress.domain` stays required with `ingress.enabled: false`** (§12.1): besides the Ingress host it feeds `APP_DOMAIN`, `WEBAUTHN_RP_ID` and `WEBAUTHN_ALLOWED_ORIGINS`. `ingress.enabled` additionally decides `RAILS_ASSUME_SSL`/`RAILS_FORCE_SSL`. - Strict `values.schema.json`, null-safe fallbacks reproducing this chart's own defaults, `replicas` + `scaleDown`, additive `env` per container through `tpl`, resources with `limitFactor`. - Wiring: `publish-sure.yml`, `sure` in the `change-app-version.yml` choice list, `ci/sure/` fixtures (throwaway Postgres 16 + Redis, dummy secrets, `INSTALL_TIMEOUT=900`), a Renovate manager on `versioning: semver`, the README chart row, and one added name in the repo-wide `AVD-KSV-0020` statement. Resource requests come from upstream's own sizing note for 0.7.4 (`docs/hosting/docker.md`): ~352 MB steady state, "at least 1 GB" with AI enabled, and the sample-data generator measured at ~680 MB **in the worker**. `512Mi` request × `limitFactor: 3` = `1536Mi` limit covers all three without reserving the large number. ## What was red, and why it is green now The install-test failed twice before it passed, and neither cause was visible in any render. Both are in the chart with the measured evidence next to them, and all three lessons are on the wiki's [Lessons learned](https://git.jk-dev.de/jlab-cluster/helm-charts/wiki/Lessons-Learned) page, because none of them is specific to this chart. **Run 1 — both containers crash-looped during boot** (`e92286b`): ``` system temporary path is world-writable: /tmp /tmp is world-writable: /tmp . is not writable: /rails ArgumentError: could not find a temporary directory ``` Ruby's `Dir.tmpdir` skips a candidate that is world-writable without a sticky bit, and a Kubernetes emptyDir is created mode 0777 with no sticky bit — so with a read-only root filesystem there was no candidate left at all. The error comes from MiniMagick, which Active Storage loads while Rails boots; nothing had asked for a temp file yet. The emptyDir root is root-owned, so its mode cannot be changed from a non-root container — but creating a subdirectory in a 0777 directory and `chmod 700`ing it needs no privilege at all. Hence the init container, and `TMPDIR=/tmp/ruby`. **Run 2 — the pod booted, and still never became Ready** (`ee62607`). Two independent causes: 1. `Readiness probe failed: Get "https://10.42.0.22:3000/registration/new": http: server gave HTTP response to HTTPS client`. The kubelet's HTTP prober **follows redirects to the same host**, and with `RAILS_ASSUME_SSL=true` Rails builds its redirect as `https://<pod-ip>:3000/...`, which Puma does not speak. The probe now runs `curl` without `-L` and compares the status code itself, accepting 2xx–3xx exactly as the kubelet would. The same log line is also the proof the probe works as intended: the redirect target `/registration/new` is the branch `User.count` decides, so the path demonstrably reaches Postgres. 2. `Pod ephemeral local storage usage exceeds the total limit of containers 60Mi` — an **eviction** roughly two minutes into every start, with a perfectly healthy container log. emptyDirs and the stdout log both count against that limit, and this image rebuilds its Bootsnap cache under `/rails/tmp` on every start because the emptyDir hides the copy baked into the image. `ephemeral-storage` is now requested at `256Mi` per app container (limit `768Mi`); `values.yaml` says in as many words that this is headroom, not a measurement. Also fixed in the same round, from the same logs: `/home/rails` is read-only, so Bundler built a throwaway home **per invocation** (including every worker probe, every 30s) — `HOME` now points at a directory the init container creates. And the CI Postgres fixture got `-d sure` on its readiness probe, which stops the server logging a missing database that nothing asked for. **Run 3 — green in 2m37s**: `OK: chart 'sure' installed and all workloads are Ready.` The pod, first start and schema creation included, was Ready about 80 seconds after `helm install`. ## Rebase note Rebased onto current `main` after #35 landed. The `sure` Renovate block was written for the new reality: automerge is global including majors, so the comment explains that the filtering has to happen at the lookup (semver marks `-alpha.N`/`-rc.N`/`-hotfix.N` unstable and rejects `stable`/`latest`/`nightly`/`sha-…` outright) rather than at the merge. Two counts that my change makes wrong were pulled up by one: `24 thin publish hulls` (README, wiki CI-CD) and `24 charts` (renovate.json5 comment, wiki chart index). ## Validated In the pipeline (the only place with a Docker daemon): - **install-test green**, which is the runtime proof: both containers Ready in a k3d cluster against throwaway Postgres 16 and Redis - lint/package, the quantity render test over all charts, and the Trivy config scan all green Locally: - `helm lint --strict sure`, also with `-f ci/sure/test-values.yaml` - `helm template` with only the required values, with the CI values, with `scaleDown: true` (renders `replicas: 0`), and with `ingress.enabled=false` — the last renders no Ingress but still requires the domain, and aborts with `A Domain/Host must be provided (ingress.domain)` when it is unset, which is the §12.1 guard doing its job - `helm package sure` → `sure-1.0.0+up0.7.4.tgz`; `ci/run-quantity-render-test.sh` over all charts: 23 exercised, 0 failed - `NOTES.txt` rendered out of a throwaway copy of the chart, since `helm template` does not render it - Upstream facts re-checked against the registry and the 0.7.4 source: `stable` and `0.7.4` are the same digest `sha256:391b2c52…`, `0.7.4` is the newest non-prerelease tag, `USER 1000:1000`, `EXPOSE 3000`, the entrypoint's migration condition, `config/storage.yml`, `config/database.yml`, `config/environments/production.rb`, the WebAuthn and Active Record encryption initializers, Sidekiq 8.0.5's identity format ## Still unverified, in those words - **Behaviour under failure.** Nobody has taken Postgres or Redis away from a *running* instance and watched the probes react. That readiness reaches the database is measured; that it drops out of the Service on an outage is inferred. - **The steady-state ephemeral-storage figure.** 256Mi is headroom chosen after an eviction at 60Mi, not a measurement of what the pod actually writes. - **The cost of a worker probe.** `bundle exec sidekiqmon` starts a Ruby process every 30s; it passes well within its 15s timeout in CI, but the CPU cost was not measured. - **Whether the emptyDir over `/rails/tmp` costs noticeable boot time** by hiding the image's precompiled Bootsnap cache. - **Whether the emptyDirs cover every path the two processes write** under real use — imports, exports and AI features were never exercised. ## Wiki Page `sure` added (outline page as the template: same sections, same table shapes, full values reference), a row in `Charts`, an entry in `_Sidebar`, the two stale publish-hull counts in `CI-CD` corrected, and three new entries on `Lessons-Learned` for the findings above. Pushed — the wiki has no PR flow.

jlab-bot added 1 commit 2026-09-22 21:54:49 +02:00

sure: new chart for the we-promise Sure app (1.0.0+up0.7.4) ... 0b36d91275

One StatefulSet with two containers - the Puma web process and a Sidekiq
worker - sharing one ReadWriteOnce PVC at /rails/storage. That shared
volume is the reason they live in one pod: splitting them would need an
RWX volume. Only the web container migrates, because the image entrypoint
runs db:prepare solely for the './bin/rails server' command; the worker is
started through args: so the entrypoint (and its jemalloc preload) stays
in place and the two can never race.

Probes are chosen for what they prove: liveness and startup use Rails'
/up, which is a boot check and touches no database, while readiness uses
'/', which runs User.count through authenticate_user! in self-hosted mode
and therefore fails while Postgres is unreachable. The worker speaks no
HTTP, so all three of its probes ask sidekiqmon whether this pod still
holds a Sidekiq heartbeat in Redis.

Three OnePasswordItem CRs (app, database, redis), ingress.domain kept
required outside the Ingress because APP_DOMAIN and the WebAuthn relying
party read it, strict values.schema.json, CI fixtures with throwaway
Postgres 16 and Redis, and a Renovate manager on semver so the
-alpha/-rc/-hotfix channel and the moving tags are never selected.

jklein added 1 commit 2026-09-22 22:19:18 +02:00

sure: give Ruby a temporary directory it accepts ... e92286b420

The first install-test run failed with both containers crash-looping
during boot:

  system temporary path is world-writable: /tmp
  /tmp is world-writable: /tmp
  . is not writable: /rails
  ArgumentError: could not find a temporary directory

raised from MiniMagick, which Active Storage loads while Rails boots.
Ruby's Dir.tmpdir skips a candidate that is world-writable without a
sticky bit, and a Kubernetes emptyDir is created mode 0777 with no sticky
bit - so with a read-only root filesystem there was no candidate left.

A tiny init container, the app's own image running as the same non-root
uid and with no capability added, now creates one private subdirectory
(mode 0700) inside each tmp emptyDir, and both containers run with
TMPDIR pointing at it. Creating an entry in a 0777 directory and chmod'ing
what one just created needs no privilege, so the hardening standard holds
unchanged.

jklein added 1 commit 2026-09-22 22:43:56 +02:00

sure: fix the readiness probe, the ephemeral-storage budget and HOME ... ee62607e2e

Second install-test run, three findings, all measured:

- Readiness on "/" failed with "http: server gave HTTP response to HTTPS
  client" on a pod that was answering correctly. The kubelet's HTTP prober
  follows redirects to the same host, and with RAILS_ASSUME_SSL=true the
  app builds its redirect as https://<pod-ip>:3000/registration/new, which
  Puma does not speak. The probe now runs curl without -L and accepts
  2xx-3xx itself. The same log also proves what the probe is for: against
  an empty database the redirect target was /registration/new, the branch
  User.count decides, so the path really does reach Postgres.
- The pod was evicted within two minutes of every start: "ephemeral local
  storage usage exceeds the total limit of containers 60Mi". emptyDirs and
  the stdout log both count there, and the image rebuilds its Bootsnap
  cache under /rails/tmp on every start because the emptyDir hides the one
  baked into the image. The request is now 256Mi (limit 768Mi) per app
  container - headroom, not a measurement.
- /home/rails is read-only, so Bundler created a throwaway home under
  TMPDIR on every invocation, worker probe included. HOME now points at a
  directory the init container creates.

The CI postgres fixture gets -d sure on its readiness probe, so the server
stops logging a missing database that nothing asked for.

jklein approved these changes 2026-09-26 23:08:49 +02:00

jklein merged commit d9267b2180 into main 2026-09-26 23:08:56 +02:00

jklein deleted branch feat/sure-chart 2026-09-26 23:08:56 +02:00

Sign in to join this conversation.

Reviewers

No Reviewers

jklein

Labels

Clear labels

inhouse-chart

Inhouse-Chart aus der JLab-Git-Paketregistry

renovate

Von Renovate angelegt

No labels

Milestone

No items

No Milestone

Assignees

Clear assignees

jklein (Justin Klein) jlab-bot (jlab-git-bot) jlab-renovate-bot (jlab-renovate-bot)

No Assignees

2 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: jlab-cluster/helm-charts#36