diff --git a/CI-CD.md b/CI-CD.md index 14d5d50..7e88d94 100644 --- a/CI-CD.md +++ b/CI-CD.md @@ -114,6 +114,8 @@ Renders each chart in the scan matrix and lists what `trivy config` finds. **It Trivy is installed from its release tarball, not via `aquasecurity/setup-trivy` — see [below](#what-a-gitea-runner-does-differently). +Exceptions live in `ci/trivy/trivyignore.yaml`, each with a mandatory `statement` — the script refuses to run without one and prints every suppressed finding together with its reason. **The trusted-registry list for AVD-KSV-0125 is a separate file on purpose**, `ci/trivy/data/ksv0125.yaml`: it is *configured* rather than suppressed, so the check keeps being able to say something about an image from a genuinely foreign source. A new registry therefore means an entry there, not an ignore — `quay.io` joined it with [keycloak](keycloak), the first chart in the repo using it. Note what the check does *not* forgive: an unqualified Docker Hub image puts the organisation in the registry position and is still reported, which is why every chart writes its image fully qualified. + ### 5. `install-test` (per changed, non-excluded chart) `ci/run-install-test.sh ` against a throwaway [k3d](https://k3d.io) cluster: apply the stub `OnePasswordItem` CRD plus the chart's fixtures, `helm install` with the CI values, and wait for every Deployment and StatefulSet to roll out. Since probes and hardening are active, "Ready" means the app actually answers. On failure the job dumps pod descriptions, events and container logs. diff --git a/Chart-Style-Guide.md b/Chart-Style-Guide.md index 80f5910..861618f 100644 --- a/Chart-Style-Guide.md +++ b/Chart-Style-Guide.md @@ -119,6 +119,8 @@ Measured on `urlaubsverwaltung` (helm 3.19.0, Kubernetes 1.35): `helm get manife - Every Deployment/StatefulSet has a `replicas` value (default `1`, path matching its section) that also accepts an explicit `0`. - Every chart has a global `scaleDown` flag (default `false`). `scaleDown: true` takes precedence over all `replicas` values, renders every workload with `replicas: 0` and suspends CronJobs. It **stops** the app without uninstalling it: PVCs, Secrets, Services and Ingress stay; setting it back brings the workloads up. - Implemented via a per-chart `.replicas` helper using an explicit nil check (`kindIs "invalid"`), because Helm's `default` would swallow an explicit `0`. +- **A chart whose default is more than one replica ships a `PodDisruptionBudget`.** `policy/v1`, named `--pdb`, `minAvailable` as a value (default `1`), selector on the workload's `app` label. [keycloak](keycloak) is the first and set the convention. It is a rule rather than a per-chart taste because of *which* failure a second replica is usually bought against: a node drain is the most common reason a pod moves — it is what a node upgrade does — and without a budget the drain is free to take every replica at once. The second replica then costs money and buys nothing. +- **And it is rendered only while the *computed* replica count is `>= 2`.** `minAvailable: 1` with exactly one replica forbids every voluntary eviction, so `kubectl drain` blocks forever on a pod nothing may move — and the message names the PDB, not the replica count, which sends the reader to the wrong file. The same applies to `scaleDown: true`, where there are no pods and the object would only describe a promise about nothing. So the condition reads the `.replicas` helper rather than the raw value, and `NOTES.txt` prints which of the two happened instead of leaving it to be inferred from `kubectl get pdb`. ## 12. Values that must not lie diff --git a/Charts.md b/Charts.md index a4b3ed1..78878a7 100644 --- a/Charts.md +++ b/Charts.md @@ -1,6 +1,6 @@ # Chart index -29 directories: 28 installable charts plus the `template-chart` scaffold. Each page holds prerequisites, an installation walkthrough, the complete list of configurable values, and the chart's operating traps. +30 directories: 29 installable charts plus the `template-chart` scaffold. Each page holds prerequisites, an installation walkthrough, the complete list of configurable values, and the chart's operating traps. **Versions are not listed here on purpose** — and not on the chart pages either. Renovate moves app versions continuously; a table of numbers in a wiki is wrong within days. Read the current pair off the chart: @@ -19,6 +19,7 @@ grep -E '^(version|appVersion):' /Chart.yaml | [garage](garage) | Garage (S3-compatible object store) | StatefulSet + **two** PVCs | off by default | **one** node at `replication_factor = 1` — which applies to the metadata too, so `metadata_fsync`, the metadata snapshots and a second PVC for them are mandatory rather than optional; `FROM scratch` image, so no shell and no exec probes ever; the RPC secret has to be an env var; readiness is `/health`, liveness and startup deliberately are not | | [gotenberg](gotenberg) | Gotenberg | Deployment + Service | no | PDF conversion for [paperless-ngx](paperless-ngx); sets every limit explicitly | | [homeassistant](homeassistant) | Home Assistant | StatefulSet | upstream-style lists | the only chart with Helm tests; starts as root for s6-overlay | +| [keycloak](keycloak) | Keycloak (OIDC / SAML identity provider) | Deployment, **2** replicas, **two init containers**, no PVC | yes, **two** (public + admin host) | **the admin hostname is not an access control** — it only decides which URLs the server generates, and upstream says so; `db`/`health-enabled`/`metrics-enabled` are *build* options, so the stock image would rewrite its own installation at start and the two init containers are what keeps `readOnlyRootFilesystem: true`; clustered via `jdbc-ping` over the database, so two releases on one database **merge**; first `PodDisruptionBudget` and first `quay.io` image in the repo | | [matrix-synapse](matrix-synapse) | Matrix Synapse | StatefulSet + nginx delegation Deployment | yes | `ingress.domain` is served in `/.well-known/matrix/*` — federation depends on it | | [minecraft-router](minecraft-router) | itzg/mc-router | Deployment + ConfigMap + NodePort | no (TCP) | `mappings` required; the chart behind the `checksum/config` outage | | [minecraft-server](minecraft-server) | itzg/minecraft-server | StatefulSet + PVC | no | only the newest Java line is maintained; `image.tagOverride` picks an older JVM | diff --git a/Lessons-Learned.md b/Lessons-Learned.md index 648a14e..e1d36a7 100644 --- a/Lessons-Learned.md +++ b/Lessons-Learned.md @@ -371,6 +371,28 @@ What makes this an entry rather than a quirk of one chart is how it presents and **Now:** before specifying `*_secret_file` for an application, check what its permission test actually masks. A `0o077` test (or any test that includes the group bits) means the file form is unavailable to a non-root pod, and the honest route is `secretKeyRef` with the price written at the value: the secret is in the process environment, readable by anyone who can `exec` into the pod. Write the **mechanism** next to it, not the conclusion — "we use the env var here" invites the next round to tidy it back into the file form, and then the pod stops starting. +## The application that rewrites its own installation at start + +The plan for [keycloak](keycloak) (Gitea #83) specified the three things this repo specifies for every chart: the stock upstream image, probes on the application's own health endpoints, and `readOnlyRootFilesystem: true`. Each is right. Together they cannot start a pod, and nothing in the plan could have shown that, because the conflict is not between any two of them — it is between all three and a property of the image. + +Keycloak splits its options into **build** options and runtime options. `db`, `health-enabled` and `metrics-enabled` are build options; the published image is augmented with their *defaults*; and supplying a different value at runtime makes the server **re-augment itself at start**, rewriting four files inside its own installation directory. Upstream describes that start without euphemism — it "creates an image that is mutable" — which is exactly what a read-only root filesystem forbids. And the escape that suggests itself is not one: *"then don't configure the database at runtime"* leaves the probes, and `health-enabled` is a build option too. **The hardening baseline and the probes were in conflict through a third thing neither mentions.** + +The repair has a trap of its own, and it is the part worth remembering, because the obvious version of it looks finished and fails: mounting an empty volume over the directory the augmentation writes does not work either. The runtime's entry point loads a serialized application file *from that directory* before it dispatches **any** command — the build command included — so an empty mount breaks the very command meant to fill it. What works is seeding: one init container copies the image's own output into the volume, a second runs the build into the now-seeded volume, and the serving container consumes it with `--optimized`. Three containers, all three with a read-only root filesystem, no documented deviation anywhere. + +**Now:** before pairing a stock image with `readOnlyRootFilesystem: true`, ask what the application does at **start** and not only at run — specifically whether any setting is baked into the image rather than read, and what happens when the running configuration disagrees with the baked one. Where that produces a write, find out *where* it writes before designing around it: inside the install tree is a different problem from a data or temp directory, and a volume mounted over an install directory hides whatever was in it. Two further rules fell out of this one and both are cheap: **build-time options belong in exactly one place in a chart**, because two places mean a disagreement that the application answers by rebuilding — and where a values key could reintroduce one, the render should **fail** naming it rather than let it reach a pod. And when a pre-build step is the answer, prefer *two hardened containers over one writable one*: a single init container with `readOnlyRootFilesystem: false` would have been shorter and would have put a documented hardening deviation into an identity provider, plus a Trivy suppression that also silences the serving container. + +## The upgrade procedure the automerge did not know about + +[keycloak](keycloak) supports a rolling update **only within a minor line**. A minor jump needs every node shut down first, the configuration and database backed up, and the cluster brought back up on the new version. Our Renovate automerges every update type, majors included, after a three-day age gate. + +Put those two sentences next to each other and the failure writes itself: a 26.8 → 26.9 app-version PR merges itself, publishes the chart, and gets rolled out pod by pod — the one procedure upstream excludes for that step. **Nothing reports it.** Both pods are Ready, both are green, and the symptom surfaces whenever two incompatible versions happen to need each other inside one Infinispan cluster. + +What makes this an entry rather than a chart note is the shape, which has nothing to do with Keycloak: **an upgrade constraint that lives in the application's documentation is invisible to the machinery that performs upgrades.** Renovate knows versions and update types. It does not know that this particular minor step is a maintenance window. Any clustered application whose versions must not be mixed has the same gap — and the gap is widest exactly where the automation is best, because a smooth pipeline is what turns "somebody will notice" into "nobody has to look". + +The decision taken here was **not** to switch automerge off for that image (Gitea #83): the global policy stands, the risk is accepted knowingly, and `main` still requires a review. That is defensible, and it moves the whole weight onto documentation — so the procedure went into the chart's `NOTES.txt` *and* onto two wiki pages, in each case next to the thing somebody will actually be looking at. + +**Now:** when a chart's application restricts how it may be upgraded, write the restriction where the upgrade is performed — the chart's `NOTES.txt`, the chart page, and the [Renovate](Renovate) page if the automation could trip it — and say explicitly that the automation does not know about it. If upstream ships a tool that decides the question, name the tool, its exact invocation **and its exit codes**, because an exit code is the only form of that knowledge a pipeline can use. And where upstream's own documentation contradicts itself about the constraint, record the contradiction instead of picking the convenient half: the honest instruction is then "run the check every time", not "patches are safe". + ## The recurring shape - **Green is a claim about the assertions that ran, not about the system.** @@ -396,3 +418,6 @@ What makes this an entry rather than a quirk of one chart is how it presents and - **A rule about two components' configuration cannot live in either one's templates.** Whatever renders first reports the error, and it reports it in its own vocabulary — so the check belongs where validation happens before rendering does. - **"Erased" and "empty" are not the same value, and they can behave oppositely.** A null override removes a key; an empty object is merged with the default. Wherever a missing key is read as *enabled*, erasing a switch turns a feature on. - **An answer that arrived by accident is worth making repeatable.** The all-charts install-test exists because one unplanned run produced a result nobody could have asked for; the fix is a way to ask the question on purpose, not a note to remember the accident. +- **Three correct requirements can still be an impossible pod.** Conflicts do not only live between pairs of settings — the stock image, the probes and a read-only root filesystem each read as right, and what they collided through was a property of the image none of them names. So the question to ask of a combination is what the application does at *start*, not whether each half is defensible. +- **A constraint that lives in documentation is invisible to the machinery.** Automation knows versions, update types and exit codes. It does not know that a particular step is a maintenance window — and the better the pipeline, the less likely anybody is standing in the path to notice. If a tool can decide it, automate on its exit code; if nothing can, write the obligation where the work is done. +- **When the fix needs a privilege, ask whether two cheap steps can replace one privileged one.** Splitting a pre-build into a seeding container plus a build container kept the hardening baseline intact across the whole pod, where the shorter single-container version would have bought a documented deviation *and* a suppression that silences the container that actually serves traffic. diff --git a/Renovate.md b/Renovate.md index 638d4a1..eb375aa 100644 --- a/Renovate.md +++ b/Renovate.md @@ -45,6 +45,28 @@ Charts whose tags carry a flavour or date suffix override `versioning` in their **And a registry that publishes a tag per commit needs a version *constraint*, not only an anchor.** `garage` is the sharp case, measured on 2026-10-03: of the 340 tags on `docker.io/dxflrs/garage` the **newest 22 are 40-character commit SHAs**, pushed almost daily, while the current release `v2.4.1` dates from 2026-09-08. A block without a constraint therefore offers a hex string as the next "update", and an `appVersion` of 40 hex characters pins a commit nobody chose — and republishes the chart for it. The same block also has to keep out `v2.0.0-rc1`, `v2.0.0-beta1` and four `v1.99.x-internal` builds, and note that **ordering does not help there**: `v1.99.3-internal` sorts above the whole `v1.3.x` line. The shape that does all of it at once is a regex versioning anchored to `^v..$` — a candidate that does not match is not a version at all, so SHA tags (no `v`, no dots) and every suffixed tag (the terminating anchor) are skipped in one rule. `stalwart` uses the identical shape against its three moving names and its `-alpine` twins. +**The counter-case, and it has to be measured just as carefully: sometimes the constraint is not needed.** [keycloak](keycloak) looks like the same trap from the outside — `quay.io/keycloak/keycloak` carries release tags (`26.8.0`), **rebuild** tags with a numeric suffix (`26.8.0-0`, `26.7.3-1`, `26.7.2-2`, where the same app version is rebuilt for a base-image or CVE fix), floating minor tags (`26.8`, `26.7`) and the floating names `latest` and `nightly`, the latter rebuilt daily. Four shapes, three of them unwanted. And plain `docker` versioning already excludes all three, so that block carries **no** `versioningTemplate`: docker versioning splits a tag at the first dash and only ever offers candidates with the **same** suffix as the current value — ours has none, so the `-N` rebuilds are invisible to it — while the floating minors fail on the missing third part and `latest`/`nightly` parse as no version at all. + +Measured rather than assumed, with Renovate 44.132.5 (`--platform=local --dry-run=lookup`, real registry) against the **committed** config: `26.8.0` → no updates and `warnings: []`; `26.7.0` → exactly one, `26.8.0` as a minor; `26.7.4` → exactly two, `26.7.5` as a patch and `26.8.0` as a minor. No run offered a rebuild tag, a floating minor, `latest` or `nightly`. + +Why that deserves a paragraph instead of an absent one: **an unnecessary constraint is not free.** It is a line that does nothing, that reads as load-bearing to the next person, and that has to stay correct as upstream's tagging changes — and the two fixtures above are exactly the guard the nginx and qbittorrent rules taught us to run, so skipping it was a decision with evidence rather than an omission. The general rule both cases share: **check the tag list, then measure what the default versioning already does with it, and only add a constraint for what is left.** + +## The upgrade procedure the automerge does not know about + +[keycloak](keycloak) supports a rolling update **only within a minor line**. A minor jump (26.8 → 26.9) requires every node shut down before any new node starts, and nothing in the cluster reports that two minor versions met in one Infinispan cluster. + +That collides with the PR policy below: `automerge: true` is global here, majors included, with a three-day age gate for patch and minor. A minor app-version PR therefore merges itself after three days, publishes the chart, and — at two replicas — is rolled out as a rolling update, which is the one procedure upstream excludes for that step. + +**The proposal to switch `automerge` off for `minor` and `major` on that one image was made and decided against** (Gitea #83). The global policy applies to keycloak like everything else, and the risk is accepted knowingly rather than overlooked. Two reasons that is defensible: `main` requires a review, so an automerged PR is still a PR somebody waved through; and a `packageRules` exception would have been the second mechanism — next to the review — that only works while somebody remembers it is there. + +So **the procedure is documented instead of prevented**, which puts an obligation on whoever reviews app-version PRs for this chart: + +- Watch for a changed **minor** in a keycloak app-version PR. The chart version's `+up` suffix makes it visible in the diff. +- Decide it with `kc.sh update-compatibility metadata --file=…` (old version, old configuration) and then `check --file=…` (new version, new configuration), passing **every** option on both calls. Automate on the exit code: `0` possible, `1` error, `2` invalid option, `3`/`4` not possible — shut down first. +- If it says shut down: `scaleDown: true`, back up database **and** configuration, bump, `scaleDown: false`. + +The chart's `NOTES.txt` carries the same sequence, so it reaches whoever runs the release and not only whoever reads this page. One thing is deliberately left open there and here: upstream's 26.6 release notes call patch-level rolling updates supported and default-on, while the 26.8.0 `update-compatibility` page says a rolling update is possible "when the Keycloak version is the same" and uses a **patch** pair as its own incompatible example. Until that is resolved, run the check for patch updates too. + ## The one digest pin `freesailarr`'s `mediathekarr` image is the only one pinned by digest, because its tags carry no version at all: `latest` is a 2024 build and the upstream README points at the moving `beta`. The built-in `helm-values` manager reads `repository`/`tag` and skips it (`skipReason: invalid-value`), so the two managers cannot produce competing PRs. diff --git a/_Sidebar.md b/_Sidebar.md index 8553ef9..a09216f 100644 --- a/_Sidebar.md +++ b/_Sidebar.md @@ -28,6 +28,7 @@ - [garage](garage) - [gotenberg](gotenberg) - [homeassistant](homeassistant) +- [keycloak](keycloak) - [matrix-synapse](matrix-synapse) - [minecraft-router](minecraft-router) - [minecraft-server](minecraft-server) diff --git a/keycloak.md b/keycloak.md new file mode 100644 index 0000000..68f7654 --- /dev/null +++ b/keycloak.md @@ -0,0 +1,329 @@ +# keycloak + +Deploys [Keycloak](https://www.keycloak.org/) — an OpenID Connect and SAML identity provider — as a **two-node cluster** against an external PostgreSQL. + +Two replicas is the point of this chart rather than a tuning option. A mail server may be away briefly because SMTP retries; a blob store has one consumer. An SSO server takes **every** dependent login down with it. And the second replica is cheap here, because node discovery runs over the database that is there anyway: `cache=ispn` plus Keycloak's default `jdbc-ping` stack means no headless Service, no JGroups DNS wiring and no extra option. + +What it costs is the thing to know before the first upgrade: **Keycloak supports a rolling update only within a minor line.** A minor jump needs the cluster shut down, and this repository's Renovate automerges every update type. See [Minor upgrades](#minor-upgrades-need-the-cluster-down) — that section is the reason this page exists as much as the values table is. + +| | | +|---|---| +| Image | `quay.io/keycloak/keycloak` — the only registry upstream publishes to, and the first `quay.io` image in this repo (hence the entry in `ci/trivy/data/ksv0125.yaml`) | +| Versions | not listed on purpose — read the current pair off `keycloak/Chart.yaml`, see [Chart index](Charts) | +| Workloads | Deployment, **2** replicas, **two init containers** — see [Why two init containers](#why-this-chart-has-two-init-containers) | +| Storage | **none.** No PVC: Keycloak 26 keeps everything in the database, user sessions included (persistent sessions are the default; volatile ones are deprecated as of 26.8) | +| Ingress | **two**: the public one and the admin hostname (switchable off). The admin one is **not** an access control — see [the first trap](#the-admin-hostname-is-not-an-access-control) | +| External dependencies | an external PostgreSQL. Nothing else | +| Secrets | 2 × `OnePasswordItem`, four fields in total | +| Network | ClusterIP Service on 8080 (`http`) and a **separate** ClusterIP Service on 9000 (`management`). The management port serves `/health/*` and optionally `/metrics` **unauthenticated** and is in no Ingress. JGroups uses 7800 pod-to-pod; no NetworkPolicy yet | +| Disruption | `PodDisruptionBudget`, `minAvailable: 1` — the **first** in this repo, and now a convention ([style guide §11](Chart-Style-Guide#11-scaling)) | +| Install-test | `ci/keycloak/` — fixtures + test values, **with two replicas** and a throwaway PostgreSQL | + +## Prerequisites + +- **An external PostgreSQL**, reachable from the cluster, with a database and a role **that can create its own schema**: Keycloak builds and migrates the entire schema itself through Liquibase on first start. A read-write grant on existing tables is not enough. +- **1Password operator** in the cluster, plus **two** items — see [What the 1Password items must contain](#what-the-1password-items-must-contain). +- **Two DNS names and a cert-manager cluster issuer**, unless the admin Ingress is switched off — one for the public endpoint, one for the administration console. +- **Nothing else.** No volume, no storage class, no cache service. + +## Installation + +The registry URL and its credentials live in 1Password (item `jlab-registry-access`: `username`, `password`, `charts-url`) and never in a repository or a shell history file. + +```bash +helm repo add jlab "$CHARTS_URL" --username "$REG_USER" --password "$REG_PASS" +helm repo update + +helm install keycloak jlab/keycloak \ + --namespace keycloak --create-namespace \ + -f my-values.yaml +``` + +Minimal `my-values.yaml`: + +```yaml +secrets: + databaseSecretRef: op://vault/keycloak-db + bootstrapAdminSecretRef: op://vault/keycloak-bootstrap-admin + +keycloak: + db: + host: postgres.example.internal + +ingress: + domain: id.example.com + clusterIssuer: letsencrypt + admin: + domain: id-admin.example.com +``` + +Five values, and four of them have no default because no default could be right. `helm install` prints the chart's `NOTES.txt`, which names the four vault fields, the verification sequence and the traps in short form. Read it — it is the per-release version of this page. + +### What the 1Password items must contain + +The chart renders two `OnePasswordItem`s; the operator turns each into a Kubernetes Secret whose **keys are the field names of the 1Password item**. A missing or misnamed field leaves the pod in `CreateContainerConfigError` — with no message naming the field. + +| Values key | Kubernetes Secret | Field | Content | +|---|---|---|---| +| `secrets.databaseSecretRef` | `-keycloak-db-secret` | **`username`** | the PostgreSQL role. It must be able to **create** its schema | +| | | **`password`** | that role's password | +| `secrets.bootstrapAdminSecretRef` | `-keycloak-bootstrap-admin-secret` | **`username`** | the **temporary** bootstrap administrator (upstream's own default name is `temp-admin`) | +| | | **`password`** | its password | + +Host, port and database name are deliberately **not** in the vault — they are plain values (`keycloak.db.host` / `.port` / `.database`), because they are not secret and because moving a database should not mean editing a vault entry. + +**Both arrive as environment variables, not as files, and that is forced rather than preferred.** Keycloak 26 has **no `_FILE` indirection for an option's value**: there is no `KC_DB_PASSWORD_FILE` and no generic `KC_