keycloak: new chart page, index, sidebar, and three rules it bought

The chart page carries the values table, the vault fields, the first-install checklist with the permanent administrator as a mandatory step, the minor-upgrade procedure, and the traps - of which two are the point of the page: hostname-admin is not an access control, and a shared database merges two clusters. Chart index counter: 30 directories / 29 installable. That follows the index TABLE, which already lists convertx - its page landed before its PR, and convertx/ does not exist on origin/main yet. So the number is one ahead of the repository on purpose, as it was before this change. Renovate: the counter-case to garage, measured rather than assumed - four tag shapes on quay.io/keycloak/keycloak and plain docker versioning already excludes the three unwanted ones, so that block carries no versioningTemplate. Written down because an unnecessary constraint is not free: it reads as load-bearing to the next person and has to stay correct as upstream's tagging changes. Plus a section on the automerge decision: switching it off for minor and major was proposed and declined, so the upgrade procedure is documented instead of prevented, with the obligation that puts on whoever reviews an app-version PR. Chart style guide section 11: a chart defaulting to more than one replica ships a PodDisruptionBudget, and it is rendered only while the computed replica count is >= 2 - minAvailable 1 with a single replica blocks every drain forever with a message that names the PDB rather than the replica count. CI/CD: the trusted-registry list for AVD-KSV-0125 is a separate file from the ignore file on purpose, so a new registry is an entry there rather than a suppression. quay.io joined it with this chart. Lessons learned, two entries and three lines in the recurring shape: - "The application that rewrites its own installation at start" - three correct requirements that cannot start a pod together, and why the obvious repair (an empty volume over the directory) breaks the command meant to fill it. - "The upgrade procedure the automerge did not know about" - the general form, which applies to any clustered application whose versions must not be mixed: a constraint living in documentation is invisible to the machinery that performs upgrades, and the better the pipeline, the less likely anybody is standing in the path to notice.
2026-10-04 17:09:56 +00:00
1 parent fc794d2659
commit 0852b72ee2
@@ -114,6 +114,8 @@ Renders each chart in the scan matrix and lists what `trivy config` finds. **It
 Trivy is installed from its release tarball, not via `aquasecurity/setup-trivy` — see [below](#what-a-gitea-runner-does-differently).
 Exceptions live in `ci/trivy/trivyignore.yaml`, each with a mandatory `statement` — the script refuses to run without one and prints every suppressed finding together with its reason. **The trusted-registry list for AVD-KSV-0125 is a separate file on purpose**, `ci/trivy/data/ksv0125.yaml`: it is *configured* rather than suppressed, so the check keeps being able to say something about an image from a genuinely foreign source. A new registry therefore means an entry there, not an ignore — `quay.io` joined it with [keycloak](keycloak), the first chart in the repo using it. Note what the check does *not* forgive: an unqualified Docker Hub image puts the organisation in the registry position and is still reported, which is why every chart writes its image fully qualified.
 ### 5. `install-test` (per changed, non-excluded chart)
 `ci/run-install-test.sh <chart>` against a throwaway [k3d](https://k3d.io) cluster: apply the stub `OnePasswordItem` CRD plus the chart's fixtures, `helm install` with the CI values, and wait for every Deployment and StatefulSet to roll out. Since probes and hardening are active, "Ready" means the app actually answers. On failure the job dumps pod descriptions, events and container logs.
@@ -119,6 +119,8 @@ Measured on `urlaubsverwaltung` (helm 3.19.0, Kubernetes 1.35): `helm get manife
 - Every Deployment/StatefulSet has a `replicas` value (default `1`, path matching its section) that also accepts an explicit `0`.
 - Every chart has a global `scaleDown` flag (default `false`). `scaleDown: true` takes precedence over all `replicas` values, renders every workload with `replicas: 0` and suspends CronJobs. It **stops** the app without uninstalling it: PVCs, Secrets, Services and Ingress stay; setting it back brings the workloads up.
 - Implemented via a per-chart `<chart>.replicas` helper using an explicit nil check (`kindIs "invalid"`), because Helm's `default` would swallow an explicit `0`.
 - **A chart whose default is more than one replica ships a `PodDisruptionBudget`.** `policy/v1`, named `<release>-<chart>-pdb`, `minAvailable` as a value (default `1`), selector on the workload's `app` label. [keycloak](keycloak) is the first and set the convention. It is a rule rather than a per-chart taste because of *which* failure a second replica is usually bought against: a node drain is the most common reason a pod moves — it is what a node upgrade does — and without a budget the drain is free to take every replica at once. The second replica then costs money and buys nothing.
 - **And it is rendered only while the *computed* replica count is `>= 2`.** `minAvailable: 1` with exactly one replica forbids every voluntary eviction, so `kubectl drain` blocks forever on a pod nothing may move — and the message names the PDB, not the replica count, which sends the reader to the wrong file. The same applies to `scaleDown: true`, where there are no pods and the object would only describe a promise about nothing. So the condition reads the `<chart>.replicas` helper rather than the raw value, and `NOTES.txt` prints which of the two happened instead of leaving it to be inferred from `kubectl get pdb`.
 ## 12. Values that must not lie
@@ -1,6 +1,6 @@
 # Chart index
-29 directories: 28 installable charts plus the `template-chart` scaffold. Each page holds prerequisites, an installation walkthrough, the complete list of configurable values, and the chart's operating traps.
+30 directories: 29 installable charts plus the `template-chart` scaffold. Each page holds prerequisites, an installation walkthrough, the complete list of configurable values, and the chart's operating traps.
 **Versions are not listed here on purpose** — and not on the chart pages either. Renovate moves app versions continuously; a table of numbers in a wiki is wrong within days. Read the current pair off the chart:
@@ -19,6 +19,7 @@ grep -E '^(version|appVersion):' <chart>/Chart.yaml
 | [garage](garage) | Garage (S3-compatible object store) | StatefulSet + **two** PVCs | off by default | **one** node at `replication_factor = 1` — which applies to the metadata too, so `metadata_fsync`, the metadata snapshots and a second PVC for them are mandatory rather than optional; `FROM scratch` image, so no shell and no exec probes ever; the RPC secret has to be an env var; readiness is `/health`, liveness and startup deliberately are not |
 | [gotenberg](gotenberg) | Gotenberg | Deployment + Service | no | PDF conversion for [paperless-ngx](paperless-ngx); sets every limit explicitly |
 | [homeassistant](homeassistant) | Home Assistant | StatefulSet | upstream-style lists | the only chart with Helm tests; starts as root for s6-overlay |
 | [keycloak](keycloak) | Keycloak (OIDC / SAML identity provider) | Deployment, **2** replicas, **two init containers**, no PVC | yes, **two** (public + admin host) | **the admin hostname is not an access control** — it only decides which URLs the server generates, and upstream says so; `db`/`health-enabled`/`metrics-enabled` are *build* options, so the stock image would rewrite its own installation at start and the two init containers are what keeps `readOnlyRootFilesystem: true`; clustered via `jdbc-ping` over the database, so two releases on one database **merge**; first `PodDisruptionBudget` and first `quay.io` image in the repo |
 | [matrix-synapse](matrix-synapse) | Matrix Synapse | StatefulSet + nginx delegation Deployment | yes | `ingress.domain` is served in `/.well-known/matrix/*` — federation depends on it |
 | [minecraft-router](minecraft-router) | itzg/mc-router | Deployment + ConfigMap + NodePort | no (TCP) | `mappings` required; the chart behind the `checksum/config` outage |
 | [minecraft-server](minecraft-server) | itzg/minecraft-server | StatefulSet + PVC | no | only the newest Java line is maintained; `image.tagOverride` picks an older JVM |
@@ -371,6 +371,28 @@ What makes this an entry rather than a quirk of one chart is how it presents and
 **Now:** before specifying `*_secret_file` for an application, check what its permission test actually masks. A `0o077` test (or any test that includes the group bits) means the file form is unavailable to a non-root pod, and the honest route is `secretKeyRef` with the price written at the value: the secret is in the process environment, readable by anyone who can `exec` into the pod. Write the **mechanism** next to it, not the conclusion — "we use the env var here" invites the next round to tidy it back into the file form, and then the pod stops starting.
 ## The application that rewrites its own installation at start
 The plan for [keycloak](keycloak) (Gitea #83) specified the three things this repo specifies for every chart: the stock upstream image, probes on the application's own health endpoints, and `readOnlyRootFilesystem: true`. Each is right. Together they cannot start a pod, and nothing in the plan could have shown that, because the conflict is not between any two of them — it is between all three and a property of the image.
 Keycloak splits its options into **build** options and runtime options. `db`, `health-enabled` and `metrics-enabled` are build options; the published image is augmented with their *defaults*; and supplying a different value at runtime makes the server **re-augment itself at start**, rewriting four files inside its own installation directory. Upstream describes that start without euphemism — it "creates an image that is mutable" — which is exactly what a read-only root filesystem forbids. And the escape that suggests itself is not one: *"then don't configure the database at runtime"* leaves the probes, and `health-enabled` is a build option too. **The hardening baseline and the probes were in conflict through a third thing neither mentions.**
 The repair has a trap of its own, and it is the part worth remembering, because the obvious version of it looks finished and fails: mounting an empty volume over the directory the augmentation writes does not work either. The runtime's entry point loads a serialized application file *from that directory* before it dispatches **any** command — the build command included — so an empty mount breaks the very command meant to fill it. What works is seeding: one init container copies the image's own output into the volume, a second runs the build into the now-seeded volume, and the serving container consumes it with `--optimized`. Three containers, all three with a read-only root filesystem, no documented deviation anywhere.
 **Now:** before pairing a stock image with `readOnlyRootFilesystem: true`, ask what the application does at **start** and not only at run — specifically whether any setting is baked into the image rather than read, and what happens when the running configuration disagrees with the baked one. Where that produces a write, find out *where* it writes before designing around it: inside the install tree is a different problem from a data or temp directory, and a volume mounted over an install directory hides whatever was in it. Two further rules fell out of this one and both are cheap: **build-time options belong in exactly one place in a chart**, because two places mean a disagreement that the application answers by rebuilding — and where a values key could reintroduce one, the render should **fail** naming it rather than let it reach a pod. And when a pre-build step is the answer, prefer *two hardened containers over one writable one*: a single init container with `readOnlyRootFilesystem: false` would have been shorter and would have put a documented hardening deviation into an identity provider, plus a Trivy suppression that also silences the serving container.
 ## The upgrade procedure the automerge did not know about
 [keycloak](keycloak) supports a rolling update **only within a minor line**. A minor jump needs every node shut down first, the configuration and database backed up, and the cluster brought back up on the new version. Our Renovate automerges every update type, majors included, after a three-day age gate.
 Put those two sentences next to each other and the failure writes itself: a 26.8 → 26.9 app-version PR merges itself, publishes the chart, and gets rolled out pod by pod — the one procedure upstream excludes for that step. **Nothing reports it.** Both pods are Ready, both are green, and the symptom surfaces whenever two incompatible versions happen to need each other inside one Infinispan cluster.
 What makes this an entry rather than a chart note is the shape, which has nothing to do with Keycloak: **an upgrade constraint that lives in the application's documentation is invisible to the machinery that performs upgrades.** Renovate knows versions and update types. It does not know that this particular minor step is a maintenance window. Any clustered application whose versions must not be mixed has the same gap — and the gap is widest exactly where the automation is best, because a smooth pipeline is what turns "somebody will notice" into "nobody has to look".
 The decision taken here was **not** to switch automerge off for that image (Gitea #83): the global policy stands, the risk is accepted knowingly, and `main` still requires a review. That is defensible, and it moves the whole weight onto documentation — so the procedure went into the chart's `NOTES.txt` *and* onto two wiki pages, in each case next to the thing somebody will actually be looking at.
 **Now:** when a chart's application restricts how it may be upgraded, write the restriction where the upgrade is performed — the chart's `NOTES.txt`, the chart page, and the [Renovate](Renovate) page if the automation could trip it — and say explicitly that the automation does not know about it. If upstream ships a tool that decides the question, name the tool, its exact invocation **and its exit codes**, because an exit code is the only form of that knowledge a pipeline can use. And where upstream's own documentation contradicts itself about the constraint, record the contradiction instead of picking the convenient half: the honest instruction is then "run the check every time", not "patches are safe".
 ## The recurring shape
 - **Green is a claim about the assertions that ran, not about the system.**
@@ -396,3 +418,6 @@ What makes this an entry rather than a quirk of one chart is how it presents and
 - **A rule about two components' configuration cannot live in either one's templates.** Whatever renders first reports the error, and it reports it in its own vocabulary — so the check belongs where validation happens before rendering does.
 - **"Erased" and "empty" are not the same value, and they can behave oppositely.** A null override removes a key; an empty object is merged with the default. Wherever a missing key is read as *enabled*, erasing a switch turns a feature on.
 - **An answer that arrived by accident is worth making repeatable.** The all-charts install-test exists because one unplanned run produced a result nobody could have asked for; the fix is a way to ask the question on purpose, not a note to remember the accident.
 - **Three correct requirements can still be an impossible pod.** Conflicts do not only live between pairs of settings — the stock image, the probes and a read-only root filesystem each read as right, and what they collided through was a property of the image none of them names. So the question to ask of a combination is what the application does at *start*, not whether each half is defensible.
 - **A constraint that lives in documentation is invisible to the machinery.** Automation knows versions, update types and exit codes. It does not know that a particular step is a maintenance window — and the better the pipeline, the less likely anybody is standing in the path to notice. If a tool can decide it, automate on its exit code; if nothing can, write the obligation where the work is done.
 - **When the fix needs a privilege, ask whether two cheap steps can replace one privileged one.** Splitting a pre-build into a seeding container plus a build container kept the hardening baseline intact across the whole pod, where the shorter single-container version would have bought a documented deviation *and* a suppression that silences the container that actually serves traffic.
@@ -45,6 +45,28 @@ Charts whose tags carry a flavour or date suffix override `versioning` in their
 **And a registry that publishes a tag per commit needs a version *constraint*, not only an anchor.** `garage` is the sharp case, measured on 2026-10-03: of the 340 tags on `docker.io/dxflrs/garage` the **newest 22 are 40-character commit SHAs**, pushed almost daily, while the current release `v2.4.1` dates from 2026-09-08. A block without a constraint therefore offers a hex string as the next "update", and an `appVersion` of 40 hex characters pins a commit nobody chose — and republishes the chart for it. The same block also has to keep out `v2.0.0-rc1`, `v2.0.0-beta1` and four `v1.99.x-internal` builds, and note that **ordering does not help there**: `v1.99.3-internal` sorts above the whole `v1.3.x` line. The shape that does all of it at once is a regex versioning anchored to `^v<major>.<minor>.<patch>$` — a candidate that does not match is not a version at all, so SHA tags (no `v`, no dots) and every suffixed tag (the terminating anchor) are skipped in one rule. `stalwart` uses the identical shape against its three moving names and its `-alpine` twins.
 **The counter-case, and it has to be measured just as carefully: sometimes the constraint is not needed.** [keycloak](keycloak) looks like the same trap from the outside — `quay.io/keycloak/keycloak` carries release tags (`26.8.0`), **rebuild** tags with a numeric suffix (`26.8.0-0`, `26.7.3-1`, `26.7.2-2`, where the same app version is rebuilt for a base-image or CVE fix), floating minor tags (`26.8`, `26.7`) and the floating names `latest` and `nightly`, the latter rebuilt daily. Four shapes, three of them unwanted. And plain `docker` versioning already excludes all three, so that block carries **no** `versioningTemplate`: docker versioning splits a tag at the first dash and only ever offers candidates with the **same** suffix as the current value — ours has none, so the `-N` rebuilds are invisible to it — while the floating minors fail on the missing third part and `latest`/`nightly` parse as no version at all.
 Measured rather than assumed, with Renovate 44.132.5 (`--platform=local --dry-run=lookup`, real registry) against the **committed** config: `26.8.0` → no updates and `warnings: []`; `26.7.0` → exactly one, `26.8.0` as a minor; `26.7.4` → exactly two, `26.7.5` as a patch and `26.8.0` as a minor. No run offered a rebuild tag, a floating minor, `latest` or `nightly`.
 Why that deserves a paragraph instead of an absent one: **an unnecessary constraint is not free.** It is a line that does nothing, that reads as load-bearing to the next person, and that has to stay correct as upstream's tagging changes — and the two fixtures above are exactly the guard the nginx and qbittorrent rules taught us to run, so skipping it was a decision with evidence rather than an omission. The general rule both cases share: **check the tag list, then measure what the default versioning already does with it, and only add a constraint for what is left.**
 ## The upgrade procedure the automerge does not know about
 [keycloak](keycloak) supports a rolling update **only within a minor line**. A minor jump (26.8 → 26.9) requires every node shut down before any new node starts, and nothing in the cluster reports that two minor versions met in one Infinispan cluster.
 That collides with the PR policy below: `automerge: true` is global here, majors included, with a three-day age gate for patch and minor. A minor app-version PR therefore merges itself after three days, publishes the chart, and — at two replicas — is rolled out as a rolling update, which is the one procedure upstream excludes for that step.
 **The proposal to switch `automerge` off for `minor` and `major` on that one image was made and decided against** (Gitea #83). The global policy applies to keycloak like everything else, and the risk is accepted knowingly rather than overlooked. Two reasons that is defensible: `main` requires a review, so an automerged PR is still a PR somebody waved through; and a `packageRules` exception would have been the second mechanism — next to the review — that only works while somebody remembers it is there.
 So **the procedure is documented instead of prevented**, which puts an obligation on whoever reviews app-version PRs for this chart:
 - Watch for a changed **minor** in a keycloak app-version PR. The chart version's `+up` suffix makes it visible in the diff.
 - Decide it with `kc.sh update-compatibility metadata --file=…` (old version, old configuration) and then `check --file=…` (new version, new configuration), passing **every** option on both calls. Automate on the exit code: `0` possible, `1` error, `2` invalid option, `3`/`4` not possible — shut down first.
 - If it says shut down: `scaleDown: true`, back up database **and** configuration, bump, `scaleDown: false`.
 The chart's `NOTES.txt` carries the same sequence, so it reaches whoever runs the release and not only whoever reads this page. One thing is deliberately left open there and here: upstream's 26.6 release notes call patch-level rolling updates supported and default-on, while the 26.8.0 `update-compatibility` page says a rolling update is possible "when the Keycloak version is the same" and uses a **patch** pair as its own incompatible example. Until that is resolved, run the check for patch updates too.
 ## The one digest pin
 `freesailarr`'s `mediathekarr` image is the only one pinned by digest, because its tags carry no version at all: `latest` is a 2024 build and the upstream README points at the moving `beta`. The built-in `helm-values` manager reads `repository`/`tag` and skips it (`skipReason: invalid-value`), so the two managers cannot produce competing PRs.
@@ -28,6 +28,7 @@
 - [garage](garage)
 - [gotenberg](gotenberg)
 - [homeassistant](homeassistant)
 - [keycloak](keycloak)
 - [matrix-synapse](matrix-synapse)
 - [minecraft-router](minecraft-router)
 - [minecraft-server](minecraft-server)
@@ -0,0 +1,329 @@
 # keycloak
 Deploys [Keycloak](https://www.keycloak.org/) — an OpenID Connect and SAML identity provider — as a **two-node cluster** against an external PostgreSQL.
 Two replicas is the point of this chart rather than a tuning option. A mail server may be away briefly because SMTP retries; a blob store has one consumer. An SSO server takes **every** dependent login down with it. And the second replica is cheap here, because node discovery runs over the database that is there anyway: `cache=ispn` plus Keycloak's default `jdbc-ping` stack means no headless Service, no JGroups DNS wiring and no extra option.
 What it costs is the thing to know before the first upgrade: **Keycloak supports a rolling update only within a minor line.** A minor jump needs the cluster shut down, and this repository's Renovate automerges every update type. See [Minor upgrades](#minor-upgrades-need-the-cluster-down) — that section is the reason this page exists as much as the values table is.
 | | |
 |---|---|
 | Image | `quay.io/keycloak/keycloak` — the only registry upstream publishes to, and the first `quay.io` image in this repo (hence the entry in `ci/trivy/data/ksv0125.yaml`) |
 | Versions | not listed on purpose — read the current pair off `keycloak/Chart.yaml`, see [Chart index](Charts) |
 | Workloads | Deployment, **2** replicas, **two init containers** — see [Why two init containers](#why-this-chart-has-two-init-containers) |
 | Storage | **none.** No PVC: Keycloak 26 keeps everything in the database, user sessions included (persistent sessions are the default; volatile ones are deprecated as of 26.8) |
 | Ingress | **two**: the public one and the admin hostname (switchable off). The admin one is **not** an access control — see [the first trap](#the-admin-hostname-is-not-an-access-control) |
 | External dependencies | an external PostgreSQL. Nothing else |
 | Secrets | 2 × `OnePasswordItem`, four fields in total |
 | Network | ClusterIP Service on 8080 (`http`) and a **separate** ClusterIP Service on 9000 (`management`). The management port serves `/health/*` and optionally `/metrics` **unauthenticated** and is in no Ingress. JGroups uses 7800 pod-to-pod; no NetworkPolicy yet |
 | Disruption | `PodDisruptionBudget`, `minAvailable: 1` — the **first** in this repo, and now a convention ([style guide §11](Chart-Style-Guide#11-scaling)) |
 | Install-test | `ci/keycloak/` — fixtures + test values, **with two replicas** and a throwaway PostgreSQL |
 ## Prerequisites
 - **An external PostgreSQL**, reachable from the cluster, with a database and a role **that can create its own schema**: Keycloak builds and migrates the entire schema itself through Liquibase on first start. A read-write grant on existing tables is not enough.
 - **1Password operator** in the cluster, plus **two** items — see [What the 1Password items must contain](#what-the-1password-items-must-contain).
 - **Two DNS names and a cert-manager cluster issuer**, unless the admin Ingress is switched off — one for the public endpoint, one for the administration console.
 - **Nothing else.** No volume, no storage class, no cache service.
 ## Installation
 The registry URL and its credentials live in 1Password (item `jlab-registry-access`: `username`, `password`, `charts-url`) and never in a repository or a shell history file.
 ```bash
 helm repo add jlab "$CHARTS_URL" --username "$REG_USER" --password "$REG_PASS"
 helm repo update
 helm install keycloak jlab/keycloak \
  --namespace keycloak --create-namespace \
  -f my-values.yaml
 ```
 Minimal `my-values.yaml`:
 ```yaml
 secrets:
  databaseSecretRef: op://vault/keycloak-db
  bootstrapAdminSecretRef: op://vault/keycloak-bootstrap-admin
 keycloak:
  db:
    host: postgres.example.internal
 ingress:
  domain: id.example.com
  clusterIssuer: letsencrypt
  admin:
    domain: id-admin.example.com
 ```
 Five values, and four of them have no default because no default could be right. `helm install` prints the chart's `NOTES.txt`, which names the four vault fields, the verification sequence and the traps in short form. Read it — it is the per-release version of this page.
 ### What the 1Password items must contain
 The chart renders two `OnePasswordItem`s; the operator turns each into a Kubernetes Secret whose **keys are the field names of the 1Password item**. A missing or misnamed field leaves the pod in `CreateContainerConfigError` — with no message naming the field.
 | Values key | Kubernetes Secret | Field | Content |
 |---|---|---|---|
 | `secrets.databaseSecretRef` | `<release>-keycloak-db-secret` | **`username`** | the PostgreSQL role. It must be able to **create** its schema |
 | | | **`password`** | that role's password |
 | `secrets.bootstrapAdminSecretRef` | `<release>-keycloak-bootstrap-admin-secret` | **`username`** | the **temporary** bootstrap administrator (upstream's own default name is `temp-admin`) |
 | | | **`password`** | its password |
 Host, port and database name are deliberately **not** in the vault — they are plain values (`keycloak.db.host` / `.port` / `.database`), because they are not secret and because moving a database should not mean editing a vault entry.
 **Both arrive as environment variables, not as files, and that is forced rather than preferred.** Keycloak 26 has **no `_FILE` indirection for an option's value**: there is no `KC_DB_PASSWORD_FILE` and no generic `KC_<OPTION>_FILE`. Checked against the 26.8.0 option table — every `KC_*_FILE` variable there is a *path* option (certificate, key store, trust store, log file, export file), i.e. "point at this file", never "read this option's value from a file". So the file form [valkey](valkey) argues for is unavailable here, as it was for [garage](garage) for a completely different reason. The residual price, stated rather than hidden: the password is in the container's process environment, readable by anyone who can `exec` into the pod. (Keycloak's own Vault SPI, `KC_VAULT=file`, is a different and larger mechanism and deliberately not used.)
 ## First install: the checklist
 **Step 4 is mandatory, not advice.**
 1. **The rollout.**
   ```bash
   kubectl -n keycloak rollout status deployment/keycloak-keycloak-deployment
   ```
   Ready means `/health/ready` answered 200, which means the database is reachable and the schema is in place. The first start is the slow one: it creates the whole schema, and with two replicas the second waits on the first's migration lock. If a pod sits in `Init`, read the init containers first — see [Why two init containers](#why-this-chart-has-two-init-containers).
 2. **Check that the cluster really formed.** This is what the second replica is for, and a silent failure here leaves two independent islands that each believe they are the cluster:
   ```bash
   kubectl -n keycloak logs deployment/keycloak-keycloak-deployment \
     | grep -i "ISPN000094\|view accepted\|members"
   ```
   Look for a view with **two** members. One member per pod means `jdbc-ping` did not find the peer.
 3. **Log in** at `https://<ingress.admin.domain>/admin` with the bootstrap credentials from the vault item.
 4. **Create a permanent administrator, then remove the temporary one.** The bootstrap account is temporary *by design* — upstream: *"the account should exist only for the duration necessary to perform operations needed to gain permanent and more secure admin access. After that, the account needs to be removed manually."* This is a change from Keycloak before 26, where `KEYCLOAK_ADMIN` created a regular administrator and there was nothing to clean up.
   1. In the `master` realm, create a user, set a password, assign the admin role.
   2. Verify you can log in as that user — before deleting anything.
   3. Delete the `temp-admin` user.
   4. `helm upgrade` with `keycloak.bootstrapAdmin.enabled: false`. That drops the variables **and** the vault item from the release.
 5. **Until step 4.4, every start logs a line about the initial administrator not being created**, because the `master` realm already exists and the bootstrap path is skipped. It is harmless — and it is exactly the kind of message somebody eventually investigates, which is why 4.4 is part of the install rather than a tidy-up.
 ## Minor upgrades need the cluster down
 **Keycloak supports a rolling update only within a minor line.** A minor jump (26.8 → 26.9) requires every node shut down before any new node starts; mixing two minor versions in one Infinispan cluster is not a supported configuration, and **nothing in the cluster reports that it happened**.
 **This repository's Renovate automerges every update type, majors included.** That was [decided deliberately](Renovate#the-upgrade-procedure-the-automerge-does-not-know-about) rather than overlooked, and it means an app-version PR that moves the minor can reach `main`, publish the chart and be rolled out as a rolling update without anybody choosing to. So the procedure is a human obligation here, not a guard rail:
 1. **Decide it with the tool, not with the version number.** With the **old** version and the **old** configuration:
   ```bash
   kc.sh update-compatibility metadata --file=/tmp/kc-compat.json
   ```
   then with the **new** version and the **new** configuration:
   ```bash
   kc.sh update-compatibility check --file=/tmp/kc-compat.json
   ```
   Pass **every** configuration option on both calls — upstream: *"Omitting any configuration options results in incomplete metadata, and could lead to a wrong reported result."* Automate on the **exit code**, which is what it is for:
   | Exit code | Meaning |
   |---|---|
   | `0` | rolling update is possible |
   | `1` | unexpected error (metadata file missing or corrupted) |
   | `2` | invalid CLI option |
   | `3` / `4` | **not** possible — the deployment must be shut down first |
 2. **If it says shut down:**
   ```bash
   helm upgrade ... --set scaleDown=true     # every pod stops
   # back up the database AND the configuration before anything else
   helm upgrade ... --set scaleDown=false    # with the new version
   ```
 **One thing is not settled, and it is recorded rather than guessed.** Upstream's 26.6 release notes call patch-level rolling updates supported and enabled by default, while the 26.8.0 page for `update-compatibility` says a rolling update is possible *"when the Keycloak version is the same"* and uses a **patch** pair (26.2.0 → 26.2.1) as its own example of an incompatible one. Those two statements cannot both be current. Until it is clear which is: **run the check for patch updates too, and trust its exit code rather than the size of the version step.**
 ## Why this chart has two init containers
 This is the part of the chart that looks like over-engineering until the four facts behind it are on the table. All four were measured for Gitea #83.
 1. **`db`, `health-enabled` and `metrics-enabled` are Keycloak *build* options**, not runtime options — each carries the tool icon in the 26.8.0 option table. (`cache` does **not**, which is why `cache=ispn` costs nothing here.) Their defaults are `dev-file`, `false` and `false`.
 2. **The published image is augmented with those defaults.** `/opt/keycloak/lib/quarkus/` in the image holds `generated-bytecode.jar`, `transformed-bytecode.jar`, `quarkus-application.dat` and `build-system.properties`; `conf/keycloak.conf` is entirely commented out.
 3. **Supplying different build options at runtime makes Keycloak re-augment itself at start, rewriting exactly those four files** — inside the root filesystem. `kc.sh` says so itself: *"only exit code 10 means that implicit reaugmentation occurred and a relaunch is needed."* Upstream's container guide names the consequence: that start *"significantly increases startup time and **creates an image that is mutable**"*. A mutable image is precisely what `readOnlyRootFilesystem: true` forbids — and *"then don't set `db`"* is no escape, because **the probes alone force it**: `health-enabled` is a build option too.
 4. **An empty `emptyDir` over that directory does not work either**, and this is the fact that is not guessable from the outside: `QuarkusEntryPoint` — the main class `kc.sh` runs for **every** command, `build` included — loads `quarkus/quarkus-application.dat` before it dispatches anything. With that file hidden by a fresh mount, `kc.sh build` cannot start.
 So the pod is:
 | container | mounts the build volume at | does | root filesystem |
 |---|---|---|---|
 | `<release>-keycloak-seed` (init) | `/quarkus-build` | `cp -R /opt/keycloak/lib/quarkus/. /quarkus-build/` | **read-only** |
 | `<release>-keycloak-build` (init) | `/opt/keycloak/lib/quarkus` | `kc.sh build` | **read-only** |
 | `<release>-keycloak-deployment` | `/opt/keycloak/lib/quarkus` | `kc.sh start --optimized` | **read-only** |
 Which is why it is two init containers and not one with a writable root filesystem: **every container in this pod keeps the full hardening baseline.** There is no documented deviation to carry, and no Trivy suppression that would also have silenced the serving container.
 Three practical consequences:
 - **Build options exist in exactly one place** (the `keycloak.buildEnv` helper, which only the build container uses). The serving container passes `start --optimized` and **no** build option at all, so there is no second list that could drift. A build option's variable name in `keycloak.env` **fails the render**, naming the variable — because the failure it prevents is a crash loop whose cause is nowhere near its message.
 - **Changing `keycloak.db.vendor`, `keycloak.metrics.enabled` or `keycloak.features` re-runs the build** on the next pod start. That is correct and costs one init-container run; it migrates no data.
 - **A pod stuck in `Init` is a build problem**, and its log is one flag away:
  ```bash
  kubectl -n keycloak logs <pod> -c <release>-keycloak-build
  ```
 ## Configurable values
 ### Global
 | Key | Type | Default | Description |
 |---|---|---|---|
 | `scaleDown` | bool | `false` | Renders the Deployment with `replicas: 0` without uninstalling. Takes precedence over `keycloak.replicas`. Also **half of the minor-upgrade procedure** above. |
 ### `secrets`
 | Key | Type | Default | Description |
 |---|---|---|---|
 | `secrets.databaseSecretRef` | string | `null` | **Required always.** `op://` path of the item carrying `username` and `password`. |
 | `secrets.bootstrapAdminSecretRef` | string | `null` | **Required** while `keycloak.bootstrapAdmin.enabled` is true (the default). |
 ### `keycloak`
 | Key | Type | Default | Description |
 |---|---|---|---|
 | `keycloak.replicas` | int | `2` | An explicit `0` is allowed and is the per-workload way to stop the release. Above `1` the rolling-update restriction applies; below `2` no PDB is rendered. |
 | `keycloak.db.vendor` | string | `postgres` | **A build option** — it reaches `kc.sh build`, never the serving container. `dev-file` and `dev-mem` are rejected by the schema: they are Keycloak's development defaults, and an identity provider on an embedded database works until the pod moves. Only `postgres` is exercised by CI. |
 | `keycloak.db.host` | string | `null` | **Required.** Enforced by `required()` in the Deployment — a `null` would render an empty `KC_DB_URL_HOST` and a pod waiting on a connection it can never make. |
 | `keycloak.db.port` | int | `5432` | |
 | `keycloak.db.database` | string | `keycloak` | |
 | `keycloak.db.pool.initialSize` / `.minSize` | int | `2` / `2` | No upstream default. A few warm connections remove the first-request latency after an idle period. |
 | `keycloak.db.pool.maxSize` | int | `20` | **Deliberately not upstream's `100`**, and the sizes are **per replica**: two replicas at 100 would ask a default PostgreSQL (`max_connections = 100`) for twice what it has, and the failure arrives under load as *"too many clients already"* rather than at install time. Raise it together with the server's `max_connections`, and multiply by `keycloak.replicas` when you do. |
 | `keycloak.bootstrapAdmin.enabled` | bool | `true` | The temporary bootstrap administrator. **Switch it off once a permanent administrator exists** — see step 4 above. |
 | `keycloak.metrics.enabled` | bool | `false` | **A build option.** Off because of the *port*, not the data: the management interface is unauthenticated here, and metrics add realm and client names to a surface nothing authenticates. Switch it on together with whatever scrapes it. |
 | `keycloak.features` | list | `[]` | Extra feature flags (`--features`). **A build option.** A list and not a comma-separated string, so two entries cannot arrive as one unknown feature name — which Keycloak rejects at *build* time, inside an init container. |
 | `keycloak.cache.clusterName` | string | `null` | `--cache-embedded-cluster-name`; `null` means Keycloak's default `ISPN`. **Read [the trap](#a-shared-database-merges-two-clusters) before setting it.** |
 | `keycloak.ports.http` | int | `8080` | |
 | `keycloak.ports.management` | int | `9000` | `/health/*` and optionally `/metrics`, **unauthenticated**. All three probes address it **by name**, so renaming the port breaks them. In no Ingress, by design. |
 | `keycloak.ports.jgroups` | int | `7800` | Declared as a `containerPort` for the record; nothing serves it and no Service exposes it. `jdbc-ping` opens direct pod-to-pod connections on it. |
 | `keycloak.resources.limitFactor` | number | `3` | Memory and ephemeral-storage limits are computed as factor × request. |
 | `keycloak.resources.requests` | object | `cpu: 500m`, `memory: 700Mi`, `ephemeral-storage: 256Mi` | **700Mi × 3 = 2100Mi, and that number is load-bearing rather than tidy**: the image sizes the JVM heap relative to the container *limit* (`-XX:MaxRAMPercentage=70`), so the limit is what the heap is computed from. Upstream: *"you should always set the memory limit"*, *"at least 750 MB"*, and *"the recommended memory limit is 2 GB"*. Lowering the request shrinks the heap proportionally. |
 | `keycloak.build.resources` | object | `cpu: 500m`, `memory: 512Mi`, `ephemeral-storage: 256Mi`, factor `3` | The two **init** containers. Transient — they are gone before the serving container starts — but they count for **scheduling**: a pod's effective request is the maximum of its init containers' and the sum of its containers'. More CPU here shortens every pod start. |
 | `keycloak.livenessProbe` | object | `httpGet /health/live` on `management`, period 10s, timeout 5s, 3 failures | **Deliberately not `/health` or `/health/ready`** — see [the probes](#the-three-probes-use-three-different-paths-on-purpose). |
 | `keycloak.readinessProbe` | object | `httpGet /health/ready` on `management`, period 10s, timeout 5s, 3 failures | Includes the database, and must. |
 | `keycloak.startupProbe` | object | `httpGet /health/started` on `management`, period 10s, 60 failures (= **600s**) | Gates liveness and readiness. The budget covers a **first** start, which creates the whole schema through Liquibase while the second replica waits on the migration lock. The Quarkus augmentation is **not** inside it — that runs in an init container, which has no probe and no deadline. |
 | `keycloak.securityContext.pod` | object | non-root **1000**, `runAsGroup` **0**, `fsGroup` `1000`, `fsGroupChangePolicy: Always`, `seccompProfile: RuntimeDefault` | **uid 1000 / gid 0 is the image's requirement, not a free choice**: it declares `keycloak:x:1000:0` in `/etc/passwd`, puts that user in the **root group**, and copies the install tree with `--chown=1000:0`. The mounted build volume has to be writable by the same pair. |
 | `keycloak.securityContext.container` | object | `readOnlyRootFilesystem: true`, all caps dropped, no privilege escalation, non-root | Applied to **every** container in the pod, both init containers included. No capability is added and none is needed: 8080, 9000 and 7800 are all above 1024 and the entrypoint is a shell script calling `java`. |
 | `keycloak.podDisruptionBudget.enabled` | bool | `true` | |
 | `keycloak.podDisruptionBudget.minAvailable` | int | `1` | **Rendered only while the computed replica count is `>= 2`** — see [the PDB trap](#a-pdb-with-one-replica-blocks-every-drain). |
 | `keycloak.env` | list | `[]` | Extra environment variables, rendered through `tpl`; `value`, `secretKeyRef` and `configMapKeyRef` forms. **Additive, not overriding** — see [§10](Chart-Style-Guide#10-environment-variables). **A build option's variable name is refused outright** (`KC_DB`, `KC_HEALTH_ENABLED`, `KC_METRICS_ENABLED`, `KC_FEATURES`, …). Useful entries: `JAVA_OPTS_APPEND`, `KC_LOG_LEVEL`, `KC_SPI_*`. |
 ### `ingress`
 | Key | Type | Default | Description |
 |---|---|---|---|
 | `ingress.enabled` | bool | `true` | Gates the **public** Ingress object only. |
 | `ingress.domain` | string | `null` | **Required even with `enabled: false`** — [§12.1](Chart-Style-Guide#12-values-that-must-not-lie) checked in both directions: it is read a second time as `KC_HOSTNAME`, and Keycloak refuses to start in production mode without a hostname (`hostname-strict` defaults to `true`). The chart renders it as the **full URL** `https://<domain>`, which upstream requires as soon as `hostname-admin` is in play. |
 | `ingress.clusterIssuer` | string | `null` | Required while **either** Ingress is enabled. Read only in the Ingress template, so switching both off frees it. |
 | `ingress.className` | string | `nginx` | |
 | `ingress.annotations` | object | `{}` | Merged with the cert-manager annotation the chart always sets. |
 | `ingress.admin.enabled` | bool | `true` | The administration console on a hostname of its own. **Not an access control** — see [the first trap](#the-admin-hostname-is-not-an-access-control). |
 | `ingress.admin.domain` | string | `null` | Required while enabled. Also rendered as `https://<domain>`. |
 | `ingress.admin.className` | string | `null` | Falls back to `ingress.className`. An **internal** ingress class here is the useful half of the admin hostname. |
 | `ingress.admin.annotations` | object | `{}` | Where an IP allow-list would go — with the caveat below firmly in mind. |
 TLS secrets: `tls-<release>-keycloak-ingress` and `tls-<release>-keycloak-admin-ingress`. Two separate certificates, because these are two hostnames that may well sit on two different ingress classes — the shared-secret form in the style guide is for several objects on **one** host.
 ## Traps
 ### The admin hostname is not an access control
 `hostname-admin` decides **which URLs the server generates** — redirects, the CSS and JS the console loads, and the API address the console itself calls. It closes nothing. Upstream, verbatim:
 > Using the `hostname-admin` option does not prevent accessing the Administration REST API endpoints via the frontend URL specified by the `hostname` option. If you want to restrict access to the Administration REST API, **you need to do it on the reverse proxy level**. Administration Console implicitly accesses the API using the URL as specified by the `hostname-admin` option.
 So `/admin/realms/...` stays reachable through `ingress.domain`, with or without the second hostname, and **an IP allow-list on the admin Ingress buys nothing on its own.** Whoever wants the administration API restricted does it on the reverse proxy in front of the **public** host.
 **The chart renders no deny rule for `/admin`, and that is a decision.** Two reasons, the second of which settled it: the console loads assets and performs redirects across more paths than `/admin`, so a rule written from the path name breaks it in ways that only show up in a browser; and a rule nobody has tested against a running console is a measure that merely looks like one, which is worse than an absent one.
 What the second hostname **is** good for: the console's URLs are generated for it, so it can live on an internal ingress class or in a different DNS zone, and operators reach it by a name that is not the public login name.
 ### A shared database merges two clusters
 Node discovery runs through the database, so — upstream — *"clusters that share the same database but should remain separate **will merge** unless they have distinct cluster names. The default cluster name is `ISPN`."* Two releases of this chart pointed at the same database therefore form **one** Infinispan cluster, silently, with no error on either side.
 If that is your situation, the first question is whether they should share a database at all — separate databases need no cluster name.
 `keycloak.cache.clusterName` exists for the case where they must, and it is **not** set per release by default even though that is the obvious fix: the same upstream page warns about the other direction — *"As invalidation of caches between clusters with distinct names will not work, this will lead to stale caches and incorrect behavior. Use only together with the `stateless` feature."* That warning is not unambiguous about which configuration it applies to, and a chart default that trades a documented merge for a possibly undocumented invalidation gap is not an improvement. So: the value is here, the trap is written down, and the default is upstream's.
 ### The management port is unauthenticated
 Port 9000 carries `/health`, `/health/live`, `/health/ready`, `/health/started` and — with `keycloak.metrics.enabled` — `/metrics`, and it carries them without authentication. It has a ClusterIP Service of its own (`<release>-keycloak-management-service`) for in-cluster scraping, and it is in **no** Ingress.
 **It must never get one, and no value should be added that would allow it.** An unauthenticated readiness endpoint on the internet tells an attacker when the database is down; `/metrics` tells them the realm and client names. The Service is separate from the serving Service precisely so that naming it is a deliberate act — on one combined Service, publishing it would be one wrong port name in an Ingress backend away.
 Upstream's own alternative, `legacy-observability-interface=true`, moves health back onto the serving port where the Ingress already is. It is explicitly *"not recommended for security reasons"* and this chart does not offer it.
 ### The three probes use three different paths on purpose
 | Probe | Path | Contains a database check? |
 |---|---|---|
 | liveness | `/health/live` | **no** |
 | readiness | `/health/ready` | **yes** |
 | startup | `/health/started` | — |
 This is [§12.3](Chart-Style-Guide#12-values-that-must-not-lie) in the one shape where upstream has already done the work: pointing liveness at `/health/ready` — or at the aggregate `/health`, which also includes the database — restarts **every pod of the cluster** for as long as the database is away. A restart cannot fix a database, so that turns a degraded login service into none at all.
 The wrong path would look entirely plausible in a diff, which is why the reason sits on the helper and not only here. And the other half of the rule holds too: readiness **must** hang on the database, because a node that cannot reach it answers every login with an error.
 ### A PDB with one replica blocks every drain
 `minAvailable: 1` with exactly **one** replica forbids every voluntary eviction, so `kubectl drain` blocks forever on a pod nothing is allowed to move — and the message names the PodDisruptionBudget rather than the replica count, which sends the reader to the wrong file.
 The chart therefore renders the PDB only while the **computed** replica count is `>= 2`, which also covers `scaleDown: true`, where there are no pods and the object would describe a promise about nothing. `NOTES.txt` prints which of the two happened, so it does not have to be inferred from `kubectl get pdb`.
 ### There is no ConfigMap, so there is no `checksum/config`
 That is not an omission. Keycloak is configured entirely through options and `KC_*` variables, so no file is rendered, mounted or hashed, and the [§9](Chart-Style-Guide#9-config-changes-must-restart-the-workload) rule has nothing to attach to. `values.yaml` states it at the top, because an absent annotation looks identical to a forgotten one.
 What **is** covered is the `op://` **path** of the two vault items, through `checksum/secret-ref`.
 ### A change inside a vault item restarts nothing
 Helm never sees it. After rotating a field:
 ```bash
 kubectl -n keycloak rollout restart deployment/keycloak-keycloak-deployment
 ```
 Repointing a `secrets.*SecretRef` at a *different* item **does** roll the pods — that is what `checksum/secret-ref` covers.
 The 1Password operator's `auto-restart` annotation is **not** set, and here that is a decision rather than a constraint: unlike the StatefulSet charts in this repo it would actually work, because the operator enumerates Deployments. But its trigger is the item's **version counter** and not its content, so a re-save with identical values rolls the whole cluster. For an identity provider, an unannounced rolling restart because somebody opened a vault entry is worse than a deliberate one. [§9](Chart-Style-Guide#9-config-changes-must-restart-the-workload) has the precedence trap if it is ever set.
 ### `jdbc-ping` needs pod-to-pod reachability, and no NetworkPolicy knows that yet
 Discovery runs through the database, but once the peers are known JGroups opens **direct** connections between pod IPs on port 7800. No Service mediates that, headless or otherwise — the replicas simply need L3 reachability to each other.
 This chart renders **no** NetworkPolicy. If one is ever added, it has to allow 7800 between the replicas, or the cluster quietly becomes two islands with two green pods.
 ## Upgrades and backup
 - **Patch updates within a minor line** are the normal case and arrive as Renovate app-version PRs. Run the compatibility check anyway — see the unresolved doc contradiction in [Minor upgrades](#minor-upgrades-need-the-cluster-down).
 - **Minor and major jumps** need the procedure in that section. The automerge does not know about it.
 - **The backup is the database.** There is no PVC and nothing on disk to snapshot: realms, clients, users, keys and sessions are all in PostgreSQL. A Keycloak backup is a database backup plus the values file, and restoring means restoring both — the realm keys live in the database, so a database restored without them signs nothing a client will accept.
 ## CI
 `ci/keycloak/` carries `test-values.yaml`, `fixtures.yaml` (throwaway PostgreSQL plus the two Secrets the operator would materialize) and `settings.env` (`INSTALL_TIMEOUT=900`).
 **The install-test runs with two replicas**, which is the single most valuable line in that fixture: the script waits for the Deployment's rollout, and a rollout is complete only when **both** pods are Ready — so cluster formation through `jdbc-ping` is inside the assertion rather than beside it. A one-replica test would be green without ever showing whether the second node is found, which is the mistake [§CI](CI-CD) was taught once already.
 It also closes three things a render cannot: that the two init containers work at all, that `readOnlyRootFilesystem: true` holds for the serving container with emptyDirs only at the three paths it needs, and that `start --optimized` accepts the pre-built installation instead of re-augmenting onto a read-only path. `metrics.enabled: true` in the fixture so the build container must produce an installation differing from the published image in **two** options rather than one — with metrics off, a build that silently did nothing would look identical.
 **What CI cannot prove**: that `hostname-admin` serves the console and its resources cleanly on the second hostname. No ingress controller answers in the test cluster and nothing drives a browser. What it does check is that both Ingress objects are accepted, that the Service port they name exists, and that Keycloak accepts both hostname values as full URLs at start.
 ## See also
 - [Chart style guide](Chart-Style-Guide) — §11 for the PodDisruptionBudget convention this chart introduced
 - [Renovate](Renovate) — why this chart needs no version constraint, and the automerge decision
 - [Lessons learned](Lessons-Learned) — *The upgrade procedure the automerge did not know about* and *The application that rewrites its own installation at start*
 - [CI/CD](CI-CD)