keycloak: extraVolumes/extraVolumeMounts, the theme route and its trap (Gitea #88)

Lessons-Learned.md

@@ -438,6 +438,18 @@ What makes it worth an entry is the failure mode rather than the mechanics. The
 
 **Now:** write the mode you actually need rather than the strictest one, and write down why the group bit is there — a mode that only works because the platform silently widens it is a mode the next hardening pass will narrow again. And when a chart hands a secret to an application as a *file*, find out what that application does when the read fails: if `EACCES` and `ENOENT` collapse into the same branch, a permission problem will present as a disabled feature, not as an error.
 
+## The mount that would have arrived after the build it was for
+
+[keycloak](keycloak) got a generic `extraVolumes` / `extraVolumeMounts` pair (Gitea #88), and the plan justified the *generic* shape — rather than a theme-specific key — with the cases that follow: extra root CAs and **SPI extensions**. The naive implementation renders the mounts on the serving container, which is where every chart in this repo mounts things. For a theme that is right. For an extension it is wrong, and wrong in the quietest available way.
+
+This chart runs `start --optimized`, so the Quarkus augmentation happens **before** the serving container exists, in a `build` init container. Upstream on adding a provider JAR: *"After that if you are using `--optimized`, you must run the `build` command in order to update the server's provider registry with the implementations from the JAR file."* A JAR that is mounted only on the serving container is **not there when the registry is written**. Nothing fails: the build succeeds, the pod becomes Ready, the file is present at runtime, and the extension simply does not exist. No error, no warning, nothing to grep for.
+
+So the mounts are rendered on the `build` init container **and** on the serving container, from **one** list — two lists would be two chances to put an extension in the wrong one, and the wrong one is silent. The price is named at the value: an extra volume is then present at build time too, which is harmless for a theme and worth knowing for a Secret.
+
+What generalizes is not the Keycloak detail but the question the plan did not ask: **where does the thing that consumes this file run?** Any chart that pre-builds, pre-compiles, pre-indexes or augments in an init container has a second consumer of its volumes, and the two consumers want different things — the runtime one wants the file *while serving*, the build one wants it *while building*. A values key whose documentation names only one of them will be used for both.
+
+**Now:** before adding a pass-through volume pair to a chart with a pre-build step, enumerate every container that could legitimately need the file, and render the mounts at each of them. Write the reason **at the value**, not only in the pull request: an occurrence that looks redundant is exactly what the next round removes while tidying up, and this one cannot be noticed by any test that checks whether the pod is Ready. And where a pass-through value only *places* a file, say so at the value — mounting a CA does not add it to a truststore, and mounting a theme does not select it — because the next reader's expectation is set by the use case the key was introduced for.
+
 ## The recurring shape
 
 - **Green is a claim about the assertions that ran, not about the system.**
@@ -469,4 +481,5 @@ What makes it worth an entry is the failure mode rather than the mechanics. The
 - **A write barrier reaches as far as the helper that implements it.** "Read-only configuration" also catches the bootstrap writes that happen once, on a first install, on the path no test walks — including the one that creates the credential for the door you kept open on purpose.
 - **Check that the platform can make the call before designing around it.** An HTTP probe is a GET; anything else is an `exec` probe, and an `exec` probe is a claim about which binaries the image still contains.
 - **The strictest file mode is not the correct one.** A mounted secret is owned by `root`, `fsGroup` supplies only the group, and an application that treats `EACCES` like "absent" turns a permission mistake into a feature that quietly switched itself off.
+- **Ask where the consumer of a file runs, not only where the application runs.** A chart with a pre-build step has two consumers of its volumes at two different times, and a mount that reaches only the serving one fails without a symptom.
 - **When the fix needs a privilege, ask whether two cheap steps can replace one privileged one.** Splitting a pre-build into a seeding container plus a build container kept the hardening baseline intact across the whole pod, where the shorter single-container version would have bought a documented deviation *and* a suppression that silences the container that actually serves traffic.

keycloak.md

@@ -172,6 +172,63 @@ Three practical consequences:
   kubectl -n keycloak logs <pod> -c <release>-keycloak-build
   ```
 
+## Getting a file into the pod: themes, providers, extra CAs
+
+There is no PVC and no ConfigMap in this chart, so until Gitea #88 nothing could put a file into this pod at all. `keycloak.extraVolumes` and `keycloak.extraVolumeMounts` are the generic way: both lists go verbatim into the pod spec, so a volume may be a ConfigMap, a Secret, a PVC or an `emptyDir` filled by an init container of your own. The chart does not decide how a theme is delivered.
+
+**The mounts are rendered twice — on the `build` init container *and* on the serving container.** One list, two places, because the two cases resolve at different times:
+
+- A **theme** is read at runtime, so the serving container is where it takes effect — even though that container serves a pre-built installation.
+- An **SPI provider** is a build-time concern. Upstream, on dropping a JAR into `providers/`: *"After that if you are using `--optimized`, you must run the `build` command in order to update the server's provider registry with the implementations from the JAR file."* That build is the init container above. **A JAR mounted only on the serving container is absent while the registry is written** — it is never registered, the file sits there at runtime, and nothing reports it.
+
+Not on the `seed` init container: it has a different layout (`/quarkus-build`) and runs no Keycloak command.
+
+The side effect of the second place, which belongs stated rather than discovered: **an extra volume is present at build time too.** For a theme that is without consequence; for a Secret it means an init container of this pod sees it as well.
+
+### It places files; it configures nothing
+
+- **An extra root CA is not trusted by being mounted.** A file in the pod is in no truststore. Keycloak reads one only from `truststore-paths`, so the mount needs `KC_TRUSTSTORE_PATHS` pointed at the mounted **directory**. That is a *runtime* option (26.8.0 option table, no build icon), so it goes into `keycloak.env` and the build-option guard does not refuse it.
+- **A theme is not activated by being mounted, and this chart cannot activate it.** The selection lives **per realm** in Keycloak's database — *Realm settings → Themes → Login theme*. Mounting makes a theme selectable, never selected.
+
+### A custom login theme
+
+The reason the issue was opened: Keycloak has **no logo field on an identity provider**. The icon on its login button comes only from a login theme, through `kcLogoIdP-<alias>` in that theme's `theme.properties`.
+
+```yaml
+keycloak:
+  extraVolumes:
+    - name: login-theme
+      configMap:
+        name: my-login-theme
+        # A theme's layout is <theme>/<type>/theme.properties, and a ConfigMap
+        # key may not contain a slash — so the subdirectory comes from
+        # items[].path, which may.
+        items:
+          - key: theme.properties
+            path: login/theme.properties
+  extraVolumeMounts:
+    - name: login-theme
+      # NOT /opt/keycloak/themes — see the trap below.
+      mountPath: /opt/keycloak/themes/my-theme
+      readOnly: true
+```
+
+with a `theme.properties` of
+
+```properties
+parent=keycloak.v2
+kcLogoIdP-my-provider = fa fa-lock
+```
+
+and then *Realm settings → Themes → Login theme → `my-theme`* plus a pod restart.
+
+Four properties of this route that cost time if they are not known:
+
+1. **A ConfigMap carries binary content only in `binaryData`** (base64). An SVG is text and fits in `data`; a PNG does not. The **1 MiB** limit per ConfigMap is ample for a theme with SVG icons and not enough for bundled web fonts — those belong in a PVC or an image of your own.
+2. **Themes are cached, and in production that is wanted.** A theme change therefore takes effect only after a pod restart. Two things add up: this chart hashes no user-supplied ConfigMap either (it renders none of its own, so there is no `checksum/config`), so editing the theme ConfigMap restarts nothing by itself. Use `kubectl -n keycloak rollout restart deployment/<release>-keycloak-deployment`. Do not configure the cache away to avoid this.
+3. **The realm selection is not the chart's** — see above.
+4. **A custom login theme can make the identity-provider buttons disappear entirely** when its template does not include the social fragment correctly (reported repeatedly upstream, `keycloak/keycloak#42711`). After switching a realm's theme, **look at the login page** — a healthy pod says nothing about whether the buttons are still there.
+
 ## Configurable values
 
 ### Global
@@ -215,7 +272,9 @@ Three practical consequences:
 | `keycloak.securityContext.container` | object | `readOnlyRootFilesystem: true`, all caps dropped, no privilege escalation, non-root | Applied to **every** container in the pod, both init containers included. No capability is added and none is needed: 8080, 9000 and 7800 are all above 1024 and the entrypoint is a shell script calling `java`. |
 | `keycloak.podDisruptionBudget.enabled` | bool | `true` | |
 | `keycloak.podDisruptionBudget.minAvailable` | int | `1` | **Rendered only while the computed replica count is `>= 2`** — see [the PDB trap](#a-pdb-with-one-replica-blocks-every-drain). |
-| `keycloak.env` | list | `[]` | Extra environment variables, rendered through `tpl`; `value`, `secretKeyRef` and `configMapKeyRef` forms. **Additive, not overriding** — see [§10](Chart-Style-Guide#10-environment-variables). **A build option's variable name is refused outright** (`KC_DB`, `KC_HEALTH_ENABLED`, `KC_METRICS_ENABLED`, `KC_FEATURES`, …). Useful entries: `JAVA_OPTS_APPEND`, `KC_LOG_LEVEL`, `KC_SPI_*`. |
+| `keycloak.env` | list | `[]` | Extra environment variables, rendered through `tpl`; `value`, `secretKeyRef` and `configMapKeyRef` forms. **Additive, not overriding** — see [§10](Chart-Style-Guide#10-environment-variables). **A build option's variable name is refused outright** (`KC_DB`, `KC_HEALTH_ENABLED`, `KC_METRICS_ENABLED`, `KC_FEATURES`, …). Useful entries: `JAVA_OPTS_APPEND`, `KC_LOG_LEVEL`, `KC_SPI_*`, and `KC_TRUSTSTORE_PATHS` for a CA mounted through `keycloak.extraVolumes`. |
+| `keycloak.extraVolumes` | list | `[]` | Additional pod volumes, rendered **verbatim** — any Kubernetes volume source. The way a file gets into this pod at all; see [Getting a file into the pod](#getting-a-file-into-the-pod-themes-providers-extra-cas). It **places files and configures nothing**: a CA still needs `KC_TRUSTSTORE_PATHS`, a theme still needs to be selected per realm. |
+| `keycloak.extraVolumeMounts` | list | `[]` | Additional mounts, rendered verbatim and **twice — on the `build` init container as well as on the serving container**. Deliberate: a theme is read at runtime, but an SPI provider JAR must be present while `kc.sh build` writes the provider registry, or it is never registered and nothing reports it. **Never mount on `/opt/keycloak/themes` itself** — see [the trap](#a-volume-on-optkeycloakthemes-hides-the-bundled-themes). |
 
 ### `ingress`
 
@@ -298,6 +357,14 @@ The wrong path would look entirely plausible in a diff, which is why the reason
 
 The chart therefore renders the PDB only while the **computed** replica count is `>= 2`, which also covers `scaleDown: true`, where there are no pods and the object would describe a promise about nothing. `NOTES.txt` prints which of the two happened, so it does not have to be inferred from `kubectl get pdb`.
 
+### A volume on `/opt/keycloak/themes` hides the bundled themes
+
+`/opt/keycloak/themes` is **not empty in the image** — `base`, `keycloak` and `keycloak.v2` live there. A volume mounted at that path hides all three, and the login page is then **broken rather than merely unstyled**: the pod stays Ready, because `/health/*` knows nothing about themes.
+
+Mount on a **subpath that does not exist in the image**, `/opt/keycloak/themes/<name>`, which hides nothing.
+
+This is the same mechanism as the other path this chart deliberately leaves alone: **`/opt/keycloak/conf` is never mounted**, because it carries `keycloak.conf` and — more importantly — the `cache-ispn.xml` that `cache=ispn` reads. An `emptyDir` there would bring the cluster up with no cache configuration. Same class of mistake, and worth learning once: **before mounting into an upstream image, check whether the directory is empty in it.**
+
 ### There is no ConfigMap, so there is no `checksum/config`
 
 That is not an omission. Keycloak is configured entirely through options and `KC_*` variables, so no file is rendered, mounted or hashed, and the [§9](Chart-Style-Guide#9-config-changes-must-restart-the-workload) rule has nothing to attach to. `values.yaml` states it at the top, because an absent annotation looks identical to a forgotten one.
@@ -330,7 +397,7 @@ This chart renders **no** NetworkPolicy. If one is ever added, it has to allow 7
 
 ## CI
 
-`ci/keycloak/` carries `test-values.yaml`, `fixtures.yaml` (throwaway PostgreSQL plus the two Secrets the operator would materialize) and `settings.env` (`INSTALL_TIMEOUT=900`).
+`ci/keycloak/` carries `test-values.yaml`, `fixtures.yaml` (throwaway PostgreSQL, the two Secrets the operator would materialize, and a minimal login-theme ConfigMap) and `settings.env` (`INSTALL_TIMEOUT=900`).
 
 **The install-test runs with two replicas**, and the rollout is complete only when **both** pods are Ready — so a second replica that cannot start, deadlocks on the schema migration lock or fights the first over the database fails the run. Measured in Gitea PR #87: **71.8 seconds** from `helm install` to both pods Ready, 0 restarts.
 
@@ -338,11 +405,15 @@ This chart renders **no** NetworkPolicy. If one is ever added, it has to allow 7
 
 It also closes three things a render cannot: that the two init containers work at all, that `readOnlyRootFilesystem: true` holds for the serving container with emptyDirs only at the three paths it needs, and that `start --optimized` accepts the pre-built installation instead of re-augmenting onto a read-only path. And because `health-enabled` and `metrics-enabled` are build options, the fact that all three probe paths answer on port 9000 **is** the proof that the build container really did its work — a build that silently did nothing would leave them 404.
 
+**The install-test mounts a theme** (Gitea #88), on the subpath `/opt/keycloak/themes/ci-theme`, so the extra-volume path is walked rather than only rendered. A green run closes three things: a mount on a subpath of a directory that *does* exist in the image is accepted alongside `readOnlyRootFilesystem: true` (the runtime has to create the mount point in a filesystem that is about to be read-only); `kc.sh build` still succeeds with the extra mount present, which is the half that matters because the mounts are rendered on the build container too; and `start --optimized` comes up with a runtime theme in place — the plan of #88 called that *"a reasoned expectation, not verified"*, and this is the verification.
+
+**What the theme fixture does *not* prove**, and it would be easy to overclaim: that the **login page renders** and that `base`, `keycloak` and `keycloak.v2` are still listed. The health endpoints answer regardless of the themes directory, and nothing in this harness makes an HTTP request to `/` or runs a command in the pod. That the subpath hides nothing is a property of the mount path, not a measurement — and `kcLogoIdP-<alias>` needs a browser plus a realm setting, neither of which exists in CI.
+
 **What CI cannot prove**: that `hostname-admin` serves the console and its resources cleanly on the second hostname. No ingress controller answers in the test cluster and nothing drives a browser. What it does check is that both Ingress objects are accepted, that the Service port they name exists, and that Keycloak accepts both hostname values as full URLs at start.
 
 ## See also
 
 - [Chart style guide](Chart-Style-Guide) — §11 for the PodDisruptionBudget convention this chart introduced
 - [Renovate](Renovate) — why this chart needs no version constraint, and the automerge decision
-- [Lessons learned](Lessons-Learned) — *The upgrade procedure the automerge did not know about* and *The application that rewrites its own installation at start*
+- [Lessons learned](Lessons-Learned) — *The upgrade procedure the automerge did not know about*, *The application that rewrites its own installation at start* and *The mount that would have arrived after the build it was for*
 - [CI/CD](CI-CD)