fix(keycloak): readiness could not fail on the database, and the measured budget
Validate Helm Charts / Detect changed charts (pull_request) Successful in 3s
Validate Helm Charts / Resource quantities (pull_request) Successful in 9s
Validate Helm Charts / Build keycloak (pull_request) Successful in 4s
Validate Helm Charts / Trivy config keycloak (pull_request) Successful in 11s
Install-test / Install-test keycloak (pull_request) Successful in 2m10s
Validate Helm Charts / Install-test (pull_request) Successful in 2m10s

Two corrections found while verifying the first CI run, not from the plan.

THE DEFAULT THAT MADE THE READINESS PROBE TOOTHLESS. Keycloak's DATABASE health
check is the one row of upstream's "Available Checks" table whose "Requires
Metrics" column says yes; Cluster, Graceful shutdown and Keycloak Initialized
say no. So with metrics-enabled=false - this chart's default until now -
/health/ready answers 200 while the database is gone, and the pod keeps taking
logins it cannot serve. The probe path was right and the aggregate behind it
contained nothing external, which is the urlaubsverwaltung defect (#218) in a
new hiding place: not a health group this time, but an option named for
observability, where nobody looks when reviewing a probe.

keycloak.metrics.enabled therefore defaults to TRUE, and its comment argues
about readiness rather than about scraping. The price is stated at the value:
/metrics then answers on the management port, unauthenticated, carrying realm
names, client ids and login counters - on a port that is ClusterIP-only and in
no Ingress. Switching it off is now documented as a decision about whether
readiness can fail at all. The CI fixture keeps it on and says why, so a green
run really does touch the database.

THE STARTUP BUDGET NOW CARRIES ITS MEASUREMENT. From the install-test of PR #87
on the Gitea docker runner (k3s v1.35.5, two replicas, empty database, image
pulled during the run): 71.8 seconds from `helm install` to BOTH pods Ready -
first at ~70.8s, second one second later, 0 restarts. The 600s budget stays,
with the reason for the headroom written down: everything in that run was the
cheap case, and a first install against a real database serializes the second
replica behind the first's Liquibase migration lock instead of letting it
arrive a second later.

AND AN OVERCLAIM OF MY OWN, REMOVED. ci/keycloak/test-values.yaml said a green
run proves jdbc-ping found the second replica. It does not: readiness reports
one node's own health, so a node that never found its peer is Ready, healthy
and alone - and two such nodes are two islands that each believe they are the
cluster. What the test does rule out is the louder half, a replica that cannot
start or cannot share the database. The Infinispan view has to be read out of
the log, which is what step 2 of NOTES.txt tells the operator to do, now with
the note that no probe and no install-test replaces that step. Measurement
point 4 of the plan therefore stays open.

Part of #83
This commit is contained in:
jklein committed 2026-10-04 17:17:09 +00:00
1 parent f784f40fd5
commit 63d0c7fed0
6 files changed
+113 -40

No files matched your search

+14 -3
View File
@@ -52,11 +52,14 @@ FIRST INSTALL - AND STEP 4 IS MANDATORY, NOT ADVICE
creates the entire schema through Liquibase, and with {{ $replicas }} replica(s)
the second waits on the first's migration lock.
2. Check that the cluster really formed - this is what the second replica is
for, and a silent failure here leaves two independent islands:
2. Check that the cluster really formed. THIS STEP IS NOT OPTIONAL AND NO
PROBE REPLACES IT: a node that never found its peer is Ready, healthy and
alone, and two such nodes are two islands that each believe they are the
cluster. The install-test cannot catch it either.
kubectl -n {{ .Release.Namespace }} logs deployment/{{ $deploy }} | grep -i "ISPN000094\|view accepted\|members"
Look for a view with {{ $replicas }} members. One member per pod means
jdbc-ping did not find the peer.
jdbc-ping did not find the peer - check that both pods can reach each
other on port {{ $settings.ports.jgroups | int64 }} and that they really share one database.
3. Log in at {{ if $adminHostname }}{{ include "keycloak.adminUrl" . }}/admin{{ else if .Values.ingress.enabled }}{{ include "keycloak.frontendUrl" . }}/admin{{ else }}the in-cluster address, port-forwarded{{ end }}
with the bootstrap credentials from the vault item above.
@@ -140,6 +143,14 @@ TRAPS WORTH KNOWING BEFORE YOU NEED THEM
unset. Read its comment in values.yaml before setting it: upstream also
warns about cache invalidation between differently-named clusters.{{ end }}
- READINESS ONLY CHECKS THE DATABASE WHILE metrics-enabled IS TRUE. Upstream's
table of health checks has a "Requires Metrics" column and the DATABASE
check is the row that says yes. So with keycloak.metrics.enabled false,
/health/ready answers 200 with the database gone and the pod keeps taking
logins it cannot serve. This release has it {{ if $settings.metrics.enabled }}ON, which is the chart default
and the reason the default is what it is.{{ else }}OFF, so the readiness probe
cannot currently detect a database outage.{{ end }}
- THERE IS NO ConfigMap IN THIS CHART, so no checksum/config annotation. That
is not an omission: Keycloak is configured entirely through options and
KC_* variables, so there is no config file whose change could fail to