fix(keycloak): readiness could not fail on the database, and the measured budget
Two corrections found while verifying the first CI run, not from the plan. THE DEFAULT THAT MADE THE READINESS PROBE TOOTHLESS. Keycloak's DATABASE health check is the one row of upstream's "Available Checks" table whose "Requires Metrics" column says yes; Cluster, Graceful shutdown and Keycloak Initialized say no. So with metrics-enabled=false - this chart's default until now - /health/ready answers 200 while the database is gone, and the pod keeps taking logins it cannot serve. The probe path was right and the aggregate behind it contained nothing external, which is the urlaubsverwaltung defect (#218) in a new hiding place: not a health group this time, but an option named for observability, where nobody looks when reviewing a probe. keycloak.metrics.enabled therefore defaults to TRUE, and its comment argues about readiness rather than about scraping. The price is stated at the value: /metrics then answers on the management port, unauthenticated, carrying realm names, client ids and login counters - on a port that is ClusterIP-only and in no Ingress. Switching it off is now documented as a decision about whether readiness can fail at all. The CI fixture keeps it on and says why, so a green run really does touch the database. THE STARTUP BUDGET NOW CARRIES ITS MEASUREMENT. From the install-test of PR #87 on the Gitea docker runner (k3s v1.35.5, two replicas, empty database, image pulled during the run): 71.8 seconds from `helm install` to BOTH pods Ready - first at ~70.8s, second one second later, 0 restarts. The 600s budget stays, with the reason for the headroom written down: everything in that run was the cheap case, and a first install against a real database serializes the second replica behind the first's Liquibase migration lock instead of letting it arrive a second later. AND AN OVERCLAIM OF MY OWN, REMOVED. ci/keycloak/test-values.yaml said a green run proves jdbc-ping found the second replica. It does not: readiness reports one node's own health, so a node that never found its peer is Ready, healthy and alone - and two such nodes are two islands that each believe they are the cluster. What the test does rule out is the louder half, a replica that cannot start or cannot share the database. The Infinispan view has to be read out of the log, which is what step 2 of NOTES.txt tells the operator to do, now with the note that no probe and no install-test replaces that step. Measurement point 4 of the plan therefore stays open. Part of #83
This commit is contained in:
1 parent
4c1cff1744
commit
320257c742
6 files changed
+113
-40
No files matched your search
@@ -24,20 +24,32 @@
|
||||
# re-augment. A disagreement between the build options and the serving
|
||||
# container would show up here as a crash loop rather than as a slow
|
||||
# start, because the path it would want to rewrite is read-only.
|
||||
# 4. jdbc-ping FINDS THE SECOND REPLICA from cache=ispn and the database
|
||||
# configuration alone - which is why this test runs TWO replicas (see
|
||||
# below) and why that is the single most valuable line in this file.
|
||||
# 4. TWO REPLICAS BOTH COME UP AND STAY UP against one database - the
|
||||
# rollout is complete only when BOTH pods are Ready, so a second replica
|
||||
# that crashed, deadlocked on the schema migration lock or fought the
|
||||
# first over the database would fail the run. Measured in Gitea PR #87:
|
||||
# 71.8s from `helm install` to both Ready, 0 restarts.
|
||||
# Plus the mechanical half: the Secret key names match what the Deployment
|
||||
# references, /health/started, /health/live and /health/ready all answer on
|
||||
# the management port (so health-enabled really was baked in by the init
|
||||
# container), the resource quantities are valid, and the schema accepts a
|
||||
# realistic non-default values file.
|
||||
#
|
||||
# WHAT IT DOES NOT PROVE, and no install-test here could:
|
||||
# WHAT IT DOES NOT PROVE, and the first entry is the one not to misread:
|
||||
# - THAT jdbc-ping ACTUALLY FOUND THE PEER. Two replicas both reaching Ready
|
||||
# is NOT that proof, and believing it was is the mistake this paragraph
|
||||
# exists to prevent: readiness reports this node's own health, so a node
|
||||
# that never found its peer is Ready, healthy and alone - and two such
|
||||
# nodes are two islands that each believe they are the cluster. The
|
||||
# Infinispan view has to be read out of the log, which is what step 2 of
|
||||
# NOTES.txt tells the operator to do. So measurement point 4 of the plan
|
||||
# stays open after this test; what the test does rule out is the louder
|
||||
# half (a second replica that cannot start or cannot share the database).
|
||||
# - that hostname-admin serves the console cleanly on a second hostname. No
|
||||
# ingress controller answers in CI and nothing drives a browser, so the
|
||||
# chart's own measurement point 5 stays open. What IS checked is that both
|
||||
# Ingress objects are accepted and that the Service port they name exists.
|
||||
# ingress controller answers in CI and nothing drives a browser, so
|
||||
# measurement point 5 stays open too. What IS checked is that both Ingress
|
||||
# objects are accepted, that the Service port they name exists, and that
|
||||
# Keycloak accepts both hostname values as full URLs at start.
|
||||
# - anything about an upgrade. The rolling-update restriction between minor
|
||||
# versions is a property of two versions meeting, and this test installs
|
||||
# one.
|
||||
@@ -75,12 +87,16 @@ keycloak:
|
||||
minSize: 1
|
||||
maxSize: 5
|
||||
|
||||
# ON against the chart's default of off, because it is a BUILD option: with
|
||||
# this true the init container has to build an installation that differs from
|
||||
# the published image in TWO options rather than one, and /metrics has to
|
||||
# appear on the management port. A green run therefore proves the build
|
||||
# arguments really reach `kc.sh build` - with metrics off, a build that
|
||||
# silently did nothing would look identical.
|
||||
# The chart default, spelled out because it is load-bearing for what this
|
||||
# test means: metrics-enabled is what puts the DATABASE check into
|
||||
# /health/ready (upstream's check table, "Requires Metrics: Yes"). With it
|
||||
# off, readiness would pass without ever touching the database, and a green
|
||||
# run would say nothing about the connection at all.
|
||||
#
|
||||
# It is also a BUILD option, so with it true the init container has to build
|
||||
# an installation differing from the published image in three options rather
|
||||
# than two - which is what makes a build that silently did nothing fail the
|
||||
# run instead of looking identical.
|
||||
metrics:
|
||||
enabled: true
|
||||
|
||||
|
||||
Reference in new issue
Block a user