fix(keycloak): readiness could not fail on the database, and the measured budget
Two corrections found while verifying the first CI run, not from the plan. THE DEFAULT THAT MADE THE READINESS PROBE TOOTHLESS. Keycloak's DATABASE health check is the one row of upstream's "Available Checks" table whose "Requires Metrics" column says yes; Cluster, Graceful shutdown and Keycloak Initialized say no. So with metrics-enabled=false - this chart's default until now - /health/ready answers 200 while the database is gone, and the pod keeps taking logins it cannot serve. The probe path was right and the aggregate behind it contained nothing external, which is the urlaubsverwaltung defect (#218) in a new hiding place: not a health group this time, but an option named for observability, where nobody looks when reviewing a probe. keycloak.metrics.enabled therefore defaults to TRUE, and its comment argues about readiness rather than about scraping. The price is stated at the value: /metrics then answers on the management port, unauthenticated, carrying realm names, client ids and login counters - on a port that is ClusterIP-only and in no Ingress. Switching it off is now documented as a decision about whether readiness can fail at all. The CI fixture keeps it on and says why, so a green run really does touch the database. THE STARTUP BUDGET NOW CARRIES ITS MEASUREMENT. From the install-test of PR #87 on the Gitea docker runner (k3s v1.35.5, two replicas, empty database, image pulled during the run): 71.8 seconds from `helm install` to BOTH pods Ready - first at ~70.8s, second one second later, 0 restarts. The 600s budget stays, with the reason for the headroom written down: everything in that run was the cheap case, and a first install against a real database serializes the second replica behind the first's Liquibase migration lock instead of letting it arrive a second later. AND AN OVERCLAIM OF MY OWN, REMOVED. ci/keycloak/test-values.yaml said a green run proves jdbc-ping found the second replica. It does not: readiness reports one node's own health, so a node that never found its peer is Ready, healthy and alone - and two such nodes are two islands that each believe they are the cluster. What the test does rule out is the louder half, a replica that cannot start or cannot share the database. The Infinispan view has to be read out of the log, which is what step 2 of NOTES.txt tells the operator to do, now with the note that no probe and no install-test replaces that step. Measurement point 4 of the plan therefore stays open. Part of #83
This commit is contained in:
1 parent
4c1cff1744
commit
320257c742
6 files changed
+113
-40
No files matched your search
+51
-19
@@ -200,17 +200,33 @@ keycloak:
|
||||
# a credential the pod no longer needs.
|
||||
enabled: true
|
||||
|
||||
# Expose /metrics on the management port (metrics-enabled, a BUILD option -
|
||||
# it goes to `kc.sh build`, so changing it re-runs the init container).
|
||||
# metrics-enabled, a BUILD option - it goes to `kc.sh build`, so changing it
|
||||
# re-runs the init containers.
|
||||
#
|
||||
# OFF, and the reason is the port rather than the data: the management
|
||||
# interface is UNAUTHENTICATED in this chart's configuration, and switching
|
||||
# metrics on adds Keycloak's full metric set - realm names, client ids, login
|
||||
# and token counters - to a surface nothing authenticates. Switch it on
|
||||
# together with whatever scrapes it, and read
|
||||
# keycloak.ports.management below first.
|
||||
# ON, AND NOT FOR THE METRICS. This value is misnamed from our point of view:
|
||||
# it also decides whether /health/ready CONTAINS A DATABASE CHECK AT ALL.
|
||||
# Upstream's table of available checks has a "Requires Metrics" column, and
|
||||
# the Database check - "Returns the status of the database connection pool" -
|
||||
# is the one row that says YES. The Cluster, Graceful shutdown and Keycloak
|
||||
# Initialized checks say no.
|
||||
#
|
||||
# So with metrics-enabled=false, a pod whose database is gone still reports
|
||||
# Ready and keeps serving errors to every login. That is precisely the shape
|
||||
# urlaubsverwaltung cost this repo two years of false confidence (#218, and
|
||||
# README guideline 12.3): the probe path was right and the thing behind it
|
||||
# contained nothing external. Here the dependency hides behind an option
|
||||
# whose name says "observability".
|
||||
#
|
||||
# The price, which is real and is why the value still exists: /metrics then
|
||||
# answers on the management port, UNAUTHENTICATED, carrying realm names,
|
||||
# client ids and login counters. That port is ClusterIP-only and in no
|
||||
# Ingress (see keycloak.ports.management), so the exposure is in-cluster.
|
||||
#
|
||||
# SWITCHING THIS OFF IS A DECISION ABOUT WHETHER READINESS CAN FAIL, not
|
||||
# about scraping. If you do it, the readiness probe stops being able to
|
||||
# detect a database outage, and nothing else in this chart replaces it.
|
||||
metrics:
|
||||
enabled: false
|
||||
enabled: true
|
||||
|
||||
# Additional Keycloak feature flags (--features), e.g. ["organization",
|
||||
# "declarative-ui"]. A BUILD option: entries here change what the init
|
||||
@@ -349,9 +365,14 @@ keycloak:
|
||||
timeoutSeconds: 5
|
||||
failureThreshold: 3
|
||||
|
||||
# READINESS: /health/ready - which DOES include the database, and must. A
|
||||
# node that cannot reach its database serves errors to every login, so no
|
||||
# Service should route to it.
|
||||
# READINESS: /health/ready - which includes the database, and must. A node
|
||||
# that cannot reach its database serves errors to every login, so no Service
|
||||
# should route to it.
|
||||
#
|
||||
# AND THAT "INCLUDES THE DATABASE" IS CONDITIONAL, which is the trap worth
|
||||
# knowing before anybody tidies up keycloak.metrics.enabled: the database
|
||||
# check is only part of the report while metrics-enabled is true (upstream's
|
||||
# check table, "Requires Metrics: Yes"). The path alone guarantees nothing.
|
||||
readinessProbe:
|
||||
httpGet:
|
||||
path: /health/ready
|
||||
@@ -364,13 +385,24 @@ keycloak:
|
||||
# endpoint. It gates liveness and readiness, which is why neither of those
|
||||
# carries an initialDelaySeconds.
|
||||
#
|
||||
# BUDGET = periodSeconds x failureThreshold = 10 x 60 = 600s, and it is
|
||||
# generous on purpose. What has to fit inside it is not only the JVM and
|
||||
# Infinispan: on a FIRST install Keycloak creates its entire schema through
|
||||
# Liquibase, and with two replicas starting at once the second waits on the
|
||||
# first's migration lock. A budget that is too short there produces a
|
||||
# CrashLoopBackOff that looks like an application fault and is really a
|
||||
# stopwatch.
|
||||
# BUDGET = periodSeconds x failureThreshold = 10 x 60 = 600s, and the gap
|
||||
# between that and the MEASURED time is deliberate rather than lazy.
|
||||
#
|
||||
# MEASURED, in the install-test of Gitea PR #87 on the Gitea docker runner
|
||||
# (k3s v1.35.5, two replicas, empty database, image pulled during the run):
|
||||
# from `helm install` to BOTH pods Ready in 71.8 seconds - the first at
|
||||
# ~70.8s, the second one second later, 0 restarts. That figure already
|
||||
# contains the image pull, both init containers and the schema creation.
|
||||
#
|
||||
# Why the budget is still eight times that: everything in the measured run
|
||||
# was the cheap case. A first install against a real database creates the
|
||||
# whole schema through Liquibase, and with two replicas the second waits on
|
||||
# the first's migration lock instead of arriving a second later; a loaded
|
||||
# node, a slower disk or a bigger existing schema all push in the same
|
||||
# direction. A budget that is too short there produces a CrashLoopBackOff
|
||||
# that looks like an application fault and is really a stopwatch - and the
|
||||
# cost of the headroom is nothing, because a pod that IS healthy becomes
|
||||
# Ready on the first probe either way.
|
||||
#
|
||||
# The Quarkus augmentation is NOT in this budget - it runs in the init
|
||||
# container, which has no probe and no deadline of its own. That is the
|
||||
|
||||
Reference in new issue
Block a user