urlaubsverwaltung: readiness must see the database, health must not lie (9.0.0+up6.7.0)
MANAGEMENT_ENDPOINT_HEALTH_GROUP_READINESS_INCLUDE=readinessState,db puts the datasource into Spring Boot's readiness group, which shipped with readinessState alone: a total database outage left the pod 1/1 Ready while every request failed. Naming the members replaces the default, so readinessState is repeated. The liveness group stays untouched on purpose, because an outage has to keep traffic away rather than add a crash loop to it. MANAGEMENT_HEALTH_MAIL_ENABLED=false removes the mail indicator, which pinned the aggregate /actuator/health at 503 for the whole life of a healthy release, because the chart's mail wiring points at localhost:1025 where nothing listens. values.yaml and NOTES.txt document what that costs and when it has to be switched back on.
This commit is contained in:
5 files changed
+155
-28
No files matched your search
@@ -4,5 +4,5 @@ description: A Helm chart to deploy urlaubsverwaltung on Kubernetes
|
||||
icon: https://raw.githubusercontent.com/urlaubsverwaltung/urlaubsverwaltung/main/src/main/resources/static/favicons/apple-touch-icon.png
|
||||
type: application
|
||||
|
||||
version: 8.1.0+up6.7.0
|
||||
version: 9.0.0+up6.7.0
|
||||
appVersion: 6.7.0
|
||||
@@ -199,33 +199,75 @@ THE MEMORY LIMIT SIZES THE JAVA HEAP — IT IS NOT JUST A CEILING
|
||||
|
||||
kubectl -n {{ .Release.Namespace }} logs deployment/{{ .Release.Name }}-{{ .Chart.Name }}-deployment | head -1
|
||||
|
||||
DO NOT HEALTH-CHECK THIS RELEASE ON /actuator/health
|
||||
That endpoint aggregates every health indicator, and one of them is the mail
|
||||
sender pointed at localhost:1025 by this chart (see below). Nothing listens
|
||||
there, so the aggregate answers 503 DOWN on a completely healthy release,
|
||||
for the lifetime of that release:
|
||||
THE THREE HEALTH PATHS AND WHAT EACH ONE ANSWERS FOR
|
||||
This chart sets two health settings in the deployment template, and they are
|
||||
what makes the three paths mean different things:
|
||||
|
||||
{"components":{ ... "mail":{"status":"DOWN"} ... },"status":"DOWN"}
|
||||
MANAGEMENT_ENDPOINT_HEALTH_GROUP_READINESS_INCLUDE=readinessState,db
|
||||
MANAGEMENT_HEALTH_MAIL_ENABLED=false
|
||||
|
||||
Everything else in it — db, ping, diskSpace, ssl — is UP. The probes this
|
||||
chart configures deliberately use the /actuator/health/liveness and
|
||||
/actuator/health/readiness GROUPS instead, which contain only the
|
||||
application's lifecycle state and answer 200. Any external monitor you point
|
||||
at this release has to use the same two paths, or it will page you forever
|
||||
about a mail server you never configured.
|
||||
/actuator/health everything the release actually runs: db,
|
||||
diskSpace, ping, ssl and the two lifecycle
|
||||
states. 200 UP on a healthy release. The mail
|
||||
indicator is NOT in it — see the mail section.
|
||||
→ point external monitoring here. This is the only path that also fails on
|
||||
diskSpace or ssl.
|
||||
|
||||
WHAT A READY POD DOES NOT PROVE
|
||||
Those two groups report lifecycle state and nothing else — the datasource is
|
||||
not in them. Measured here: with the database scaled away underneath a
|
||||
running pod, both groups kept answering 200, the pod stayed 1/1 Ready and was
|
||||
never restarted. The database is verified exactly once, at boot, by the
|
||||
migration.
|
||||
/actuator/health/readiness readinessState AND db. Goes DOWN (503) on a
|
||||
database outage, which takes the pod out of Ready
|
||||
and stops traffic reaching it.
|
||||
→ the readinessProbe.
|
||||
|
||||
Know the resulting picture in advance: during a database outage the pod stays
|
||||
Ready, its endpoint keeps receiving traffic{{ if .Values.ingress.enabled }}, the Ingress keeps routing to
|
||||
it{{ end }}, and users get application errors while Kubernetes reports everything
|
||||
healthy. Nothing in the pod's status or restart count points at the database.
|
||||
Look there first anyway.
|
||||
/actuator/health/liveness livenessState alone, deliberately WITHOUT db.
|
||||
Stays 200 through a database outage.
|
||||
→ the livenessProbe and the startupProbe.
|
||||
|
||||
The split is the point: a database outage has to keep traffic away, but
|
||||
restarting the pod does not bring the database back. Adding db to the
|
||||
liveness group would turn every outage into a crash loop on top of the
|
||||
outage. Measured on this chart, with the database scaled to zero underneath a
|
||||
running pod: readiness 503, liveness 200, pod 0/1 Ready, restart count 0.
|
||||
|
||||
How readiness actually fails is worth knowing before you tune it. The db
|
||||
indicator does not answer 503 quickly — it waits out the JDBC connection
|
||||
timeout first, measured at 30 seconds against a database whose Service has no
|
||||
endpoints at all. readinessProbe.timeoutSeconds is 5, so the kubelet gives up
|
||||
long before the answer arrives and logs
|
||||
|
||||
Readiness probe failed: ... context deadline exceeded
|
||||
|
||||
rather than a 503. The pod leaves Ready either way, and faster; the 503 is
|
||||
what a client with patience sees. Raising timeoutSeconds past the JDBC
|
||||
timeout would swap the timeout for the 503 and delay the reaction — there is
|
||||
no reason to.
|
||||
|
||||
WHAT A READY POD DOES AND DOES NOT PROVE
|
||||
Ready now means the pod can reach the database — that is checked on every
|
||||
readiness period, not once at boot. It still does not mean the identity
|
||||
provider is reachable: OIDC is resolved once at startup (see below) and no
|
||||
health indicator watches it afterwards, so a provider that goes down after
|
||||
the rollout leaves a Ready pod on which nobody can log in.
|
||||
|
||||
A database outage now shows up in the pod's status: READY drops to 0/1 while
|
||||
STATUS stays Running and RESTARTS stays put. That combination — running, not
|
||||
restarting, not Ready — is the signature of a dependency outage rather than
|
||||
of a broken release. The reason is in the endpoint, not in the log:
|
||||
|
||||
kubectl -n {{ .Release.Namespace }} get pod -l app={{ .Release.Name }}-{{ .Chart.Name }}-deployment
|
||||
kubectl -n {{ .Release.Namespace }} port-forward svc/{{ .Release.Name }}-{{ .Chart.Name }}-service 8080:80
|
||||
curl -s localhost:8080/actuator/health
|
||||
|
||||
The body names components only when show-details is on; the chart does not
|
||||
set it, so the aggregate answers {"status":"DOWN","groups":[...]} and the
|
||||
status code carries the message. To see WHICH component is down, add
|
||||
|
||||
urlaubsverwaltung.env:
|
||||
- name: MANAGEMENT_ENDPOINT_HEALTH_SHOW_DETAILS
|
||||
value: always
|
||||
|
||||
and know what that publishes to anyone who can reach the endpoint: the
|
||||
database product, the container's filesystem path and free bytes, the
|
||||
certificate chains, and the stack-trace class of whatever failed.
|
||||
|
||||
Note also that there is no shell in this image: "kubectl exec" cannot open
|
||||
one, and neither can a debug sidecar sharing the process namespace run the
|
||||
@@ -241,10 +283,34 @@ MAIL AND THE APPLICATION URL ARE PLACEHOLDERS UNTIL YOU SET THEM
|
||||
UV_MAIL_APPLICATIONURL = http://localhost:8080
|
||||
|
||||
Nothing in this pod listens on localhost:1025, so no notification mail leaves
|
||||
the release and the mail health indicator stays DOWN. UV_MAIL_APPLICATIONURL
|
||||
is what the app puts into the links inside those mails — it is NOT derived
|
||||
from ingress.domain. Left as it is, every link points at the reader's own
|
||||
machine.
|
||||
the release. UV_MAIL_APPLICATIONURL is what the app puts into the links
|
||||
inside those mails — it is NOT derived from ingress.domain. Left as it is,
|
||||
every link points at the reader's own machine.
|
||||
|
||||
BECAUSE that wiring is a placeholder, the chart sets
|
||||
|
||||
MANAGEMENT_HEALTH_MAIL_ENABLED=false
|
||||
|
||||
and the mail indicator is gone from /actuator/health. Otherwise the aggregate
|
||||
would answer 503 DOWN on a perfectly healthy release for its whole lifetime,
|
||||
and any monitor pointed at it would page forever about a mail server nobody
|
||||
configured.
|
||||
|
||||
THE PART THAT BITES LATER: the day you configure real mail delivery, switch
|
||||
the indicator back on in the SAME change —
|
||||
|
||||
urlaubsverwaltung.env:
|
||||
- name: MANAGEMENT_HEALTH_MAIL_ENABLED
|
||||
value: "true"
|
||||
|
||||
Left at false it does not report a false alarm, it reports nothing: the
|
||||
endpoint answers 200 while the mail server is unreachable and notifications
|
||||
are silently not delivered. The setting is only honest while there is no mail
|
||||
server to be honest about.
|
||||
|
||||
Note that this variable repeats a name the chart already sets, so per the
|
||||
rule below it can only be added at INSTALL time — putting it into an already
|
||||
installed release makes the next upgrade fail in the API server.
|
||||
|
||||
urlaubsverwaltung.env IS ADDITIVE ONLY, AND A COLLISION BREAKS THE UPGRADE
|
||||
The entries are appended AFTER the chart's own list{{ if $envNames }} (this release appends:
|
||||
|
||||
@@ -50,6 +50,32 @@ spec:
|
||||
secretKeyRef:
|
||||
name: {{ .Release.Name }}-{{ .Chart.Name }}-db-secret
|
||||
key: password
|
||||
# Health indicators and probe groups
|
||||
#
|
||||
# Spring Boot creates the "liveness" and "readiness" groups by
|
||||
# itself here (measured: /actuator/health answers
|
||||
# {"groups":["liveness","readiness"]} with nothing configured), but
|
||||
# the readiness group ships with readinessState ALONE. Without the
|
||||
# line below a total database outage leaves /actuator/health/readiness
|
||||
# at 200 and the pod 1/1 Ready while every request fails (#218).
|
||||
# Naming the group's members REPLACES the default, so readinessState
|
||||
# has to be repeated. "db" is the datasource indicator's real name,
|
||||
# read off the running endpoint.
|
||||
- name: MANAGEMENT_ENDPOINT_HEALTH_GROUP_READINESS_INCLUDE
|
||||
value: "readinessState,db"
|
||||
# The liveness group is deliberately NOT extended: a database
|
||||
# outage has to keep traffic away, but restarting the pod does not
|
||||
# bring the database back — it would only add a crash loop to the
|
||||
# outage.
|
||||
#
|
||||
# The mail indicator is switched off because the mail wiring below
|
||||
# points at localhost:1025, where nothing listens. Left on, it holds
|
||||
# the aggregate /actuator/health at 503 DOWN for the whole life of a
|
||||
# healthy release (#217). Anyone who later configures real mail
|
||||
# delivery has to switch it back on via urlaubsverwaltung.env, or
|
||||
# the endpoint stays blind to a broken mail server.
|
||||
- name: MANAGEMENT_HEALTH_MAIL_ENABLED
|
||||
value: "false"
|
||||
# Mail
|
||||
- name: UV_MAIL_FROM
|
||||
value: urlaubsverwaltung@example.org
|
||||
|
||||
@@ -45,6 +45,39 @@ urlaubsverwaltung:
|
||||
capabilities:
|
||||
drop:
|
||||
- ALL
|
||||
# HEALTH INDICATORS — TWO SETTINGS THE CHART FIXES IN THE TEMPLATE
|
||||
#
|
||||
# The chart sets these two as plain environment variables, not as values,
|
||||
# because urlaubsverwaltung.env below already overrides anything:
|
||||
#
|
||||
# MANAGEMENT_ENDPOINT_HEALTH_GROUP_READINESS_INCLUDE=readinessState,db
|
||||
# puts the datasource into the readiness group, so a database outage
|
||||
# takes the pod out of Ready instead of leaving it serving errors (#218).
|
||||
# The liveness group stays untouched on purpose — an outage must keep
|
||||
# traffic away, not restart the pod in a loop.
|
||||
# MANAGEMENT_HEALTH_MAIL_ENABLED=false
|
||||
# removes the mail indicator, which otherwise pins the aggregate
|
||||
# /actuator/health at 503 DOWN forever, because the chart's mail wiring
|
||||
# points at localhost:1025 where nothing listens (#217).
|
||||
#
|
||||
# THE CONSEQUENCE TO KNOW BEFORE IT BITES: whoever later configures real mail
|
||||
# delivery (SPRING_MAIL_HOST/SPRING_MAIL_PORT via urlaubsverwaltung.env) MUST
|
||||
# switch the indicator back on in the same change:
|
||||
#
|
||||
# env:
|
||||
# - name: MANAGEMENT_HEALTH_MAIL_ENABLED
|
||||
# value: "true"
|
||||
#
|
||||
# Left at false, /actuator/health keeps answering 200 while the configured
|
||||
# mail server is unreachable and no notification mail is delivered. The
|
||||
# setting trades a permanently false alarm for silence, and silence is only
|
||||
# correct as long as there is nothing to report about.
|
||||
#
|
||||
# That override repeats a name the chart already sets, and env is additive:
|
||||
# it renders the variable twice (the later entry wins in the container), and
|
||||
# adding it to an ALREADY INSTALLED release makes the next upgrade fail in
|
||||
# the API server. See the "env IS ADDITIVE ONLY" section of NOTES.txt — the
|
||||
# switch has to go in at install time, not at upgrade time.
|
||||
livenessProbe:
|
||||
httpGet:
|
||||
port: 8080
|
||||
|
||||
Reference in new issue
Block a user