diff --git a/README.md b/README.md index 2b82990..f658db2 100644 --- a/README.md +++ b/README.md @@ -270,6 +270,8 @@ Reference implementation: `template-chart/values.schema.json` and `template-char **3. Do not hang liveness on an external dependency where a dependency-free path exists.** `zipline` deliberately probes `/` for liveness and `/api/healthcheck` only for readiness and startup, so a database outage cannot restart the pod. Where no such path exists — `outline` returns 500 on `/` when the database is down — all three probes may share the endpoint, but the reason belongs in a comment on the helper, or the next reader will "fix" the inconsistency. + The other half of the same rule: readiness **must** hang on the dependency, and a separate path is not proof that it does. `urlaubsverwaltung` probed `/actuator/health/liveness` and `/actuator/health/readiness` for two years and still stayed 1/1 Ready through a total database outage (#218), because Spring Boot's `readiness` group contains `readinessState` **alone** — the split paths were right, the group content was empty of anything external. Whatever aggregates behind a probe path (Spring health groups, a `/healthz` a framework assembles for you) has to be read off the **running** endpoint, not off the path name; the chart names the members explicitly (`MANAGEMENT_ENDPOINT_HEALTH_GROUP_READINESS_INCLUDE=readinessState,db`) so the composition is in the template rather than in an upstream default. + **4. A fallback reproduces *this chart's* defaults, not the repo standard.** Five charts deviate from the hardening baseline on purpose. A "cleaned up" standard default is not tidying, it is an outage: `calibre-web-automated` must keep `readOnlyRootFilesystem: false`, because in read-only mode the image disables its PUID/PGID remapping and the runtime uid silently moves from 1000 to 911 — **making existing volume data inaccessible**. The same applies to non-standard values that look like mistakes: `satisfactory` keeps `limitFactor: 2` (that is what holds the memory limit at the published 8Gi) and `fsGroupChangePolicy: OnRootMismatch` (which avoids a recursive chown of multi-GB game files on every start). **5. A probe default carries the probe's context, not just its timings.** Two different symptoms, both from real charts: `teamspeak-server` addresses its TCP port by the name `file-transfer` because the voice service is UDP and cannot be probed — a default without that name leaves the workload **with no working probe at all**. `patchmon` sends a `Host` header because the server answers **403** to any request whose Host is neither loopback nor a CORS_ORIGIN host — a default without it means the pod never becomes Ready. Port names, `Host` headers and `exec` commands belong in the fallback; where the context is dynamic, pass the root context in and rebuild it (patchmon derives the header from `ingress.domain`). diff --git a/urlaubsverwaltung/Chart.yaml b/urlaubsverwaltung/Chart.yaml index 0beab97..c05627d 100644 --- a/urlaubsverwaltung/Chart.yaml +++ b/urlaubsverwaltung/Chart.yaml @@ -4,5 +4,5 @@ description: A Helm chart to deploy urlaubsverwaltung on Kubernetes icon: https://raw.githubusercontent.com/urlaubsverwaltung/urlaubsverwaltung/main/src/main/resources/static/favicons/apple-touch-icon.png type: application -version: 8.1.0+up6.7.0 +version: 9.0.0+up6.7.0 appVersion: 6.7.0 diff --git a/urlaubsverwaltung/templates/NOTES.txt b/urlaubsverwaltung/templates/NOTES.txt index f88020c..8ec6286 100644 --- a/urlaubsverwaltung/templates/NOTES.txt +++ b/urlaubsverwaltung/templates/NOTES.txt @@ -199,33 +199,75 @@ THE MEMORY LIMIT SIZES THE JAVA HEAP — IT IS NOT JUST A CEILING kubectl -n {{ .Release.Namespace }} logs deployment/{{ .Release.Name }}-{{ .Chart.Name }}-deployment | head -1 -DO NOT HEALTH-CHECK THIS RELEASE ON /actuator/health - That endpoint aggregates every health indicator, and one of them is the mail - sender pointed at localhost:1025 by this chart (see below). Nothing listens - there, so the aggregate answers 503 DOWN on a completely healthy release, - for the lifetime of that release: +THE THREE HEALTH PATHS AND WHAT EACH ONE ANSWERS FOR + This chart sets two health settings in the deployment template, and they are + what makes the three paths mean different things: - {"components":{ ... "mail":{"status":"DOWN"} ... },"status":"DOWN"} + MANAGEMENT_ENDPOINT_HEALTH_GROUP_READINESS_INCLUDE=readinessState,db + MANAGEMENT_HEALTH_MAIL_ENABLED=false - Everything else in it — db, ping, diskSpace, ssl — is UP. The probes this - chart configures deliberately use the /actuator/health/liveness and - /actuator/health/readiness GROUPS instead, which contain only the - application's lifecycle state and answer 200. Any external monitor you point - at this release has to use the same two paths, or it will page you forever - about a mail server you never configured. + /actuator/health everything the release actually runs: db, + diskSpace, ping, ssl and the two lifecycle + states. 200 UP on a healthy release. The mail + indicator is NOT in it — see the mail section. + → point external monitoring here. This is the only path that also fails on + diskSpace or ssl. -WHAT A READY POD DOES NOT PROVE - Those two groups report lifecycle state and nothing else — the datasource is - not in them. Measured here: with the database scaled away underneath a - running pod, both groups kept answering 200, the pod stayed 1/1 Ready and was - never restarted. The database is verified exactly once, at boot, by the - migration. + /actuator/health/readiness readinessState AND db. Goes DOWN (503) on a + database outage, which takes the pod out of Ready + and stops traffic reaching it. + → the readinessProbe. - Know the resulting picture in advance: during a database outage the pod stays - Ready, its endpoint keeps receiving traffic{{ if .Values.ingress.enabled }}, the Ingress keeps routing to - it{{ end }}, and users get application errors while Kubernetes reports everything - healthy. Nothing in the pod's status or restart count points at the database. - Look there first anyway. + /actuator/health/liveness livenessState alone, deliberately WITHOUT db. + Stays 200 through a database outage. + → the livenessProbe and the startupProbe. + + The split is the point: a database outage has to keep traffic away, but + restarting the pod does not bring the database back. Adding db to the + liveness group would turn every outage into a crash loop on top of the + outage. Measured on this chart, with the database scaled to zero underneath a + running pod: readiness 503, liveness 200, pod 0/1 Ready, restart count 0. + + How readiness actually fails is worth knowing before you tune it. The db + indicator does not answer 503 quickly — it waits out the JDBC connection + timeout first, measured at 30 seconds against a database whose Service has no + endpoints at all. readinessProbe.timeoutSeconds is 5, so the kubelet gives up + long before the answer arrives and logs + + Readiness probe failed: ... context deadline exceeded + + rather than a 503. The pod leaves Ready either way, and faster; the 503 is + what a client with patience sees. Raising timeoutSeconds past the JDBC + timeout would swap the timeout for the 503 and delay the reaction — there is + no reason to. + +WHAT A READY POD DOES AND DOES NOT PROVE + Ready now means the pod can reach the database — that is checked on every + readiness period, not once at boot. It still does not mean the identity + provider is reachable: OIDC is resolved once at startup (see below) and no + health indicator watches it afterwards, so a provider that goes down after + the rollout leaves a Ready pod on which nobody can log in. + + A database outage now shows up in the pod's status: READY drops to 0/1 while + STATUS stays Running and RESTARTS stays put. That combination — running, not + restarting, not Ready — is the signature of a dependency outage rather than + of a broken release. The reason is in the endpoint, not in the log: + + kubectl -n {{ .Release.Namespace }} get pod -l app={{ .Release.Name }}-{{ .Chart.Name }}-deployment + kubectl -n {{ .Release.Namespace }} port-forward svc/{{ .Release.Name }}-{{ .Chart.Name }}-service 8080:80 + curl -s localhost:8080/actuator/health + + The body names components only when show-details is on; the chart does not + set it, so the aggregate answers {"status":"DOWN","groups":[...]} and the + status code carries the message. To see WHICH component is down, add + + urlaubsverwaltung.env: + - name: MANAGEMENT_ENDPOINT_HEALTH_SHOW_DETAILS + value: always + + and know what that publishes to anyone who can reach the endpoint: the + database product, the container's filesystem path and free bytes, the + certificate chains, and the stack-trace class of whatever failed. Note also that there is no shell in this image: "kubectl exec" cannot open one, and neither can a debug sidecar sharing the process namespace run the @@ -241,10 +283,34 @@ MAIL AND THE APPLICATION URL ARE PLACEHOLDERS UNTIL YOU SET THEM UV_MAIL_APPLICATIONURL = http://localhost:8080 Nothing in this pod listens on localhost:1025, so no notification mail leaves - the release and the mail health indicator stays DOWN. UV_MAIL_APPLICATIONURL - is what the app puts into the links inside those mails — it is NOT derived - from ingress.domain. Left as it is, every link points at the reader's own - machine. + the release. UV_MAIL_APPLICATIONURL is what the app puts into the links + inside those mails — it is NOT derived from ingress.domain. Left as it is, + every link points at the reader's own machine. + + BECAUSE that wiring is a placeholder, the chart sets + + MANAGEMENT_HEALTH_MAIL_ENABLED=false + + and the mail indicator is gone from /actuator/health. Otherwise the aggregate + would answer 503 DOWN on a perfectly healthy release for its whole lifetime, + and any monitor pointed at it would page forever about a mail server nobody + configured. + + THE PART THAT BITES LATER: the day you configure real mail delivery, switch + the indicator back on in the SAME change — + + urlaubsverwaltung.env: + - name: MANAGEMENT_HEALTH_MAIL_ENABLED + value: "true" + + Left at false it does not report a false alarm, it reports nothing: the + endpoint answers 200 while the mail server is unreachable and notifications + are silently not delivered. The setting is only honest while there is no mail + server to be honest about. + + Note that this variable repeats a name the chart already sets, so per the + rule below it can only be added at INSTALL time — putting it into an already + installed release makes the next upgrade fail in the API server. urlaubsverwaltung.env IS ADDITIVE ONLY, AND A COLLISION BREAKS THE UPGRADE The entries are appended AFTER the chart's own list{{ if $envNames }} (this release appends: diff --git a/urlaubsverwaltung/templates/urlaubsverwaltung.deployment.yaml b/urlaubsverwaltung/templates/urlaubsverwaltung.deployment.yaml index 0b7755b..f7c1bad 100644 --- a/urlaubsverwaltung/templates/urlaubsverwaltung.deployment.yaml +++ b/urlaubsverwaltung/templates/urlaubsverwaltung.deployment.yaml @@ -50,6 +50,32 @@ spec: secretKeyRef: name: {{ .Release.Name }}-{{ .Chart.Name }}-db-secret key: password + # Health indicators and probe groups + # + # Spring Boot creates the "liveness" and "readiness" groups by + # itself here (measured: /actuator/health answers + # {"groups":["liveness","readiness"]} with nothing configured), but + # the readiness group ships with readinessState ALONE. Without the + # line below a total database outage leaves /actuator/health/readiness + # at 200 and the pod 1/1 Ready while every request fails (#218). + # Naming the group's members REPLACES the default, so readinessState + # has to be repeated. "db" is the datasource indicator's real name, + # read off the running endpoint. + - name: MANAGEMENT_ENDPOINT_HEALTH_GROUP_READINESS_INCLUDE + value: "readinessState,db" + # The liveness group is deliberately NOT extended: a database + # outage has to keep traffic away, but restarting the pod does not + # bring the database back — it would only add a crash loop to the + # outage. + # + # The mail indicator is switched off because the mail wiring below + # points at localhost:1025, where nothing listens. Left on, it holds + # the aggregate /actuator/health at 503 DOWN for the whole life of a + # healthy release (#217). Anyone who later configures real mail + # delivery has to switch it back on via urlaubsverwaltung.env, or + # the endpoint stays blind to a broken mail server. + - name: MANAGEMENT_HEALTH_MAIL_ENABLED + value: "false" # Mail - name: UV_MAIL_FROM value: urlaubsverwaltung@example.org diff --git a/urlaubsverwaltung/values.yaml b/urlaubsverwaltung/values.yaml index a83750f..5933670 100644 --- a/urlaubsverwaltung/values.yaml +++ b/urlaubsverwaltung/values.yaml @@ -45,6 +45,39 @@ urlaubsverwaltung: capabilities: drop: - ALL + # HEALTH INDICATORS — TWO SETTINGS THE CHART FIXES IN THE TEMPLATE + # + # The chart sets these two as plain environment variables, not as values, + # because urlaubsverwaltung.env below already overrides anything: + # + # MANAGEMENT_ENDPOINT_HEALTH_GROUP_READINESS_INCLUDE=readinessState,db + # puts the datasource into the readiness group, so a database outage + # takes the pod out of Ready instead of leaving it serving errors (#218). + # The liveness group stays untouched on purpose — an outage must keep + # traffic away, not restart the pod in a loop. + # MANAGEMENT_HEALTH_MAIL_ENABLED=false + # removes the mail indicator, which otherwise pins the aggregate + # /actuator/health at 503 DOWN forever, because the chart's mail wiring + # points at localhost:1025 where nothing listens (#217). + # + # THE CONSEQUENCE TO KNOW BEFORE IT BITES: whoever later configures real mail + # delivery (SPRING_MAIL_HOST/SPRING_MAIL_PORT via urlaubsverwaltung.env) MUST + # switch the indicator back on in the same change: + # + # env: + # - name: MANAGEMENT_HEALTH_MAIL_ENABLED + # value: "true" + # + # Left at false, /actuator/health keeps answering 200 while the configured + # mail server is unreachable and no notification mail is delivered. The + # setting trades a permanently false alarm for silence, and silence is only + # correct as long as there is nothing to report about. + # + # That override repeats a name the chart already sets, and env is additive: + # it renders the variable twice (the later entry wins in the container), and + # adding it to an ALREADY INSTALLED release makes the next upgrade fail in + # the API server. See the "env IS ADDITIVE ONLY" section of NOTES.txt — the + # switch has to go in at install time, not at upgrade time. livenessProbe: httpGet: port: 8080