Install-test / Install-test apache-tika (pull_request) Successful in 1m14s
Install-test / Install-test rss-bridge (pull_request) Successful in 54s
Install-test / Install-test patchmon (pull_request) Successful in 1m35s
Install-test / Install-test bentopdf (pull_request) Successful in 1m2s
Validate Helm Charts / Detect changed charts (pull_request) Successful in 3s
Validate Helm Charts / Resource quantities (pull_request) Successful in 13s
Validate Helm Charts / Selector match (pull_request) Successful in 10s
Validate Helm Charts / Bump level (pull_request) Successful in 4s
Validate Helm Charts / YAML scalars (pull_request) Successful in 10s
Validate Helm Charts / NetworkPolicy (pull_request) Failing after 15s
Install-test / Install-test bulwark (pull_request) Successful in 1m25s
Validate Helm Charts / HTTPRoute match (pull_request) Successful in 9s
Validate Helm Charts / Build apache-tika (pull_request) Successful in 6s
Validate Helm Charts / Build bentopdf (pull_request) Successful in 5s
Validate Helm Charts / Trivy config apache-tika (pull_request) Successful in 14s
Validate Helm Charts / Build bulwark (pull_request) Successful in 5s
Validate Helm Charts / Build calibre-web-automated (pull_request) Successful in 6s
Validate Helm Charts / Build gotenberg (pull_request) Successful in 7s
Validate Helm Charts / Build rallly (pull_request) Successful in 13s
Validate Helm Charts / Build searxng (pull_request) Successful in 17s
Validate Helm Charts / Build sure (pull_request) Successful in 18s
Validate Helm Charts / Build template-chart (pull_request) Successful in 6s
Validate Helm Charts / Build cyberchef (pull_request) Successful in 5s
Validate Helm Charts / Build homeassistant (pull_request) Successful in 7s
Validate Helm Charts / Build keycloak (pull_request) Successful in 11s
Validate Helm Charts / Build outline (pull_request) Successful in 8s
Validate Helm Charts / Build patchmon (pull_request) Successful in 6s
Validate Helm Charts / Build rss-bridge (pull_request) Successful in 5s
Install-test / Install-test rallly (pull_request) Successful in 1m39s
Validate Helm Charts / Build urlaubsverwaltung (pull_request) Successful in 5s
Validate Helm Charts / Build zipline (pull_request) Successful in 5s
Validate Helm Charts / Build voucher-vault (pull_request) Successful in 6s
Validate Helm Charts / Build ytdlp-web-player (pull_request) Successful in 6s
Validate Helm Charts / Trivy config homeassistant (pull_request) Successful in 11s
Validate Helm Charts / Trivy config bentopdf (pull_request) Successful in 15s
Validate Helm Charts / Trivy config bulwark (pull_request) Successful in 13s
Validate Helm Charts / Trivy config calibre-web-automated (pull_request) Successful in 16s
Validate Helm Charts / Trivy config cyberchef (pull_request) Successful in 26s
Validate Helm Charts / Trivy config gotenberg (pull_request) Successful in 19s
Validate Helm Charts / Trivy config patchmon (pull_request) Successful in 13s
Validate Helm Charts / Trivy config keycloak (pull_request) Successful in 19s
Validate Helm Charts / Trivy config outline (pull_request) Successful in 12s
Validate Helm Charts / Trivy config rallly (pull_request) Successful in 16s
Validate Helm Charts / Trivy config sure (pull_request) Successful in 13s
Validate Helm Charts / Trivy config template-chart (pull_request) Successful in 14s
Validate Helm Charts / Trivy config urlaubsverwaltung (pull_request) Successful in 21s
Validate Helm Charts / Trivy config rss-bridge (pull_request) Successful in 17s
Validate Helm Charts / Trivy config searxng (pull_request) Successful in 11s
Validate Helm Charts / Trivy config voucher-vault (pull_request) Successful in 15s
Validate Helm Charts / Trivy config ytdlp-web-player (pull_request) Successful in 13s
Validate Helm Charts / Trivy config zipline (pull_request) Successful in 13s
Install-test / Install-test calibre-web-automated (pull_request) Successful in 2m8s
Install-test / Install-test cyberchef (pull_request) Successful in 1m9s
Install-test / Install-test gotenberg (pull_request) Successful in 1m40s
Install-test / Install-test homeassistant (pull_request) Successful in 2m30s
Install-test / Install-test keycloak (pull_request) Successful in 1m58s
Install-test / Install-test searxng (pull_request) Successful in 1m5s
Install-test / Install-test outline (pull_request) Successful in 1m57s
Install-test / Install-test sure (pull_request) Successful in 2m30s
Install-test / Install-test urlaubsverwaltung (pull_request) Successful in 1m38s
Install-test / Install-test voucher-vault (pull_request) Successful in 2m6s
Install-test / Install-test ytdlp-web-player (pull_request) Successful in 1m5s
Install-test / Install-test zipline (pull_request) Successful in 1m50s
Validate Helm Charts / Install-test (pull_request) Successful in 29m30s
Major. This is the richest policy of the sweep and the one with the largest
blast radius, and it is the only chart whose three ports fall into three
different classes - so it is where the convention is visible in full:
http (8080) NOT addressed by any probe, so it CAN be restricted by
source, and it is: own namespace, allowedNamespaces, and
the ingress controller. The only real inbound restriction
in this whole rollout, possible only because the probes use
a different port.
management (9000) the port all three probes address, so it stays open inside
the cluster. That matters here specifically: this surface
carries health and metrics UNAUTHENTICATED, and "no
Ingress" was its only protection before this object. The
policy turns "anything that can route to the pod" into
"anything inside the cluster"; probeSources closes the rest.
jgroups (7800) the cluster bus, restricted to the pods this policy itself
selects - a podSelector with no namespaceSelector, the
narrowest rule the object can express.
All four port numbers come from keycloak.settings rather than being written
down twice, so moving keycloak.ports.http moves the policy with it. And the
egress rule for the database is the CONFIGURED port, keycloak.db.port - this is
the one chart in the sweep that does not have to guess, because its database
port is a value instead of a field inside an opaque secret.
The outbound cluster-bus rule is not a duplicate of the inbound one: policyTypes
lists Egress, so a replica dialling its peer is subject to this policy too, and
without it the bus would be open inbound and blocked outbound - a cluster that
never forms, with two nodes that each look healthy.
THE BLAST RADIUS IS WRITTEN INTO THE CHART, because nothing in CI can catch it:
this release is the cluster's identity provider, no ingress controller runs in
the install-test, and the probes use 9000 - so the pod becomes Ready whether or
not the http rule is right. NOTES.txt therefore asks for two checks after the
first upgrade: log in, and look at whether any consumer uses the in-cluster
Service address instead of the public issuer URL (those namespaces need
allowedNamespaces). Not allowed at all, and named as the likely first surprise:
SMTP, because the mail server is configured per realm where the chart cannot
see it.
The test's overlay guard is tightened in the same commit, and keycloak is why.
It counted any source-restricted rule, and this chart restricts two of them in
its own defaults - so the guard would have been satisfied while the probe
branch, the only branch an overlay exists to reach, was never rendered. It now
counts restricted PROBE rules. All twelve failure paths were re-run against the
change.
Part of #5
443 lines
26 KiB
Plaintext
443 lines
26 KiB
Plaintext
{{- /*
|
|
NOTHING IN CI READS THIS FILE. `helm template` does not emit notes and no
|
|
workflow or script under ci/ reads it, so every claim below carries
|
|
review-only assurance - it is checked by being read, not by a stage. A
|
|
rendering ERROR here does fail `helm template`, which is all the pipeline can
|
|
see of it.
|
|
*/ -}}
|
|
{{- $settings := fromYaml (include "keycloak.settings" .) -}}
|
|
{{- $replicas := int (include "keycloak.replicas" (dict "scaleDown" .Values.scaleDown "replicas" .Values.keycloak.replicas)) -}}
|
|
{{- $adminHostname := eq (include "keycloak.adminHostnameEnabled" .) "true" -}}
|
|
{{- $httpRoute := .Values.httpRoute | default dict -}}
|
|
{{- $httpRouteAdmin := $httpRoute.admin | default dict -}}
|
|
{{- $hostnames := $httpRoute.hostnames | default list -}}
|
|
{{- if not $hostnames -}}
|
|
{{- $hostnames = list .Values.ingress.domain -}}
|
|
{{- end -}}
|
|
{{- $adminHostnames := $httpRouteAdmin.hostnames | default list -}}
|
|
{{- if not $adminHostnames -}}
|
|
{{- $adminHostnames = list (.Values.ingress.admin | default dict).domain -}}
|
|
{{- end -}}
|
|
{{- $realmHosts := (fromYaml (include "keycloak.settings" .)).realmHosts | default list -}}
|
|
{{- $deploy := printf "%s-%s-deployment" .Release.Name .Chart.Name -}}
|
|
{{- $svc := printf "%s-%s-service" .Release.Name .Chart.Name -}}
|
|
Keycloak {{ .Chart.AppVersion }} installed as release {{ .Release.Name }} in namespace {{ .Release.Namespace }}.
|
|
|
|
{{ $replicas }} replica(s), cache=ispn with jdbc-ping discovery over the database,
|
|
external PostgreSQL at {{ $settings.db.host | default "<keycloak.db.host NOT SET>" }}:{{ $settings.db.port | int64 }}/{{ $settings.db.database }}.
|
|
|
|
ENDPOINTS - {{ if .Values.ingress.enabled }}INGRESS MODE (ingress.enabled: true, httpRoute.enabled: false){{ else if $httpRoute.enabled }}GATEWAY MODE (httpRoute.enabled: true, ingress.enabled: false){{ else }}NEITHER INGRESS NOR HTTPROUTE{{ end }}
|
|
{{ if .Values.ingress.enabled }} Public: {{ include "keycloak.frontendUrl" . }}
|
|
{{ else if $httpRoute.enabled }}{{ range $hostnames }} Public: https://{{ . }}
|
|
{{ end }}{{ else }} Public: no Ingress and no HTTPRoute (ingress.enabled and httpRoute.enabled are
|
|
both false) - in-cluster only
|
|
{{ end }}{{ if not $adminHostname }} Admin: no admin hostname (ingress.admin.enabled: false) - the console is
|
|
served on the public host and Keycloak generates its URLs for it
|
|
{{ else if $httpRoute.enabled }}{{ range $adminHostnames }} Admin: https://{{ . }}/admin
|
|
{{ end }}{{ else if .Values.ingress.enabled }} Admin: {{ include "keycloak.adminUrl" . }}/admin
|
|
{{ else }} Admin: {{ include "keycloak.adminUrl" . }}/admin is what KC_HOSTNAME_ADMIN says, and
|
|
NOTHING PUBLISHES IT. ingress.admin.enabled is true, so the console
|
|
still generates its URLs for that host, but with both modes off
|
|
there is no Ingress and no HTTPRoute to carry it. Use the
|
|
port-forward below, or switch a mode on.
|
|
{{ end }} In-cluster: http://{{ $svc }}.{{ .Release.Namespace }}.svc.cluster.local:{{ $settings.ports.http | int64 }}
|
|
|
|
Management port {{ $settings.ports.management | int64 }} is NOT published and must never be: it serves
|
|
/health, /health/live, /health/ready, /health/started{{ if $settings.metrics.enabled }} and /metrics{{ end }}
|
|
WITHOUT authentication. It has a ClusterIP Service
|
|
({{ .Release.Name }}-{{ .Chart.Name }}-management-service) for in-cluster scraping only.
|
|
{{ if $realmHosts }}
|
|
REALM HOSTNAMES - {{ len $realmHosts }} further name(s), each landing in ONE realm
|
|
|
|
The object names carry the LIST INDEX and not the realm, so this is the only
|
|
place that says which is which. Reordering keycloak.realmHosts therefore
|
|
renames these objects and recreates them.
|
|
{{ range $i, $entry := $realmHosts }}
|
|
https://{{ $entry.host }}
|
|
realm {{ $entry.realm }} (bare / -> /realms/{{ $entry.realm }}/account/)
|
|
object {{ $.Release.Name }}-{{ $.Chart.Name }}-realm-{{ if $httpRoute.enabled }}httproute{{ else }}ingress{{ end }}{{ if $i }}-{{ $i }}{{ end }}
|
|
{{- if not $httpRoute.enabled }}
|
|
cert {{ $entry.tlsSecretName }}{{ if $entry.clusterIssuer }}, requested by THIS object ({{ $entry.clusterIssuer }}){{ else }}, NOT requested by this object - it must already exist or be issued elsewhere{{ end }}
|
|
{{- end }}
|
|
{{ end }}
|
|
{{- /*
|
|
No "and nothing publishes them" branch here, deliberately: a filled
|
|
keycloak.realmHosts with both modes off ABORTS the render in
|
|
keycloak.ingress.yaml, so this block can only ever be reached with a mode on.
|
|
A branch for an impossible state reads like a state somebody has seen.
|
|
*/ -}}
|
|
{{- end }}
|
|
{{ if $httpRoute.enabled }}
|
|
GATEWAY MODE - WHAT THIS CHART DOES NOT RENDER, AND WHAT YOU MUST SET YOURSELF
|
|
|
|
This release renders {{ add (ternary 2 1 $adminHostname) (len $realmHosts) }} HTTPRoute(s) and NOTHING else of the routing path -
|
|
no Gateway, no class, no Certificate, no policy object.
|
|
|
|
{{ .Release.Name }}-{{ .Chart.Name }}-httproute {{ join ", " $hostnames }}
|
|
{{- if $adminHostname }}
|
|
{{ .Release.Name }}-{{ .Chart.Name }}-admin-httproute {{ join ", " $adminHostnames }}
|
|
{{- end }}
|
|
{{- range $i, $entry := $realmHosts }}
|
|
{{ $.Release.Name }}-{{ $.Chart.Name }}-realm-httproute{{ if $i }}-{{ $i }}{{ end }} {{ $entry.host }} (realm {{ $entry.realm }})
|
|
{{- end }}
|
|
|
|
Three things therefore belong to the cluster, and if any is missing the
|
|
hostnames above simply do not answer:
|
|
|
|
1. The Gateway-API CRDs. Without them this install would have failed, so
|
|
they are present - nothing to check.
|
|
2. The Gateway(s) named in parentRefs
|
|
{{- range $httpRoute.parentRefs }}
|
|
{{ .name }}{{ if .namespace }} in namespace {{ .namespace }}{{ end }}{{ if .sectionName }}, listener {{ .sectionName }}{{ end }} (public)
|
|
{{- end }}
|
|
{{- if $adminHostname }}
|
|
{{- range ($httpRouteAdmin.parentRefs | default $httpRoute.parentRefs) }}
|
|
{{ .name }}{{ if .namespace }} in namespace {{ .namespace }}{{ end }}{{ if .sectionName }}, listener {{ .sectionName }}{{ end }} (admin)
|
|
{{- end }}
|
|
{{- end }}
|
|
{{- range $i, $entry := $realmHosts }}
|
|
{{- range ($entry.parentRefs | default $httpRoute.parentRefs) }}
|
|
{{ .name }}{{ if .namespace }} in namespace {{ .namespace }}{{ end }}{{ if .sectionName }}, listener {{ .sectionName }}{{ end }} (realm {{ $entry.realm }})
|
|
{{- end }}
|
|
{{- end }}
|
|
must exist AND their listeners must allow routes from this namespace
|
|
({{ .Release.Namespace }}) in allowedRoutes. A rejected attachment is visible only
|
|
in the route's status:
|
|
|
|
kubectl -n {{ .Release.Namespace }} get httproute {{ .Release.Name }}-{{ .Chart.Name }}-httproute -o yaml
|
|
|
|
Read status.parents[].conditions - "Accepted: True" and
|
|
"ResolvedRefs: True" is what a working attachment looks like. An empty
|
|
status means no controller has looked at the route at all.
|
|
3. The TLS certificates for those hostnames, on the Gateway's listeners.
|
|
ingress.clusterIssuer is NOT used in this mode and is not required: the
|
|
chart renders no Certificate here, so there is no
|
|
tls-{{ .Release.Name }}-{{ .Chart.Name }}-ingress secret and nothing for cert-manager to do
|
|
for this release.
|
|
{{- if $realmHosts }}
|
|
The realm hostnames above are part of that: their tlsSecretName values
|
|
are not read in this mode either. Where several of them share a DNS zone,
|
|
ONE wildcard listener for that zone replaces one certificate per name -
|
|
which is the one thing this mode makes cheaper here rather than dearer.
|
|
{{- end }}
|
|
|
|
AND THE ONE YOU WILL NOT NOTICE UNTIL A LOGIN FAILS: THE PROXY BUFFERS.
|
|
|
|
Keycloak's responses carry large headers - JWTs, SAML assertions, long
|
|
Set-Cookie chains during the login flow. In Ingress mode that is handled by
|
|
|
|
nginx.ingress.kubernetes.io/proxy-buffer-size: 16k
|
|
nginx.ingress.kubernetes.io/proxy-buffers-number: 4
|
|
|
|
which this chart has never set: they come from ingress.annotations and
|
|
ingress.admin.annotations, i.e. from whoever installs it. IN THIS MODE THEY
|
|
ARE GONE and there is NO counterpart in the Gateway API - buffer sizing is
|
|
implementation policy, not routing, and this chart renders no policy object
|
|
on purpose (it would bind the chart to one Gateway implementation).
|
|
|
|
The failure mode, so you recognise it: the PROXY answers 502 while Keycloak
|
|
logs the request as successful. Nothing says a chart value was ignored,
|
|
because no chart value was involved. Set the equivalent next to the Gateway,
|
|
in your implementation's own object, BEFORE you point DNS at it - and raise
|
|
it, do not just match 16k, if you use long-lived tokens or SAML.
|
|
|
|
ingress.annotations and ingress.admin.annotations are inert here in general:
|
|
the Gateway API has no annotation channel for controller behaviour. An IP
|
|
allow-list for the admin host is the other thing that has to move - and read
|
|
the first trap below before relying on it in either mode.
|
|
{{ if $realmHosts }}
|
|
AND THE SAME LOSS ONCE MORE, ON THE REALM HOSTNAMES - where it is likelier to
|
|
be met rather than merely possible.
|
|
|
|
Each entry of keycloak.realmHosts carries its own annotations block in Ingress
|
|
mode, and in this mode every one of them is gone: {{ len $realmHosts }} hostname(s) here, none
|
|
of which can configure a proxy buffer through this chart any more. What the
|
|
entries CAN still express is the routing - the redirect into the realm is a
|
|
rule of the route and survives the mode change intact.
|
|
|
|
Why these hostnames are the awkward ones: they exist to put somebody in an
|
|
ACCOUNT CONSOLE. That is a logged-in session from the first request after the
|
|
redirect, so the large Authorization and Set-Cookie headers are the normal
|
|
traffic on these names rather than an edge case the way they are on a login
|
|
host that mostly serves anonymous requests. A buffer sized for the default
|
|
therefore fails here FIRST, and it fails as a 502 from the Gateway on a
|
|
hostname whose Ingress never had a chart value to point at - so there is
|
|
nothing to grep for and nothing to un-set.
|
|
|
|
Set the buffer equivalent for these hostnames too, in the Gateway
|
|
implementation's own object, and set it before the names resolve there.
|
|
{{- end }}
|
|
|
|
THE ADMIN HOSTNAME'S "INTERNAL CLASS" MOVES TO THE PARENT. What
|
|
ingress.admin.className does in the other mode, httpRoute.admin.parentRefs
|
|
does here: point the admin route at an internal Gateway. It falls back to
|
|
httpRoute.parentRefs, so leaving it empty puts both hostnames on the same
|
|
Gateway.
|
|
{{ end }}
|
|
THE 1PASSWORD ITEMS, AND WHICH FIELDS THEY MUST CARRY
|
|
The operator materializes one Secret key per item FIELD, so a missing or
|
|
misnamed field leaves the pod in CreateContainerConfigError - with no message
|
|
naming the field.
|
|
|
|
{{ .Values.secrets.databaseSecretRef | default "<secrets.databaseSecretRef NOT SET>" }}
|
|
username the PostgreSQL role
|
|
password that role's password
|
|
The role must be able to CREATE its schema: Keycloak builds and migrates
|
|
the whole schema itself on first start.
|
|
{{ if $settings.bootstrapAdmin.enabled }}
|
|
{{ .Values.secrets.bootstrapAdminSecretRef | default "<secrets.bootstrapAdminSecretRef NOT SET>" }}
|
|
username the TEMPORARY bootstrap administrator (upstream default name:
|
|
temp-admin)
|
|
password its password
|
|
{{ else }}
|
|
(no bootstrap admin item - keycloak.bootstrapAdmin.enabled is false, which is
|
|
the correct state once a permanent administrator exists)
|
|
{{ end }}
|
|
Both arrive as ENVIRONMENT VARIABLES, not as files, and that is forced:
|
|
Keycloak 26 has no _FILE form for an option's value. The price: the password
|
|
is in the container's process environment.
|
|
|
|
FIRST INSTALL - AND STEP 4 IS MANDATORY, NOT ADVICE
|
|
|
|
1. kubectl -n {{ .Release.Namespace }} rollout status deployment/{{ $deploy }}
|
|
Ready means /health/ready answered 200, which means the database is
|
|
reachable and the schema is in place. The first start is the slow one: it
|
|
creates the entire schema through Liquibase, and with {{ $replicas }} replica(s)
|
|
the second waits on the first's migration lock.
|
|
|
|
2. Check that the cluster really formed. THIS STEP IS NOT OPTIONAL AND NO
|
|
PROBE REPLACES IT: a node that never found its peer is Ready, healthy and
|
|
alone, and two such nodes are two islands that each believe they are the
|
|
cluster. The install-test cannot catch it either.
|
|
kubectl -n {{ .Release.Namespace }} logs deployment/{{ $deploy }} | grep -i "ISPN000094\|view accepted\|members"
|
|
Look for a view with {{ $replicas }} members. One member per pod means
|
|
jdbc-ping did not find the peer - check that both pods can reach each
|
|
other on port {{ $settings.ports.jgroups | int64 }} and that they really share one database.
|
|
|
|
3. Log in at {{ if and $adminHostname (not $httpRoute.enabled) }}{{ include "keycloak.adminUrl" . }}/admin{{ else if $adminHostname }}https://{{ first $adminHostnames }}/admin{{ else if .Values.ingress.enabled }}{{ include "keycloak.frontendUrl" . }}/admin{{ else if $httpRoute.enabled }}https://{{ first $hostnames }}/admin{{ else }}the in-cluster address, port-forwarded{{ end }}
|
|
with the bootstrap credentials from the vault item above.
|
|
|
|
4. CREATE A PERMANENT ADMINISTRATOR, then remove the temporary one.
|
|
The bootstrap account is TEMPORARY by design - upstream: "the account
|
|
should exist only for the duration necessary to perform operations needed
|
|
to gain permanent and more secure admin access. After that, the account
|
|
needs to be removed manually." This is different from Keycloak before 26,
|
|
where KEYCLOAK_ADMIN created a regular administrator and there was nothing
|
|
to clean up.
|
|
a) In the master realm, create a user, set a password, assign the
|
|
realm-admin / admin role.
|
|
b) Verify you can log in as that user.
|
|
c) Delete the temp-admin user.
|
|
d) helm upgrade with keycloak.bootstrapAdmin.enabled: false
|
|
|
|
5. Until step 4d, EVERY START LOGS a line about the initial administrator not
|
|
being created, because the master realm already exists and the bootstrap
|
|
path is skipped. It is harmless. It is also the kind of message somebody
|
|
eventually investigates, which is why 4d is part of the install and not a
|
|
tidy-up.
|
|
|
|
MINOR UPGRADES NEED THE CLUSTER DOWN - AND THE AUTOMERGE DOES NOT KNOW THAT
|
|
|
|
Keycloak supports a rolling update only WITHIN a minor line. A minor jump
|
|
(26.8 -> 26.9) requires every node shut down before any new node starts;
|
|
mixing two minor versions in one Infinispan cluster is not a supported
|
|
configuration, and nothing in the cluster reports that it happened.
|
|
|
|
THIS REPOSITORY'S RENOVATE AUTOMERGES EVERY UPDATE TYPE, MAJORS INCLUDED.
|
|
That is a deliberate, accepted risk and not an oversight (Gitea #83), but it
|
|
means an app-version PR that moves the minor can reach main, publish the
|
|
chart and be rolled out as a rolling update without anybody deciding to.
|
|
Watch app-version PRs for a changed minor, and if one has landed, take this
|
|
path rather than letting the rollout happen:
|
|
|
|
1. Decide it with the tool instead of the version number. With the OLD
|
|
version and the OLD configuration:
|
|
kc.sh update-compatibility metadata --file=/tmp/kc-compat.json
|
|
then with the NEW version and the NEW configuration:
|
|
kc.sh update-compatibility check --file=/tmp/kc-compat.json
|
|
Pass EVERY configuration option on both calls - upstream: "Omitting any
|
|
configuration options results in incomplete metadata, and could lead to
|
|
a wrong reported result".
|
|
Exit codes, which is what to automate on: 0 rolling update possible,
|
|
1 unexpected error, 2 invalid CLI option, 3 and 4 not possible - the
|
|
deployment must be shut down.
|
|
|
|
2. If it says shut down:
|
|
helm upgrade ... --set scaleDown=true # every pod stops
|
|
# back up the database AND the configuration before anything else
|
|
helm upgrade ... --set scaleDown=false # with the new version
|
|
|
|
A NOTE ON WHAT IS NOT SETTLED: upstream's 26.6 release notes call patch-level
|
|
rolling updates supported and active by default, while the 26.8.0 page for
|
|
update-compatibility says a rolling update is possible "when the Keycloak
|
|
version is the same" and uses a PATCH pair (26.2.0 -> 26.2.1) as its example
|
|
of an incompatible one. Those two cannot both be current. Until it is clear,
|
|
run the check for patch updates too and trust its exit code rather than the
|
|
size of the version step.
|
|
|
|
TRAPS WORTH KNOWING BEFORE YOU NEED THEM
|
|
|
|
- THE ADMIN HOSTNAME IS NOT AN ACCESS CONTROL. hostname-admin decides which
|
|
URLs the server generates, nothing more. Upstream: "Using the
|
|
hostname-admin option does not prevent accessing the Administration REST
|
|
API endpoints via the frontend URL specified by the hostname option. If you
|
|
want to restrict access to the Administration REST API, you need to do it
|
|
on the reverse proxy level." So /admin/realms/... is reachable through the
|
|
public host whatever you put on the admin host, and an IP allow-list on the
|
|
admin Ingress restricts nothing on its own. This chart renders no deny rule
|
|
for /admin - see values.yaml for why not.{{ if $httpRoute.enabled }} In GATEWAY MODE there is not
|
|
even a place in this chart to put one: no annotation channel, and no
|
|
implementation-specific policy object by design. The restriction belongs
|
|
next to the Gateway, which is where it can also be tested.{{ end }}
|
|
|
|
- A SHARED DATABASE MERGES TWO CLUSTERS. Node discovery runs through the
|
|
database, so - upstream - "clusters that share the same database but should
|
|
remain separate will merge unless they have distinct cluster names. The
|
|
default cluster name is ISPN." Two releases of this chart on one database
|
|
therefore form ONE cluster, silently.
|
|
{{ with $settings.cache.clusterName }}This release sets cache-embedded-cluster-name={{ . }}.{{ else }}This release uses the default name (ISPN) - keycloak.cache.clusterName is
|
|
unset. Read its comment in values.yaml before setting it: upstream also
|
|
warns about cache invalidation between differently-named clusters.{{ end }}
|
|
|
|
- READINESS ONLY CHECKS THE DATABASE WHILE metrics-enabled IS TRUE. Upstream's
|
|
table of health checks has a "Requires Metrics" column and the DATABASE
|
|
check is the row that says yes. So with keycloak.metrics.enabled false,
|
|
/health/ready answers 200 with the database gone and the pod keeps taking
|
|
logins it cannot serve. This release has it {{ if $settings.metrics.enabled }}ON, which is the chart default
|
|
and the reason the default is what it is.{{ else }}OFF, so the readiness probe
|
|
cannot currently detect a database outage.{{ end }}
|
|
|
|
- THERE IS NO ConfigMap IN THIS CHART, so no checksum/config annotation. That
|
|
is not an omission: Keycloak is configured entirely through options and
|
|
KC_* variables, so there is no config file whose change could fail to
|
|
restart anything.
|
|
|
|
- A CHANGE INSIDE A VAULT ITEM RESTARTS NOTHING. Helm never sees it. After
|
|
rotating a field:
|
|
kubectl -n {{ .Release.Namespace }} rollout restart deployment/{{ $deploy }}
|
|
(Repointing a secrets.*SecretRef at a different item DOES roll the pods -
|
|
that is covered by checksum/secret-ref.)
|
|
|
|
- THE POD HAS TWO INIT CONTAINERS AND NEEDS BOTH. db, health-enabled and
|
|
metrics-enabled are Keycloak BUILD options, the published image is built
|
|
with their defaults, and a runtime change makes Keycloak rewrite
|
|
/opt/keycloak/lib/quarkus at start - which readOnlyRootFilesystem: true
|
|
forbids. So one init container seeds that directory into a volume and the
|
|
next runs `kc.sh build` into it; the serving container then runs
|
|
`start --optimized`. If a pod is stuck in Init, read those logs first:
|
|
kubectl -n {{ .Release.Namespace }} logs <pod> -c {{ .Release.Name }}-{{ .Chart.Name }}-build
|
|
Build options are set ONLY through keycloak.db.vendor,
|
|
keycloak.metrics.enabled and keycloak.features. Putting one into
|
|
keycloak.env fails the render on purpose.
|
|
|
|
- POOL SIZES ARE PER REPLICA. keycloak.db.pool.maxSize is {{ $settings.db.pool.maxSize | int64 }}, so with the
|
|
current {{ $replicas }} replica(s) this release can open up to {{ mul ($settings.db.pool.maxSize | int64) $replicas }} connections.
|
|
Upstream's default is 100 per node, which with two nodes exceeds a default
|
|
PostgreSQL's max_connections of 100. Raise it together with the server,
|
|
never alone.
|
|
{{ if and $settings.podDisruptionBudget.enabled (ge $replicas 2) }}
|
|
- A PodDisruptionBudget IS in place ({{ .Release.Name }}-{{ .Chart.Name }}-pdb, minAvailable
|
|
{{ $settings.podDisruptionBudget.minAvailable | int64 }}), so a node drain cannot take all replicas at once.
|
|
{{ else if $settings.podDisruptionBudget.enabled }}
|
|
- NO PodDisruptionBudget was rendered, because the effective replica count is
|
|
{{ $replicas }}. That is deliberate: minAvailable 1 with a single replica forbids every
|
|
voluntary eviction, so `kubectl drain` would block forever on a pod nothing
|
|
may move - and the error would name the PDB rather than the replica count.
|
|
Run two replicas to get the budget.
|
|
{{ else }}
|
|
- NO PodDisruptionBudget (keycloak.podDisruptionBudget.enabled is false). A
|
|
node drain may take every replica at once.
|
|
{{ end }}
|
|
WHAT THIS CHART DOES NOT RENDER
|
|
|
|
- No PVC and no StatefulSet: Keycloak 26 keeps everything in the database,
|
|
user sessions included (persistent sessions are the default; volatile ones
|
|
are deprecated as of 26.8). There is no disk state to keep.
|
|
- No NetworkPolicy. jdbc-ping opens direct pod-to-pod connections on port
|
|
{{ $settings.ports.jgroups | int64 }}, so any policy added later has to allow that between the
|
|
replicas, or the cluster silently becomes two islands.
|
|
- No deny rule for /admin on the public Ingress{{ if $httpRoute.enabled }} or route{{ end }}, and no restriction of
|
|
the administration REST API. That is reverse-proxy work - see the first
|
|
trap.
|
|
- No Gateway, no GatewayClass and no listener, in either mode. Those are
|
|
platform objects and live in cluster-config; this chart renders an
|
|
HTTPRoute at most, which is the part that looks the same on every Gateway
|
|
implementation.
|
|
{{- if .Values.networkPolicy.enabled }}
|
|
|
|
NETWORKPOLICY IS ACTIVE — WHAT IT CAPS, AND WHAT IT DOES NOT
|
|
{{ .Release.Name }}-{{ .Chart.Name }}-networkpolicy selects this workload's pods.
|
|
Inbound, and all three ports differ:
|
|
- http ({{ $settings.ports.http | int64 }}): RESTRICTED BY SOURCE — this namespace, anything in
|
|
networkPolicy.allowedNamespaces, and the ingress controller. No probe uses
|
|
this port, which is what makes the restriction possible.
|
|
- management ({{ $settings.ports.management | int64 }}): open inside the cluster, because all three probes
|
|
address it. Its health and metrics surface is UNAUTHENTICATED, and before
|
|
this object the only thing protecting it was that no Ingress pointed at
|
|
it. Narrow it with networkPolicy.probeSources.
|
|
- jgroups ({{ $settings.ports.jgroups | int64 }}): this release's own pods only.
|
|
Outbound: DNS, TCP {{ $settings.db.port | int64 }} (the CONFIGURED database port — this chart derives it
|
|
rather than guessing), {{ $settings.ports.jgroups | int64 }} (the cluster bus, outbound as well as in) and 443{{ with .Values.networkPolicy.egress }}{{ with .extraPorts }},
|
|
plus TCP {{ join ", " . }}{{ end }}{{ end }}.
|
|
|
|
TWO THINGS TO CHECK AFTER THE FIRST UPGRADE, because nothing else will.
|
|
|
|
1. LOG IN. This release is the cluster's identity provider, and the http rule
|
|
is the only source-restricted inbound rule in the repository. If the
|
|
ingress-controller source is wrong for this cluster, every login
|
|
everywhere stops — and no part of CI can see it, because no ingress
|
|
controller runs there and the probes use the management port, so the pod
|
|
becomes Ready either way. The default was measured against the live
|
|
cluster (DaemonSet rke2-traefik in kube-system, which serves the Gateway
|
|
API here too), but measured labels are not proven traffic.
|
|
2. CHECK YOUR CONSUMERS' ISSUER URLS. An application that reaches Keycloak on
|
|
its in-cluster Service address rather than its public URL is a pod in
|
|
another namespace, and the http rule blocks it. Those namespaces go into
|
|
networkPolicy.allowedNamespaces. Consumers using the public issuer URL
|
|
arrive through Traefik and are already covered.
|
|
|
|
AND MAIL IS NOT ALLOWED. Keycloak's SMTP server is configured per realm,
|
|
which this chart never sees, so verification and password-reset mail needs its
|
|
port (587 or 465) in networkPolicy.egress.extraPorts. The symptom otherwise is
|
|
mail that is never delivered, with a healthy pod behind it.
|
|
|
|
Three limits belong in the same breath, because none of them is visible in
|
|
the object itself:
|
|
|
|
- EGRESS IS BY PORT, NEVER BY DESTINATION. A NetworkPolicy matches CIDRs and
|
|
selectors and cannot express a hostname, so every egress rule permits its
|
|
port to ANY address. That is a limit of the object, not a shortcut — this
|
|
cluster's CNI (Canal: Flannel plus Calico) has no FQDN rule either. An
|
|
outbound port the chart does not know about is BLOCKED: add it to
|
|
networkPolicy.egress.extraPorts, and expect to find out through an
|
|
application-level connection error rather than through this object.
|
|
{{- if .Values.networkPolicy.probeSources }}
|
|
- THE PROBE PATH IS RESTRICTED in this release, because
|
|
networkPolicy.probeSources is set to
|
|
{{ join ", " .Values.networkPolicy.probeSources }}
|
|
Probes come from the IP of the WORKER NODE running the pod. If none of those
|
|
CIDRs covers the node this pod lands on, the kubelet's probes are dropped
|
|
and THE POD NEVER BECOMES READY — the rollout then times out and the symptom
|
|
looks like a broken application. Compare
|
|
|
|
kubectl -n {{ .Release.Namespace }} describe pod -l app={{ .Release.Name }}-{{ .Chart.Name }}-deployment
|
|
|
|
against `kubectl get node -o wide`.
|
|
{{- else }}
|
|
- THE PROBED PORT IS OPEN INSIDE THE CLUSTER. networkPolicy.probeSources is
|
|
empty, so that rule carries no source selector at all and any pod the CNI
|
|
routes to can reach it. Not from the internet — but this is NOT pod-level
|
|
isolation. Narrowing it means listing the worker node IPs in
|
|
networkPolicy.probeSources, because the kubelet probes from the node's own
|
|
address and the chart cannot know it; a wrong guess there leaves the pod
|
|
permanently un-Ready.
|
|
{{- end }}
|
|
- IT IS ONLY A BOUNDARY WHERE THE CNI ENFORCES POLICIES. Where it does not,
|
|
this object is accepted and filters nothing, so everything reads as
|
|
configured while nothing is filtered. `kubectl -n {{ .Release.Namespace }} get networkpolicy`
|
|
shows the object exists; only a connection attempt from an unrelated
|
|
namespace shows that it bites.
|
|
|
|
Switch it off with networkPolicy.enabled: false.
|
|
{{- end }}
|