{{- /* NOTHING IN CI READS THIS FILE. `helm template` does not emit notes and no workflow or script under ci/ reads it, so every claim below carries review-only assurance - it is checked by being read, not by a stage. A rendering ERROR here does fail `helm template`, which is all the pipeline can see of it. */ -}} {{- $settings := fromYaml (include "keycloak.settings" .) -}} {{- $replicas := int (include "keycloak.replicas" (dict "scaleDown" .Values.scaleDown "replicas" .Values.keycloak.replicas)) -}} {{- $adminHostname := eq (include "keycloak.adminHostnameEnabled" .) "true" -}} {{- $httpRoute := .Values.httpRoute | default dict -}} {{- $httpRouteAdmin := $httpRoute.admin | default dict -}} {{- $hostnames := $httpRoute.hostnames | default list -}} {{- if not $hostnames -}} {{- $hostnames = list .Values.ingress.domain -}} {{- end -}} {{- $adminHostnames := $httpRouteAdmin.hostnames | default list -}} {{- if not $adminHostnames -}} {{- $adminHostnames = list (.Values.ingress.admin | default dict).domain -}} {{- end -}} {{- $realmHosts := (fromYaml (include "keycloak.settings" .)).realmHosts | default list -}} {{- $deploy := printf "%s-%s-deployment" .Release.Name .Chart.Name -}} {{- $svc := printf "%s-%s-service" .Release.Name .Chart.Name -}} Keycloak {{ .Chart.AppVersion }} installed as release {{ .Release.Name }} in namespace {{ .Release.Namespace }}. {{ $replicas }} replica(s), cache=ispn with jdbc-ping discovery over the database, external PostgreSQL at {{ $settings.db.host | default "" }}:{{ $settings.db.port | int64 }}/{{ $settings.db.database }}. ENDPOINTS - {{ if .Values.ingress.enabled }}INGRESS MODE (ingress.enabled: true, httpRoute.enabled: false){{ else if $httpRoute.enabled }}GATEWAY MODE (httpRoute.enabled: true, ingress.enabled: false){{ else }}NEITHER INGRESS NOR HTTPROUTE{{ end }} {{ if .Values.ingress.enabled }} Public: {{ include "keycloak.frontendUrl" . }} {{ else if $httpRoute.enabled }}{{ range $hostnames }} Public: https://{{ . }} {{ end }}{{ else }} Public: no Ingress and no HTTPRoute (ingress.enabled and httpRoute.enabled are both false) - in-cluster only {{ end }}{{ if not $adminHostname }} Admin: no admin hostname (ingress.admin.enabled: false) - the console is served on the public host and Keycloak generates its URLs for it {{ else if $httpRoute.enabled }}{{ range $adminHostnames }} Admin: https://{{ . }}/admin {{ end }}{{ else if .Values.ingress.enabled }} Admin: {{ include "keycloak.adminUrl" . }}/admin {{ else }} Admin: {{ include "keycloak.adminUrl" . }}/admin is what KC_HOSTNAME_ADMIN says, and NOTHING PUBLISHES IT. ingress.admin.enabled is true, so the console still generates its URLs for that host, but with both modes off there is no Ingress and no HTTPRoute to carry it. Use the port-forward below, or switch a mode on. {{ end }} In-cluster: http://{{ $svc }}.{{ .Release.Namespace }}.svc.cluster.local:{{ $settings.ports.http | int64 }} Management port {{ $settings.ports.management | int64 }} is NOT published and must never be: it serves /health, /health/live, /health/ready, /health/started{{ if $settings.metrics.enabled }} and /metrics{{ end }} WITHOUT authentication. It has a ClusterIP Service ({{ .Release.Name }}-{{ .Chart.Name }}-management-service) for in-cluster scraping only. {{ if $realmHosts }} REALM HOSTNAMES - {{ len $realmHosts }} further name(s), each landing in ONE realm The object names carry the LIST INDEX and not the realm, so this is the only place that says which is which. Reordering keycloak.realmHosts therefore renames these objects and recreates them. {{ range $i, $entry := $realmHosts }} https://{{ $entry.host }} realm {{ $entry.realm }} (bare / -> /realms/{{ $entry.realm }}/account/) object {{ $.Release.Name }}-{{ $.Chart.Name }}-realm-{{ if $httpRoute.enabled }}httproute{{ else }}ingress{{ end }}{{ if $i }}-{{ $i }}{{ end }} {{- if not $httpRoute.enabled }} cert {{ $entry.tlsSecretName }}{{ if $entry.clusterIssuer }}, requested by THIS object ({{ $entry.clusterIssuer }}){{ else }}, NOT requested by this object - it must already exist or be issued elsewhere{{ end }} {{- end }} {{ end }} {{- /* No "and nothing publishes them" branch here, deliberately: a filled keycloak.realmHosts with both modes off ABORTS the render in keycloak.ingress.yaml, so this block can only ever be reached with a mode on. A branch for an impossible state reads like a state somebody has seen. */ -}} {{- end }} {{ if $httpRoute.enabled }} GATEWAY MODE - WHAT THIS CHART DOES NOT RENDER, AND WHAT YOU MUST SET YOURSELF This release renders {{ add (ternary 2 1 $adminHostname) (len $realmHosts) }} HTTPRoute(s) and NOTHING else of the routing path - no Gateway, no class, no Certificate, no policy object. {{ .Release.Name }}-{{ .Chart.Name }}-httproute {{ join ", " $hostnames }} {{- if $adminHostname }} {{ .Release.Name }}-{{ .Chart.Name }}-admin-httproute {{ join ", " $adminHostnames }} {{- end }} {{- range $i, $entry := $realmHosts }} {{ $.Release.Name }}-{{ $.Chart.Name }}-realm-httproute{{ if $i }}-{{ $i }}{{ end }} {{ $entry.host }} (realm {{ $entry.realm }}) {{- end }} Three things therefore belong to the cluster, and if any is missing the hostnames above simply do not answer: 1. The Gateway-API CRDs. Without them this install would have failed, so they are present - nothing to check. 2. The Gateway(s) named in parentRefs {{- range $httpRoute.parentRefs }} {{ .name }}{{ if .namespace }} in namespace {{ .namespace }}{{ end }}{{ if .sectionName }}, listener {{ .sectionName }}{{ end }} (public) {{- end }} {{- if $adminHostname }} {{- range ($httpRouteAdmin.parentRefs | default $httpRoute.parentRefs) }} {{ .name }}{{ if .namespace }} in namespace {{ .namespace }}{{ end }}{{ if .sectionName }}, listener {{ .sectionName }}{{ end }} (admin) {{- end }} {{- end }} {{- range $i, $entry := $realmHosts }} {{- range ($entry.parentRefs | default $httpRoute.parentRefs) }} {{ .name }}{{ if .namespace }} in namespace {{ .namespace }}{{ end }}{{ if .sectionName }}, listener {{ .sectionName }}{{ end }} (realm {{ $entry.realm }}) {{- end }} {{- end }} must exist AND their listeners must allow routes from this namespace ({{ .Release.Namespace }}) in allowedRoutes. A rejected attachment is visible only in the route's status: kubectl -n {{ .Release.Namespace }} get httproute {{ .Release.Name }}-{{ .Chart.Name }}-httproute -o yaml Read status.parents[].conditions - "Accepted: True" and "ResolvedRefs: True" is what a working attachment looks like. An empty status means no controller has looked at the route at all. 3. The TLS certificates for those hostnames, on the Gateway's listeners. ingress.clusterIssuer is NOT used in this mode and is not required: the chart renders no Certificate here, so there is no tls-{{ .Release.Name }}-{{ .Chart.Name }}-ingress secret and nothing for cert-manager to do for this release. {{- if $realmHosts }} The realm hostnames above are part of that: their tlsSecretName values are not read in this mode either. Where several of them share a DNS zone, ONE wildcard listener for that zone replaces one certificate per name - which is the one thing this mode makes cheaper here rather than dearer. {{- end }} AND THE ONE YOU WILL NOT NOTICE UNTIL A LOGIN FAILS: THE PROXY BUFFERS. Keycloak's responses carry large headers - JWTs, SAML assertions, long Set-Cookie chains during the login flow. In Ingress mode that is handled by nginx.ingress.kubernetes.io/proxy-buffer-size: 16k nginx.ingress.kubernetes.io/proxy-buffers-number: 4 which this chart has never set: they come from ingress.annotations and ingress.admin.annotations, i.e. from whoever installs it. IN THIS MODE THEY ARE GONE and there is NO counterpart in the Gateway API - buffer sizing is implementation policy, not routing, and this chart renders no policy object on purpose (it would bind the chart to one Gateway implementation). The failure mode, so you recognise it: the PROXY answers 502 while Keycloak logs the request as successful. Nothing says a chart value was ignored, because no chart value was involved. Set the equivalent next to the Gateway, in your implementation's own object, BEFORE you point DNS at it - and raise it, do not just match 16k, if you use long-lived tokens or SAML. ingress.annotations and ingress.admin.annotations are inert here in general: the Gateway API has no annotation channel for controller behaviour. An IP allow-list for the admin host is the other thing that has to move - and read the first trap below before relying on it in either mode. {{ if $realmHosts }} AND THE SAME LOSS ONCE MORE, ON THE REALM HOSTNAMES - where it is likelier to be met rather than merely possible. Each entry of keycloak.realmHosts carries its own annotations block in Ingress mode, and in this mode every one of them is gone: {{ len $realmHosts }} hostname(s) here, none of which can configure a proxy buffer through this chart any more. What the entries CAN still express is the routing - the redirect into the realm is a rule of the route and survives the mode change intact. Why these hostnames are the awkward ones: they exist to put somebody in an ACCOUNT CONSOLE. That is a logged-in session from the first request after the redirect, so the large Authorization and Set-Cookie headers are the normal traffic on these names rather than an edge case the way they are on a login host that mostly serves anonymous requests. A buffer sized for the default therefore fails here FIRST, and it fails as a 502 from the Gateway on a hostname whose Ingress never had a chart value to point at - so there is nothing to grep for and nothing to un-set. Set the buffer equivalent for these hostnames too, in the Gateway implementation's own object, and set it before the names resolve there. {{- end }} THE ADMIN HOSTNAME'S "INTERNAL CLASS" MOVES TO THE PARENT. What ingress.admin.className does in the other mode, httpRoute.admin.parentRefs does here: point the admin route at an internal Gateway. It falls back to httpRoute.parentRefs, so leaving it empty puts both hostnames on the same Gateway. {{ end }} THE 1PASSWORD ITEMS, AND WHICH FIELDS THEY MUST CARRY The operator materializes one Secret key per item FIELD, so a missing or misnamed field leaves the pod in CreateContainerConfigError - with no message naming the field. {{ .Values.secrets.databaseSecretRef | default "" }} username the PostgreSQL role password that role's password The role must be able to CREATE its schema: Keycloak builds and migrates the whole schema itself on first start. {{ if $settings.bootstrapAdmin.enabled }} {{ .Values.secrets.bootstrapAdminSecretRef | default "" }} username the TEMPORARY bootstrap administrator (upstream default name: temp-admin) password its password {{ else }} (no bootstrap admin item - keycloak.bootstrapAdmin.enabled is false, which is the correct state once a permanent administrator exists) {{ end }} Both arrive as ENVIRONMENT VARIABLES, not as files, and that is forced: Keycloak 26 has no _FILE form for an option's value. The price: the password is in the container's process environment. FIRST INSTALL - AND STEP 4 IS MANDATORY, NOT ADVICE 1. kubectl -n {{ .Release.Namespace }} rollout status deployment/{{ $deploy }} Ready means /health/ready answered 200, which means the database is reachable and the schema is in place. The first start is the slow one: it creates the entire schema through Liquibase, and with {{ $replicas }} replica(s) the second waits on the first's migration lock. 2. Check that the cluster really formed. THIS STEP IS NOT OPTIONAL AND NO PROBE REPLACES IT: a node that never found its peer is Ready, healthy and alone, and two such nodes are two islands that each believe they are the cluster. The install-test cannot catch it either. kubectl -n {{ .Release.Namespace }} logs deployment/{{ $deploy }} | grep -i "ISPN000094\|view accepted\|members" Look for a view with {{ $replicas }} members. One member per pod means jdbc-ping did not find the peer - check that both pods can reach each other on port {{ $settings.ports.jgroups | int64 }} and that they really share one database. 3. Log in at {{ if and $adminHostname (not $httpRoute.enabled) }}{{ include "keycloak.adminUrl" . }}/admin{{ else if $adminHostname }}https://{{ first $adminHostnames }}/admin{{ else if .Values.ingress.enabled }}{{ include "keycloak.frontendUrl" . }}/admin{{ else if $httpRoute.enabled }}https://{{ first $hostnames }}/admin{{ else }}the in-cluster address, port-forwarded{{ end }} with the bootstrap credentials from the vault item above. 4. CREATE A PERMANENT ADMINISTRATOR, then remove the temporary one. The bootstrap account is TEMPORARY by design - upstream: "the account should exist only for the duration necessary to perform operations needed to gain permanent and more secure admin access. After that, the account needs to be removed manually." This is different from Keycloak before 26, where KEYCLOAK_ADMIN created a regular administrator and there was nothing to clean up. a) In the master realm, create a user, set a password, assign the realm-admin / admin role. b) Verify you can log in as that user. c) Delete the temp-admin user. d) helm upgrade with keycloak.bootstrapAdmin.enabled: false 5. Until step 4d, EVERY START LOGS a line about the initial administrator not being created, because the master realm already exists and the bootstrap path is skipped. It is harmless. It is also the kind of message somebody eventually investigates, which is why 4d is part of the install and not a tidy-up. MINOR UPGRADES NEED THE CLUSTER DOWN - AND THE AUTOMERGE DOES NOT KNOW THAT Keycloak supports a rolling update only WITHIN a minor line. A minor jump (26.8 -> 26.9) requires every node shut down before any new node starts; mixing two minor versions in one Infinispan cluster is not a supported configuration, and nothing in the cluster reports that it happened. THIS REPOSITORY'S RENOVATE AUTOMERGES EVERY UPDATE TYPE, MAJORS INCLUDED. That is a deliberate, accepted risk and not an oversight (Gitea #83), but it means an app-version PR that moves the minor can reach main, publish the chart and be rolled out as a rolling update without anybody deciding to. Watch app-version PRs for a changed minor, and if one has landed, take this path rather than letting the rollout happen: 1. Decide it with the tool instead of the version number. With the OLD version and the OLD configuration: kc.sh update-compatibility metadata --file=/tmp/kc-compat.json then with the NEW version and the NEW configuration: kc.sh update-compatibility check --file=/tmp/kc-compat.json Pass EVERY configuration option on both calls - upstream: "Omitting any configuration options results in incomplete metadata, and could lead to a wrong reported result". Exit codes, which is what to automate on: 0 rolling update possible, 1 unexpected error, 2 invalid CLI option, 3 and 4 not possible - the deployment must be shut down. 2. If it says shut down: helm upgrade ... --set scaleDown=true # every pod stops # back up the database AND the configuration before anything else helm upgrade ... --set scaleDown=false # with the new version A NOTE ON WHAT IS NOT SETTLED: upstream's 26.6 release notes call patch-level rolling updates supported and active by default, while the 26.8.0 page for update-compatibility says a rolling update is possible "when the Keycloak version is the same" and uses a PATCH pair (26.2.0 -> 26.2.1) as its example of an incompatible one. Those two cannot both be current. Until it is clear, run the check for patch updates too and trust its exit code rather than the size of the version step. TRAPS WORTH KNOWING BEFORE YOU NEED THEM - THE ADMIN HOSTNAME IS NOT AN ACCESS CONTROL. hostname-admin decides which URLs the server generates, nothing more. Upstream: "Using the hostname-admin option does not prevent accessing the Administration REST API endpoints via the frontend URL specified by the hostname option. If you want to restrict access to the Administration REST API, you need to do it on the reverse proxy level." So /admin/realms/... is reachable through the public host whatever you put on the admin host, and an IP allow-list on the admin Ingress restricts nothing on its own. This chart renders no deny rule for /admin - see values.yaml for why not.{{ if $httpRoute.enabled }} In GATEWAY MODE there is not even a place in this chart to put one: no annotation channel, and no implementation-specific policy object by design. The restriction belongs next to the Gateway, which is where it can also be tested.{{ end }} - A SHARED DATABASE MERGES TWO CLUSTERS. Node discovery runs through the database, so - upstream - "clusters that share the same database but should remain separate will merge unless they have distinct cluster names. The default cluster name is ISPN." Two releases of this chart on one database therefore form ONE cluster, silently. {{ with $settings.cache.clusterName }}This release sets cache-embedded-cluster-name={{ . }}.{{ else }}This release uses the default name (ISPN) - keycloak.cache.clusterName is unset. Read its comment in values.yaml before setting it: upstream also warns about cache invalidation between differently-named clusters.{{ end }} - READINESS ONLY CHECKS THE DATABASE WHILE metrics-enabled IS TRUE. Upstream's table of health checks has a "Requires Metrics" column and the DATABASE check is the row that says yes. So with keycloak.metrics.enabled false, /health/ready answers 200 with the database gone and the pod keeps taking logins it cannot serve. This release has it {{ if $settings.metrics.enabled }}ON, which is the chart default and the reason the default is what it is.{{ else }}OFF, so the readiness probe cannot currently detect a database outage.{{ end }} - THERE IS NO ConfigMap IN THIS CHART, so no checksum/config annotation. That is not an omission: Keycloak is configured entirely through options and KC_* variables, so there is no config file whose change could fail to restart anything. - A CHANGE INSIDE A VAULT ITEM RESTARTS NOTHING. Helm never sees it. After rotating a field: kubectl -n {{ .Release.Namespace }} rollout restart deployment/{{ $deploy }} (Repointing a secrets.*SecretRef at a different item DOES roll the pods - that is covered by checksum/secret-ref.) - THE POD HAS TWO INIT CONTAINERS AND NEEDS BOTH. db, health-enabled and metrics-enabled are Keycloak BUILD options, the published image is built with their defaults, and a runtime change makes Keycloak rewrite /opt/keycloak/lib/quarkus at start - which readOnlyRootFilesystem: true forbids. So one init container seeds that directory into a volume and the next runs `kc.sh build` into it; the serving container then runs `start --optimized`. If a pod is stuck in Init, read those logs first: kubectl -n {{ .Release.Namespace }} logs -c {{ .Release.Name }}-{{ .Chart.Name }}-build Build options are set ONLY through keycloak.db.vendor, keycloak.metrics.enabled and keycloak.features. Putting one into keycloak.env fails the render on purpose. - POOL SIZES ARE PER REPLICA. keycloak.db.pool.maxSize is {{ $settings.db.pool.maxSize | int64 }}, so with the current {{ $replicas }} replica(s) this release can open up to {{ mul ($settings.db.pool.maxSize | int64) $replicas }} connections. Upstream's default is 100 per node, which with two nodes exceeds a default PostgreSQL's max_connections of 100. Raise it together with the server, never alone. {{ if and $settings.podDisruptionBudget.enabled (ge $replicas 2) }} - A PodDisruptionBudget IS in place ({{ .Release.Name }}-{{ .Chart.Name }}-pdb, minAvailable {{ $settings.podDisruptionBudget.minAvailable | int64 }}), so a node drain cannot take all replicas at once. {{ else if $settings.podDisruptionBudget.enabled }} - NO PodDisruptionBudget was rendered, because the effective replica count is {{ $replicas }}. That is deliberate: minAvailable 1 with a single replica forbids every voluntary eviction, so `kubectl drain` would block forever on a pod nothing may move - and the error would name the PDB rather than the replica count. Run two replicas to get the budget. {{ else }} - NO PodDisruptionBudget (keycloak.podDisruptionBudget.enabled is false). A node drain may take every replica at once. {{ end }} WHAT THIS CHART DOES NOT RENDER - No PVC and no StatefulSet: Keycloak 26 keeps everything in the database, user sessions included (persistent sessions are the default; volatile ones are deprecated as of 26.8). There is no disk state to keep. - No NetworkPolicy. jdbc-ping opens direct pod-to-pod connections on port {{ $settings.ports.jgroups | int64 }}, so any policy added later has to allow that between the replicas, or the cluster silently becomes two islands. - No deny rule for /admin on the public Ingress{{ if $httpRoute.enabled }} or route{{ end }}, and no restriction of the administration REST API. That is reverse-proxy work - see the first trap. - No Gateway, no GatewayClass and no listener, in either mode. Those are platform objects and live in cluster-config; this chart renders an HTTPRoute at most, which is the part that looks the same on every Gateway implementation. {{- if .Values.networkPolicy.enabled }} NETWORKPOLICY IS ACTIVE — WHAT IT CAPS, AND WHAT IT DOES NOT {{ .Release.Name }}-{{ .Chart.Name }}-networkpolicy selects this workload's pods. Inbound, and all three ports differ: - http ({{ $settings.ports.http | int64 }}): RESTRICTED BY SOURCE — this namespace, anything in networkPolicy.allowedNamespaces, and the ingress controller. No probe uses this port, which is what makes the restriction possible. - management ({{ $settings.ports.management | int64 }}): open inside the cluster, because all three probes address it. Its health and metrics surface is UNAUTHENTICATED, and before this object the only thing protecting it was that no Ingress pointed at it. Narrow it with networkPolicy.probeSources. - jgroups ({{ $settings.ports.jgroups | int64 }}): this release's own pods only. Outbound: DNS, TCP {{ $settings.db.port | int64 }} (the CONFIGURED database port — this chart derives it rather than guessing), {{ $settings.ports.jgroups | int64 }} (the cluster bus, outbound as well as in) and 443{{ with .Values.networkPolicy.egress }}{{ with .extraPorts }}, plus TCP {{ join ", " . }}{{ end }}{{ end }}. TWO THINGS TO CHECK AFTER THE FIRST UPGRADE, because nothing else will. 1. LOG IN. This release is the cluster's identity provider, and the http rule is the only source-restricted inbound rule in the repository. If the ingress-controller source is wrong for this cluster, every login everywhere stops — and no part of CI can see it, because no ingress controller runs there and the probes use the management port, so the pod becomes Ready either way. The default was measured against the live cluster (DaemonSet rke2-traefik in kube-system, which serves the Gateway API here too), but measured labels are not proven traffic. 2. CHECK YOUR CONSUMERS' ISSUER URLS. An application that reaches Keycloak on its in-cluster Service address rather than its public URL is a pod in another namespace, and the http rule blocks it. Those namespaces go into networkPolicy.allowedNamespaces. Consumers using the public issuer URL arrive through Traefik and are already covered. AND MAIL IS NOT ALLOWED. Keycloak's SMTP server is configured per realm, which this chart never sees, so verification and password-reset mail needs its port (587 or 465) in networkPolicy.egress.extraPorts. The symptom otherwise is mail that is never delivered, with a healthy pod behind it. Three limits belong in the same breath, because none of them is visible in the object itself: - EGRESS IS BY PORT, NEVER BY DESTINATION. A NetworkPolicy matches CIDRs and selectors and cannot express a hostname, so every egress rule permits its port to ANY address. That is a limit of the object, not a shortcut — this cluster's CNI (Canal: Flannel plus Calico) has no FQDN rule either. An outbound port the chart does not know about is BLOCKED: add it to networkPolicy.egress.extraPorts, and expect to find out through an application-level connection error rather than through this object. {{- if .Values.networkPolicy.probeSources }} - THE PROBE PATH IS RESTRICTED in this release, because networkPolicy.probeSources is set to {{ join ", " .Values.networkPolicy.probeSources }} Probes come from the IP of the WORKER NODE running the pod. If none of those CIDRs covers the node this pod lands on, the kubelet's probes are dropped and THE POD NEVER BECOMES READY — the rollout then times out and the symptom looks like a broken application. Compare kubectl -n {{ .Release.Namespace }} describe pod -l app={{ .Release.Name }}-{{ .Chart.Name }}-deployment against `kubectl get node -o wide`. {{- else }} - THE PROBED PORT IS OPEN INSIDE THE CLUSTER. networkPolicy.probeSources is empty, so that rule carries no source selector at all and any pod the CNI routes to can reach it. Not from the internet — but this is NOT pod-level isolation. Narrowing it means listing the worker node IPs in networkPolicy.probeSources, because the kubelet probes from the node's own address and the chart cannot know it; a wrong guess there leaves the pod permanently un-Ready. {{- end }} - IT IS ONLY A BOUNDARY WHERE THE CNI ENFORCES POLICIES. Where it does not, this object is accepted and filters nothing, so everything reads as configured while nothing is filtered. `kubectl -n {{ .Release.Namespace }} get networkpolicy` shows the object exists; only a connection attempt from an unrelated namespace shows that it bites. Switch it off with networkPolicy.enabled: false. {{- end }}