helm-charts/keycloak/templates/NOTES.txt

{{- /*
NOTHING IN CI READS THIS FILE. `helm template` does not emit notes and no
workflow or script under ci/ reads it, so every claim below carries
review-only assurance - it is checked by being read, not by a stage. A
rendering ERROR here does fail `helm template`, which is all the pipeline can
see of it.
*/ -}}
{{- $settings := fromYaml (include "keycloak.settings" .) -}}
{{- $replicas := int (include "keycloak.replicas" (dict "scaleDown" .Values.scaleDown "replicas" .Values.keycloak.replicas)) -}}
{{- $adminHostname := eq (include "keycloak.adminHostnameEnabled" .) "true" -}}
{{- $httpRoute := .Values.httpRoute | default dict -}}
{{- $httpRouteAdmin := $httpRoute.admin | default dict -}}
{{- $hostnames := $httpRoute.hostnames | default list -}}
{{- if not $hostnames -}}
{{- $hostnames = list .Values.ingress.domain -}}
{{- end -}}
{{- $adminHostnames := $httpRouteAdmin.hostnames | default list -}}
{{- if not $adminHostnames -}}
{{- $adminHostnames = list (.Values.ingress.admin | default dict).domain -}}
{{- end -}}
{{- $realmHosts := (fromYaml (include "keycloak.settings" .)).realmHosts | default list -}}
{{- $deploy := printf "%s-%s-deployment" .Release.Name .Chart.Name -}}
{{- $svc := printf "%s-%s-service" .Release.Name .Chart.Name -}}
Keycloak {{ .Chart.AppVersion }} installed as release {{ .Release.Name }} in namespace {{ .Release.Namespace }}.

{{ $replicas }} replica(s), cache=ispn with jdbc-ping discovery over the database,
external PostgreSQL at {{ $settings.db.host | default "<keycloak.db.host NOT SET>" }}:{{ $settings.db.port | int64 }}/{{ $settings.db.database }}.

ENDPOINTS - {{ if .Values.ingress.enabled }}INGRESS MODE (ingress.enabled: true, httpRoute.enabled: false){{ else if $httpRoute.enabled }}GATEWAY MODE (httpRoute.enabled: true, ingress.enabled: false){{ else }}NEITHER INGRESS NOR HTTPROUTE{{ end }}
{{ if .Values.ingress.enabled }}  Public:  {{ include "keycloak.frontendUrl" . }}
{{ else if $httpRoute.enabled }}{{ range $hostnames }}  Public:  https://{{ . }}
{{ end }}{{ else }}  Public:  no Ingress and no HTTPRoute (ingress.enabled and httpRoute.enabled are
           both false) - in-cluster only
{{ end }}{{ if not $adminHostname }}  Admin:   no admin hostname (ingress.admin.enabled: false) - the console is
           served on the public host and Keycloak generates its URLs for it
{{ else if $httpRoute.enabled }}{{ range $adminHostnames }}  Admin:   https://{{ . }}/admin
{{ end }}{{ else if .Values.ingress.enabled }}  Admin:   {{ include "keycloak.adminUrl" . }}/admin
{{ else }}  Admin:   {{ include "keycloak.adminUrl" . }}/admin is what KC_HOSTNAME_ADMIN says, and
           NOTHING PUBLISHES IT. ingress.admin.enabled is true, so the console
           still generates its URLs for that host, but with both modes off
           there is no Ingress and no HTTPRoute to carry it. Use the
           port-forward below, or switch a mode on.
{{ end }}  In-cluster: http://{{ $svc }}.{{ .Release.Namespace }}.svc.cluster.local:{{ $settings.ports.http | int64 }}

  Management port {{ $settings.ports.management | int64 }} is NOT published and must never be: it serves
  /health, /health/live, /health/ready, /health/started{{ if $settings.metrics.enabled }} and /metrics{{ end }}
  WITHOUT authentication. It has a ClusterIP Service
  ({{ .Release.Name }}-{{ .Chart.Name }}-management-service) for in-cluster scraping only.
{{ if $realmHosts }}
REALM HOSTNAMES - {{ len $realmHosts }} further name(s), each landing in ONE realm

  The object names carry the LIST INDEX and not the realm, so this is the only
  place that says which is which. Reordering keycloak.realmHosts therefore
  renames these objects and recreates them.
{{ range $i, $entry := $realmHosts }}
    https://{{ $entry.host }}
      realm      {{ $entry.realm }}  (bare / -> /realms/{{ $entry.realm }}/account/)
      object     {{ $.Release.Name }}-{{ $.Chart.Name }}-realm-{{ if $httpRoute.enabled }}httproute{{ else }}ingress{{ end }}{{ if $i }}-{{ $i }}{{ end }}
{{- if not $httpRoute.enabled }}
      cert       {{ $entry.tlsSecretName }}{{ if $entry.clusterIssuer }}, requested by THIS object ({{ $entry.clusterIssuer }}){{ else }}, NOT requested by this object - it must already exist or be issued elsewhere{{ end }}
{{- end }}
{{ end }}
{{- /*
No "and nothing publishes them" branch here, deliberately: a filled
keycloak.realmHosts with both modes off ABORTS the render in
keycloak.ingress.yaml, so this block can only ever be reached with a mode on.
A branch for an impossible state reads like a state somebody has seen.
*/ -}}
{{- end }}
{{ if $httpRoute.enabled }}
GATEWAY MODE - WHAT THIS CHART DOES NOT RENDER, AND WHAT YOU MUST SET YOURSELF

  This release renders {{ add (ternary 2 1 $adminHostname) (len $realmHosts) }} HTTPRoute(s) and NOTHING else of the routing path -
  no Gateway, no class, no Certificate, no policy object.

    {{ .Release.Name }}-{{ .Chart.Name }}-httproute         {{ join ", " $hostnames }}
{{- if $adminHostname }}
    {{ .Release.Name }}-{{ .Chart.Name }}-admin-httproute   {{ join ", " $adminHostnames }}
{{- end }}
{{- range $i, $entry := $realmHosts }}
    {{ $.Release.Name }}-{{ $.Chart.Name }}-realm-httproute{{ if $i }}-{{ $i }}{{ end }}   {{ $entry.host }}   (realm {{ $entry.realm }})
{{- end }}

  Three things therefore belong to the cluster, and if any is missing the
  hostnames above simply do not answer:

    1. The Gateway-API CRDs. Without them this install would have failed, so
       they are present - nothing to check.
    2. The Gateway(s) named in parentRefs
{{- range $httpRoute.parentRefs }}
         {{ .name }}{{ if .namespace }} in namespace {{ .namespace }}{{ end }}{{ if .sectionName }}, listener {{ .sectionName }}{{ end }}   (public)
{{- end }}
{{- if $adminHostname }}
{{- range ($httpRouteAdmin.parentRefs | default $httpRoute.parentRefs) }}
         {{ .name }}{{ if .namespace }} in namespace {{ .namespace }}{{ end }}{{ if .sectionName }}, listener {{ .sectionName }}{{ end }}   (admin)
{{- end }}
{{- end }}
{{- range $i, $entry := $realmHosts }}
{{- range ($entry.parentRefs | default $httpRoute.parentRefs) }}
         {{ .name }}{{ if .namespace }} in namespace {{ .namespace }}{{ end }}{{ if .sectionName }}, listener {{ .sectionName }}{{ end }}   (realm {{ $entry.realm }})
{{- end }}
{{- end }}
       must exist AND their listeners must allow routes from this namespace
       ({{ .Release.Namespace }}) in allowedRoutes. A rejected attachment is visible only
       in the route's status:

         kubectl -n {{ .Release.Namespace }} get httproute {{ .Release.Name }}-{{ .Chart.Name }}-httproute -o yaml

       Read status.parents[].conditions - "Accepted: True" and
       "ResolvedRefs: True" is what a working attachment looks like. An empty
       status means no controller has looked at the route at all.
    3. The TLS certificates for those hostnames, on the Gateway's listeners.
       ingress.clusterIssuer is NOT used in this mode and is not required: the
       chart renders no Certificate here, so there is no
       tls-{{ .Release.Name }}-{{ .Chart.Name }}-ingress secret and nothing for cert-manager to do
       for this release.
{{- if $realmHosts }}
       The realm hostnames above are part of that: their tlsSecretName values
       are not read in this mode either. Where several of them share a DNS zone,
       ONE wildcard listener for that zone replaces one certificate per name -
       which is the one thing this mode makes cheaper here rather than dearer.
{{- end }}

  AND THE ONE YOU WILL NOT NOTICE UNTIL A LOGIN FAILS: THE PROXY BUFFERS.

  Keycloak's responses carry large headers - JWTs, SAML assertions, long
  Set-Cookie chains during the login flow. In Ingress mode that is handled by

    nginx.ingress.kubernetes.io/proxy-buffer-size:    16k
    nginx.ingress.kubernetes.io/proxy-buffers-number: 4

  which this chart has never set: they come from ingress.annotations and
  ingress.admin.annotations, i.e. from whoever installs it. IN THIS MODE THEY
  ARE GONE and there is NO counterpart in the Gateway API - buffer sizing is
  implementation policy, not routing, and this chart renders no policy object
  on purpose (it would bind the chart to one Gateway implementation).

  The failure mode, so you recognise it: the PROXY answers 502 while Keycloak
  logs the request as successful. Nothing says a chart value was ignored,
  because no chart value was involved. Set the equivalent next to the Gateway,
  in your implementation's own object, BEFORE you point DNS at it - and raise
  it, do not just match 16k, if you use long-lived tokens or SAML.

  ingress.annotations and ingress.admin.annotations are inert here in general:
  the Gateway API has no annotation channel for controller behaviour. An IP
  allow-list for the admin host is the other thing that has to move - and read
  the first trap below before relying on it in either mode.
{{ if $realmHosts }}
  AND THE SAME LOSS ONCE MORE, ON THE REALM HOSTNAMES - where it is likelier to
  be met rather than merely possible.

  Each entry of keycloak.realmHosts carries its own annotations block in Ingress
  mode, and in this mode every one of them is gone: {{ len $realmHosts }} hostname(s) here, none
  of which can configure a proxy buffer through this chart any more. What the
  entries CAN still express is the routing - the redirect into the realm is a
  rule of the route and survives the mode change intact.

  Why these hostnames are the awkward ones: they exist to put somebody in an
  ACCOUNT CONSOLE. That is a logged-in session from the first request after the
  redirect, so the large Authorization and Set-Cookie headers are the normal
  traffic on these names rather than an edge case the way they are on a login
  host that mostly serves anonymous requests. A buffer sized for the default
  therefore fails here FIRST, and it fails as a 502 from the Gateway on a
  hostname whose Ingress never had a chart value to point at - so there is
  nothing to grep for and nothing to un-set.

  Set the buffer equivalent for these hostnames too, in the Gateway
  implementation's own object, and set it before the names resolve there.
{{- end }}

  THE ADMIN HOSTNAME'S "INTERNAL CLASS" MOVES TO THE PARENT. What
  ingress.admin.className does in the other mode, httpRoute.admin.parentRefs
  does here: point the admin route at an internal Gateway. It falls back to
  httpRoute.parentRefs, so leaving it empty puts both hostnames on the same
  Gateway.
{{ end }}
THE 1PASSWORD ITEMS, AND WHICH FIELDS THEY MUST CARRY
The operator materializes one Secret key per item FIELD, so a missing or
misnamed field leaves the pod in CreateContainerConfigError - with no message
naming the field.

  {{ .Values.secrets.databaseSecretRef | default "<secrets.databaseSecretRef NOT SET>" }}
    username      the PostgreSQL role
    password      that role's password
    The role must be able to CREATE its schema: Keycloak builds and migrates
    the whole schema itself on first start.
{{ if $settings.bootstrapAdmin.enabled }}
  {{ .Values.secrets.bootstrapAdminSecretRef | default "<secrets.bootstrapAdminSecretRef NOT SET>" }}
    username      the TEMPORARY bootstrap administrator (upstream default name:
                  temp-admin)
    password      its password
{{ else }}
  (no bootstrap admin item - keycloak.bootstrapAdmin.enabled is false, which is
   the correct state once a permanent administrator exists)
{{ end }}
  Both arrive as ENVIRONMENT VARIABLES, not as files, and that is forced:
  Keycloak 26 has no _FILE form for an option's value. The price: the password
  is in the container's process environment.

FIRST INSTALL - AND STEP 4 IS MANDATORY, NOT ADVICE

  1. kubectl -n {{ .Release.Namespace }} rollout status deployment/{{ $deploy }}
     Ready means /health/ready answered 200, which means the database is
     reachable and the schema is in place. The first start is the slow one: it
     creates the entire schema through Liquibase, and with {{ $replicas }} replica(s)
     the second waits on the first's migration lock.

  2. Check that the cluster really formed. THIS STEP IS NOT OPTIONAL AND NO
     PROBE REPLACES IT: a node that never found its peer is Ready, healthy and
     alone, and two such nodes are two islands that each believe they are the
     cluster. The install-test cannot catch it either.
       kubectl -n {{ .Release.Namespace }} logs deployment/{{ $deploy }} | grep -i "ISPN000094\|view accepted\|members"
     Look for a view with {{ $replicas }} members. One member per pod means
     jdbc-ping did not find the peer - check that both pods can reach each
     other on port {{ $settings.ports.jgroups | int64 }} and that they really share one database.

  3. Log in at {{ if and $adminHostname (not $httpRoute.enabled) }}{{ include "keycloak.adminUrl" . }}/admin{{ else if $adminHostname }}https://{{ first $adminHostnames }}/admin{{ else if .Values.ingress.enabled }}{{ include "keycloak.frontendUrl" . }}/admin{{ else if $httpRoute.enabled }}https://{{ first $hostnames }}/admin{{ else }}the in-cluster address, port-forwarded{{ end }}
     with the bootstrap credentials from the vault item above.

  4. CREATE A PERMANENT ADMINISTRATOR, then remove the temporary one.
     The bootstrap account is TEMPORARY by design - upstream: "the account
     should exist only for the duration necessary to perform operations needed
     to gain permanent and more secure admin access. After that, the account
     needs to be removed manually." This is different from Keycloak before 26,
     where KEYCLOAK_ADMIN created a regular administrator and there was nothing
     to clean up.
       a) In the master realm, create a user, set a password, assign the
          realm-admin / admin role.
       b) Verify you can log in as that user.
       c) Delete the temp-admin user.
       d) helm upgrade with keycloak.bootstrapAdmin.enabled: false

  5. Until step 4d, EVERY START LOGS a line about the initial administrator not
     being created, because the master realm already exists and the bootstrap
     path is skipped. It is harmless. It is also the kind of message somebody
     eventually investigates, which is why 4d is part of the install and not a
     tidy-up.

MINOR UPGRADES NEED THE CLUSTER DOWN - AND THE AUTOMERGE DOES NOT KNOW THAT

  Keycloak supports a rolling update only WITHIN a minor line. A minor jump
  (26.8 -> 26.9) requires every node shut down before any new node starts;
  mixing two minor versions in one Infinispan cluster is not a supported
  configuration, and nothing in the cluster reports that it happened.

  THIS REPOSITORY'S RENOVATE AUTOMERGES EVERY UPDATE TYPE, MAJORS INCLUDED.
  That is a deliberate, accepted risk and not an oversight (Gitea #83), but it
  means an app-version PR that moves the minor can reach main, publish the
  chart and be rolled out as a rolling update without anybody deciding to.
  Watch app-version PRs for a changed minor, and if one has landed, take this
  path rather than letting the rollout happen:

    1. Decide it with the tool instead of the version number. With the OLD
       version and the OLD configuration:
         kc.sh update-compatibility metadata --file=/tmp/kc-compat.json
       then with the NEW version and the NEW configuration:
         kc.sh update-compatibility check --file=/tmp/kc-compat.json
       Pass EVERY configuration option on both calls - upstream: "Omitting any
       configuration options results in incomplete metadata, and could lead to
       a wrong reported result".
       Exit codes, which is what to automate on: 0 rolling update possible,
       1 unexpected error, 2 invalid CLI option, 3 and 4 not possible - the
       deployment must be shut down.

    2. If it says shut down:
         helm upgrade ... --set scaleDown=true      # every pod stops
         # back up the database AND the configuration before anything else
         helm upgrade ... --set scaleDown=false     # with the new version

  A NOTE ON WHAT IS NOT SETTLED: upstream's 26.6 release notes call patch-level
  rolling updates supported and active by default, while the 26.8.0 page for
  update-compatibility says a rolling update is possible "when the Keycloak
  version is the same" and uses a PATCH pair (26.2.0 -> 26.2.1) as its example
  of an incompatible one. Those two cannot both be current. Until it is clear,
  run the check for patch updates too and trust its exit code rather than the
  size of the version step.

TRAPS WORTH KNOWING BEFORE YOU NEED THEM

  - THE ADMIN HOSTNAME IS NOT AN ACCESS CONTROL. hostname-admin decides which
    URLs the server generates, nothing more. Upstream: "Using the
    hostname-admin option does not prevent accessing the Administration REST
    API endpoints via the frontend URL specified by the hostname option. If you
    want to restrict access to the Administration REST API, you need to do it
    on the reverse proxy level." So /admin/realms/... is reachable through the
    public host whatever you put on the admin host, and an IP allow-list on the
    admin Ingress restricts nothing on its own. This chart renders no deny rule
    for /admin - see values.yaml for why not.{{ if $httpRoute.enabled }} In GATEWAY MODE there is not
    even a place in this chart to put one: no annotation channel, and no
    implementation-specific policy object by design. The restriction belongs
    next to the Gateway, which is where it can also be tested.{{ end }}

  - A SHARED DATABASE MERGES TWO CLUSTERS. Node discovery runs through the
    database, so - upstream - "clusters that share the same database but should
    remain separate will merge unless they have distinct cluster names. The
    default cluster name is ISPN." Two releases of this chart on one database
    therefore form ONE cluster, silently.
    {{ with $settings.cache.clusterName }}This release sets cache-embedded-cluster-name={{ . }}.{{ else }}This release uses the default name (ISPN) - keycloak.cache.clusterName is
    unset. Read its comment in values.yaml before setting it: upstream also
    warns about cache invalidation between differently-named clusters.{{ end }}

  - READINESS ONLY CHECKS THE DATABASE WHILE metrics-enabled IS TRUE. Upstream's
    table of health checks has a "Requires Metrics" column and the DATABASE
    check is the row that says yes. So with keycloak.metrics.enabled false,
    /health/ready answers 200 with the database gone and the pod keeps taking
    logins it cannot serve. This release has it {{ if $settings.metrics.enabled }}ON, which is the chart default
    and the reason the default is what it is.{{ else }}OFF, so the readiness probe
    cannot currently detect a database outage.{{ end }}

  - THERE IS NO ConfigMap IN THIS CHART, so no checksum/config annotation. That
    is not an omission: Keycloak is configured entirely through options and
    KC_* variables, so there is no config file whose change could fail to
    restart anything.

  - A CHANGE INSIDE A VAULT ITEM RESTARTS NOTHING. Helm never sees it. After
    rotating a field:
      kubectl -n {{ .Release.Namespace }} rollout restart deployment/{{ $deploy }}
    (Repointing a secrets.*SecretRef at a different item DOES roll the pods -
    that is covered by checksum/secret-ref.)

  - THE POD HAS TWO INIT CONTAINERS AND NEEDS BOTH. db, health-enabled and
    metrics-enabled are Keycloak BUILD options, the published image is built
    with their defaults, and a runtime change makes Keycloak rewrite
    /opt/keycloak/lib/quarkus at start - which readOnlyRootFilesystem: true
    forbids. So one init container seeds that directory into a volume and the
    next runs `kc.sh build` into it; the serving container then runs
    `start --optimized`. If a pod is stuck in Init, read those logs first:
      kubectl -n {{ .Release.Namespace }} logs <pod> -c {{ .Release.Name }}-{{ .Chart.Name }}-build
    Build options are set ONLY through keycloak.db.vendor,
    keycloak.metrics.enabled and keycloak.features. Putting one into
    keycloak.env fails the render on purpose.

  - POOL SIZES ARE PER REPLICA. keycloak.db.pool.maxSize is {{ $settings.db.pool.maxSize | int64 }}, so with the
    current {{ $replicas }} replica(s) this release can open up to {{ mul ($settings.db.pool.maxSize | int64) $replicas }} connections.
    Upstream's default is 100 per node, which with two nodes exceeds a default
    PostgreSQL's max_connections of 100. Raise it together with the server,
    never alone.
{{ if and $settings.podDisruptionBudget.enabled (ge $replicas 2) }}
  - A PodDisruptionBudget IS in place ({{ .Release.Name }}-{{ .Chart.Name }}-pdb, minAvailable
    {{ $settings.podDisruptionBudget.minAvailable | int64 }}), so a node drain cannot take all replicas at once.
{{ else if $settings.podDisruptionBudget.enabled }}
  - NO PodDisruptionBudget was rendered, because the effective replica count is
    {{ $replicas }}. That is deliberate: minAvailable 1 with a single replica forbids every
    voluntary eviction, so `kubectl drain` would block forever on a pod nothing
    may move - and the error would name the PDB rather than the replica count.
    Run two replicas to get the budget.
{{ else }}
  - NO PodDisruptionBudget (keycloak.podDisruptionBudget.enabled is false). A
    node drain may take every replica at once.
{{ end }}
WHAT THIS CHART DOES NOT RENDER

  - No PVC and no StatefulSet: Keycloak 26 keeps everything in the database,
    user sessions included (persistent sessions are the default; volatile ones
    are deprecated as of 26.8). There is no disk state to keep.
  - No NetworkPolicy. jdbc-ping opens direct pod-to-pod connections on port
    {{ $settings.ports.jgroups | int64 }}, so any policy added later has to allow that between the
    replicas, or the cluster silently becomes two islands.
  - No deny rule for /admin on the public Ingress{{ if $httpRoute.enabled }} or route{{ end }}, and no restriction of
    the administration REST API. That is reverse-proxy work - see the first
    trap.
  - No Gateway, no GatewayClass and no listener, in either mode. Those are
    platform objects and live in cluster-config; this chart renders an
    HTTPRoute at most, which is the part that looks the same on every Gateway
    implementation.