helm-charts/keycloak/templates/NOTES.txt

{{- $settings := fromYaml (include "keycloak.settings" .) -}}
{{- $replicas := int (include "keycloak.replicas" (dict "scaleDown" .Values.scaleDown "replicas" .Values.keycloak.replicas)) -}}
{{- $adminHostname := eq (include "keycloak.adminHostnameEnabled" .) "true" -}}
{{- $deploy := printf "%s-%s-deployment" .Release.Name .Chart.Name -}}
{{- $svc := printf "%s-%s-service" .Release.Name .Chart.Name -}}
Keycloak {{ .Chart.AppVersion }} installed as release {{ .Release.Name }} in namespace {{ .Release.Namespace }}.

{{ $replicas }} replica(s), cache=ispn with jdbc-ping discovery over the database,
external PostgreSQL at {{ $settings.db.host | default "<keycloak.db.host NOT SET>" }}:{{ $settings.db.port | int64 }}/{{ $settings.db.database }}.

ENDPOINTS
{{ if .Values.ingress.enabled }}  Public:  {{ include "keycloak.frontendUrl" . }}
{{ else }}  Public:  no Ingress (ingress.enabled: false) - in-cluster only
{{ end }}{{ if $adminHostname }}  Admin:   {{ include "keycloak.adminUrl" . }}/admin
{{ else }}  Admin:   no admin hostname (ingress.admin.enabled: false) - the console is
           served on the public host and Keycloak generates its URLs for it
{{ end }}  In-cluster: http://{{ $svc }}.{{ .Release.Namespace }}.svc.cluster.local:{{ $settings.ports.http | int64 }}

  Management port {{ $settings.ports.management | int64 }} is NOT published and must never be: it serves
  /health, /health/live, /health/ready, /health/started{{ if $settings.metrics.enabled }} and /metrics{{ end }}
  WITHOUT authentication. It has a ClusterIP Service
  ({{ .Release.Name }}-{{ .Chart.Name }}-management-service) for in-cluster scraping only.

THE 1PASSWORD ITEMS, AND WHICH FIELDS THEY MUST CARRY
The operator materializes one Secret key per item FIELD, so a missing or
misnamed field leaves the pod in CreateContainerConfigError - with no message
naming the field.

  {{ .Values.secrets.databaseSecretRef | default "<secrets.databaseSecretRef NOT SET>" }}
    username      the PostgreSQL role
    password      that role's password
    The role must be able to CREATE its schema: Keycloak builds and migrates
    the whole schema itself on first start.
{{ if $settings.bootstrapAdmin.enabled }}
  {{ .Values.secrets.bootstrapAdminSecretRef | default "<secrets.bootstrapAdminSecretRef NOT SET>" }}
    username      the TEMPORARY bootstrap administrator (upstream default name:
                  temp-admin)
    password      its password
{{ else }}
  (no bootstrap admin item - keycloak.bootstrapAdmin.enabled is false, which is
   the correct state once a permanent administrator exists)
{{ end }}
  Both arrive as ENVIRONMENT VARIABLES, not as files, and that is forced:
  Keycloak 26 has no _FILE form for an option's value. The price: the password
  is in the container's process environment.

FIRST INSTALL - AND STEP 4 IS MANDATORY, NOT ADVICE

  1. kubectl -n {{ .Release.Namespace }} rollout status deployment/{{ $deploy }}
     Ready means /health/ready answered 200, which means the database is
     reachable and the schema is in place. The first start is the slow one: it
     creates the entire schema through Liquibase, and with {{ $replicas }} replica(s)
     the second waits on the first's migration lock.

  2. Check that the cluster really formed. THIS STEP IS NOT OPTIONAL AND NO
     PROBE REPLACES IT: a node that never found its peer is Ready, healthy and
     alone, and two such nodes are two islands that each believe they are the
     cluster. The install-test cannot catch it either.
       kubectl -n {{ .Release.Namespace }} logs deployment/{{ $deploy }} | grep -i "ISPN000094\|view accepted\|members"
     Look for a view with {{ $replicas }} members. One member per pod means
     jdbc-ping did not find the peer - check that both pods can reach each
     other on port {{ $settings.ports.jgroups | int64 }} and that they really share one database.

  3. Log in at {{ if $adminHostname }}{{ include "keycloak.adminUrl" . }}/admin{{ else if .Values.ingress.enabled }}{{ include "keycloak.frontendUrl" . }}/admin{{ else }}the in-cluster address, port-forwarded{{ end }}
     with the bootstrap credentials from the vault item above.

  4. CREATE A PERMANENT ADMINISTRATOR, then remove the temporary one.
     The bootstrap account is TEMPORARY by design - upstream: "the account
     should exist only for the duration necessary to perform operations needed
     to gain permanent and more secure admin access. After that, the account
     needs to be removed manually." This is different from Keycloak before 26,
     where KEYCLOAK_ADMIN created a regular administrator and there was nothing
     to clean up.
       a) In the master realm, create a user, set a password, assign the
          realm-admin / admin role.
       b) Verify you can log in as that user.
       c) Delete the temp-admin user.
       d) helm upgrade with keycloak.bootstrapAdmin.enabled: false

  5. Until step 4d, EVERY START LOGS a line about the initial administrator not
     being created, because the master realm already exists and the bootstrap
     path is skipped. It is harmless. It is also the kind of message somebody
     eventually investigates, which is why 4d is part of the install and not a
     tidy-up.

MINOR UPGRADES NEED THE CLUSTER DOWN - AND THE AUTOMERGE DOES NOT KNOW THAT

  Keycloak supports a rolling update only WITHIN a minor line. A minor jump
  (26.8 -> 26.9) requires every node shut down before any new node starts;
  mixing two minor versions in one Infinispan cluster is not a supported
  configuration, and nothing in the cluster reports that it happened.

  THIS REPOSITORY'S RENOVATE AUTOMERGES EVERY UPDATE TYPE, MAJORS INCLUDED.
  That is a deliberate, accepted risk and not an oversight (Gitea #83), but it
  means an app-version PR that moves the minor can reach main, publish the
  chart and be rolled out as a rolling update without anybody deciding to.
  Watch app-version PRs for a changed minor, and if one has landed, take this
  path rather than letting the rollout happen:

    1. Decide it with the tool instead of the version number. With the OLD
       version and the OLD configuration:
         kc.sh update-compatibility metadata --file=/tmp/kc-compat.json
       then with the NEW version and the NEW configuration:
         kc.sh update-compatibility check --file=/tmp/kc-compat.json
       Pass EVERY configuration option on both calls - upstream: "Omitting any
       configuration options results in incomplete metadata, and could lead to
       a wrong reported result".
       Exit codes, which is what to automate on: 0 rolling update possible,
       1 unexpected error, 2 invalid CLI option, 3 and 4 not possible - the
       deployment must be shut down.

    2. If it says shut down:
         helm upgrade ... --set scaleDown=true      # every pod stops
         # back up the database AND the configuration before anything else
         helm upgrade ... --set scaleDown=false     # with the new version

  A NOTE ON WHAT IS NOT SETTLED: upstream's 26.6 release notes call patch-level
  rolling updates supported and active by default, while the 26.8.0 page for
  update-compatibility says a rolling update is possible "when the Keycloak
  version is the same" and uses a PATCH pair (26.2.0 -> 26.2.1) as its example
  of an incompatible one. Those two cannot both be current. Until it is clear,
  run the check for patch updates too and trust its exit code rather than the
  size of the version step.

TRAPS WORTH KNOWING BEFORE YOU NEED THEM

  - THE ADMIN HOSTNAME IS NOT AN ACCESS CONTROL. hostname-admin decides which
    URLs the server generates, nothing more. Upstream: "Using the
    hostname-admin option does not prevent accessing the Administration REST
    API endpoints via the frontend URL specified by the hostname option. If you
    want to restrict access to the Administration REST API, you need to do it
    on the reverse proxy level." So /admin/realms/... is reachable through the
    public host whatever you put on the admin host, and an IP allow-list on the
    admin Ingress restricts nothing on its own. This chart renders no deny rule
    for /admin - see values.yaml for why not.

  - A SHARED DATABASE MERGES TWO CLUSTERS. Node discovery runs through the
    database, so - upstream - "clusters that share the same database but should
    remain separate will merge unless they have distinct cluster names. The
    default cluster name is ISPN." Two releases of this chart on one database
    therefore form ONE cluster, silently.
    {{ with $settings.cache.clusterName }}This release sets cache-embedded-cluster-name={{ . }}.{{ else }}This release uses the default name (ISPN) - keycloak.cache.clusterName is
    unset. Read its comment in values.yaml before setting it: upstream also
    warns about cache invalidation between differently-named clusters.{{ end }}

  - READINESS ONLY CHECKS THE DATABASE WHILE metrics-enabled IS TRUE. Upstream's
    table of health checks has a "Requires Metrics" column and the DATABASE
    check is the row that says yes. So with keycloak.metrics.enabled false,
    /health/ready answers 200 with the database gone and the pod keeps taking
    logins it cannot serve. This release has it {{ if $settings.metrics.enabled }}ON, which is the chart default
    and the reason the default is what it is.{{ else }}OFF, so the readiness probe
    cannot currently detect a database outage.{{ end }}

  - THERE IS NO ConfigMap IN THIS CHART, so no checksum/config annotation. That
    is not an omission: Keycloak is configured entirely through options and
    KC_* variables, so there is no config file whose change could fail to
    restart anything.

  - A CHANGE INSIDE A VAULT ITEM RESTARTS NOTHING. Helm never sees it. After
    rotating a field:
      kubectl -n {{ .Release.Namespace }} rollout restart deployment/{{ $deploy }}
    (Repointing a secrets.*SecretRef at a different item DOES roll the pods -
    that is covered by checksum/secret-ref.)

  - THE POD HAS TWO INIT CONTAINERS AND NEEDS BOTH. db, health-enabled and
    metrics-enabled are Keycloak BUILD options, the published image is built
    with their defaults, and a runtime change makes Keycloak rewrite
    /opt/keycloak/lib/quarkus at start - which readOnlyRootFilesystem: true
    forbids. So one init container seeds that directory into a volume and the
    next runs `kc.sh build` into it; the serving container then runs
    `start --optimized`. If a pod is stuck in Init, read those logs first:
      kubectl -n {{ .Release.Namespace }} logs <pod> -c {{ .Release.Name }}-{{ .Chart.Name }}-build
    Build options are set ONLY through keycloak.db.vendor,
    keycloak.metrics.enabled and keycloak.features. Putting one into
    keycloak.env fails the render on purpose.

  - POOL SIZES ARE PER REPLICA. keycloak.db.pool.maxSize is {{ $settings.db.pool.maxSize | int64 }}, so with the
    current {{ $replicas }} replica(s) this release can open up to {{ mul ($settings.db.pool.maxSize | int64) $replicas }} connections.
    Upstream's default is 100 per node, which with two nodes exceeds a default
    PostgreSQL's max_connections of 100. Raise it together with the server,
    never alone.
{{ if and $settings.podDisruptionBudget.enabled (ge $replicas 2) }}
  - A PodDisruptionBudget IS in place ({{ .Release.Name }}-{{ .Chart.Name }}-pdb, minAvailable
    {{ $settings.podDisruptionBudget.minAvailable | int64 }}), so a node drain cannot take all replicas at once.
{{ else if $settings.podDisruptionBudget.enabled }}
  - NO PodDisruptionBudget was rendered, because the effective replica count is
    {{ $replicas }}. That is deliberate: minAvailable 1 with a single replica forbids every
    voluntary eviction, so `kubectl drain` would block forever on a pod nothing
    may move - and the error would name the PDB rather than the replica count.
    Run two replicas to get the budget.
{{ else }}
  - NO PodDisruptionBudget (keycloak.podDisruptionBudget.enabled is false). A
    node drain may take every replica at once.
{{ end }}
WHAT THIS CHART DOES NOT RENDER

  - No PVC and no StatefulSet: Keycloak 26 keeps everything in the database,
    user sessions included (persistent sessions are the default; volatile ones
    are deprecated as of 26.8). There is no disk state to keep.
  - No NetworkPolicy. jdbc-ping opens direct pod-to-pod connections on port
    {{ $settings.ports.jgroups | int64 }}, so any policy added later has to allow that between the
    replicas, or the cluster silently becomes two islands.
  - No deny rule for /admin on the public Ingress, and no restriction of the
    administration REST API. That is reverse-proxy work - see the first trap.