helm-charts/matrix-synapse/templates/NOTES.txt

{{- /*
NOTHING IN CI READS THIS FILE. `helm template` does not emit notes and no
workflow or script under ci/ reads it, so every claim below carries
review-only assurance - it is checked by being read, not by a stage. A
rendering ERROR here does fail `helm template`, which is all the pipeline can
see of it.
*/ -}}
{{- $replicas := include "matrix-synapse.replicas" (dict "scaleDown" .Values.scaleDown "replicas" .Values.matrix.replicas) -}}
{{- $delegation := .Values.matrix.delegation | default dict -}}
{{- $httpRoute := .Values.httpRoute | default dict -}}
{{- $httpRouteDelegation := $httpRoute.delegation | default dict -}}
{{- /*
"Published" rather than "has an Ingress": the two modes are alternatives, and
every paragraph below that used to ask ingress.enabled was really asking
whether anything publishes this release. Gitea #114, and the sure.publishedOverTls
/ freesailarr.published shape.
*/ -}}
{{- $published := or .Values.ingress.enabled $httpRoute.enabled -}}
{{- $hostnames := $httpRoute.hostnames | default list -}}
{{- if not $hostnames -}}
{{- $hostnames = list .Values.ingress.domain -}}
{{- end -}}
{{- $delegationHostnames := $httpRouteDelegation.hostnames | default list -}}
{{- if not $delegationHostnames -}}
{{- $delegationHostnames = list $delegation.serverUrl -}}
{{- end -}}
{{- $delegationReplicas := include "matrix-synapse.replicas" (dict "scaleDown" .Values.scaleDown "replicas" $delegation.replicas) -}}
{{- $res := fromYaml (include "matrix-synapse.effectiveResources" .Values.matrix.resources) -}}
{{- $eph := index ($res.limits | default dict) "ephemeral-storage" | default "the computed ephemeral-storage limit" -}}
{{- $startup := fromYaml (include "matrix-synapse.startupProbe" .Values.matrix.startupProbe) -}}
{{- $budget := "" -}}
{{- if and (kindIs "float64" $startup.periodSeconds) (kindIs "float64" $startup.failureThreshold) -}}
{{- $budget = printf "%.0f seconds" (mulf $startup.periodSeconds $startup.failureThreshold) -}}
{{- end -}}
{{- $sfs := printf "%s-%s-sfs" .Release.Name .Chart.Name -}}
{{- $pod := printf "%s-0" $sfs -}}
Matrix Synapse {{ .Chart.AppVersion }} — release {{ .Release.Name }} in namespace {{ .Release.Namespace }}.

{{- if .Values.ingress.enabled }}

ACCESS — INGRESS MODE (ingress.enabled: true, httpRoute.enabled: false)
  https://{{ .Values.ingress.domain }}

  That host carries the whole client and federation API: the Ingress sends "/"
  on it to the Synapse Service, which forwards port 80 to the container's
  8008. The certificate is requested by cert-manager through the
  "{{ .Values.ingress.clusterIssuer }}" cluster issuer; until it has been issued the browser
  shows a warning and federation fails. Follow it with

    kubectl -n {{ .Release.Namespace }} get certificate tls-{{ .Release.Name }}-{{ .Chart.Name }}-ingress
{{- else if $httpRoute.enabled }}

ACCESS — GATEWAY MODE (httpRoute.enabled: true, ingress.enabled: false)
{{- range $hostnames }}
  https://{{ . }}
{{- end }}

  That host carries the whole client and federation API: the HTTPRoute sends
  "/" on it to the Synapse Service, which forwards port 80 to the container's
  8008.

  THIS RELEASE RENDERS {{ if $delegation.enabled }}TWO HTTPRoutes{{ else }}ONE HTTPRoute{{ end }} AND NOTHING ELSE of the routing
  path — no Gateway, no class, no Certificate, no policy object.

    {{ .Release.Name }}-{{ .Chart.Name }}-httproute              {{ join ", " $hostnames }}
{{- if $delegation.enabled }}
    {{ .Release.Name }}-{{ .Chart.Name }}-delegation-httproute   {{ join ", " $delegationHostnames }}
{{- end }}

  Three things therefore belong to the cluster, and if any is missing the
  hostname does not answer — which for Matrix means federation is down, not
  just the web client:

    1. The Gateway-API CRDs. Without them this install would have failed, so
       they are present — nothing to check.
    2. The Gateway(s) named in parentRefs
{{- range $httpRoute.parentRefs }}
         {{ .name }}{{ if .namespace }} in namespace {{ .namespace }}{{ end }}{{ if .sectionName }}, listener {{ .sectionName }}{{ end }}   (homeserver)
{{- end }}
{{- if $delegation.enabled }}
{{- range ($httpRouteDelegation.parentRefs | default $httpRoute.parentRefs) }}
         {{ .name }}{{ if .namespace }} in namespace {{ .namespace }}{{ end }}{{ if .sectionName }}, listener {{ .sectionName }}{{ end }}   (delegation)
{{- end }}
{{- end }}
       must exist AND their listeners must allow routes from this namespace
       ({{ .Release.Namespace }}) in allowedRoutes. A rejected attachment is visible only
       in the route's status:

         kubectl -n {{ .Release.Namespace }} get httproute {{ .Release.Name }}-{{ .Chart.Name }}-httproute -o yaml

       Read status.parents[].conditions — "Accepted: True" and
       "ResolvedRefs: True" is what a working attachment looks like. An empty
       status means no controller has looked at the route at all.
    3. {{ if $delegation.enabled }}The TLS certificates — one per hostname, on the Gateway's listeners,
       or one wildcard listener covering both{{ else }}The TLS certificate for the hostname, on the Gateway's listener{{ end }}.
       ingress.clusterIssuer is NOT used in this mode and is not required: the
       chart renders no Certificate here, so there is no
       tls-{{ .Release.Name }}-{{ .Chart.Name }}-ingress secret{{ if $delegation.enabled }} and no
       tls-{{ .Release.Name }}-{{ .Chart.Name }}-delegation secret{{ end }} and nothing for
       cert-manager to do for this release. MATRIX FEDERATION IS HTTPS-ONLY,
       so a listener without a valid certificate for these names is a
       federation outage that no probe in this release can see.

  THE UPLOAD LIMIT IS NOT ENFORCED BY ANYTHING THIS CHART RENDERS.

  In Ingress mode an operator sets
  nginx.ingress.kubernetes.io/proxy-body-size through ingress.annotations —
  this installation runs with 50m — because Synapse's media repository takes
  uploads and nginx's default of 1m rejects nearly all of them. IN THIS MODE
  THAT IS GONE: the Gateway API has no body-size field, it is implementation
  policy rather than routing, and this chart renders no policy object on
  purpose (it would bind the chart to one Gateway implementation).

  The ceiling in force is the smaller of max_upload_size in your
  homeserver.yaml — which this chart cannot see, it comes from 1Password —
  and whatever the GATEWAY enforces. Neither number anyone can point at is
  necessarily the one that decides. Set the Gateway's half, in your
  implementation's own object, to match max_upload_size, BEFORE you point DNS
  at it.

  AND ONE THING NOT TO RE-CREATE. The live Ingress also carries a
  configuration-snippet of three more_set_headers directives
  (X-Forwarded-For $remote_addr, X-Forwarded-Proto $scheme, Host). It LOOKS
  like what Synapse's x_forwarded setting depends on. It is not:
  more_set_headers is the headers-more module's OUTPUT-header directive, so
  those three headers go on the RESPONSE TO THE CLIENT and Synapse never sees
  them. What Synapse reads with x_forwarded: true is the REQUEST's
  X-Forwarded-For, which the proxy supplies by itself. Do not spend effort
  rebuilding this at the Gateway, and do not read its absence as the cause of
  a rate-limiting or federation problem. (This is a reading of the directive's
  documented semantics; nothing here A/B-tested a Synapse instance.)

  ingress.annotations is inert in this mode in general: the Gateway API has no
  annotation channel for controller behaviour.
{{- else }}

ACCESS — NEITHER INGRESS NOR HTTPROUTE
  ingress.enabled and httpRoute.enabled are both false, so this release has no
  Ingress, no HTTPRoute and no certificate; ingress.clusterIssuer is not
  needed.
{{- if $delegation.enabled }}
  ingress.domain still is: the delegation nginx renders it into the
  /.well-known/matrix/* documents, so it stays the host every other Matrix
  server is told to talk to, not merely an Ingress entry.
{{- else }}
  With the delegation off as well, ingress.domain is read by nothing in this
  release — no object rendered here carries it.
{{- end }}
  Reach the server inside the cluster with

    kubectl -n {{ .Release.Namespace }} port-forward svc/{{ .Release.Name }}-{{ .Chart.Name }}-service 8008:80

  and point a client at http://127.0.0.1:8008/. Nothing in this release
  terminates TLS, and Matrix federation is HTTPS-only: no remote server can
  reach this one until something in front of the cluster serves this Service
  over HTTPS{{ if .Values.ingress.domain }} as https://{{ .Values.ingress.domain }}{{ end }}.
{{- end }}
{{- if $delegation.enabled }}

DELEGATION — TWO HOSTNAMES THAT ARE NOT INTERCHANGEABLE
  matrix.delegation.enabled is true, so a second nginx Deployment serves the
  two .well-known documents, filled in from ingress.domain:

    /.well-known/matrix/server
      → {"m.server": "{{ .Values.ingress.domain }}:443"}
    /.well-known/matrix/client
      → {"m.homeserver": {"base_url": "https://{{ .Values.ingress.domain }}"}}
{{- if $published }}

  {{ if $httpRoute.enabled }}A SECOND HTTPRoute publishes{{ else }}The Ingress publishes{{ end }} them on a SECOND host. Read that split
  carefully, because the chart cannot check it for you:

    matrix.delegation.serverUrl ({{ $delegation.serverUrl }})
      is the DELEGATING host — the domain in every user ID (@user:{{ $delegation.serverUrl }}).
      Remote servers look up .well-known here, so this is the name that must
      equal the server_name of the running homeserver.
    ingress.domain ({{ .Values.ingress.domain }})
      is where Synapse actually answers. It appears inside the documents
      above and needs no relation to the server name.

  Only the path /.well-known/matrix is routed to nginx on {{ join ", " $delegationHostnames }};
  everything else on that host is not served by this release.
{{- if $httpRoute.enabled }}

  AND THAT CO-EXISTENCE IS EASIER HERE THAN IT WAS. The server name is usually
  an apex domain that already serves a website, and under ingress-nginx two
  Ingress objects cannot both claim one hostname — the admission webhook
  rejects the second — which is why the delegation rule had to sit inside this
  chart's single Ingress object. HTTPRoutes attached to the same listener for
  the same hostname are MERGED by the implementation, longest match winning,
  so this route may sit beside whatever owns that apex without either side
  knowing about the other.

  Both hosts need a certificate ON THE GATEWAY'S LISTENERS — this chart renders
  none in this mode — and a listener without a valid certificate for the server
  name is a federation outage, not a cosmetic problem.
{{- else }}
  Both hosts need
  a certificate — the second one is tls-{{ .Release.Name }}-{{ .Chart.Name }}-delegation — and
  cert-manager only issues it once {{ $delegation.serverUrl }} resolves to this cluster.
  A pending certificate there is a federation outage, not a cosmetic problem.
{{- end }}

  Check what is actually being served, from outside the cluster:

    curl -s https://{{ first $delegationHostnames }}/.well-known/matrix/server
{{- else }}

  Nothing publishes them, though: ingress.enabled and httpRoute.enabled are
  both false, so the documents
  exist only on the in-cluster Service {{ .Release.Name }}-{{ .Chart.Name }}-delegation-service, and
  matrix.delegation.serverUrl ({{ $delegation.serverUrl }}) is rendered into no object of this
  release. Whatever terminates TLS in front of the cluster has to route
  https://<server name>/.well-known/matrix to that Service, or federation
  never finds this homeserver. Inspect the documents meanwhile with

    kubectl -n {{ .Release.Namespace }} port-forward svc/{{ .Release.Name }}-{{ .Chart.Name }}-delegation-service 8080:80
{{- end }}

  The nginx image is pinned separately at {{ $delegation.image.repository }}:{{ $delegation.image.tag }} — it is the one
  image in this chart that does not follow the chart's appVersion.
{{- else }}

NO DELEGATION IN THIS RELEASE — FEDERATION LIKELY DOES NOT WORK
  matrix.delegation.enabled is false, so nothing here serves
  /.well-known/matrix/server. A remote server that finds no such document
  falls back to port 8448 of the server name — and this release exposes no
  8448 anywhere: the container publishes 8008 only, and the Service maps port
  80 onto it. Unless the delegation is served by something else (a static
  .well-known on the server-name host, or an SRV record), federation stays
  broken{{ if $published }} while clients on {{ first $hostnames }} work fine{{ end }}.
  Setting matrix.delegation.enabled to true and matrix.delegation.serverUrl
  to the server name lets this chart serve it.
{{- end }}
{{- if or (eq $replicas "0") (and $delegation.enabled (eq $delegationReplicas "0")) }}

SCALED TO ZERO
{{- if .Values.scaleDown }}
  scaleDown: true — every workload of this release runs with replicas: 0.
{{- else }}
{{- if eq $replicas "0" }}
  matrix.replicas: 0 — the Synapse StatefulSet runs with replicas: 0.
{{- end }}
{{- if and $delegation.enabled (eq $delegationReplicas "0") }}
  matrix.delegation.replicas: 0 — the delegation nginx runs with replicas: 0,
  so the .well-known documents 404 while remote servers keep asking for them.
{{- end }}
{{- end }}
  The PVC, the Secret and the Services stay in place, so no data is lost{{ if .Values.ingress.enabled }} and
  the certificates are not re-issued{{ else if $httpRoute.enabled }} and the
  HTTPRoutes stay in place, so the hostnames keep pointing at Services with no
  endpoints — the Gateway answers with an error of its own rather than with
  nothing, and remote homeservers see a failing .well-known instead of an
  unresolvable name{{ end }}. Set it back and upgrade again to
  return.
{{- end }}

SECRETS — WHAT THE 1PASSWORD ITEM MUST CARRY
  secrets.configSecretRef → {{ .Values.secrets.configSecretRef }}
  → OnePasswordItem and Secret {{ .Release.Name }}-{{ .Chart.Name }}-config-secret

  The operator copies the item's FIELDS into that Secret, one key per field
  name, and this chart reads exactly one of them. The item therefore needs a
  field named literally

    homeserver.yaml

  holding the COMPLETE Synapse configuration file; it is mounted read-only at
  /data/homeserver.yaml. A field spelled differently produces a Secret without
  that key, the pod never starts, and the reason appears only as a volume
  error in "kubectl -n {{ .Release.Namespace }} describe pod {{ $pod }}" — the
  Secret itself exists, so the operator looks as if it had done its job.

  Four settings inside that file are dictated by this chart's mounts, and
  getting them wrong is not a config error the app reports clearly:

    media_store_path: /media_store     the second PVC subPath. The root
                                       filesystem is read-only, so a path
                                       outside /data, /media_store and /tmp
                                       fails on the first upload.
    signing_key_path: under /data      the federation identity. Losing it
                                       makes this server a stranger to every
                                       server that ever spoke to it.
    log_config: under /data            anywhere else is unwritable.
    a listener on port 8008, type http — the Service and all three probes
                                       target that port by name.

  This chart deploys NO database. Whatever "database:" names — an external
  PostgreSQL, or an SQLite file, which then has to live under /data — must
  exist and be reachable from namespace {{ .Release.Namespace }}.

  Note that /data is the PVC and homeserver.yaml is mounted ON TOP of it: a
  homeserver.yaml already lying in the volume (from a "synapse generate" run,
  for instance) is shadowed and never read. What runs is the item's content.

CHANGING THE ITEM DOES NOT RESTART SYNAPSE
  Pointing secrets.configSecretRef at a DIFFERENT item changes the rendered
  StatefulSet and rolls the pod. Changing the content INSIDE the same item
  does not: Helm never sees that content, so nothing in the pod template
  changes — and because the file is a subPath mount, the kubelet does not
  refresh it in the running container either. Both halves have to be undone
  by hand:

    kubectl -n {{ .Release.Namespace }} rollout restart statefulset/{{ $sfs }}

  The operator's own "auto-restart" annotation is no substitute here: it
  restarts Deployments, and this workload is a StatefulSet. The delegation
  nginx is the opposite case — its ConfigMap is hashed into the pod template
  (checksum/config), so a changed ingress.domain rolls it automatically.

THE SERVER NAME IS DECIDED ONCE, AND NOT BY matrix.serverName
  matrix.serverName ({{ .Values.matrix.serverName }}) is rendered into the SYNAPSE_SERVER_NAME
  environment variable. The server that actually runs reads its name from
  "server_name:" in the mounted homeserver.yaml — the env var matters to the
  image only when it generates a configuration, which is not what happens
  here. Changing matrix.serverName alone rolls the pod and changes nothing
  else.

  And the name cannot be changed afterwards at all: it is baked into every
  user ID, room ID and event this server has ever signed. A server that needs
  a different name is a new server with a new database, not an upgrade. Decide
  it before the first start — together with matrix.delegation.serverUrl, which
  has to match it.
{{- if and $published $delegation.enabled .Values.matrix.serverName $delegation.serverUrl }}
{{- if ne (printf "%v" .Values.matrix.serverName) (printf "%v" $delegation.serverUrl) }}

  MISMATCH IN THIS RELEASE: matrix.serverName is "{{ .Values.matrix.serverName }}" but the
  delegation is published on "{{ $delegation.serverUrl }}". Remote servers resolve
  .well-known on the SERVER NAME, so unless something outside this chart
  serves it on "{{ .Values.matrix.serverName }}", the nginx deployed here is
  never asked. Both values name the same host in a working setup.
{{- end }}
{{- end }}

WHAT A READY POD PROVES — AND WHAT IT DOES NOT
  All three Synapse probes hit /health, which answers 200 as soon as the HTTP
  listener is up. It says nothing about the database, the signing key or
  federation: a homeserver.yaml pointing at an unreachable database usually
  fails at start (CrashLoopBackOff, log only), but every problem that appears
  later leaves the pod Ready. The startup probe covers the first start{{ if $budget }}
  ({{ $budget }}), which is where schema migrations and cache warmup happen{{ end }}:

    kubectl -n {{ .Release.Namespace }} logs {{ $pod }}
{{- if $delegation.enabled }}

  The delegation probe is weaker still. It requests
  /.well-known/matrix/server, and that endpoint is a static "return 200" in
  the generated nginx.conf — it passes with a wrong ingress.domain baked into
  the document. Read the body, not the probe.
{{- end }}

THE VOLUME BELONGS TO THIS RELEASE
  /data (database when SQLite, signing key, log config) and /media_store (all
  uploaded media) are two subPaths of ONE ReadWriteOnce claim,
  {{ .Release.Name }}-{{ .Chart.Name }}-pvc ({{ .Values.storage.amount }}{{ if .Values.storage.storageClass }} from storageClass {{ .Values.storage.storageClass }}{{ end }}). It is an
  ordinary PVC in this release, not a StatefulSet volumeClaimTemplate, and it
  carries no resource-policy annotation: "helm uninstall {{ .Release.Name }}" deletes it, and
  with a Delete reclaim policy the media and the signing key go with it. Add
  the annotation to the live PVC afterwards and it still will not help — Helm
  reads that annotation from the release manifest. Back the volume up first.

  Growing storage.amount only takes effect if the StorageClass allows volume
  expansion; shrinking it is refused by Kubernetes and fails the next upgrade
  on the PVC instead of touching the data. Media is what grows here, and it
  grows without an upper bound unless the homeserver.yaml sets retention.

  The container runs as uid/gid 1000 with fsGroup 1000 — the gosu default the
  upstream image used before this chart pinned the user, so data written by
  an older release stays readable. Files in the volume that belong to
  another uid — restored from a backup, written by an older release — are not
  readable, and the signing key (mode 0600) is the first thing to fail.

  Everything else the process writes has to go to /tmp, which is an emptyDir
  and counts against the container's ephemeral-storage limit ({{ $eph }}).
  Raise matrix.resources.requests.ephemeralStorage to move it; the limit
  follows through matrix.resources.limitFactor.
{{- if gt (atoi $replicas) 1 }}

MORE THAN ONE SYNAPSE REPLICA WILL NOT WORK
  matrix.replicas is {{ $replicas }}, but every pod of this StatefulSet mounts the same
  ReadWriteOnce claim {{ .Release.Name }}-{{ .Chart.Name }}-pvc and the same homeserver.yaml — there is
  no volumeClaimTemplate and no per-pod configuration. Additional pods either
  stay Pending on a node without the volume, or start as a second full
  homeserver on the same data. Scaling Synapse means running workers with
  distinct configurations, which this chart does not deploy.
{{- end }}
{{- if .Values.matrix.env }}

matrix.env IS APPENDED, NOT MERGED
  Its {{ len .Values.matrix.env }} entr{{ if eq (len .Values.matrix.env) 1 }}y lands{{ else }}ies land{{ end }} after the three the chart sets itself —
  SYNAPSE_SERVER_NAME, SYNAPSE_REPORT_STATS ("no") and TZ ({{ .Values.matrix.timezone }}) — so
  repeating one of those names leaves two entries of the same name in the pod
  spec. Values are rendered through tpl and may contain template expressions.
  Anything sensitive belongs in valueFrom.secretKeyRef: a plain "value" is
  stored in the pod spec and readable by everyone who can read pods in
  {{ .Release.Namespace }}. Note that Synapse takes its real configuration from
  homeserver.yaml, not from the environment.
{{- end }}