helm-charts/bulwark/templates/NOTES.txt

{{- /*
NOTHING IN CI READS THIS FILE. `helm template` does not emit notes and no
workflow or script under ci/ reads it, so every claim below carries
review-only assurance - it is checked by being read, not by a stage. A
rendering ERROR here does fail `helm template`, which is all the pipeline can
see of it.
*/ -}}
{{- $settings := fromYaml (include "bulwark.settings" .) -}}
{{- $httpRoute := .Values.httpRoute | default dict -}}
{{- $hostnames := $httpRoute.hostnames | default list -}}
{{- if not $hostnames -}}
{{- $hostnames = list .Values.ingress.domain -}}
{{- end -}}
{{- $replicas := int (include "bulwark.replicas" (dict "scaleDown" .Values.scaleDown "replicas" $settings.replicas)) -}}
{{- $sfs := printf "%s-%s-sfs" .Release.Name .Chart.Name -}}
{{- $svc := printf "%s-%s-service" .Release.Name .Chart.Name -}}
{{- $url := include "bulwark.publicUrl" . -}}
Bulwark Webmail {{ .Chart.AppVersion }} installed as release {{ .Release.Name }} in namespace {{ .Release.Namespace }}.

{{ $replicas }} replica(s). This is a CLIENT: it holds no mail and has no database.
The messages and the accounts belong to the Stalwart server at
  {{ $settings.jmapServerUrl }}
If that server is away, this release is still Ready and still green - see
"WHAT THE PROBES DO NOT TELL YOU" below.

ENDPOINTS - {{ if .Values.ingress.enabled }}INGRESS MODE (ingress.enabled: true, httpRoute.enabled: false){{ else if $httpRoute.enabled }}GATEWAY MODE (httpRoute.enabled: true, ingress.enabled: false){{ else }}NEITHER INGRESS NOR HTTPROUTE{{ end }}
{{ if or .Values.ingress.enabled $httpRoute.enabled }}  Web:        {{ $url }}
{{ else }}  Web:        no Ingress and no HTTPRoute (ingress.enabled and httpRoute.enabled are
              both false) - in-cluster only
{{ end }}  In-cluster: http://{{ $svc }}.{{ .Release.Namespace }}.svc.cluster.local:{{ $settings.port | int64 }}
{{ if or .Values.ingress.enabled $httpRoute.enabled }}  Admin:      {{ $url }}/admin/login
{{ else }}  Admin:      /admin/login on the in-cluster address above. With nothing published, reach it with
                kubectl -n {{ .Release.Namespace }} port-forward statefulset/{{ $sfs }} 3000:{{ $settings.port | int64 }}
              and open http://localhost:3000/admin/login
{{ end }}{{ if or .Values.ingress.enabled $httpRoute.enabled }}
  Bulwark is served at the ROOT of that host and cannot be moved under a path:
  NEXT_PUBLIC_BASE_PATH is compiled into the image.{{ if $httpRoute.enabled }} That is also why the
  HTTPRoute carries no URLRewrite filter - there is no prefix to strip.{{ end }}
{{ end }}{{ if $httpRoute.enabled }}
GATEWAY MODE - WHAT THIS CHART DOES NOT RENDER, AND WHAT YOU MUST SET YOURSELF

  This release renders ONE HTTPRoute and NOTHING else of the routing path - no
  Gateway, no class, no Certificate, no policy object.

    {{ .Release.Name }}-{{ .Chart.Name }}-httproute   {{ join ", " $hostnames }}

  Three things therefore belong to the cluster, and if any is missing the
  hostname above simply does not answer:

    1. The Gateway-API CRDs. Without them this install would have failed, so
       they are present - nothing to check.
    2. The Gateway named in httpRoute.parentRefs
{{- range $httpRoute.parentRefs }}
         {{ .name }}{{ if .namespace }} in namespace {{ .namespace }}{{ end }}{{ if .sectionName }}, listener {{ .sectionName }}{{ end }}
{{- end }}
       must exist AND its listener must allow routes from this namespace
       ({{ .Release.Namespace }}) in allowedRoutes. A rejected attachment is visible only
       in the route's status:

         kubectl -n {{ .Release.Namespace }} get httproute {{ .Release.Name }}-{{ .Chart.Name }}-httproute -o yaml

       Read status.parents[].conditions - "Accepted: True" and
       "ResolvedRefs: True" is what a working attachment looks like. An empty
       status means no controller has looked at the route at all.
    3. The TLS certificate for the hostname, on the Gateway's listener.
       ingress.clusterIssuer is NOT used in this mode and is not required: the
       chart renders no Certificate here, so there is no
       tls-{{ .Release.Name }}-{{ .Chart.Name }}-ingress secret and nothing for cert-manager to do
       for this release.

  THE UPLOAD LIMIT IS NOT ENFORCED BY ANYTHING THIS CHART RENDERS.

  In Ingress mode an operator sets nginx.ingress.kubernetes.io/proxy-body-size
  through ingress.annotations - this installation runs with 25m - because
  nginx's default of 1m rejects almost every attachment. IN THIS MODE THAT IS
  GONE: the Gateway API has no body-size field, it is implementation policy
  rather than routing, and this chart renders no policy object on purpose (it
  would bind the chart to one Gateway implementation).

  So the limit in force is THE GATEWAY'S, whatever it happens to be, and no
  value of this chart has any say in it. The failure mode on a webmail client
  is worth picturing before it happens: a user has written the mail, attached
  the file and pressed send, and the 413 comes back FROM THE PROXY. It reads
  like Bulwark being broken, nothing mentions a limit, and the draft is the
  thing at risk.

  Set the equivalent next to the Gateway, in your implementation's own object,
  BEFORE you point DNS at it - and size it to what the MAIL SERVER accepts
  (the Stalwart release above), not to what looks generous here. A webmail
  client that accepts a 40 MB attachment the mail server will refuse has only
  moved the failure later.

  ingress.annotations is inert in this mode in general: the Gateway API has no
  annotation channel for controller behaviour.
{{ end }}
THE 1PASSWORD ITEMS, AND WHICH FIELDS THEY MUST CARRY
The operator materializes one Secret key per item FIELD. A missing or misnamed
field here does NOT stop the pod - the two file-form secrets simply read as
absent - so the symptom is a feature that is quietly off rather than an error.

  {{ .Values.secrets.appSecretRef | default "<secrets.appSecretRef NOT SET>" }}
    session-secret    openssl rand -base64 32
                      MANDATORY. Persistent sessions and the encryption of the
                      synchronized settings both hang on it, and the second
                      fails SILENTLY without it: settings sync is computed as
                      "enabled AND a secret exists", so with no secret it is
                      simply off and nothing says so.
    admin-password    the admin dashboard's initial password. CONSUMED ONCE -
                      see step 5 below.
{{ if $settings.oauth.enabled }}
  {{ .Values.secrets.oauthSecretRef | default "<secrets.oauthSecretRef NOT SET>" }}
    oauth-client-secret   the confidential OIDC client's secret, as issued by
                          the identity provider when the client was registered.
{{ else }}
  (no OAuth item - bulwark.oauth.enabled is false, so sign-in is by password)
{{ end }}
  The file-form secret{{ if $settings.oauth.enabled }}s are{{ else }} is{{ end }} mounted as a DIRECTORY, not with subPath, and
  re-read on every use - so rotating {{ if $settings.oauth.enabled }}either field{{ else }}that field{{ end }} in the vault takes effect
  WITHOUT a restart. admin-password is the exception and no restart helps it.

{{ if $settings.oauth.enabled }}
BEFORE YOU INSTALL - THE OIDC CLIENT HAS TO EXIST FIRST
{{ if $settings.oauth.only }}
  THIS IS NOT ADVICE AND IT IS NOT A FOLLOW-UP STEP. bulwark.oauth.only is ON,
  which means the password form is hidden, which means that WITHOUT A WORKING
  OIDC CLIENT THE FIRST SIGN-IN IS IMPOSSIBLE. There is no fallback for users.
{{ end }}
  Register a CONFIDENTIAL client with the provider at
    {{ $settings.oauth.issuerUrl }}
  client id:     {{ $settings.oauth.clientId }}
  secret:        into the vault item's oauth-client-secret field

  THE REDIRECT URI CARRIES THE UI LANGUAGE, AND THAT IS THE ONE THING THAT WILL
  COST YOU THE FIRST SIGN-IN IF YOU REGISTER A SINGLE URI. Bulwark builds it as
    {{ $url }}/<locale>/auth/callback
  where <locale> is the language the VISITOR'S BROWSER resolved to - so a German
  browser sends /de/auth/callback and an English one /en/auth/callback, from the
  same deployment. The language is not in the visible URL (the locale prefix is
  "never" in this image), which is exactly why this is surprising: the address
  bar shows no language and the redirect_uri contains one.

  Register ONE PER LANGUAGE YOU EXPECT, or a wildcard if the provider allows it.
  On Keycloak the wildcard is the practical answer:
    {{ $url }}/*
  Otherwise the 27 locales Bulwark supports are:
    ar ca cs da de en es fa fr he hu it ja ko lv mn nb nl pl pt ro ru sk tr uk
    zh zh-TW

  If a sign-in fails with a redirect_uri mismatch, read the URI out of the
  failed request rather than guessing which language it was - that is the value
  the provider compares byte for byte.

  POST-LOGOUT, IF YOU USE IT: bulwark.oauth.endSession is {{ $settings.oauth.endSession }}, and a
  post-logout redirect URI is only sent when OAUTH_POST_LOGOUT_REDIRECT_URI is
  set (this chart does not render it). If you add it through bulwark.env, it has
  to be registered with the provider too.

  The issuer must be reachable BY THE POD, and the endpoints it advertises must
  share the issuer's origin or be publicly routable - Bulwark rejects a
  discovered endpoint on a private address from a different origin. A
  split-horizon DNS setup needs OAUTH_ALLOW_PRIVATE_ENDPOINTS through
  bulwark.env; this chart does not render it.
{{ end }}
FIRST INSTALL

  1. kubectl -n {{ .Release.Namespace }} rollout status statefulset/{{ $sfs }}
     Ready means the HTTP server is accepting. It does NOT mean the mail server
     is reachable - nothing in this chart checks that.

  2. Confirm the setup wizard really is OFF. It is, as long as
     bulwark.jmapServerUrl is set, and the chart refuses to render without it -
     but this is the one thing worth seeing rather than trusting, because a
     wizard that ran once changes the rules for good (step 4):
       kubectl -n {{ .Release.Namespace }} logs statefulset/{{ $sfs }} | grep -i "SETUP REQUIRED"
     Nothing found is the correct result. A "SETUP REQUIRED" banner with a token
     means Bulwark is in bootstrap mode and something is wrong with the value.

  3. Sign in{{ if $settings.oauth.only }} through the identity provider{{ end }} at {{ $url }}

  4. READ THIS BEFORE YOU OPEN THE ADMIN DASHBOARD. The dashboard writes
     OVERRIDES into config.json on the volume, and an override BEATS this chart
     for that key - for good, and silently.

     Priority is: admin override  >  environment variable  >  default.

     So if you save a server URL in the dashboard, changing bulwark.jmapServerUrl
     and running helm upgrade afterwards does nothing and reports success. The
     same is true of every other key the dashboard can write. What the chart
     sets is a DEFAULT, not a setting.

     To see which keys are currently overridden:
       kubectl -n {{ .Release.Namespace }} exec {{ $sfs }}-0 -- cat /app/data/admin/config.json
     An empty object, or a file that does not exist, means this chart is in
     force.

  5. THE ADMIN PASSWORD IS CONSUMED EXACTLY ONCE. On the first start it is
     hashed into /app/data/admin/admin.json and the environment variable is
     never read again. Rotating the vault field afterwards has NO EFFECT -
     changing the password goes through the dashboard, or by deleting admin.json
     from the volume and restarting. Know that before you treat the vault field
     as the source of truth.

IF NOBODY CAN SIGN IN - THE WAY BACK, AND IT IS NOT A helm upgrade

  Read this now, not when it happens, because the obvious repair does not work.

  The situation: the identity provider is down, or its configuration is wrong,
  and bulwark.oauth.only hides the password form. No user can get in.

  WHAT DOES NOT WORK RELIABLY: helm upgrade with bulwark.oauth.only=false.
  oauthOnly is an ordinary config key, and every part of the app reads it
  through the priority chain above - not from the environment. If the admin
  dashboard's OAuth setup has ever been used, "oauthOnly": true is sitting in
  config.json on the volume, and from then on the chart's value is ignored. The
  upgrade succeeds and changes nothing.

  WHAT DOES WORK, and it does not depend on the identity provider:

    a) Open {{ $url }}/admin/login and sign in with the admin-password field
       from the vault item. This door has its own password and is not behind
       OIDC.{{ if eq $settings.stalwartAdminAccess "auto" }} (bulwark.stalwartAdminAccess is "auto", so a Stalwart
       admin also gets in here without that password.){{ end }}
    b) In the dashboard, REVERT the override - the dashboard's own "remove
       override" on the OAuth settings. That is what puts the chart's value back
       in charge.

  IF THE DASHBOARD ITSELF IS UNREACHABLE, the last resort is the volume:
    kubectl -n {{ .Release.Namespace }} exec {{ $sfs }}-0 -- cat /app/data/admin/config.json
    # remove the "oauthOnly" key (and any other override in the way), then
    kubectl -n {{ .Release.Namespace }} rollout restart statefulset/{{ $sfs }}
  Deleting admin.json in the same directory resets the admin password to
  whatever the vault field currently holds, on the next start.

WHAT THE PROBES DO NOT TELL YOU

  All three probes are a plain HTTP GET on /api/health, and that endpoint returns
  200 whatever state the application is in - for an unauthenticated caller the
  body even says "healthy" unconditionally. So a green pod here means: the
  process is alive, HTTP is being accepted, the event loop is free, and Next.js
  can route.

  IT SAYS NOTHING ABOUT THE MAIL SERVER. With Stalwart completely gone, this
  release is Ready, green, and serves a web app that cannot fetch a single
  message. Monitor the stalwart release for that - there is no endpoint here
  that could cover it, and the one that would (/api/setup/test-jmap) sits behind
  the setup token this chart disables.

  It also says nothing about the reported state: /api/health reports "degraded"
  on high V8 heap usage, but only to an authenticated caller asking for details,
  and always with HTTP 200. A memory limit set too low therefore produces a pod
  that is permanently degraded and permanently Ready.

BACKUPS - THE VOLUME ALONE IS NOT ONE, AND IT MAY NEED PROTECTING ITSELF

  Two statements, both of them about {{ .Release.Name }}-{{ .Chart.Name }}-pvc:

  1. A BACKUP OF THE VOLUME ALONE IS NOT A BACKUP.{{ if $settings.settingsSync }} The synchronized user
     settings on it are encrypted with session-secret.{{ else }} The user settings are not
     synchronized in this release, but persistent sessions still are.{{ end }} Lose the vault
     item and that data is unreadable out of a perfectly intact volume. Back up
     the 1Password item together with the volume, or the restore is a directory
     of ciphertext.

  2. A BACKUP OF THE VOLUME MAY CONTAIN A SECRET IN CLEARTEXT. If anybody has
     used the dashboard's OAuth setup, the OIDC client secret is written as
     PLAINTEXT into config.json on this volume. So the backup may need the same
     protection the vault gets. Check with:
       kubectl -n {{ .Release.Namespace }} exec {{ $sfs }}-0 -- grep -l oauthClientSecret /app/data/admin/config.json

TRAPS WORTH KNOWING BEFORE YOU NEED THEM

  - WEB PUSH GOES THROUGH THE VENDOR'S RELAY until somebody changes it, and the
    change is NOT a chart value. The built-in default is
    https://notifications.relay.bulwarkmail.org and it is compiled into the
    image. A self-hosted relay CAN be made the effective default and even
    mandatory for all users - through the settings policy in the admin
    dashboard, because that setting has no environment variable at all. So this
    is a choice you have, not a limitation you are stuck with; it just lives in
    the dashboard.

  - THERE IS NO ConfigMap IN THIS CHART, so no checksum/config annotation, and
    that is not an omission: there is no rendered config file whose change could
    fail to restart anything. Repointing secrets.* at a different vault item
    DOES roll the pod (checksum/secret-ref). Changing a value INSIDE an item
    does not - but for the two file-form secrets it does not need to, because
    they are re-read on every use.
{{ if $settings.adminConfigReadOnly }}
  - bulwark.adminConfigReadOnly IS ON. The chart is authoritative and no
    override can appear - and three things are frozen with it: the settings
    policy (so Web Push cannot be moved to your own relay any more), any
    override that was ALREADY on the volume (it cannot be removed either), and -
    if this release has never started with it off - the admin password, which
    means the admin dashboard never comes up and its login answers HTTP 500
    while this pod stays Ready.
    If you have just enabled this on a fresh install, turn it off, let the pod
    start once so admin.json is written, and only then turn it back on.
{{ else }}
  - bulwark.adminConfigReadOnly is OFF, which is the default and keeps the
    dashboard working. The cost is the precedence trap of step 4: the volume can
    override this chart at any time. Turning it on makes the chart authoritative
    but freezes the settings policy, freezes any override already present, and
    must never be the state of a FIRST install - see values.yaml for the
    sequence.
{{ end }}
  - THE SETTINGS POLICY, THE THEMES AND THE BRANDING ARE NOT IN THIS CHART.
    They live in policy.json on the volume and are set in the dashboard. This
    chart neither renders nor backs them up beyond the volume itself.

WHAT THIS CHART DOES NOT RENDER

  - No ConfigMap (see above) and no NetworkPolicy. The pod needs egress to the
    JMAP server and to the OIDC issuer; a policy added later has to allow both,
    or sign-in breaks in a way that looks like a provider outage.
  - No PodDisruptionBudget: with one replica a minAvailable of 1 would forbid
    every voluntary eviction and make `kubectl drain` hang forever on a pod
    nothing may move (README, guideline 11).
  - No sub-path Ingress and no sub-path route, and no value that could add
    one - the base path is compiled into the image. Hence also no URLRewrite
    filter in Gateway mode.
  - No proxy-body-size annotation. Attachment uploads go through the Ingress and
    nginx's default of 1m rejects larger ones with a 413 that reads like an
    application fault; the right number is the mail server's limit, which this
    chart does not know. Set it in ingress.annotations{{ if $httpRoute.enabled }} - except that in GATEWAY
    MODE there is no annotation to set and no counterpart at all, which is the
    section above{{ end }}.
  - No Gateway, no GatewayClass and no listener, in either mode. Those are
    platform objects and live in cluster-config; this chart renders an
    HTTPRoute at most, which is the part that looks the same on every Gateway
    implementation.
{{- if .Values.networkPolicy.enabled }}

NETWORKPOLICY IS ACTIVE — WHAT IT CAPS, AND WHAT IT DOES NOT
  {{ .Release.Name }}-{{ .Chart.Name }}-networkpolicy selects this workload's pods.
  Inbound: TCP 3000 only. Outbound: DNS and TCP 443 — the JMAP server and the
  OIDC provider{{ with .Values.networkPolicy.egress }}{{ with .extraPorts }}, plus TCP {{ join ", " . }}{{ end }}{{ end }}.

  CHECK THE JMAP PORT. 443 is where a published JMAP endpoint lives. A
  bulwark.jmapServerUrl pointing at Stalwart's in-cluster HTTP port (8080), or
  at anything else, is BLOCKED by this policy — and because Bulwark's probes
  only test its own port, the pod becomes Ready and then serves an empty,
  broken webmail with nothing in the events to explain it. Add the real port to
  networkPolicy.egress.extraPorts.

  Three limits belong in the same breath, because none of them is visible in
  the object itself:

  - EGRESS IS BY PORT, NEVER BY DESTINATION. A NetworkPolicy matches CIDRs and
    selectors and cannot express a hostname, so every egress rule permits its
    port to ANY address. That is a limit of the object, not a shortcut — this
    cluster's CNI (Canal: Flannel plus Calico) has no FQDN rule either. An
    outbound port the chart does not know about is BLOCKED: add it to
    networkPolicy.egress.extraPorts, and expect to find out through an
    application-level connection error rather than through this object.
{{- if .Values.networkPolicy.probeSources }}
  - THE PROBE PATH IS RESTRICTED in this release, because
    networkPolicy.probeSources is set to
      {{ join ", " .Values.networkPolicy.probeSources }}
    Probes come from the IP of the WORKER NODE running the pod. If none of those
    CIDRs covers the node this pod lands on, the kubelet's probes are dropped
    and THE POD NEVER BECOMES READY — the rollout then times out and the symptom
    looks like a broken application. Compare

      kubectl -n {{ .Release.Namespace }} describe pod -l app={{ .Release.Name }}-{{ .Chart.Name }}-sfs

    against `kubectl get node -o wide`.
{{- else }}
  - THE PROBED PORT IS OPEN INSIDE THE CLUSTER. networkPolicy.probeSources is
    empty, so that rule carries no source selector at all and any pod the CNI
    routes to can reach it. Not from the internet — but this is NOT pod-level
    isolation. Narrowing it means listing the worker node IPs in
    networkPolicy.probeSources, because the kubelet probes from the node's own
    address and the chart cannot know it; a wrong guess there leaves the pod
    permanently un-Ready.
{{- end }}
  - IT IS ONLY A BOUNDARY WHERE THE CNI ENFORCES POLICIES. Where it does not,
    this object is accepted and filters nothing, so everything reads as
    configured while nothing is filtered. `kubectl -n {{ .Release.Namespace }} get networkpolicy`
    shows the object exists; only a connection attempt from an unrelated
    namespace shows that it bites.

  Switch it off with networkPolicy.enabled: false.
{{- end }}