jlab-cluster/helm-charts
3
0 Fork 0
Code Issues 15 Pull Requests 3 Actions Packages Wiki Activity
Page: bulwark
Pages
CI CD Chart Style Guide Charts Forge and Tooling Home Lessons Learned Local Development New Chart Renovate Versioning Workflow apache tika bentopdf bulwark calibre web automated convertx cyberchef freesailarr garage gotenberg homeassistant keycloak matrix synapse minecraft router minecraft server outline paperless ngx patchmon rallly rss bridge samba satisfactory searxng stalwart sure teamspeak server template chart urlaubsverwaltung valkey voucher vault ytdlp web player zipline
Clone
4
bulwark
Justin Klein edited this page 2026-10-14 14:57:00 +00:00
Unescape Escape
Table of Contents
This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

bulwark

Deploys Bulwark Webmail — a JMAP webmail client (mail, calendar, contacts, files) — as the browser front end for a stalwart mail server.

It is a client, and that is the first thing to internalise. It holds no mail, has no database, no cache and no broker. The messages, the mailboxes and the accounts belong to the Stalwart release named by bulwark.jmapServerUrl; upstream puts it plainly: "Stalwart is the mail server: it holds the messages, speaks SMTP, and owns the accounts." The practical consequence runs through this whole page — with the mail server completely gone, this release is Ready, green, and useless. Nothing here can detect that, and no probe could.

The second thing to internalise is in Traps: the volume beats the chart. Every value this chart sets is a default that the admin dashboard may already be overriding, silently, forever. That section is the reason this page exists as much as the values table is. Read it before you open /admin, not after.
Image ghcr.io/bulwarkmail/webmail, AGPL-3.0-only, node:24-alpine with a Next.js standalone build. Upstream removes npm and npx from the runtime layer and runs apk upgrade in the build — the image is in good shape
Versions not listed on purpose — read the current pair off bulwark/Chart.yaml, see Chart index. Note that the registry publishes every release twice, 1.12.0 and v1.12.0 for the same image; the chart pins the form without the v, and Renovate keeps it in that family
Workloads one StatefulSet, 1 replica — capped at 1, see Storage
Storage one PVC at /app/data, carrying all four of Bulwark's directories as subdirectories
Routing two mutually exclusive modes, Ingress (default) or Gateway API (httpRoute.enabled) — one object in either mode, at the root of a hostname. A sub-path is impossible, which is also why the route carries no rewrite filter — see Three options are compiled in and Routing. In Gateway mode the upload limit is not enforced
External dependencies a Stalwart (or other JMAP) server, and an OIDC provider (keycloak) while bulwark.oauth.enabled is true. No database, no cache, no broker
Secrets 2 × OnePasswordItem, three fields in total — two of them as files, which is the better path garage and keycloak did not have
Network ClusterIP Service on 3000 (http) plus a headless Service the StatefulSet needs for its serviceName. No NetworkPolicy
Config files none rendered. No ConfigMap, so no checksum/config — see the trap
Install-test ci/bulwark/ — the cheapest in the repo: two dummy Secrets and a selector-less Service, no infrastructure at all

Prerequisites

  • A reachable JMAP endpoint. In-cluster that is a Stalwart release's http Service; a published one is its HTTPS URL. It does not have to be reachable at install time — Bulwark connects per request — but the value must be set.
  • 1Password operator in the cluster plus two items — see What the 1Password items must contain.
  • An OIDC client, registered BEFORE the first install, while bulwark.oauth.enabled is true. This is not a follow-up step: with bulwark.oauth.only (the default) the password form is hidden, so without a working client the first sign-in is impossible. And read the redirect URI carries the UI language before you register it — a single redirect URI is the likeliest reason a first sign-in fails.
  • One DNS name and a cert-manager cluster issuer. In Gateway mode the cluster issuer is not needed from this release: the certificate sits on the Gateway's listener. What is needed instead is the Gateway-API CRDs plus a Gateway whose listener admits routes from this namespace — and a body-size limit configured next to that Gateway, because this chart's recommendation does not reach it (why).
  • A StorageClass, or the cluster's default.

Installation

The registry URL and its credentials live in 1Password (item jlab-registry-access: username, password, charts-url) and never in a repository or a shell history file.

helm repo add jlab "$CHARTS_URL" --username "$REG_USER" --password "$REG_PASS"
helm repo update

helm install bulwark jlab/bulwark \
  --namespace bulwark --create-namespace \
  -f my-values.yaml

Minimal my-values.yaml:

secrets:
  appSecretRef: op://vault/bulwark-app
  oauthSecretRef: op://vault/bulwark-oauth

bulwark:
  jmapServerUrl: http://stalwart-stalwart-full-http.stalwart.svc.cluster.local:8080
  oauth:
    clientId: bulwark
    issuerUrl: https://id.example.com/realms/main

storage:
  amount: 1Gi

ingress:
  domain: webmail.example.com
  clusterIssuer: letsencrypt

Seven values, and six of them have no default because no default could be right. The chart refuses to render without jmapServerUrl, without the two OIDC values while OIDC is on, and without the vault references — each with a message naming the value and why. helm install then prints NOTES.txt, which is the per-release version of this page: the vault fields, the redirect URI to register, the first-install checklist and the way out of a lockout.

What the 1Password items must contain

The chart renders two OnePasswordItems; the operator turns each into a Kubernetes Secret whose keys are the field names of the 1Password item.

Values key Kubernetes Secret Field Arrives as Content
secrets.appSecretRef <release>-bulwark-app-secret session-secret file, SESSION_SECRET_FILE 32 random bytes, base64: openssl rand -base64 32. Mandatory
admin-password env var, ADMIN_PASSWORD the admin dashboard's initial password. Consumed exactly once
secrets.oauthSecretRef <release>-bulwark-oauth-secret oauth-client-secret file, OAUTH_CLIENT_SECRET_FILE the confidential client's secret, as issued by the provider

Two items and not one, split by audience. The first two fields are this release's own and nobody else knows them. The OIDC client secret is not ours — the identity provider creates it, rotates it, and also holds it — so a separate item lets it be repointed and rotated without touching the other two. Same reasoning as garage's split between its RPC secret and its S3 key.

A missing or misnamed field here does not stop the pod, and that is worth knowing because it inverts the usual symptom. The two file-form fields are mounted from an optional volume (so that a first install does not hang in ContainerCreating while the operator catches up), so a typo yields a missing file and a perfectly healthy pod: settings sync switches itself off and every token exchange sends an empty secret. Only admin-password fails loudly — a wrong name there leaves the container in CreateContainerConfigError.

Why session-secret is mandatory rather than recommended

Two things hang on it, and one fails silently without it:

  • persistent "remember me" sessions, and
  • the encryption of the synchronized user settings on the volume.

app/api/config/route.ts computes settingsSyncEnabled: configManager.get(...) && hasSessionSecret(). So with bulwark.settingsSync: true (the default) and no secret, synchronization is simply off, with no error anywhere.

admin-password is consumed exactly once

initAdminPassword returns as soon as admin.json exists on the volume and never looks at the environment variable again; on the first start it hashes it with scrypt and persists the hash. Rotating the vault field afterwards has no effect — restart or not. A reset goes through the dashboard itself, or by deleting /app/data/admin/admin.json and restarting.

And the reason it is set at all is not a secret leak. That was the assumption going in and it is wrong: without ADMIN_PASSWORD Bulwark generates no random password and writes nothing to stdout — lib/admin/password.ts logs Admin dashboard disabled (no ADMIN_PASSWORD set) and the dashboard stays off. The only random value in the project is the bootstrap setup token, which goes into the state directory and is never logged. So there is no STALWART_RECOVERY_ADMIN situation here (stalwart, Gitea #65). It is set because the admin dashboard is the only way back out of an OIDC lockout — switching it off means taking away the door.

Storage: one volume, and why

One PVC at /app/data. Bulwark keeps four directories under it and creates all four itself on start (ensureConfigDir / ensureStateDir, both mkdir -p) — it has to, because the image's own mkdir ran in a layer the mount covers.

Directory Holds Must survive
/app/data/admin config.json, policy.json, admin.json (the password hash), plugin-config/, plugins/, themes/, branding uploads yes
/app/data/admin-state admin-state.json (login timestamps), audit.log, the bootstrap setup token yes — the audit trail cannot be reconstructed
/app/data/settings the encrypted per-account user settings yes, while settingsSync is on
/app/data/telemetry consent record and instance id only to keep them stable across image upgrades

One volume and not four, because a snapshot is atomic per volume and never across volumes — the durability lesson garage was built on. Split up, a backup could capture an audit log recording logins the configuration does not know about yet, or settings encrypted against a secret the configuration has since been repointed from. There is no argument on the other side: this is a few hundred kilobytes of JSON plus an append-only log.

What actually grows is audit.log, and it grows with admin activity rather than with mail volume. The application does not rotate it. On a long-lived release that is the number to watch, and 1Gi is generous for everything else.

One replica follows from the volume, not from the application. ReadWriteOnce means two pods cannot both mount it, so the schema caps bulwark.replicas at 1. An RWX volume would not help either: nothing in Bulwark coordinates two processes over that directory, and the admin config is read into memory once at start, so two pods would hold two divergent views of it. The PVC name is never changed — a renamed PVC provisions a new, empty volume and orphans admin.json and the audit log.

First install: the checklist

  1. Wait for the rollout.

    kubectl -n <ns> rollout status statefulset/<release>-bulwark-sfs

    Ready means the HTTP server is accepting. It does not mean the mail server is reachable — nothing in this chart checks that.

  2. Confirm the setup wizard really is off.

    kubectl -n <ns> logs statefulset/<release>-bulwark-sfs | grep -i "SETUP REQUIRED"

    Nothing found is the correct result. A SETUP REQUIRED banner with a token means Bulwark is in bootstrap mode — see the wizard is the thing to keep switched off.

  3. Sign in through the identity provider at https://<ingress.domain>.

  4. Read the volume beats the chart before you open /admin. This is the step people skip and then spend an afternoon on.

  5. Know that the admin password is already spent (above). If you intend a different one, change it in the dashboard now rather than in the vault later.

  6. Set the settings policy you want, including the push relay, before you ever consider bulwark.adminConfigReadOnly — that value freezes the policy permanently.

If nobody can sign in: the way back

Read this now, not when it happens, because the obvious repair does not work.

The situation: the identity provider is down or misconfigured, bulwark.oauth.only hides the password form, and no user can get in.

What does not work

helm upgrade with bulwark.oauth.only: false.

oauthOnly is an ordinary key of CONFIG_ENV_MAP, and every consumer reads it through configManager.get — app/api/auth/session/route.ts, app/api/auth/verify/route.ts, app/api/config/route.ts; not one of them reads process.env. The admin dashboard's own OAuth setup writes oauthOnly: true into config.json when it is used. So once that override is on the volume, the chart's value is ignored, the upgrade reports success, and nothing says why.

This is the precedence trap sitting inside the rescue instruction itself, which is why it is called out here rather than in a footnote.

What does work

Neither step depends on the identity provider.

  1. Open https://<ingress.domain>/admin/login and sign in with the admin-password field from the vault item. This door has its own password and is not behind OIDC.
  2. Remove the override — the dashboard's revert, i.e. DELETE on /api/admin/config, which calls configManager.removeAdminOverride. After that the chart's value applies again.

With no Ingress, reach the dashboard with kubectl -n <ns> port-forward statefulset/<release>-bulwark-sfs 3000:3000.

If the dashboard itself is unreachable

The last resort is the volume:

kubectl -n <ns> exec <release>-bulwark-sfs-0 -- cat /app/data/admin/config.json
# remove the "oauthOnly" key (and any other override in the way), then
kubectl -n <ns> rollout restart statefulset/<release>-bulwark-sfs

Deleting admin.json in the same directory resets the admin password to whatever the vault field currently holds, on the next start.

Configurable values

Global

Value Default Notes
scaleDown false true renders the StatefulSet with replicas: 0 and overrides bulwark.replicas. PVC, Secrets, Service and Ingress stay

secrets

Value Default Notes
secrets.appSecretRef null required. op:// path; fields session-secret, admin-password
secrets.oauthSecretRef null required while bulwark.oauth.enabled. op:// path; field oauth-client-secret

bulwark

Value Default Notes
bulwark.replicas 1 0 or 1 only; the schema caps it. See Storage
bulwark.port 3000 What the image binds. The Service and all three probes address the port by name (http), so the number is free and the name is not
bulwark.jmapServerUrl null required. The mail server's address and the switch that disables the setup wizard
bulwark.appName null APP_NAME; null leaves upstream's "Webmail"
bulwark.stalwartFeatures true Stalwart-specific UI: password change, Sieve rules. Does not gate the server-side credential passthrough, which has its own upstream variable this chart does not render
bulwark.stalwartAdminAccess auto auto / password / off. Who may open the admin dashboard — read the note below
bulwark.settingsSync true User preferences follow the user. Puts user data on the volume, encrypted with the session secret
bulwark.adminConfigReadOnly false Makes the chart authoritative. Three-part price — see its own section
bulwark.telemetry false BULWARK_TELEMETRY. Upstream's default is already off; what this buys is the lock (see below)
bulwark.updateCheck false BULWARK_UPDATE_CHECK. Unlike telemetry this is on unless switched off, so the value does change behaviour
bulwark.logLevel info error / warn / info / debug
bulwark.logFormat json Against upstream's text, because the logs go to a collector
bulwark.oauth.enabled true Gates the OIDC values, the second vault item and the secret file
bulwark.oauth.only true Hides the password form. The riskiest default in this chart — see the way back
bulwark.oauth.clientId null required while enabled; the render fails without it
bulwark.oauth.issuerUrl null required while enabled. For keycloak: https://<host>/realms/<realm>
bulwark.oauth.scopes null Replaces the scope list rather than adding to it. null keeps upstream's. Use OAUTH_EXTRA_SCOPES through bulwark.env to add
bulwark.oauth.endSession true Signing out also ends the provider's session
bulwark.writableCache.tmpSizeLimit 256Mi sizeLimit of the /tmp emptyDir
bulwark.writableCache.nextCacheSizeLimit 512Mi sizeLimit of the /app/.next/cache emptyDir
bulwark.resources.limitFactor 3 Memory and ephemeral-storage limits = factor × request; no CPU limit unless set
bulwark.resources.requests 100m / 256Mi / 64Mi The memory limit is what V8's heap limit derives from — see the probe trap
bulwark.livenessProbe GET /api/health 10s / 5s / 3
bulwark.readinessProbe GET /api/health 10s / 5s / 3. Carries no statement about Stalwart
bulwark.startupProbe GET /api/health 5s / 5s / 24 → a 120s budget. Gates the other two
bulwark.securityContext.pod uid/gid/fsGroup 1001 The image's own user (adduser --uid 1001 nextjs); fsGroupChangePolicy: Always
bulwark.securityContext.container hardened readOnlyRootFilesystem: true, all caps dropped, no privilege escalation
bulwark.env [] Additive. And only the middle of three layers — see the trap below

storage

Value Default Notes
storage.amount 1Gi Deliberately not null-safe: an erased size yields an invalid PVC, which fails loudly
storage.storageClass null null uses the cluster default. ReadWriteOnce

ingress

Value Default Notes
ingress.enabled true Mutually exclusive with httpRoute.enabled. Gating it off frees clusterIssuer; it frees domain only if httpRoute is off as well — the route reads the domain too. This page used to say nothing outside the Ingress template functionally reads the domain; that was true until the Gateway support landed
ingress.domain null Required while either mode is on. Bulwark is served at the root of it. Also the HTTPRoute's hostname while httpRoute.hostnames is empty, and the host NOTES.txt builds the OIDC redirect URI from
ingress.clusterIssuer null Required while the Ingress is enabled. Unused and unnecessary in Gateway mode
ingress.className nginx No counterpart in Gateway mode — the class sits on the Gateway
ingress.annotations {} Merged with the cert-manager annotation. Consider proxy-body-size — see the trap. No counterpart in Gateway mode: the Gateway API has no annotation channel, so this block is inert there, and for this chart that is the one annotation the installation actually runs on

httpRoute

The Gateway-API alternative to the Ingress, off by default. Added with group 3 of the Gateway-API sweep (Gitea #114); the form is the pilot's and is binding for every chart (style guide §4).

Value Default Notes
httpRoute.enabled false Renders <release>-bulwark-httproute instead of the Ingress. Mutually exclusive with ingress.enabled — with both true the render aborts naming both values
httpRoute.parentRefs [] The Gateway(s) the route attaches to — Gateway API ParentReference entries (name, and optionally namespace, sectionName, port, group, kind), passed through unchanged. Required while httpRoute.enabled is true: the render aborts on an empty list. The schema requires name on every entry and is otherwise left open, because Kubernetes owns that schema and adds fields to it across releases
httpRoute.hostnames [] Empty means [ingress.domain], so the hostname has a single source of truth. If you set it, register that origin with the identity provider first: NOTES.txt derives the OIDC redirect URI from the active mode's hostname, and a login returning to an unregistered origin is rejected at the provider, which says nothing about this chart

One rule, at /, with no URLRewrite filter — the application cannot be served under a sub-path at all, so there is nothing to strip.

networkPolicy

Inbound 3000 only. Outbound DNS and TCP 443, which covers both of Bulwark's outbound dependencies: the JMAP server named by bulwark.jmapServerUrl, and the OAuth/OIDC provider.

Value Type Default Meaning
networkPolicy.enabled bool true Renders the policy. Only a boundary where the CNI enforces one — unenforced, the API server accepts it and nothing is filtered, so everything reads as configured.
networkPolicy.allowedNamespaces list [] Namespaces whose pods may reach the restricted ports, additionally to the release's own. Matched on the kubernetes.io/metadata.name label Kubernetes sets itself — no labelling work, and no typo protection either: a name that does not exist matches nothing and is silently ineffective. Namespace-granular, so an entry opens the port to every pod in it.
networkPolicy.probeSources list [] CIDRs allowed to reach the probed port. Empty means open, not blocked — the rule is then rendered with no from at all. Probes come from the IP of the worker node running the pod, which the chart cannot know; a list that misses it drops every probe and the pod never becomes Ready.
networkPolicy.ingressController object kube-system / app.kubernetes.io/name: rke2-traefik The controller allowed to reach the routed port once probeSources restricts it. The two keys render as one from entry, i.e. an AND — which is why this is not a line in allowedNamespaces, where it would open the port to every pod in kube-system. Default measured on the cluster, not assumed.
networkPolicy.egress.extraPorts list [] Additional outbound TCP ports, to any destination. The escape hatch from the egress cap, and the only alternative to switching the policy off.

Operating notes

Routing: Ingress or Gateway API

Two mutually exclusive modes, and exactly one of them may be on. ingress.enabled ships true, httpRoute.enabled ships false.

Rule Why
ingress.enabled and httpRoute.enabled cannot both be true — the render aborts naming both value paths two data paths onto one Service under one hostname is a conflict at DNS level already. For Bulwark it also breaks the login: the OIDC redirect URI registered with the provider names one origin, so a session that starts on one data path and returns on the other fails on the state cookie. A migration runs over a second hostname — and that second origin has to be registered as a redirect URI too
httpRoute.parentRefs must not be empty — the render aborts a route without a parent attaches to no Gateway. The object is created, it is schema-valid, no controller rejects it, and it routes nothing: no event, no condition, and the only symptom is a hostname that does not answer
bulwark.publicUrl follows the active mode the helper NOTES.txt prints the access URL and the OIDC redirect URI from. The two modes' hostnames differ exactly when httpRoute.hostnames is set, which is how a migration runs — printing ingress.domain there would hand the operator a redirect URI for the host being migrated away from, and that failure surfaces as a rejected login at the provider rather than as anything about this chart

What this chart is not, and #114 said otherwise

Worth having written down, because the issue's measured table and the CI test's own header both expected something else of this chart:

  • It has ONE Ingress object, not two. keycloak is the chart with two.
  • It has no rewrite-target annotation and cannot have one. NEXT_PUBLIC_BASE_PATH is compiled into the bundle (Three options are compiled in), so a sub-path deployment needs a rebuilt image, and stripping a prefix at the proxy would produce a page that loads and then fetches everything from the wrong place. Every occurrence of the string rewrite-target in this chart — values.yaml, the schema, the Ingress template — is an explanation of exactly that, and a git grep for the string counted those refutations as occurrences.
  • Consequently the #118 prefix/redirect assurance does not apply here. That check pairs a prefix-stripping URLRewrite with an Exact-match RequestRedirect on the bare prefix; with no rewrite on a prefix other than /, its precondition is never met. There is no bare prefix to redirect.

What Gateway mode costs this chart

Measured on the live Ingress on 2026-10-12:

nginx.ingress.kubernetes.io/proxy-body-size: 25m

The chart does not set it — values.yaml recommends exactly that value and explains why the chart cannot pick it (the right number is the mail server's, which this chart does not know), so the 25m is the operator's, supplied through ingress.annotations. It is the single annotation this installation depends on, and in Gateway mode it is gone: the Gateway API has no body-size field, that is implementation policy rather than routing, and the chart renders no implementation-specific policy object on purpose (style guide §4).

So the limit in force becomes the Gateway's, whatever it happens to be, and no value of this chart has any say in it. The failure mode on a webmail client is worth picturing before it happens: the user has written the mail, attached the file and pressed send, and the 413 comes from the proxy. It reads like Bulwark being broken, nothing mentions a limit, and the draft is the thing at risk.

Set the equivalent next to the Gateway before pointing DNS at it, and size it to what the mail server accepts (stalwart) rather than to what looks generous here — a webmail client that accepts a 40 MB attachment the mail server will refuse has only moved the failure later. The Gateway branch of NOTES.txt prints this.

Check the JMAP port — this is where the cap fails most quietly

443 is where a published JMAP endpoint lives. If bulwark.jmapServerUrl points straight at Stalwart's in-cluster HTTP port (8080), or anywhere else, that traffic is blocked — and Bulwark's probes only test its own port.

So the pod comes up Ready and then serves an empty, broken webmail, with nothing in the events to say why. Put the real port in networkPolicy.egress.extraPorts; 8080 is the entry this chart needs most often.

A NetworkPolicy matches CIDRs and selectors and never hostnames, and this cluster's CNI (Canal — Flannel plus Calico) has no FQDN rule, so an egress rule can only ever name a port and not a destination.

The probed port is open inside the cluster by default

networkPolicy.probeSources is empty out of the box, so the rule for the port the probes address is rendered with no from selector at all — any pod the CNI routes to can reach it. Not from the internet (the port carries no NodePort), but this is not pod-level isolation, and reading the object as if it were is the mistake to avoid.

It is that way on purpose. The kubelet's liveness, readiness and startup probes originate from the IP of the worker node running the pod — an address no chart can know and must not guess. A CIDR that excludes the real node blocks every probe: the pod never becomes Ready, the rollout times out, and the symptom reads as a broken application rather than as a policy. Shipping open is the honest default; narrowing it means listing the worker node IPs yourself.

Note the companion trap: the moment probeSources is non-empty the rule is restricted, so everything else that needs the port has to be in the same rule. The chart adds the release's own namespace, ingressController and allowedNamespaces for exactly that reason.

The policy is only a boundary where the CNI enforces one

The API server accepts a NetworkPolicy whatever the cluster does with it. Where the CNI does not enforce policies, everything looks configured and nothing is filtered — which is worse than no policy, because it invites the assumption that something is being filtered. Verify by connecting from an unrelated namespace rather than by reading it off the manifest.

Measured for this cluster: the CNI is Canal (Flannel for the network, Calico for policy), 6/6, and it does enforce. Measured for CI: k3s brings its own policy controller and enforces too — but it exempts the kubelet's probe traffic, so a blocked probe path does not turn the install-test red. What CI does catch is anything the application itself needs.

Traps

The volume beats the chart, for every value, silently

The single most important thing on this page. lib/admin/config-manager.ts resolves a config value in this order:

admin override (config.json on the PVC)  >  environment variable  >  default

Every setting this chart renders is therefore a default that the admin dashboard may already be overriding. The dashboard may write any key of CONFIG_ENV_MAP (app/api/admin/config/route.ts validates against exactly that key list), and nothing in the UI says that it has just disconnected the chart. A helm upgrade with a changed value then changes nothing, reports success, and leaves no trace of why.

Upstream documents this for the server URL only — "A server URL saved by the wizard takes priority over the environment variable" — and that wording is the trap, because it reads like a special case. It is the behaviour of every key.

Exactly three values escape it, all verified:

Value Why it escapes
SESSION_SECRET / SESSION_SECRET_FILE getSessionSecret() reads the environment first and the admin override last — the reverse of the normal order
ADMIN_PASSWORD initAdminPassword() reads the environment only (and only once)
BULWARK_TELEMETRY effectiveConsent() takes the environment over the state file, and upstream blocks the admin UI from changing it while the override is set

The OAuth client secret is not one of them, which is the one most likely to be assumed: getGlobalClientSecret() checks the admin override first. If the dashboard's OAuth setup has ever run, config.json carries oauthClientSecret in plaintext and that value beats the mounted file forever.

To see what is overridden right now:

kubectl -n <ns> exec <release>-bulwark-sfs-0 -- cat /app/data/admin/config.json

An empty object, or no file, means the chart is in force.

adminConfigReadOnly makes the chart authoritative, and costs more than it looks

bulwark.adminConfigReadOnly: true sets ADMIN_CONFIG_READONLY, which refuses every write to the admin config directory at the application layer. No override can appear, so the chart's values always apply and the trap above is closed for good.

It defaults to false because the price is threefold, and the third part was measured rather than assumed:

  1. Policy, themes and branding freeze too. assertWritable (lib/admin/paths.ts) guards setPolicy, markSetupComplete, the plugin and theme registries and the branding upload — not just the config override. So the settings policy becomes unchangeable, and with it the only way to point Web Push at your own relay.
  2. An override that already exists is frozen in place, not removed. removeAdminOverride is guarded by the same check. Turning this on while config.json already carries oauthOnly: true makes that lockout permanent and removes the documented repair in the same move.
  3. On a fresh install it takes the admin dashboard away entirely — i.e. exactly the door this chart keeps a password for. initAdminPassword persists the hash through writeConfigData, which begins with assertWritable('save admin password'). With no admin.json on the volume yet, that throws: the startup hook logs Admin dashboard init skipped, the retry inside /api/admin/auth throws again, and the login endpoint answers HTTP 500. The pod is Ready and serving mail the whole time, so nothing reports it.

The only safe sequence, which is also upstream's own framing ("Operators set ADMIN_CONFIG_READONLY=true after running the setup wizard"):

  1. install with it false and let the pod start once, so admin.json exists;
  2. set every policy you want, push relay included — after step 4 you cannot;
  3. check config.json for an override you still need removed;
  4. then helm upgrade with it true.

The chart cannot verify step 1 for you: it has no way to know what is on the volume.

Who can open the admin dashboard, and therefore write overrides

bulwark.stalwartAdminAccess keeps upstream's default auto, and the consequence is stated rather than left to be found: with auto, anybody who is an admin on the Stalwart side is signed into /admin without the Bulwark admin password. Since the dashboard is what writes the overrides from the trap above — and what can lock the login path — that is a wider door than it looks.

It is kept because it is usually the same person. If it is not: password narrows it to whoever holds the vault field, and off also removes the shield from the UI. Both require an admin password to be configured, which this chart always does.

The redirect URI carries the UI language

This is the likeliest reason a first sign-in fails, and it is invisible from the address bar.

Bulwark builds its OIDC redirect URI as

https://<ingress.domain>/<locale>/auth/callback

where <locale> is the language the visitor's browser resolved to. A German browser sends /de/auth/callback, an English one /en/auth/callback — from the same deployment. The locale prefix mode in the published image is never, so the language is not in the visible URL, which is exactly why this surprises people: the address bar shows no language and the redirect_uri contains one. Upstream's own admin helper registers one URI per locale for this reason.

Register one per language you expect, or a wildcard where the provider allows it. On Keycloak the wildcard is the practical answer:

https://<ingress.domain>/*

The 27 supported locales are ar ca cs da de en es fa fr he hu it ja ko lv mn nb nl pl pt ro ru sk tr uk zh zh-TW.

If a sign-in fails with a redirect_uri mismatch, read the URI out of the failed request rather than guessing which language it was.

The wizard is the thing to keep switched off

Bulwark has a first-run setup wizard that "chooses OAuth or password sign-in, generates a session secret and sets the admin password" and writes the result into the config directory. bulwark.jmapServerUrl disables it: detectSetupState() returns env-managed as soon as the variable is non-empty, and in that state /setup is not served at all.

The chart refuses to render without that value, and the reason is sharper than "the mail server is unset": an empty JMAP_SERVER_URL puts the app into bootstrap, serves the wizard, and whatever is clicked through there lands in config.json — from which moment the volume beats every value the chart sets, permanently. An unset value does not produce a broken release, it produces a silently different one.

The probes say nothing about the mail server

All three probes are a plain HTTP GET on /api/health.

Not HEAD, although Bulwark offers one and it would be cheaper: a Kubernetes HTTPGetAction has no method field and the kubelet always issues a GET. The only alternative would be an exec probe calling an HTTP client, and the image contains neither curl nor wget (npm and npx are removed from the runtime layer), so it would mean starting a Node process every few seconds. The GET handler does gather diagnostics, which makes it marginally more capable than HEAD, not less.

What the probe really carries: the Node process is alive, the HTTP server accepts, the event loop is not blocked, Next.js can route. An OOM kill, a crash, a wedged event loop and a container that never finished booting all fail it. That is genuinely more than urlaubsverwaltung's endpoint carried (#218).

What it does not carry:

  • The reported state. The success path returns a hard 200 whatever it computed, and for an unauthenticated caller the body is hard-coded to healthy (status: detailed ? status : 'healthy'). degraded appears only with ?detailed=true and a caller passing canSeeInstanceDetails. Upstream tests exactly that: lib/__tests__/health-route.test.ts expects HTTP 200 while payload.status is degraded. The only 503 this endpoint can produce comes from its catch block, i.e. when process.memoryUsage() itself throws.
  • Anything about Stalwart. Bulwark holds no mail and this endpoint asks the JMAP server nothing. With the mail server completely gone, the probe is green and the pod stays Ready. Monitor the stalwart release for that; nothing here can.

There is no better endpoint: /api/setup/test-jmap would really test the connection, but it sits behind the bootstrap setup token, which this chart disables.

A consequence of the first point worth pairing with the resources: /api/health reports degraded on high V8 heap usage, and the heap limit derives from the container memory limit. So a limit set too low produces a pod that is permanently degraded, permanently Ready, and says so only to an authenticated caller asking for details.

Backups: the volume alone is not one, and it may need protecting itself

Two statements, both about <release>-bulwark-pvc, and they pull in opposite directions:

  1. A backup of the volume alone is not a backup. The synchronized user settings on it are encrypted with session-secret. Lose the vault item and that data is unreadable out of a perfectly intact volume. The 1Password item belongs in the backup just as much as the volume does.
  2. A backup of the volume may contain a secret in cleartext. app/api/admin/oauth/setup/route.ts writes oauthClientSecret as plaintext into config.json on the PVC. So if anybody has used the dashboard's OAuth setup, the volume backup may need the same protection the vault gets.

Check for the second with:

kubectl -n <ns> exec <release>-bulwark-sfs-0 -- grep -l oauthClientSecret /app/data/admin/config.json

Secret rotation works without a restart — because of one mount detail

Both file-form secrets are mounted from a projected volume without subPath, and that is load-bearing rather than stylistic.

getSessionSecret() and getGlobalClientSecret() both call readFileEnv on every use, with no cache. So a rotated field in the vault is live as soon as the operator has written the Secret — no pod restart. The kubelet never refreshes a subPath mount, so a subPath here would silently freeze both secrets at their first value while the vault, the Secret and helm upgrade all looked correct.

The mode is 0440, not 0400: files in a secret volume are owned by root, and what gives uid 1001 access is fsGroup, which only supplies the group. Without a group read bit the files would be unreadable — and silently so, since readFileEnv answers an EACCES exactly as it answers a missing file.

Two exceptions to "rotation just works":

  • admin-password is consumed once and no restart helps (above).
  • The OAuth client secret is beaten by an admin override in config.json if one exists (above).

garage's file-permission trap does not apply here, and it was checked rather than assumed, because it is what forced garage onto an environment variable: lib/read-file-env.ts is readFileSync plus try/catch and nothing else — no statSync, no mode & 0o077. Bulwark does not reject a group-readable secret file, which is what makes the file form usable at all.

There is no ConfigMap, so there is no checksum/config

Not an omission. Bulwark is configured through environment variables plus the two JSON files it writes itself onto the PVC, so there is no rendered config file whose change could fail to restart the workload (style guide §9). values.yaml states this explicitly so it does not read as a gap.

What is covered is the op:// paths, through checksum/secret-ref: repointing secrets.appSecretRef or secrets.oauthSecretRef at a different item rolls the pod. A change inside an item does not — Helm never sees it — but for the two file-form secrets it does not need to (above).

The 1Password operator's auto-restart annotation is not set and must not be: against operator 1.12.0 it enumerates Deployments only, so on a StatefulSet it is a silent no-op advertising a guarantee it does not provide.

Web Push goes through the vendor's relay until you change it — and that is not a chart value

The obvious reading of the build-time variable is wrong, so this is stated carefully.

NEXT_PUBLIC_PUSH_RELAY_URL is compiled into the bundle, so the chart cannot move the built-in entry, which is https://notifications.relay.bulwarkmail.org. It would be wrong to stop there and call Web Push "not changeable with the published image": that variable only sets the fallback. lib/push-relays.ts resolves the effective relay as resolveDefaultRelayUrl(policy), which takes policy.pushRelayUrl whenever it is a valid URL; with pushRelayUrlLocked: true every user is pinned to it.

So, two-part:

  • Without further action, Web Push for this release goes through the vendor's relay. For a mail client that is worth knowing, and it stays in the users' relay list either way, because the list is built from the bundle.
  • A self-hosted relay can be made the effective default and even mandatory — through policy.json, i.e. the admin dashboard, not a chart value. There is deliberately no value for it: pushRelayUrl lives in SettingsPolicy and has no entry in CONFIG_ENV_MAP, so no environment variable reaches it and a chart value would be a switch wired to nothing.

And the interaction: adminConfigReadOnly freezes setPolicy, so turning that on before setting the relay makes this change impossible afterwards.

Three options are compiled into the image

Upstream: "These are compiled into the bundle and cannot be changed at runtime … To change build-time variables with the published Docker image, rebuild with matching --build-arg." Building our own image is outside what this repository does — the same limit as garage and keycloak.

Variable Consequence
NEXT_PUBLIC_BASE_PATH Bulwark is served only at the root of a hostname. No sub-path, and a rewrite-target annotation does not help — the bundle's own asset URLs are built for the root
NEXT_PUBLIC_DEFAULT_LOCALE The fallback UI language stays en. A visitor with a matching Accept-Language gets their own; everybody else gets English
NEXT_PUBLIC_PUSH_RELAY_URL Only the built-in default entry — see the section above, which is the part that is not a limitation

Attachment uploads need proxy-body-size

File attachments and the Files view upload through the Ingress, and nginx's default proxy-body-size of 1m rejects anything larger with a 413 that reads like an application fault. The chart does not set it, because the right number is the mail server's limit and the chart does not know it:

ingress:
  annotations:
    nginx.ingress.kubernetes.io/proxy-body-size: "25m"

That is what the live installation runs on, measured 2026-10-12 — the recommendation is not hypothetical. In Gateway mode this annotation does not exist and has no counterpart; the limit then comes from the Gateway and has to be set there. See What Gateway mode costs this chart.

bulwark.env is additive, and only the middle layer

Additive means additive: an entry repeating a name the chart already sets does not replace it. At install both reach the pod and the later one wins; on an existing release the pair collapses to the chart's own value, because Kubernetes merges env by name — and helm upgrade still reports success (style guide §10).

And the sharper caveat specific to this application: an environment variable is only the middle of three layers. Anything set here is a default an admin override may already be beating. Genuinely useful entries, all CONFIG_ENV_MAP keys the chart does not render: APP_DESCRIPTION, the LOGIN_* branding variables, COOKIE_SAME_SITE, STALWART_JMAP_PASSTHROUGH_ENABLED, OAUTH_EXTRA_SCOPES, WOPI_CLIENT_URL.

Upgrades and backup

  • A plain app-version bump needs nothing special: no database, no migration, no schema. Renovate opens it and ci/renovate-bump-chart.sh bumps the chart version — since Gitea #128 at the app's own level, so an app patch is a chart patch and an app minor a chart minor (see Versioning).
  • Back up the PVC and the appSecretRef vault item together. Neither is sufficient alone — see Backups.
  • Before any upgrade worth the caution, read config.json off the volume. It tells you which of the chart's values are actually in force.
  • scaleDown: true stops the release without uninstalling it.

CI

ci/bulwark/ is the cheapest install-test in the repo: two dummy Secrets and a selector-less Service for the JMAP endpoint — a name that resolves and then refuses the connection. No database, no cache, no mail server, and deliberately no stub workload (a fixture Deployment has to roll out before the chart installs, so it would be one more thing able to fail a run that is not about it).

What a green run proves: readOnlyRootFilesystem: true holds with exactly the two emptyDirs the chart renders (/tmp and /app/.next/cache) — the set was derived from next.config.ts and can only be proven by starting the real image; the probes can really be satisfied, i.e. /api/health exists in the built bundle and answers 200 unauthenticated; uid/gid 1001 plus fsGroup let Bulwark create its four subdirectories in a volume it did not make; and ADMIN_PASSWORD is really hashed into admin.json inside the startup budget.

What it does not prove, and the list matters:

  • anything about JMAP or OIDC — nothing answers either address, which is precisely the limitation the readiness probe cannot see. The test demonstrates that limitation rather than closing it.
  • the precedence trap. config.json never exists in a CI run, so the chart's single most important operating caveat is not exercised. It cannot be: it needs an admin with a browser.
  • that the two file-form secret keys are spelled right. The projected volume is optional: true by design, so a misspelled key yields a missing file and a healthy pod. Only admin-password is proven.
  • that no emptyDir is superfluous. A missing writable path fails loudly; an unnecessary one does not. That half stays open.

The Gateway mode is held at render level, by ci/httproute/bulwark.yaml plus ci/run-httproute-match-test.sh, which runs over every chart on every PR (CI/CD). The overlay switches this chart into Gateway mode; 2 assertions — parentRefs not empty, and the backendRefs entry resolving to the serving Service of the same render by port number, because backendRefs takes a PortNumber and cannot name the port http the way the Ingress does. An overlay rather than a fixture change, because the install-test runs one configuration per chart and this chart's fixture installs the StatefulSet with its PVC and both vault items.

What that does not cover: that the hostname answers — no Gateway controller runs in CI, so no route reaches Accepted: True — and the proxy-body-size: 25m this installation depends on, which has no counterpart in that mode at all. That loss is documented in values.yaml and printed by NOTES.txt, and documentation is not something a test can assert. Nothing in CI reads NOTES.txt: helm template does not emit it and no script under ci/ reads it, so its text carries review-only assurance.

See also

helm-charts wiki

How we work

How it is built

How charts are written

Charts