diff --git a/Charts.md b/Charts.md index 78878a7..1f0d657 100644 --- a/Charts.md +++ b/Charts.md @@ -1,6 +1,8 @@ # Chart index -30 directories: 29 installable charts plus the `template-chart` scaffold. Each page holds prerequisites, an installation walkthrough, the complete list of configurable values, and the chart's operating traps. +**30 chart directories in the repository: 29 installable charts plus the `template-chart` scaffold.** Counted against `*/Chart.yaml` on 2026-10-05 rather than carried forward, because the previous number had drifted. Each page holds prerequisites, an installation walkthrough, the complete list of configurable values, and the chart's operating traps. + +> **The table below has 31 rows, and the extra one is a known discrepancy rather than a miscount.** It still lists [convertx](convertx), for which there is **no chart directory in this repository**. Flagged here instead of being deleted: removing a page and a row is a decision about whether that chart is gone, planned or moved, and that is not a documentation fix. Whoever knows the answer should resolve it — and then this note and the row go together. **Versions are not listed here on purpose** — and not on the chart pages either. Renovate moves app versions continuously; a table of numbers in a wiki is wrong within days. Read the current pair off the chart: @@ -12,6 +14,7 @@ grep -E '^(version|appVersion):' /Chart.yaml |---|---|---|---|---| | [apache-tika](apache-tika) | Apache Tika | Deployment + Service | no | content extraction for [paperless-ngx](paperless-ngx) | | [bentopdf](bentopdf) | BentoPDF | Deployment + ConfigMap | yes | `config.json` always rendered, `subPath`-mounted into the web root | +| [bulwark](bulwark) | Bulwark Webmail (JMAP client for [stalwart](stalwart)) | StatefulSet + **one** PVC | yes, at a host **root** only | **the volume beats the chart, for every value, silently** — `admin override > env var > default`, the admin dashboard may write any key, and a `helm upgrade` can succeed and change nothing. Only three values escape it. The way out of an OIDC lockout is therefore **not** `helm upgrade` with `oauth.only: false` but removing the override in the dashboard — which is why `ADMIN_PASSWORD` is set. Both secrets use the **file** form, mounted without `subPath` so rotation needs no restart. Probes carry **nothing** about the mail server: with Stalwart gone the pod is green | | [calibre-web-automated](calibre-web-automated) | Calibre-Web Automated | StatefulSet + extra consume PVC | yes | keeps `readOnlyRootFilesystem: false` on purpose — see [§12.4](Chart-Style-Guide#12-values-that-must-not-lie) | | [convertx](convertx) | ConvertX | StatefulSet + data PVC | yes | **the only chart that overrides `command` to bypass its image entrypoint** — that is what keeps the hardening baseline intact on a rootful image, at the price of re-reading the entrypoint at every app-version bump; probes use `/healthcheck` because `/` is a 302 | | [cyberchef](cyberchef) | CyberChef | Deployment | yes | stateless; the simplest chart in the repo and a good first read | diff --git a/Lessons-Learned.md b/Lessons-Learned.md index ee16e70..04a7d54 100644 --- a/Lessons-Learned.md +++ b/Lessons-Learned.md @@ -395,6 +395,49 @@ The decision taken here was **not** to switch automerge off for that image (Gite **Now:** when a chart's application restricts how it may be upgraded, write the restriction where the upgrade is performed — the chart's `NOTES.txt`, the chart page, and the [Renovate](Renovate) page if the automation could trip it — and say explicitly that the automation does not know about it. If upstream ships a tool that decides the question, name the tool, its exact invocation **and its exit codes**, because an exit code is the only form of that knowledge a pipeline can use. And where upstream's own documentation contradicts itself about the constraint, record the contradiction instead of picking the convenient half: the honest instruction is then "run the check every time", not "patches are safe". +## The rescue instruction that ran through the fault it was repairing + +The approved plan for [bulwark](bulwark) (Gitea #89) accepted a real risk knowingly — sign-in is OIDC-only, so a broken identity provider locks everybody out — and it did the right thing by writing down the way back. The way back was: `helm upgrade` with `OAUTH_ONLY=false`, *"the only door that does not depend on Keycloak"*. + +Reading the source showed that door can be locked **exactly when it is needed**. + +The application resolves every configuration value as `admin override > environment variable > default`, where the override is a file on the volume written by its own admin dashboard. The plan knew this — it had documented it, correctly, for the server URL, because that is the one case upstream's documentation mentions. What it had not noticed is that the rule is not about the server URL: it is the behaviour of **every** key, `oauthOnly` included, and the dashboard's OAuth setup writes exactly that key when somebody uses it. So on any installation where the dashboard had been used, the documented repair would have changed nothing, reported success, and left no trace of why. + +The actual repair is one layer further in: open the dashboard (its own password, not the provider's) and **remove** the override, which is what hands control back to the chart. Same door, different handle — and the chart now sets an admin password *because* that is the door, rather than for the reason the issue originally gave. + +**Now:** when a plan documents a recovery path, check it against the same mechanism the fault runs through. A recovery step is the one instruction nobody rehearses, so it is the one that must be derived rather than assumed — and a precedence rule that upstream documents for a single value should be read as a property of the system until something proves it is a special case. Where the recovery depends on a component staying reachable, say which component that is and what keeps it alive, because the next hardening pass will otherwise switch it off as unused. + +## The read-only switch whose reach exceeded its name + +The same chart gained a value to close that precedence trap: a flag that refuses all writes to the application's configuration directory, making the chart authoritative. Its price was understood as "the settings policy freezes too" — true, and already a reason for defaulting it off. + +Measuring found two more, and the second one is the reason this is an entry: + +- the function that **removes** an override is guarded by the same check, so turning the flag on does not clear an existing override — it **freezes it in place**, which converts a recoverable lockout into a permanent one and removes the repair in the same move; +- the admin password is persisted through the same guarded write, so on a **fresh** install the flag prevents the password file from ever being created. The dashboard then never comes up and its login endpoint answers HTTP 500 — while the pod is `Ready` and serving traffic the whole time. The flag intended to protect the configuration would have removed the only way back into it. + +None of that is visible in a render, and none of it would have failed a pipeline. + +**Now:** when a switch says "read-only", find every call site of the check it sets, not just the one the documentation names. A write barrier in an application tends to be implemented once and applied broadly, so its blast radius is whatever shares that helper — typically including bootstrap writes that happen exactly once, on a path nobody tests because it only exists on a first install. And where a flag is only safe in a particular order relative to the first start, the order **is** the documentation: say which state must already exist on disk before it is switched on, and say that the chart cannot verify it. + +## The probe method you do not get to choose + +The same plan specified `HEAD /api/health` for all three probes, reasoning — correctly — that the application offers a HEAD handler and that it is the cheapest call the endpoint supports. + +**A Kubernetes probe cannot send HEAD.** The API type is `HTTPGetAction`; it has no method field, and the kubelet always issues a GET. There is no value anywhere in a pod spec that changes it. The only way to issue a HEAD is an `exec` probe calling an HTTP client — and in a minimal image that often means there is no client to call: this one ships neither `curl` nor `wget`, with `npm` and `npx` deliberately removed, so it would have meant starting a language runtime every few seconds against an endpoint whose GET is already cheap. + +The outcome was harmless because GET is marginally *more* capable here than HEAD. That is luck, not design: had the application's GET been the expensive variant — a handler that touches a database, or one that only answers 200 when fully warmed — the plan's "cheapest form" would have silently become "the heaviest form", at the probe interval. + +**Now:** when choosing a health endpoint, check that the *platform* can make the call you are designing around before writing it into a plan. HTTP probes are GET, period; a method, a body or a non-trivial header set is an `exec` probe, and an `exec` probe is a promise about what binaries the image contains. And when the achievable form differs from the planned one, say which it is at the value — otherwise the next reader sees a comment describing HEAD next to a spec that performs GET and has to re-derive which one is true. + +## The secret file the container could not read + +A mode of `0400` on a mounted secret looks like the careful choice. In a pod it can lock out the very process that needs the file: **files in a secret volume are owned by `root`**, and what gives a non-root container access is `fsGroup`, which only supplies the **group**. With no group read bit, the application's own uid is denied. + +What makes it worth an entry is the failure mode rather than the mechanics. The kubelet ORs a group-read bit onto secret volumes once `fsGroup` is set, so `0400` is in practice corrected to `0440` — meaning the mistake usually *works*, for a reason nowhere in the chart. And where it does not work, the application in question answered `EACCES` exactly as it answered "file absent": a feature switched itself off, a credential went out empty, and nothing logged a permission error. (The opposite trap is already on this page under *A permission check that assumes it owns the filesystem* — [garage](garage) **rejects** a group-readable secret file. Two applications, opposite requirements, same mount.) + +**Now:** write the mode you actually need rather than the strictest one, and write down why the group bit is there — a mode that only works because the platform silently widens it is a mode the next hardening pass will narrow again. And when a chart hands a secret to an application as a *file*, find out what that application does when the read fails: if `EACCES` and `ENOENT` collapse into the same branch, a permission problem will present as a disabled feature, not as an error. + ## The recurring shape - **Green is a claim about the assertions that ran, not about the system.** @@ -422,4 +465,8 @@ The decision taken here was **not** to switch automerge off for that image (Gite - **An answer that arrived by accident is worth making repeatable.** The all-charts install-test exists because one unplanned run produced a result nobody could have asked for; the fix is a way to ask the question on purpose, not a note to remember the accident. - **Three correct requirements can still be an impossible pod.** Conflicts do not only live between pairs of settings — the stock image, the probes and a read-only root filesystem each read as right, and what they collided through was a property of the image none of them names. So the question to ask of a combination is what the application does at *start*, not whether each half is defensible. - **A constraint that lives in documentation is invisible to the machinery.** Automation knows versions, update types and exit codes. It does not know that a particular step is a maintenance window — and the better the pipeline, the less likely anybody is standing in the path to notice. If a tool can decide it, automate on its exit code; if nothing can, write the obligation where the work is done. +- **Check the recovery path against the mechanism the fault runs through.** A repair nobody rehearses is the one most likely to be assumed — and a precedence rule documented for one value is usually a property of the whole system, not a special case. +- **A write barrier reaches as far as the helper that implements it.** "Read-only configuration" also catches the bootstrap writes that happen once, on a first install, on the path no test walks — including the one that creates the credential for the door you kept open on purpose. +- **Check that the platform can make the call before designing around it.** An HTTP probe is a GET; anything else is an `exec` probe, and an `exec` probe is a claim about which binaries the image still contains. +- **The strictest file mode is not the correct one.** A mounted secret is owned by `root`, `fsGroup` supplies only the group, and an application that treats `EACCES` like "absent" turns a permission mistake into a feature that quietly switched itself off. - **When the fix needs a privilege, ask whether two cheap steps can replace one privileged one.** Splitting a pre-build into a seeding container plus a build container kept the hardening baseline intact across the whole pod, where the shorter single-container version would have bought a documented deviation *and* a suppression that silences the container that actually serves traffic. diff --git a/_Sidebar.md b/_Sidebar.md index a09216f..f1e203b 100644 --- a/_Sidebar.md +++ b/_Sidebar.md @@ -21,6 +21,7 @@ **Charts** - [apache-tika](apache-tika) - [bentopdf](bentopdf) +- [bulwark](bulwark) - [calibre-web-automated](calibre-web-automated) - [convertx](convertx) - [cyberchef](cyberchef) diff --git a/bulwark.md b/bulwark.md new file mode 100644 index 0000000..d55fda9 --- /dev/null +++ b/bulwark.md @@ -0,0 +1,437 @@ +# bulwark + +Deploys [Bulwark Webmail](https://bulwarkmail.org/) — a JMAP webmail client (mail, calendar, contacts, files) — as the browser front end for a [stalwart](stalwart) mail server. + +**It is a client, and that is the first thing to internalise.** It holds no mail, has no database, no cache and no broker. The messages, the mailboxes and the accounts belong to the Stalwart release named by `bulwark.jmapServerUrl`; upstream puts it plainly: *"Stalwart is the mail server: it holds the messages, speaks SMTP, and owns the accounts."* The practical consequence runs through this whole page — **with the mail server completely gone, this release is `Ready`, green, and useless.** Nothing here can detect that, and no probe could. + +**The second thing to internalise is in [Traps](#the-volume-beats-the-chart-for-every-value-silently): the volume beats the chart.** Every value this chart sets is a *default* that the admin dashboard may already be overriding, silently, forever. That section is the reason this page exists as much as the values table is. Read it before you open `/admin`, not after. + +| | | +|---|---| +| Image | `ghcr.io/bulwarkmail/webmail`, AGPL-3.0-only, `node:24-alpine` with a Next.js standalone build. Upstream removes `npm` and `npx` from the runtime layer and runs `apk upgrade` in the build — the image is in good shape | +| Versions | not listed on purpose — read the current pair off `bulwark/Chart.yaml`, see [Chart index](Charts). Note that the registry publishes **every release twice**, `1.12.0` and `v1.12.0` for the same image; the chart pins the form **without** the `v`, and [Renovate](Renovate) keeps it in that family | +| Workloads | one StatefulSet, **1** replica — capped at 1, see [Storage](#storage-one-volume-and-why) | +| Storage | **one** PVC at `/app/data`, carrying all four of Bulwark's directories as subdirectories | +| Ingress | one, at the **root of a hostname**. A sub-path is impossible — see [Three options are compiled in](#three-options-are-compiled-into-the-image) | +| External dependencies | a Stalwart (or other JMAP) server, and an OIDC provider ([keycloak](keycloak)) while `bulwark.oauth.enabled` is true. **No database, no cache, no broker** | +| Secrets | 2 × `OnePasswordItem`, three fields in total — two of them as **files**, which is the better path [garage](garage) and [keycloak](keycloak) did not have | +| Network | ClusterIP Service on 3000 (`http`) plus a headless Service the StatefulSet needs for its `serviceName`. No NetworkPolicy | +| Config files | **none rendered.** No ConfigMap, so no `checksum/config` — see [the trap](#there-is-no-configmap-so-there-is-no-checksumconfig) | +| Install-test | `ci/bulwark/` — the cheapest in the repo: two dummy Secrets and a selector-less Service, no infrastructure at all | + +## Prerequisites + +- **A reachable JMAP endpoint.** In-cluster that is a Stalwart release's `http` Service; a published one is its HTTPS URL. It does **not** have to be reachable at install time — Bulwark connects per request — but the value must be set. +- **1Password operator** in the cluster plus **two** items — see [What the 1Password items must contain](#what-the-1password-items-must-contain). +- **An OIDC client, registered BEFORE the first install**, while `bulwark.oauth.enabled` is true. This is not a follow-up step: with `bulwark.oauth.only` (the default) the password form is hidden, so without a working client **the first sign-in is impossible**. And read [the redirect URI carries the UI language](#the-redirect-uri-carries-the-ui-language) before you register it — a single redirect URI is the likeliest reason a first sign-in fails. +- **One DNS name and a cert-manager cluster issuer.** +- **A StorageClass**, or the cluster's default. + +## Installation + +The registry URL and its credentials live in 1Password (item `jlab-registry-access`: `username`, `password`, `charts-url`) and never in a repository or a shell history file. + +```bash +helm repo add jlab "$CHARTS_URL" --username "$REG_USER" --password "$REG_PASS" +helm repo update + +helm install bulwark jlab/bulwark \ + --namespace bulwark --create-namespace \ + -f my-values.yaml +``` + +Minimal `my-values.yaml`: + +```yaml +secrets: + appSecretRef: op://vault/bulwark-app + oauthSecretRef: op://vault/bulwark-oauth + +bulwark: + jmapServerUrl: http://stalwart-stalwart-full-http.stalwart.svc.cluster.local:8080 + oauth: + clientId: bulwark + issuerUrl: https://id.example.com/realms/main + +storage: + amount: 1Gi + +ingress: + domain: webmail.example.com + clusterIssuer: letsencrypt +``` + +Seven values, and six of them have no default because no default could be right. The chart **refuses to render** without `jmapServerUrl`, without the two OIDC values while OIDC is on, and without the vault references — each with a message naming the value and why. `helm install` then prints `NOTES.txt`, which is the per-release version of this page: the vault fields, the redirect URI to register, the first-install checklist and the way out of a lockout. + +### What the 1Password items must contain + +The chart renders two `OnePasswordItem`s; the operator turns each into a Kubernetes Secret whose **keys are the field names of the 1Password item**. + +| Values key | Kubernetes Secret | Field | Arrives as | Content | +|---|---|---|---|---| +| `secrets.appSecretRef` | `-bulwark-app-secret` | **`session-secret`** | **file**, `SESSION_SECRET_FILE` | 32 random bytes, base64: `openssl rand -base64 32`. **Mandatory** | +| | | **`admin-password`** | env var, `ADMIN_PASSWORD` | the admin dashboard's initial password. **Consumed exactly once** | +| `secrets.oauthSecretRef` | `-bulwark-oauth-secret` | **`oauth-client-secret`** | **file**, `OAUTH_CLIENT_SECRET_FILE` | the confidential client's secret, as issued by the provider | + +**Two items and not one, split by audience.** The first two fields are this release's own and nobody else knows them. The OIDC client secret is **not ours** — the identity provider creates it, rotates it, and also holds it — so a separate item lets it be repointed and rotated without touching the other two. Same reasoning as [garage](garage)'s split between its RPC secret and its S3 key. + +**A missing or misnamed field here does not stop the pod**, and that is worth knowing because it inverts the usual symptom. The two file-form fields are mounted from an *optional* volume (so that a first install does not hang in `ContainerCreating` while the operator catches up), so a typo yields a **missing file** and a perfectly healthy pod: settings sync switches itself off and every token exchange sends an empty secret. Only `admin-password` fails loudly — a wrong name there leaves the container in `CreateContainerConfigError`. + +#### Why `session-secret` is mandatory rather than recommended + +Two things hang on it, and one fails **silently** without it: + +- persistent "remember me" sessions, and +- the **encryption of the synchronized user settings** on the volume. + +`app/api/config/route.ts` computes `settingsSyncEnabled: configManager.get(...) && hasSessionSecret()`. So with `bulwark.settingsSync: true` (the default) and no secret, synchronization is simply **off**, with no error anywhere. + +#### `admin-password` is consumed exactly once + +`initAdminPassword` returns as soon as `admin.json` exists on the volume and never looks at the environment variable again; on the first start it hashes it with scrypt and persists the hash. **Rotating the vault field afterwards has no effect** — restart or not. A reset goes through the dashboard itself, or by deleting `/app/data/admin/admin.json` and restarting. + +And the reason it is set at all is **not** a secret leak. That was the assumption going in and it is wrong: without `ADMIN_PASSWORD` Bulwark generates **no** random password and writes **nothing** to stdout — `lib/admin/password.ts` logs `Admin dashboard disabled (no ADMIN_PASSWORD set)` and the dashboard stays off. The only random value in the project is the bootstrap setup token, which goes into the state directory and is never logged. So there is no `STALWART_RECOVERY_ADMIN` situation here ([stalwart](stalwart), Gitea #65). It is set because **the admin dashboard is the only way back out of an OIDC lockout** — switching it off means taking away the door. + +## Storage: one volume, and why + +One PVC at `/app/data`. Bulwark keeps four directories under it and creates all four itself on start (`ensureConfigDir` / `ensureStateDir`, both `mkdir -p`) — it has to, because the image's own `mkdir` ran in a layer the mount covers. + +| Directory | Holds | Must survive | +|---|---|---| +| `/app/data/admin` | `config.json`, `policy.json`, `admin.json` (the password hash), `plugin-config/`, `plugins/`, `themes/`, branding uploads | **yes** | +| `/app/data/admin-state` | `admin-state.json` (login timestamps), **`audit.log`**, the bootstrap setup token | **yes** — the audit trail cannot be reconstructed | +| `/app/data/settings` | the encrypted per-account user settings | yes, while `settingsSync` is on | +| `/app/data/telemetry` | consent record and instance id | only to keep them stable across image upgrades | + +**One volume and not four, because a snapshot is atomic per volume and never across volumes** — the durability lesson [garage](garage) was built on. Split up, a backup could capture an audit log recording logins the configuration does not know about yet, or settings encrypted against a secret the configuration has since been repointed from. There is no argument on the other side: this is a few hundred kilobytes of JSON plus an append-only log. + +**What actually grows is `audit.log`**, and it grows with admin activity rather than with mail volume. The application does not rotate it. On a long-lived release that is the number to watch, and `1Gi` is generous for everything else. + +**One replica follows from the volume, not from the application.** `ReadWriteOnce` means two pods cannot both mount it, so the schema caps `bulwark.replicas` at 1. An RWX volume would not help either: nothing in Bulwark coordinates two processes over that directory, and the admin config is read into memory **once** at start, so two pods would hold two divergent views of it. The PVC name is never changed — a renamed PVC provisions a new, empty volume and orphans `admin.json` and the audit log. + +## First install: the checklist + +1. **Wait for the rollout.** + ```bash + kubectl -n rollout status statefulset/-bulwark-sfs + ``` + `Ready` means the HTTP server is accepting. It does **not** mean the mail server is reachable — nothing in this chart checks that. + +2. **Confirm the setup wizard really is off.** + ```bash + kubectl -n logs statefulset/-bulwark-sfs | grep -i "SETUP REQUIRED" + ``` + **Nothing found is the correct result.** A `SETUP REQUIRED` banner with a token means Bulwark is in bootstrap mode — see [the wizard is the thing to keep switched off](#the-wizard-is-the-thing-to-keep-switched-off). + +3. **Sign in** through the identity provider at `https://`. + +4. **Read [the volume beats the chart](#the-volume-beats-the-chart-for-every-value-silently) before you open `/admin`.** This is the step people skip and then spend an afternoon on. + +5. **Know that the admin password is already spent** (above). If you intend a different one, change it in the dashboard now rather than in the vault later. + +6. **Set the settings policy you want, including the push relay**, *before* you ever consider `bulwark.adminConfigReadOnly` — that value freezes the policy permanently. + +## If nobody can sign in: the way back + +Read this now, not when it happens, because **the obvious repair does not work.** + +The situation: the identity provider is down or misconfigured, `bulwark.oauth.only` hides the password form, and no user can get in. + +### What does not work + +`helm upgrade` with `bulwark.oauth.only: false`. + +`oauthOnly` is an ordinary key of `CONFIG_ENV_MAP`, and **every** consumer reads it through `configManager.get` — `app/api/auth/session/route.ts`, `app/api/auth/verify/route.ts`, `app/api/config/route.ts`; not one of them reads `process.env`. The admin dashboard's own OAuth setup writes `oauthOnly: true` into `config.json` when it is used. So once that override is on the volume, the chart's value is **ignored**, the upgrade reports success, and nothing says why. + +This is the precedence trap sitting inside the rescue instruction itself, which is why it is called out here rather than in a footnote. + +### What does work + +Neither step depends on the identity provider. + +1. **Open `https:///admin/login`** and sign in with the `admin-password` field from the vault item. This door has its own password and is not behind OIDC. +2. **Remove the override** — the dashboard's revert, i.e. `DELETE` on `/api/admin/config`, which calls `configManager.removeAdminOverride`. After that the chart's value applies again. + +With no Ingress, reach the dashboard with `kubectl -n port-forward statefulset/-bulwark-sfs 3000:3000`. + +### If the dashboard itself is unreachable + +The last resort is the volume: + +```bash +kubectl -n exec -bulwark-sfs-0 -- cat /app/data/admin/config.json +# remove the "oauthOnly" key (and any other override in the way), then +kubectl -n rollout restart statefulset/-bulwark-sfs +``` + +Deleting `admin.json` in the same directory resets the admin password to whatever the vault field currently holds, on the next start. + +## Configurable values + +### Global + +| Value | Default | Notes | +|---|---|---| +| `scaleDown` | `false` | `true` renders the StatefulSet with `replicas: 0` and overrides `bulwark.replicas`. PVC, Secrets, Service and Ingress stay | + +### `secrets` + +| Value | Default | Notes | +|---|---|---| +| `secrets.appSecretRef` | `null` | **required.** `op://` path; fields `session-secret`, `admin-password` | +| `secrets.oauthSecretRef` | `null` | **required while `bulwark.oauth.enabled`.** `op://` path; field `oauth-client-secret` | + +### `bulwark` + +| Value | Default | Notes | +|---|---|---| +| `bulwark.replicas` | `1` | `0` or `1` only; the schema caps it. See [Storage](#storage-one-volume-and-why) | +| `bulwark.port` | `3000` | What the image binds. The Service and all three probes address the port **by name** (`http`), so the number is free and the name is not | +| `bulwark.jmapServerUrl` | `null` | **required.** The mail server's address **and** the switch that disables the setup wizard | +| `bulwark.appName` | `null` | `APP_NAME`; `null` leaves upstream's "Webmail" | +| `bulwark.stalwartFeatures` | `true` | Stalwart-specific UI: password change, Sieve rules. Does **not** gate the server-side credential passthrough, which has its own upstream variable this chart does not render | +| `bulwark.stalwartAdminAccess` | `auto` | `auto` / `password` / `off`. Who may open the admin dashboard — read [the note below](#who-can-open-the-admin-dashboard-and-therefore-write-overrides) | +| `bulwark.settingsSync` | `true` | User preferences follow the user. Puts **user data** on the volume, encrypted with the session secret | +| `bulwark.adminConfigReadOnly` | `false` | Makes the chart authoritative. **Three-part price — see [its own section](#adminconfigreadonly-makes-the-chart-authoritative-and-costs-more-than-it-looks)** | +| `bulwark.telemetry` | `false` | `BULWARK_TELEMETRY`. Upstream's default is already off; what this buys is the **lock** (see below) | +| `bulwark.updateCheck` | `false` | `BULWARK_UPDATE_CHECK`. Unlike telemetry this is **on** unless switched off, so the value does change behaviour | +| `bulwark.logLevel` | `info` | `error` / `warn` / `info` / `debug` | +| `bulwark.logFormat` | `json` | Against upstream's `text`, because the logs go to a collector | +| `bulwark.oauth.enabled` | `true` | Gates the OIDC values, the second vault item and the secret file | +| `bulwark.oauth.only` | `true` | Hides the password form. **The riskiest default in this chart** — see [the way back](#if-nobody-can-sign-in-the-way-back) | +| `bulwark.oauth.clientId` | `null` | **required while enabled**; the render fails without it | +| `bulwark.oauth.issuerUrl` | `null` | **required while enabled.** For keycloak: `https:///realms/` | +| `bulwark.oauth.scopes` | `null` | **Replaces** the scope list rather than adding to it. `null` keeps upstream's. Use `OAUTH_EXTRA_SCOPES` through `bulwark.env` to add | +| `bulwark.oauth.endSession` | `true` | Signing out also ends the provider's session | +| `bulwark.writableCache.tmpSizeLimit` | `256Mi` | `sizeLimit` of the `/tmp` emptyDir | +| `bulwark.writableCache.nextCacheSizeLimit` | `512Mi` | `sizeLimit` of the `/app/.next/cache` emptyDir | +| `bulwark.resources.limitFactor` | `3` | Memory and ephemeral-storage limits = factor × request; no CPU limit unless set | +| `bulwark.resources.requests` | `100m` / `256Mi` / `64Mi` | The memory **limit** is what V8's heap limit derives from — see [the probe trap](#the-probes-say-nothing-about-the-mail-server) | +| `bulwark.livenessProbe` | GET `/api/health` | 10s / 5s / 3 | +| `bulwark.readinessProbe` | GET `/api/health` | 10s / 5s / 3. **Carries no statement about Stalwart** | +| `bulwark.startupProbe` | GET `/api/health` | 5s / 5s / 24 → a 120s budget. Gates the other two | +| `bulwark.securityContext.pod` | uid/gid/fsGroup **1001** | The image's own user (`adduser --uid 1001 nextjs`); `fsGroupChangePolicy: Always` | +| `bulwark.securityContext.container` | hardened | `readOnlyRootFilesystem: true`, all caps dropped, no privilege escalation | +| `bulwark.env` | `[]` | Additive. **And only the middle of three layers** — see the trap below | + +### `storage` + +| Value | Default | Notes | +|---|---|---| +| `storage.amount` | `1Gi` | Deliberately **not** null-safe: an erased size yields an invalid PVC, which fails loudly | +| `storage.storageClass` | `null` | `null` uses the cluster default. `ReadWriteOnce` | + +### `ingress` + +| Value | Default | Notes | +|---|---|---| +| `ingress.enabled` | `true` | Gating it off frees **both** `domain` and `clusterIssuer` — checked in both directions; nothing outside the Ingress template functionally reads the domain | +| `ingress.domain` | `null` | Required while enabled. Bulwark is served at the **root** of it | +| `ingress.clusterIssuer` | `null` | Required while enabled | +| `ingress.className` | `nginx` | | +| `ingress.annotations` | `{}` | Merged with the cert-manager annotation. **Consider `proxy-body-size`** — see the trap | + +## Traps + +### The volume beats the chart, for every value, silently + +**The single most important thing on this page.** `lib/admin/config-manager.ts` resolves a config value in this order: + +``` +admin override (config.json on the PVC) > environment variable > default +``` + +Every setting this chart renders is therefore a **default** that the admin dashboard may already be overriding. The dashboard may write **any** key of `CONFIG_ENV_MAP` (`app/api/admin/config/route.ts` validates against exactly that key list), and nothing in the UI says that it has just disconnected the chart. A `helm upgrade` with a changed value then changes nothing, reports success, and leaves no trace of why. + +Upstream documents this for the server URL only — *"A server URL saved by the wizard takes priority over the environment variable"* — and that wording is the trap, because it reads like a special case. It is the behaviour of **every** key. + +**Exactly three values escape it**, all verified: + +| Value | Why it escapes | +|---|---| +| `SESSION_SECRET` / `SESSION_SECRET_FILE` | `getSessionSecret()` reads the environment **first** and the admin override last — the reverse of the normal order | +| `ADMIN_PASSWORD` | `initAdminPassword()` reads the environment only (and only once) | +| `BULWARK_TELEMETRY` | `effectiveConsent()` takes the environment over the state file, and upstream **blocks the admin UI** from changing it while the override is set | + +**The OAuth client secret is not one of them**, which is the one most likely to be assumed: `getGlobalClientSecret()` checks the admin override *first*. If the dashboard's OAuth setup has ever run, `config.json` carries `oauthClientSecret` in plaintext and that value beats the mounted file forever. + +To see what is overridden right now: + +```bash +kubectl -n exec -bulwark-sfs-0 -- cat /app/data/admin/config.json +``` + +An empty object, or no file, means the chart is in force. + +### `adminConfigReadOnly` makes the chart authoritative, and costs more than it looks + +`bulwark.adminConfigReadOnly: true` sets `ADMIN_CONFIG_READONLY`, which refuses every write to the admin **config** directory at the application layer. No override can appear, so the chart's values always apply and the trap above is closed for good. + +**It defaults to `false` because the price is threefold, and the third part was measured rather than assumed:** + +1. **Policy, themes and branding freeze too.** `assertWritable` (`lib/admin/paths.ts`) guards `setPolicy`, `markSetupComplete`, the plugin and theme registries and the branding upload — not just the config override. So the settings policy becomes unchangeable, and with it **the only way to point Web Push at your own relay**. +2. **An override that already exists is frozen in place, not removed.** `removeAdminOverride` is guarded by the same check. Turning this on while `config.json` already carries `oauthOnly: true` makes that lockout **permanent** and removes the documented repair in the same move. +3. **On a fresh install it takes the admin dashboard away entirely** — i.e. exactly the door this chart keeps a password for. `initAdminPassword` persists the hash through `writeConfigData`, which begins with `assertWritable('save admin password')`. With no `admin.json` on the volume yet, that throws: the startup hook logs `Admin dashboard init skipped`, the retry inside `/api/admin/auth` throws again, and **the login endpoint answers HTTP 500**. The pod is `Ready` and serving mail the whole time, so nothing reports it. + +**The only safe sequence**, which is also upstream's own framing (*"Operators set `ADMIN_CONFIG_READONLY=true` after running the setup wizard"*): + +1. install with it `false` and let the pod start once, so `admin.json` exists; +2. set every policy you want, **push relay included** — after step 4 you cannot; +3. check `config.json` for an override you still need removed; +4. then `helm upgrade` with it `true`. + +The chart cannot verify step 1 for you: it has no way to know what is on the volume. + +### Who can open the admin dashboard, and therefore write overrides + +`bulwark.stalwartAdminAccess` keeps upstream's default `auto`, and the consequence is stated rather than left to be found: with `auto`, **anybody who is an admin on the Stalwart side is signed into `/admin` without the Bulwark admin password**. Since the dashboard is what writes the overrides from the trap above — and what can lock the login path — that is a wider door than it looks. + +It is kept because it is usually the same person. If it is not: `password` narrows it to whoever holds the vault field, and `off` also removes the shield from the UI. Both require an admin password to be configured, which this chart always does. + +### The redirect URI carries the UI language + +This is the likeliest reason a first sign-in fails, and it is invisible from the address bar. + +Bulwark builds its OIDC redirect URI as + +``` +https:////auth/callback +``` + +where `` is the language **the visitor's browser resolved to**. A German browser sends `/de/auth/callback`, an English one `/en/auth/callback` — from the same deployment. The locale prefix mode in the published image is `never`, so the language is **not** in the visible URL, which is exactly why this surprises people: the address bar shows no language and the `redirect_uri` contains one. Upstream's own admin helper registers one URI per locale for this reason. + +Register one per language you expect, or a wildcard where the provider allows it. On Keycloak the wildcard is the practical answer: + +``` +https:///* +``` + +The 27 supported locales are `ar ca cs da de en es fa fr he hu it ja ko lv mn nb nl pl pt ro ru sk tr uk zh zh-TW`. + +If a sign-in fails with a `redirect_uri` mismatch, read the URI out of the failed request rather than guessing which language it was. + +### The wizard is the thing to keep switched off + +Bulwark has a first-run setup wizard that *"chooses OAuth or password sign-in, generates a session secret and sets the admin password"* and writes the result into the config directory. `bulwark.jmapServerUrl` disables it: `detectSetupState()` returns `env-managed` as soon as the variable is non-empty, and in that state `/setup` is not served at all. + +**The chart refuses to render without that value**, and the reason is sharper than "the mail server is unset": an empty `JMAP_SERVER_URL` puts the app into `bootstrap`, serves the wizard, and whatever is clicked through there lands in `config.json` — from which moment the volume beats every value the chart sets, permanently. An unset value does not produce a broken release, it produces a **silently different** one. + +### The probes say nothing about the mail server + +All three probes are a plain HTTP **GET** on `/api/health`. + +**Not `HEAD`, although Bulwark offers one and it would be cheaper**: a Kubernetes `HTTPGetAction` has no method field and the kubelet always issues a GET. The only alternative would be an exec probe calling an HTTP client, and the image contains neither `curl` nor `wget` (`npm` and `npx` are removed from the runtime layer), so it would mean starting a Node process every few seconds. The GET handler does gather diagnostics, which makes it marginally **more** capable than HEAD, not less. + +**What the probe really carries:** the Node process is alive, the HTTP server accepts, the event loop is not blocked, Next.js can route. An OOM kill, a crash, a wedged event loop and a container that never finished booting all fail it. That is genuinely more than [urlaubsverwaltung](urlaubsverwaltung)'s endpoint carried (#218). + +**What it does not carry:** + +- **The reported state.** The success path returns a hard `200` whatever it computed, and for an unauthenticated caller the body is hard-coded to `healthy` (`status: detailed ? status : 'healthy'`). `degraded` appears only with `?detailed=true` **and** a caller passing `canSeeInstanceDetails`. Upstream tests exactly that: `lib/__tests__/health-route.test.ts` expects HTTP 200 while `payload.status` is `degraded`. The only `503` this endpoint can produce comes from its catch block, i.e. when `process.memoryUsage()` itself throws. +- **Anything about Stalwart.** Bulwark holds no mail and this endpoint asks the JMAP server nothing. **With the mail server completely gone, the probe is green and the pod stays `Ready`.** Monitor the [stalwart](stalwart) release for that; nothing here can. + +There is no better endpoint: `/api/setup/test-jmap` would really test the connection, but it sits behind the bootstrap setup token, which this chart disables. + +A consequence of the first point worth pairing with the resources: `/api/health` reports `degraded` on high V8 heap usage, and the heap limit derives from the **container memory limit**. So a limit set too low produces a pod that is permanently degraded, permanently `Ready`, and says so only to an authenticated caller asking for details. + +### Backups: the volume alone is not one, and it may need protecting itself + +Two statements, both about `-bulwark-pvc`, and they pull in opposite directions: + +1. **A backup of the volume alone is not a backup.** The synchronized user settings on it are encrypted with `session-secret`. Lose the vault item and that data is unreadable out of a perfectly intact volume. **The 1Password item belongs in the backup just as much as the volume does.** +2. **A backup of the volume may contain a secret in cleartext.** `app/api/admin/oauth/setup/route.ts` writes `oauthClientSecret` as **plaintext** into `config.json` on the PVC. So if anybody has used the dashboard's OAuth setup, the volume backup may need the same protection the vault gets. + +Check for the second with: + +```bash +kubectl -n exec -bulwark-sfs-0 -- grep -l oauthClientSecret /app/data/admin/config.json +``` + +### Secret rotation works without a restart — because of one mount detail + +Both file-form secrets are mounted from a **projected volume without `subPath`**, and that is load-bearing rather than stylistic. + +`getSessionSecret()` and `getGlobalClientSecret()` both call `readFileEnv` on **every use**, with no cache. So a rotated field in the vault is live as soon as the operator has written the Secret — **no pod restart**. The kubelet never refreshes a `subPath` mount, so a `subPath` here would silently freeze both secrets at their first value while the vault, the Secret and `helm upgrade` all looked correct. + +The mode is `0440`, not `0400`: files in a secret volume are owned by **root**, and what gives uid 1001 access is `fsGroup`, which only supplies the group. Without a group read bit the files would be unreadable — and silently so, since `readFileEnv` answers an `EACCES` exactly as it answers a missing file. + +Two exceptions to "rotation just works": + +- **`admin-password`** is consumed once and no restart helps (above). +- **The OAuth client secret** is beaten by an admin override in `config.json` if one exists (above). + +**[garage](garage)'s file-permission trap does not apply here**, and it was checked rather than assumed, because it is what forced garage onto an environment variable: `lib/read-file-env.ts` is `readFileSync` plus `try/catch` and nothing else — no `statSync`, no `mode & 0o077`. Bulwark does not *reject* a group-readable secret file, which is what makes the file form usable at all. + +### There is no ConfigMap, so there is no `checksum/config` + +Not an omission. Bulwark is configured through environment variables plus the two JSON files it writes itself onto the PVC, so there is no rendered config file whose change could fail to restart the workload ([style guide §9](Chart-Style-Guide#9-config-changes-must-restart-the-workload)). `values.yaml` states this explicitly so it does not read as a gap. + +What **is** covered is the `op://` **paths**, through `checksum/secret-ref`: repointing `secrets.appSecretRef` or `secrets.oauthSecretRef` at a different item rolls the pod. A change *inside* an item does not — Helm never sees it — but for the two file-form secrets it does not need to (above). + +The 1Password operator's auto-restart annotation is **not** set and must not be: against operator 1.12.0 it enumerates Deployments only, so on a StatefulSet it is a silent no-op advertising a guarantee it does not provide. + +### Web Push goes through the vendor's relay until you change it — and that is not a chart value + +The obvious reading of the build-time variable is wrong, so this is stated carefully. + +`NEXT_PUBLIC_PUSH_RELAY_URL` is compiled into the bundle, so the chart cannot move the **built-in** entry, which is `https://notifications.relay.bulwarkmail.org`. It would be wrong to stop there and call Web Push "not changeable with the published image": that variable only sets the **fallback**. `lib/push-relays.ts` resolves the effective relay as `resolveDefaultRelayUrl(policy)`, which takes `policy.pushRelayUrl` whenever it is a valid URL; with `pushRelayUrlLocked: true` every user is pinned to it. + +So, two-part: + +- **Without further action**, Web Push for this release goes through the vendor's relay. For a mail client that is worth knowing, and it stays in the users' relay list either way, because the list is built from the bundle. +- **A self-hosted relay can be made the effective default and even mandatory** — through `policy.json`, i.e. the **admin dashboard**, not a chart value. There is deliberately no value for it: `pushRelayUrl` lives in `SettingsPolicy` and has no entry in `CONFIG_ENV_MAP`, so no environment variable reaches it and a chart value would be a switch wired to nothing. + +**And the interaction:** `adminConfigReadOnly` freezes `setPolicy`, so turning that on before setting the relay makes this change impossible afterwards. + +### Three options are compiled into the image + +Upstream: *"These are compiled into the bundle and cannot be changed at runtime … To change build-time variables with the published Docker image, rebuild with matching `--build-arg`."* Building our own image is outside what this repository does — the same limit as [garage](garage) and [keycloak](keycloak). + +| Variable | Consequence | +|---|---| +| `NEXT_PUBLIC_BASE_PATH` | Bulwark is served only at the **root of a hostname**. No sub-path, and a `rewrite-target` annotation does not help — the bundle's own asset URLs are built for the root | +| `NEXT_PUBLIC_DEFAULT_LOCALE` | The fallback UI language stays `en`. A visitor with a matching `Accept-Language` gets their own; everybody else gets English | +| `NEXT_PUBLIC_PUSH_RELAY_URL` | Only the built-in **default** entry — see the section above, which is the part that is *not* a limitation | + +### Attachment uploads need `proxy-body-size` + +File attachments and the Files view upload through the Ingress, and nginx's default `proxy-body-size` of `1m` rejects anything larger with a **413 that reads like an application fault**. The chart does not set it, because the right number is the mail server's limit and the chart does not know it: + +```yaml +ingress: + annotations: + nginx.ingress.kubernetes.io/proxy-body-size: "25m" +``` + +### `bulwark.env` is additive, and only the middle layer + +Additive means additive: an entry repeating a name the chart already sets does not replace it. At install both reach the pod and the later one wins; on an **existing** release the pair collapses to the chart's own value, because Kubernetes merges `env` by name — and `helm upgrade` still reports success ([style guide §10](Chart-Style-Guide#10-environment-variables)). + +And the sharper caveat specific to this application: an environment variable is only the **middle** of three layers. Anything set here is a default an admin override may already be beating. Genuinely useful entries, all `CONFIG_ENV_MAP` keys the chart does not render: `APP_DESCRIPTION`, the `LOGIN_*` branding variables, `COOKIE_SAME_SITE`, `STALWART_JMAP_PASSTHROUGH_ENABLED`, `OAUTH_EXTRA_SCOPES`, `WOPI_CLIENT_URL`. + +## Upgrades and backup + +- **A plain app-version bump** is a chart patch and needs nothing special: no database, no migration, no schema. Renovate opens it and `ci/renovate-bump-chart.sh` bumps the chart version. +- **Back up the PVC and the `appSecretRef` vault item together.** Neither is sufficient alone — see [Backups](#backups-the-volume-alone-is-not-one-and-it-may-need-protecting-itself). +- **Before any upgrade worth the caution**, read `config.json` off the volume. It tells you which of the chart's values are actually in force. +- `scaleDown: true` stops the release without uninstalling it. + +## CI + +`ci/bulwark/` is the cheapest install-test in the repo: two dummy Secrets and a **selector-less Service** for the JMAP endpoint — a name that resolves and then refuses the connection. No database, no cache, no mail server, and deliberately no stub workload (a fixture Deployment has to roll out before the chart installs, so it would be one more thing able to fail a run that is not about it). + +**What a green run proves:** `readOnlyRootFilesystem: true` holds with exactly the two emptyDirs the chart renders (`/tmp` and `/app/.next/cache`) — the set was *derived* from `next.config.ts` and can only be *proven* by starting the real image; the probes can really be satisfied, i.e. `/api/health` exists in the built bundle and answers 200 unauthenticated; uid/gid 1001 plus `fsGroup` let Bulwark create its four subdirectories in a volume it did not make; and `ADMIN_PASSWORD` is really hashed into `admin.json` inside the startup budget. + +**What it does not prove**, and the list matters: + +- **anything about JMAP or OIDC** — nothing answers either address, which is precisely the limitation the readiness probe cannot see. The test *demonstrates* that limitation rather than closing it. +- **the precedence trap.** `config.json` never exists in a CI run, so the chart's single most important operating caveat is not exercised. It cannot be: it needs an admin with a browser. +- **that the two file-form secret keys are spelled right.** The projected volume is `optional: true` by design, so a misspelled key yields a missing file and a healthy pod. Only `admin-password` is proven. +- **that no emptyDir is superfluous.** A missing writable path fails loudly; an unnecessary one does not. That half stays open. + +## See also + +- [stalwart](stalwart) — the mail server this is a client for. The two are only useful together. +- [keycloak](keycloak) — the identity provider behind `bulwark.oauth.*`. +- [garage](garage) — where the one-volume-per-snapshot rule and the secret-file permission trap come from. +- [Lessons learned](Lessons-Learned) — the rules this chart bought. +- [Chart style guide](Chart-Style-Guide) · [Chart index](Charts) diff --git a/stalwart.md b/stalwart.md index bd81917..811bcaf 100644 --- a/stalwart.md +++ b/stalwart.md @@ -519,5 +519,6 @@ What it does prove is the chart's own responsibility, and the first item is why - [Chart style guide](Chart-Style-Guide) — the conventions this chart follows and the two it deviates from - [Lessons learned](Lessons-Learned) — the source-IP mechanism behind NodePort, which recurs in every chart whose application evaluates the client address +- [bulwark](bulwark) — a **webmail client for this server**: mail, calendar, contacts and files in a browser, over JMAP. A separate release pointed at this one's JMAP endpoint, and the two are only useful together. Two things worth carrying across before installing it: it is a pure client, so **nothing in either chart notices the other being away** — with this server gone, Bulwark is `Ready` and green — and its own configuration can be overridden from its volume, silently, which is the first thing its page explains - [valkey](valkey) — the chart this one takes as a subchart, and the one to install separately when it should outlive this release - [Chart index](Charts)