# Bulwark Webmail - a JMAP webmail client (mail, calendar, contacts, files) for # a Stalwart mail server. It is a CLIENT: it holds no mail, has no database, no # cache and no broker. The messages, the accounts and SMTP are stalwart's # (Gitea #65), reached over the network by bulwark.jmapServerUrl. # # READ THIS FIRST. Four properties shape every value below, and not one of them # is visible from the outside (Gitea #89). Everything here was measured against # the source at tag 1.12.0, not taken from the documentation - the documentation # was wrong about two of them. # # 1. THE VOLUME BEATS THE CHART, FOR EVERY VALUE, SILENTLY. # This is the single most important sentence in this file. # lib/admin/config-manager.ts resolves a config value in this order: # # admin override (config.json on the PVC) > environment variable > default # # Every setting this chart renders as an environment variable is therefore a # DEFAULT that the admin dashboard may already be overriding. The override is # a key in config.json on the PVC, the dashboard may write ANY key of # CONFIG_ENV_MAP (app/api/admin/config/route.ts validates against exactly # that key list), and nothing in the dashboard says that it has just # disconnected the chart. A `helm upgrade` with a changed value then changes # nothing, reports success, and leaves no trace of why. # # The documentation names this only for the server URL ("A server URL saved # by the wizard takes priority over the environment variable"). It is not # special to the server URL. It is the behaviour of every key. # # THERE ARE EXACTLY THREE EXCEPTIONS, all verified, and they are the only # values in this chart that the volume cannot silently beat: # - SESSION_SECRET / SESSION_SECRET_FILE: getSessionSecret() # (lib/auth/session-secret.ts) reads the environment FIRST and the admin # override last - the reverse of the normal order. # - ADMIN_PASSWORD: initAdminPassword() reads the environment only. (It is # still consumed exactly once - see secrets.appSecretRef.) # - BULWARK_TELEMETRY: effectiveConsent() takes the environment over the # state file, and upstream blocks the admin UI from changing it while the # override is set. # The OAuth client secret is NOT an exception, which is the one most likely to # be assumed - see secrets.oauthSecretRef. # # The way out, if you want the chart to be authoritative, is # bulwark.adminConfigReadOnly - and it has a price that is bigger than it # looks. Read it there before setting it. # # 2. THERE IS NO ConfigMap IN THIS CHART, AND THAT IS NOT AN OMISSION. # Bulwark is configured entirely through environment variables plus the two # JSON files it writes itself onto the PVC (config.json, policy.json). No # config file is rendered, mounted or hashed here, so the repo's # checksum/config rule (#82) has nothing to attach to: there is no ConfigMap # whose change could fail to restart the workload. What IS covered is the # op:// PATHS of the vault items, through checksum/secret-ref on the pod # template, so repointing a secrets.* value really does roll the pod. A # change INSIDE an item is not covered and cannot be - Helm never sees it - # and for the two FILE-form secrets it does not need to be: both are re-read # on every use, so a rotation takes effect without a restart (point 3). # # 3. THE TWO FILE-FORM SECRETS ARE MOUNTED WITHOUT subPath, ON PURPOSE. # SESSION_SECRET_FILE and OAUTH_CLIENT_SECRET_FILE both exist (upstream calls # them the file form of SESSION_SECRET and OAUTH_CLIENT_SECRET), and taking # them keeps both values out of the container's process environment, which is # what garage (#81) and keycloak (#83) could not do because neither # application offered a file form at all. Here the file form is also # ROTATABLE: readFileEnv() is called on every use with no cache, so a changed # field in the vault item takes effect without a pod restart - but only # because the Secret is mounted as a whole directory. The kubelet never # refreshes a subPath mount, so a subPath here would silently freeze both # secrets at their first value while everything else looked correct. # ADMIN_PASSWORD has NO file form and stays an environment variable. # # 4. THREE OPTIONS ARE BAKED IN AT BUILD TIME AND THIS CHART CANNOT MOVE THEM. # Upstream: "These are compiled into the bundle and cannot be changed at # runtime ... To change build-time variables with the published Docker image, # rebuild with matching --build-arg." Building our own image is outside what # this repository does - the same limit as garage (#81) and keycloak (#83). # - NEXT_PUBLIC_BASE_PATH: so Bulwark is served at the ROOT of a hostname. # There is no sub-path deployment and no value here that could make one. # - NEXT_PUBLIC_DEFAULT_LOCALE: the fallback UI language stays "en". A # visitor with a matching Accept-Language gets their own language; # everybody else gets English. # - NEXT_PUBLIC_PUSH_RELAY_URL: see bulwark.pushRelay below. This one is # NOT the disclosure it looks like. # Scale the whole chart down without uninstalling it: scaleDown: true renders # the StatefulSet with replicas: 0 and takes precedence over bulwark.replicas. # The PVC, the Secrets, the Service and the Ingress stay in place; # scaleDown: false brings the workload back up. scaleDown: false secrets: # op:// path of the 1Password item carrying THIS RELEASE'S OWN secrets. # REQUIRED - the chart does not render without it. # # Must carry the fields: # session-secret -> mounted as a file, read via SESSION_SECRET_FILE # admin-password -> ADMIN_PASSWORD (environment variable, no file form) # # WHY session-secret IS MANDATORY AND NOT OPTIONAL. Two things hang on it, and # one of them fails silently without it: persistent "remember me" sessions, # and the encryption of the synchronized user settings. app/api/config/route.ts # computes `settingsSyncEnabled: configManager.get(...) && hasSessionSecret()` # - so with bulwark.settingsSync true and no secret, synchronization is # switched OFF with no error anywhere. Generate it with # openssl rand -base64 32 # # AND THE BACKUP STATEMENT, WHICH IS TWO-PART AND IS THE SHARPEST THING ON # THIS PAGE. It belongs wherever this release's backups are described: # # (a) A BACKUP OF THE VOLUME ALONE IS NOT A BACKUP. The synchronized # settings on the PVC are encrypted with session-secret. Lose the vault # item and they are unreadable out of a perfectly intact volume. The # secret has to be in the backup just as much as the volume is. # (b) A BACKUP OF THE VOLUME MAY CONTAIN A SECRET IN CLEARTEXT. If anybody # has ever used the admin dashboard's OAuth setup, # app/api/admin/oauth/setup/route.ts has written oauthClientSecret as # PLAINTEXT into config.json on the PVC. So the volume backup is not # merely useless without the vault - it may itself need the protection # the vault gets. # # WHY admin-password IS SET AT ALL, and the reason is NOT the one the issue # originally gave. Measured: without ADMIN_PASSWORD, Bulwark does NOT generate # a random password and writes NOTHING to stdout - lib/admin/password.ts # simply logs "Admin dashboard disabled (no ADMIN_PASSWORD set)" and the # dashboard stays off. (The only random value in the project is the bootstrap # setup token, which is written into the state directory and never logged.) So # there is no leak into the pod log here and the analogy to stalwart's # STALWART_RECOVERY_ADMIN (#65) does not hold. # # The reason it is set is point 1 at the top of this file combined with # bulwark.oauth.only: THE ADMIN DASHBOARD IS THE ONLY WAY BACK OUT OF AN # OIDC LOCKOUT. Switching it off means taking away the door. # # THE FIELD IS CONSUMED EXACTLY ONCE, which is the part that surprises people # later: initAdminPassword() returns as soon as admin.json exists on the PVC # and never looks at the variable again; on the first start it hashes it with # scrypt and persists the hash. ROTATING THE VAULT FIELD AFTERWARDS HAS NO # EFFECT. A reset goes through the dashboard itself, or by deleting admin.json # from the PVC. Same cause as point 1: the volume beats the chart. appSecretRef: null # op:// path of the 1Password item carrying the OIDC CLIENT SECRET. # Required while bulwark.oauth.enabled is true (the default). # # Must carry the field: # oauth-client-secret -> mounted as a file, read via OAUTH_CLIENT_SECRET_FILE # # A SEPARATE ITEM FROM appSecretRef, BY AUDIENCE: this value is not ours, it # is created by the identity provider when the client is registered (keycloak, # #83), it is rotated there, and it is the one of the three that another # system also knows. Two items mean it can be repointed and rotated without # touching this release's own secrets. Same reasoning as garage's split # between its RPC secret and its S3 key. # # ROTATION NEEDS NO RESTART - with one condition the chart satisfies. # getGlobalClientSecret() (lib/oauth/token-exchange.ts) calls readFileEnv on # EVERY token exchange with no cache, so a new value in the vault is live as # soon as the operator has written it. The condition is the whole-directory # mount of point 3; a subPath mount would never be refreshed. # # AND THE ONE WAY A ROTATION STILL DOES NOTHING, which is point 1 again and is # why this is not filed under "rotation just works": getGlobalClientSecret # checks the ADMIN OVERRIDE FIRST. If the admin dashboard's OAuth setup has # ever run, config.json on the PVC carries oauthClientSecret in plaintext and # that value wins over this file - forever, and with no error. The repair is # the same as for every other override: remove it in the dashboard (see # bulwark.oauth.only). # # #81 DOES NOT APPLY HERE, and it was checked rather than assumed, because it # is exactly what forced garage onto an environment variable: lib/read-file-env.ts # is readFileSync plus try/catch and nothing else - no statSync, no # `mode & 0o077` permission check. The group-read bit the kubelet ORs onto # every file of a secret volume once fsGroup is set is therefore harmless, and # the file form is load-bearing rather than merely available. oauthSecretRef: null bulwark: # ONE, and the schema caps it there. The reason is this chart's single # ReadWriteOnce PVC, not anything about the application: two pods cannot both # mount it, so a second replica would sit in ContainerCreating forever. An RWX # volume would not help either - nothing in Bulwark coordinates two processes # over that directory, and the admin config is read into memory once at start # (configManager.load() in instrumentation.node.ts), so two pods would hold # two divergent views of it. # # An explicit 0 is allowed and is the per-workload way to stop the release; # the chart-global scaleDown flag does the same for everything at once. replicas: 1 # THE JMAP SERVER - the stalwart release this client talks to. REQUIRED, and # it does two things at once, which is why it has no default. # # (a) it is the address of the mail server. Without it there is no mail. # (b) IT SWITCHES OFF THE FIRST-RUN SETUP WIZARD. detectSetupState() # (lib/setup/state.ts) returns "env-managed" as soon as this variable is # non-empty, and in that state /setup is not served at all. That is what # makes this chart declarative instead of "install, then click through a # wizard whose result lands on a volume". # # An in-cluster stalwart release is reached at its http Service, e.g. # http://-stalwart--http..svc.cluster.local:8080 # A published one is reached at its public https URL. Either works - this is # server-to-server, the browser never sees it. # # THE TRAP, AND IT IS THE ONE THE UPSTREAM DOCUMENTATION ACTUALLY WARNS ABOUT: # a server URL saved in the admin dashboard WINS over this value (point 1 at # the top). If the dashboard has ever been used to set it, changing this value # and running `helm upgrade` moves nothing, succeeds, and says nothing. Read # the effective value back off the running app rather than off the chart: # kubectl -n exec -- cat /app/data/admin/config.json # An absent jmapServerUrl key in that file means this value is in force. jmapServerUrl: null # The product name in the UI and the PWA. null leaves upstream's "Webmail". appName: null # STALWART_FEATURES - the Stalwart-specific parts of the UI (password change, # Sieve rule management). True, which is upstream's default and is the point # of pairing this chart with stalwart at all. # # Note what it does NOT gate: the server-side credential-bearing passthrough # at /api/account/stalwart/jmap has its own switch upstream # (STALWART_JMAP_PASSTHROUGH_ENABLED, default true). This chart does not render # it; an installation that wants the UI features but not the passthrough sets # it through bulwark.env and should read the precedence note on that value # first. stalwartFeatures: true # WHO MAY OPEN THE ADMIN DASHBOARD, and this value matters more here than its # name suggests, because after bulwark.oauth.only the dashboard is the ONLY # door back into a broken login path. # # Upstream's three modes (STALWART_ADMIN_ACCESS): # auto a Stalwart admin sees the shield and is signed into /admin # WITHOUT the Bulwark admin password. Upstream's default and the # legacy behaviour. # password a Stalwart admin sees the shield but must enter the Bulwark # admin password like everybody else. # off Stalwart admin status is ignored; /admin is reachable only via # /admin/login with the Bulwark admin password. # # "auto", i.e. upstream's default, is kept DELIBERATELY and the consequence is # stated rather than left to be discovered: with it, anybody who is an admin # on the stalwart side can write config overrides here - and an override is # exactly the thing that can disconnect this chart from its own values # (point 1) or lock the login path (bulwark.oauth.only). That is usually the # same person, which is why it is not changed; if it is not, "password" # narrows the door to whoever holds the vault item, and "off" also removes the # shield from the UI. Both require admin-password to be configured - which # this chart always does - or the dashboard would become unreachable. stalwartAdminAccess: auto # SETTINGS_SYNC_ENABLED - per-account user preferences follow the user between # browsers instead of living in one browser's local storage. # # ON. The consequence is that /app/data/settings on the PVC carries USER DATA, # encrypted with the session secret - which is where the two-part backup # statement at secrets.appSecretRef comes from. It also means the secret is # mandatory rather than recommended: without it this switch is ignored, # silently. settingsSync: true # ADMIN_CONFIG_READONLY - refuse every write to the admin CONFIG directory at # the application layer, which makes the values in this file authoritative and # closes the precedence trap of point 1 for good: with no override able to # appear, bulwark.oauth.only (and every other value here) always takes effect. # # OFF BY DEFAULT, AND THE PRICE IS WHY. It is not one cost but three, and the # third was measured rather than inferred - it is bigger than the plan for # this chart assumed: # # (a) POLICY, THEMES AND BRANDING FREEZE TOO. assertWritable # (lib/admin/paths.ts) guards setPolicy, markSetupComplete, the plugin # and theme registries and the branding upload - not just the config # override. So the settings policy becomes unchangeable, and with it the # ONLY way to point Web Push at your own relay (see bulwark.pushRelay). # (b) AN OVERRIDE THAT ALREADY EXISTS IS FROZEN IN PLACE, not removed. # removeAdminOverride is guarded by the same assertWritable. Turning this # on while config.json already carries `oauthOnly: true` therefore makes # that lockout permanent and takes away the documented repair at the same # time. If you are turning this on, read config.json first. # (c) ON A FRESH INSTALL IT TAKES THE ADMIN DASHBOARD AWAY ENTIRELY - i.e. # exactly the door this chart keeps a password for. initAdminPassword # persists the hashed ADMIN_PASSWORD by calling writeConfigData, which # begins with assertWritable('save admin password'). With no admin.json # on the volume yet, that throws: the startup hook logs "Admin dashboard # init skipped", the retry inside /api/admin/auth throws again, and the # login endpoint answers HTTP 500. The pod is Ready and serving mail the # whole time, so nothing reports it. # # THE ONLY SAFE SEQUENCE, which is also upstream's own framing ("Operators set # ADMIN_CONFIG_READONLY=true after running the setup wizard"): # 1. install with this false, let the pod start once, so admin.json exists # 2. set every policy you want (push relay included) - after step 3 you # cannot # 3. check that config.json carries no override you still need removed # 4. then helm upgrade with this true # The chart cannot verify step 1 for you: it has no way to know what is on the # volume. NOTES.txt repeats the sequence. adminConfigReadOnly: false oauth: # OIDC SIGN-IN against the identity provider (keycloak, #83). ON. enabled: true # OAUTH_ONLY - hide the password form entirely, so the identity provider is # the only way for a USER to sign in. ON, and this is the chart's riskiest # default, taken deliberately with the way back out written down. # # THE RISK: if the provider is down or the OIDC configuration is wrong, # nobody reaches their mail, and there is no password form to fall back to. # # THE WAY BACK, AND IT IS NOT WHAT IT LOOKS LIKE. `helm upgrade` with this # set to false is NOT reliably the repair, and believing it was would be the # trap of point 1 sitting inside the rescue instruction itself: oauthOnly is # an ordinary key of CONFIG_ENV_MAP, every consumer reads it through # configManager.get (app/api/auth/session/route.ts, # app/api/auth/verify/route.ts, app/api/config/route.ts - none of them reads # process.env), and the admin dashboard's OAuth setup writes # `oauthOnly: true` into config.json when it is used. Once that override is # on the PVC, this value is ignored and the upgrade reports success. # # WHAT REALLY WORKS, and it does not depend on the identity provider: # OPEN THE ADMIN DASHBOARD (its own password, from the vault item) AND # REMOVE THE OVERRIDE - the dashboard's revert, i.e. DELETE on # /api/admin/config, which calls configManager.removeAdminOverride. # After that the chart's value applies again. If the dashboard itself is # unreachable, the remaining repair is editing or deleting config.json on # the PVC directly. Both are in NOTES.txt. # # AND THE PRE-INSTALL STEP THIS DEFAULT MAKES MANDATORY: the OIDC client # must be registered with the provider BEFORE the first install, not after. # With oauth.only true and no working client, the very first sign-in is # impossible. NOTES.txt prints the redirect URI to register. only: true # OAUTH_CLIENT_ID. Required while oauth.enabled is true. clientId: null # OAUTH_ISSUER_URL - the issuer, from which Bulwark fetches the discovery # document. Required while oauth.enabled is true. For keycloak (#83) this is # https:///realms/ # # It must be reachable BY THE POD and the endpoints it advertises must share # the issuer's origin or be public: getDiscoveryValidator # (lib/oauth/token-exchange.ts) rejects a discovered endpoint on a private # address from a different origin unless OAUTH_ALLOW_PRIVATE_ENDPOINTS is # set. The chart does not render that escape hatch - a split-horizon setup # that needs it can add it through bulwark.env. issuerUrl: null # OAUTH_SCOPES - replaces the scope list Bulwark would otherwise request. # null leaves upstream's default, which is the right starting point; an # incomplete list here is a login that succeeds and then has no identity to # work with. scopes: null # OAUTH_END_SESSION - signing out of Bulwark also ends the provider's # session through its end_session_endpoint. True, upstream's default: on a # shared browser, a sign-out that leaves the provider signed in means the # next click is straight back in. Set it false where the provider is shared # with applications that should stay signed in. endSession: true # WEB PUSH RELAY - read this before writing it down as a disclosure, because # the obvious reading of the build-time variable is wrong and was corrected # against the source. # # NEXT_PUBLIC_PUSH_RELAY_URL is baked into the bundle (point 4 at the top), so # this chart cannot move the BUILT-IN entry, which is upstream's hosted relay # at https://notifications.relay.bulwarkmail.org. It would be wrong to stop # there and declare Web Push "not changeable with the published image": that # variable only sets the FALLBACK. lib/push-relays.ts resolves the effective # relay as `resolveDefaultRelayUrl(policy)`, which takes policy.pushRelayUrl # whenever it is a valid URL and uses the built-in one only otherwise; with # policy.pushRelayUrlLocked an operator can additionally pin every user to it. # # So the correct, two-part statement, and both halves matter: # - WITHOUT further action, Web Push for this release goes through the # VENDOR'S relay. For a mail client that is a fact worth knowing, and it # stays in the users' relay list either way, because the list is built # from the bundle. # - A SELF-HOSTED RELAY CAN BE MADE THE EFFECTIVE DEFAULT AND MANDATORY - # but through policy.json, i.e. the ADMIN DASHBOARD, not through a chart # value. There is deliberately no value here: pushRelayUrl lives in # SettingsPolicy and has no entry in CONFIG_ENV_MAP, so no environment # variable reaches it and a chart value would be a switch wired to # nothing. # # AND THE INTERACTION TO KEEP IN MIND: bulwark.adminConfigReadOnly freezes # setPolicy, so turning that on before setting the relay makes this change # impossible afterwards. Order matters - see the sequence at that value. # # THERE IS DELIBERATELY NO VALUE UNDER THIS COMMENT. A `pushRelayUrl:` key # here would render nothing and change nothing, which is the exact defect # guideline 12 exists to remove - a key that looks like configuration and does # nothing. The knowledge belongs in the file an operator reads; the switch # belongs where it actually is, in the dashboard. # BULWARK_TELEMETRY - upstream's anonymous telemetry. # # FALSE, AND THAT IS NOT MERELY RESTATING THE IMAGE'S DEFAULT, which is the # reason this value exists at all. Measured in lib/telemetry/state.ts: consent # defaults to 'off' and telemetry really is opt-in (the comment in # instrumentation.node.ts calling it "on by default" is wrong - DEFAULTS carries # `consent: 'off'`). What rendering the variable buys is the LOCK: # effectiveConsent() takes the environment over the state file, and upstream # blocks the admin UI from changing it while the override is active ("UI changes # are blocked when env override is active"). So this is one of the very few # settings in this chart where the chart beats the volume rather than the other # way round - see the exception list in point 1 at the top of this file. # # Setting it true opts in; /app/data/telemetry on the PVC then holds the # consent record and the instance id, which is why that directory is inside the # single volume rather than an emptyDir. telemetry: false # BULWARK_UPDATE_CHECK - an hourly version check against # version.telemetry.bulwarkmail.org. # # FALSE, AND HERE THE DEFAULT REALLY IS THE OTHER WAY: unlike telemetry, this # check is ON unless switched off (lib/version-check/state.ts, disabledByEnv() # recognizes only off/false/0/no - an absent variable means enabled). So this # value does change behaviour. # # Off for two reasons: a mail client should not phone home from a private # cluster unless somebody chose it, and Renovate already watches the image tag # (renovate.json5) - so the check would answer outbound a question that is # already answered inbound, and answer it to a third party. updateCheck: false # LOG_LEVEL - error, warn, info or debug. logLevel: info # LOG_FORMAT - text or json. JSON here, against upstream's default of text, # because the logs go to a cluster log collector rather than to a terminal. logFormat: json # The HTTP port. 3000, which is what the image binds (ENV PORT=3000, # EXPOSE 3000) and what HOSTNAME=0.0.0.0 makes reachable. Both the Service and # all three probes address this port BY NAME ("http"), so changing the number # here is safe and renaming the port is not. port: 3000 resources: # Limits are optional: memory and ephemeral-storage limits default to # limitFactor x the request, an explicitly set limit always wins, and a CPU # limit is only set when configured explicitly - CPU is compressible, so # leaving it unlimited throttles instead of OOM-killing. # # 256Mi x 3 = 768Mi. A Next.js standalone server serving a SPA and proxying # JMAP holds no mail and caches no mailbox, so the footprint is the Node # heap plus the image-optimization cache - not a function of how much mail # the users have. # # THE MEMORY LIMIT IS LOAD-BEARING IN ONE DIRECTION, though: /api/health # reports `degraded` once V8 heap usage crosses its warning threshold, and # the heap limit is derived from the container limit. That report is not # visible to the probes (see bulwark.readinessProbe), so a limit set too low # produces a pod that is permanently degraded, permanently Ready, and says # so only to an authenticated caller asking for details. # # The ephemeral-storage request covers the two emptyDirs (/tmp and the # Next.js cache) plus container logs. It does NOT cover /app/data, which is # the PVC - see storage.amount. limitFactor: 3 requests: cpu: 100m memory: 256Mi ephemeral-storage: 64Mi # ALL THREE PROBES ARE AN HTTP GET ON /api/health, and the limit of that is # written at the readiness probe below rather than left for somebody to assume. # # A NOTE ON GET VERSUS HEAD, so nobody "fixes" this back. Bulwark offers a HEAD # handler on this path and it would be the cheaper call - but a KUBERNETES PROBE # CANNOT SEND HEAD. The API type is HTTPGetAction, it has no method field, and # the kubelet always issues a GET. The only way to send a HEAD is an exec probe # calling an HTTP client, and this image has neither curl nor wget (npm and npx # are removed from the runtime layer), so it would mean starting a Node process # every few seconds instead of one cheap HTTP request. The GET handler does # gather diagnostics, which makes it marginally MORE capable than HEAD, not # less: it can answer 503 when the diagnostics themselves throw. # # LIVENESS: the process is alive, accepting HTTP, its event loop is free and # Next.js can route. An OOM kill, a crash and a wedged event loop all fail it. livenessProbe: httpGet: path: /api/health port: http periodSeconds: 10 timeoutSeconds: 5 failureThreshold: 3 # READINESS: the same endpoint, and HERE IS THE LIMIT, stated at the value # because a probe path is not a promise (README guideline 12.3; #218 and # Gitea #83 are the two times this repo paid for assuming otherwise). # # WHAT THIS PROBE CARRIES: the Node process is alive, the HTTP server accepts, # the event loop is not blocked, Next.js routes. That is real and it is not # nothing. # # WHAT IT DOES NOT CARRY, measured in app/api/health/route.ts at tag 1.12.0: # - THE REPORTED STATE. The success path returns a hard `status: 200` # regardless of what it computed, and for an unauthenticated caller the # body is hard-coded to "healthy" as well # (`status: detailed ? status : 'healthy'`); "degraded" appears only with # ?detailed=true AND a caller that passes canSeeInstanceDetails. Upstream # tests precisely this - lib/__tests__/health-route.test.ts expects HTTP # 200 while payload.status is "degraded". A probe is unauthenticated and # asks for no details, so it sees 200 and "healthy" always. The only 503 # this endpoint can produce comes from its catch block, i.e. when # process.memoryUsage() or v8.getHeapStatistics() itself throws. # - ANYTHING ABOUT STALWART. Bulwark holds no mail and this endpoint asks the # JMAP server nothing. WITH THE MAIL SERVER COMPLETELY GONE, THIS PROBE IS # GREEN AND THE POD STAYS Ready. Monitor the stalwart release for that; # nothing here can. # # AND WHY IT IS STILL THIS ENDPOINT: there is no better one. # /api/setup/test-jmap really would test the connection, but it lives behind # the bootstrap setup token and this chart disables the wizard, so it is # unreachable. Pointing readiness at "/" instead would only add Next.js # rendering to the same set of guarantees, and would start failing on an # unauthenticated redirect rather than on anything about the mail path. readinessProbe: httpGet: path: /api/health port: http periodSeconds: 10 timeoutSeconds: 5 failureThreshold: 3 # STARTUP: gates the other two, which is why neither carries an # initialDelaySeconds. # # BUDGET = periodSeconds x failureThreshold = 5 x 24 = 120s. What that has to # cover is not the Next.js server, which starts in about a second: it is # attaching and mounting the network volume, and the instrumentation hook that # runs before the first request is served (instrumentation.node.ts migrates # the legacy admin layout, loads config.json and policy.json off the volume, # and on a FIRST install hashes ADMIN_PASSWORD with scrypt and writes # admin.json - the scrypt hash being the one deliberately slow step). # # NOT MEASURED AGAINST A RUNNING POD AT THE TIME OF WRITING, and said so # rather than presented as calibrated: the budget is derived from what the # start does, with head room, and the cost of head room is nothing because a # pod that IS healthy becomes Ready on the first probe either way. startupProbe: httpGet: path: /api/health port: http periodSeconds: 5 timeoutSeconds: 5 failureThreshold: 24 securityContext: pod: runAsNonRoot: true # uid/gid 1001, NOT this repo's 20000, and that is the image's own user # rather than a free choice: the Dockerfile creates # addgroup --system --gid 1001 nodejs # adduser --system --uid 1001 nextjs # and copies the application tree plus /app/data with --chown=nextjs:nodejs. # 1001 therefore OWNS everything the process reads, and the four data # subdirectories are chowned to it in the image. Read off the Dockerfile at # tag 1.12.0, not guessed. # # fsGroupChangePolicy: Always, the repo standard, and here it is the right # one rather than the default one: /app/data holds a handful of JSON files, # an audit log and a small settings tree, so the recursive chown is # cheap - unlike garage's block store or satisfactory's game files, where # it is the reason both deviate to OnRootMismatch. It also matters on the # FIRST start, because a freshly provisioned volume arrives owned by root # and Bulwark has to create its four subdirectories in it. runAsUser: 1001 runAsGroup: 1001 fsGroup: 1001 fsGroupChangePolicy: Always seccompProfile: type: RuntimeDefault container: allowPrivilegeEscalation: false # TRUE, with two emptyDirs carrying the writes that are not state. Which # paths those are, and how the set was established, is at # bulwark.writableCache below - it is the one part of this chart that was # derived from the source and then PROVEN by the install-test rather than # the other way round. readOnlyRootFilesystem: true runAsNonRoot: true capabilities: drop: - ALL # EMPTY, and it can stay empty: the process binds 3000, which is above # 1024, and the entrypoint is `node server.js` with no file # capabilities - so there is no execve-time capability check like # stalwart's. add: [] # THE WRITABLE PATHS OUTSIDE THE VOLUME, as emptyDirs, because # readOnlyRootFilesystem is true and Next.js writes in two places that are not # application state. # # HOW THIS SET WAS ESTABLISHED, and the honest order of events: it was DERIVED # from next.config.ts at tag 1.12.0 - `output: "standalone"`, and # `images: { unoptimized: true }` applies only in the isLite branch, so the # published image has image optimization ACTIVE and writes its cache under # /app/.next/cache - and it was then PROVEN by the install-test, which starts # the real image with this exact hardening and requires the pod to become # Ready. A missing writable path here is not a subtle defect: it is an EACCES # or EROFS at the first write. # # /tmp the conventional scratch path. Node, Next.js and the # image-optimization pipeline all use it. # /app/.next/cache Next.js's own cache directory in standalone mode: the # optimized-image cache and the fetch cache. It is a # SUBDIRECTORY of .next, which is why it can be an # emptyDir at all - /app/.next/static holds the build # output and must stay readable, and mounting the cache # path does not shadow it. # # Both are CACHES and neither is state: losing them on a restart costs one # round of image re-optimization. Do not "simplify" either one onto the PVC - # a cache on a ReadWriteOnce network volume is slower than regenerating it, # and it would grow the only volume that has to be backed up. writableCache: # Size limits on both, so a runaway cache fills an emptyDir instead of the # node's disk. These are the sizeLimit of each emptyDir; the pod's # ephemeral-storage request/limit (bulwark.resources) is the other half of # the same accounting and the two should move together. tmpSizeLimit: 256Mi nextCacheSizeLimit: 512Mi # Additive environment entries for the Bulwark container, rendered through tpl # and supporting the value, secretKeyRef and configMapKeyRef forms. # # ADDITIVE MEANS ADDITIVE: an entry repeating a name the chart already sets # does not replace it. At install time both reach the pod and the later one # wins; on an EXISTING release the pair collapses to the chart's own value, # because Kubernetes merges env by name - and helm upgrade still reports # success. So do not use this to override JMAP_SERVER_URL, OAUTH_ONLY or # SESSION_SECRET_FILE; change the values that produce them. # # AND THE SECOND, SHARPER CAVEAT, SPECIFIC TO THIS APPLICATION: an environment # variable is only the MIDDLE of three layers (point 1 at the top of this # file). Anything set here is a default that an admin override in config.json # on the PVC may already be beating, silently. An entry here is therefore not # a guarantee - unless bulwark.adminConfigReadOnly is on, and that has its own # price. # # Genuinely useful here, all of them CONFIG_ENV_MAP keys this chart does not # render: APP_DESCRIPTION, LOGIN_COMPANY_NAME and the other LOGIN_* branding # variables, COOKIE_SAME_SITE, STALWART_JMAP_PASSTHROUGH_ENABLED, # OAUTH_EXTRA_SCOPES, WOPI_CLIENT_URL. env: [] storage: # THE ONE PVC, mounted at /app/data, and "one" is the decision rather than the # obvious shape - Bulwark keeps FOUR directories under it and they could each # have their own volume: # # /app/data/admin config.json, policy.json, admin.json (the password # hash), plugins, themes, branding uploads # /app/data/admin-state admin-state.json (login timestamps), audit.log, the # bootstrap setup token # /app/data/settings the encrypted per-account user settings # /app/data/telemetry consent and instance id # # ONE VOLUME, BECAUSE A SNAPSHOT IS ATOMIC PER VOLUME AND NEVER ACROSS THEM - # the lesson garage (#81) was built on. Split over four, a backup could # capture an audit log that records logins the configuration does not know # about yet, or settings encrypted against a secret the config has since been # repointed from. One volume keeps configuration, runtime state, audit trail # and user settings consistent with each other, and there is no performance # argument on the other side: this is a few hundred kilobytes of JSON plus an # append-only log. # # Bulwark creates all four subdirectories itself on start (ensureConfigDir / # ensureStateDir, both mkdir -p), which it has to, because the image's own # mkdir ran in a layer that this mount covers. # # THE PVC IS DELIBERATELY NOT null-SAFE: an erased size yields an invalid PVC # that is rejected at apply time, which is loud. A chart-side fallback would # silently size somebody's volume - and on an existing release it could not # help anyway, because a PVC cannot shrink. Same reasoning as garage's storage # block and satisfactory's documented exception (Gitea #39). # # IT IS ALSO NEVER RENAMED: a renamed PVC makes Kubernetes provision a new, # empty volume and orphans the data - including admin.json, which is the only # copy of the admin password hash, and the audit log, which by its nature # cannot be reconstructed. # # 1Gi is generous for the JSON files and the settings tree; what actually # grows is audit.log, and it grows with admin activity rather than with mail # volume. There is no rotation for it in the application, so this is the value # to watch on a long-lived release. amount: 1Gi # null uses the cluster's default StorageClass. ReadWriteOnce is enough and is # what the single replica needs; see bulwark.replicas for why more than one is # not on offer. storageClass: null ingress: # ON by default: a webmail client nobody can reach is not a webmail client. enabled: true # The public hostname. REQUIRED, AND REQUIRED EVEN WITH ingress.enabled: # false - this is guideline 12.1, checked in both directions for this chart. # The value is read a SECOND time outside the Ingress template: NOTES.txt # derives the OIDC redirect URI from it, and that URI is what has to be # registered with the identity provider before the first install. Gating it # away would produce a release that renders cleanly and whose operator is told # to register `https:///auth/callback`. # # BULWARK IS SERVED AT THE ROOT OF THIS HOST AND CANNOT BE SERVED UNDER A # PATH. NEXT_PUBLIC_BASE_PATH is compiled into the bundle (point 4 at the top # of this file), so a sub-path deployment would need a rebuilt image. The # Ingress rule is therefore "/" and there is no value to change it - a # rewrite-target annotation would not help either, because the bundle's own # asset URLs are built for the root. domain: null # cert-manager cluster issuer. Required while the Ingress is enabled. clusterIssuer: null className: nginx # Extra annotations, merged with the cert-manager annotation the chart always # sets. # # ONE WORTH KNOWING ABOUT FOR THIS APPLICATION: file attachments and the Files # view upload through this Ingress, and nginx's default # proxy-body-size (1m) will reject anything larger with a 413 that looks like # an application fault. Set # nginx.ingress.kubernetes.io/proxy-body-size: "25m" # (or whatever the mail server accepts) to match stalwart's own limit. The # chart does not set it, because the right number is the mail server's and # this chart does not know it. annotations: {}