jlab-cluster/helm-charts
3
0 Fork 0
Code Issues 15 Pull Requests 3 Actions Packages Wiki Activity

garage: new chart page, index row, sidebar entry, two lessons 

The page carries the values table, the first-install checklist for both paths
(self-configuring with --single-node, and the manual layout sequence) and a
traps section: the metadata gap in a volume backup, metadata_fsync being off
upstream, the two-snapshot restore window, the non-idempotency of
layout apply and why single-node mode does not have it, replication factor 1
applying to the metadata, the scratch image that rules out exec probes for
good, and why the RPC secret has to be an environment variable.

Two Lessons-Learned entries, both in their general form rather than as garage
notes: "the snapshot that was not a restore point" repeats at every embedded
database on a volume with snapshots, and "a permission check that assumes it
owns the filesystem" at every application offering a *_secret_file with a
0o077 mode test.

Chart index count 28 -> 29 directories.
jklein committed 2026-10-03 20:36:12 +00:00
1 parent 43aef80b2e
commit c9c99757c1
4 files changed
+399 -1
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

Charts.md
+2 -1
@@ -1,6 +1,6 @@
# Chart index
28 directories: 27 installable charts plus the `template-chart` scaffold. Each page holds prerequisites, an installation walkthrough, the complete list of configurable values, and the chart's operating traps.
29 directories: 28 installable charts plus the `template-chart` scaffold. Each page holds prerequisites, an installation walkthrough, the complete list of configurable values, and the chart's operating traps.
**Versions are not listed here on purpose** — and not on the chart pages either. Renovate moves app versions continuously; a table of numbers in a wiki is wrong within days. Read the current pair off the chart:
@@ -16,6 +16,7 @@ grep -E '^(version|appVersion):' <chart>/Chart.yaml
| [convertx](convertx) | ConvertX | StatefulSet + data PVC | yes | **the only chart that overrides `command` to bypass its image entrypoint** — that is what keeps the hardening baseline intact on a rootful image, at the price of re-reading the entrypoint at every app-version bump; probes use `/healthcheck` because `/` is a 302 |
| [cyberchef](cyberchef) | CyberChef | Deployment | yes | stateless; the simplest chart in the repo and a good first read |
| [freesailarr](freesailarr) | gluetun VPN media stack, ten containers | single-pod StatefulSet | per app | the exception to many rules: no icon, no app-version-driven tags, two ingress shapes |
| [garage](garage) | Garage (S3-compatible object store) | StatefulSet + **two** PVCs | off by default | **one** node at `replication_factor = 1` — which applies to the metadata too, so `metadata_fsync`, the metadata snapshots and a second PVC for them are mandatory rather than optional; `FROM scratch` image, so no shell and no exec probes ever; the RPC secret has to be an env var; readiness is `/health`, liveness and startup deliberately are not |
| [gotenberg](gotenberg) | Gotenberg | Deployment + Service | no | PDF conversion for [paperless-ngx](paperless-ngx); sets every limit explicitly |
| [homeassistant](homeassistant) | Home Assistant | StatefulSet | upstream-style lists | the only chart with Helm tests; starts as root for s6-overlay |
| [matrix-synapse](matrix-synapse) | Matrix Synapse | StatefulSet + nginx delegation Deployment | yes | `ingress.domain` is served in `/.well-known/matrix/*` — federation depends on it |
Lessons-Learned.md
+31
@@ -343,6 +343,34 @@ A parse error, in somebody else's template, naming neither the value that caused
**And the one nobody was looking for: erasing the dependency switch turns the subchart ON.** `valkeySubchart: null` does not mean "the default applies". A null override **deletes** the key, Helm then cannot resolve the `condition:` path, and an **unresolvable condition counts as enabled** — so the block that reads like "off" produces a Valkey with no password configured, reported only by the subchart's own error. Everywhere else in these charts an erased block falls back to the chart defaults ([§12](Chart-Style-Guide#12-values-that-must-not-lie), issue #99); the dependency switch is the one block where that is backwards, so the schema now **requires** it. An empty `{}` is fine, because Helm merges that with the default — which is precisely why the dangerous case is easy to miss: the two spellings look equivalent and behave oppositely.
## The snapshot that was not a restore point
The plan for [garage](garage) (Gitea #81) rested on a backup story that read as complete: the data lives on an encrypted Longhorn volume, Longhorn snapshots it and backs the snapshot up off-site, and the backup target only ever sees ciphertext because the backup engine reads *below* the decryption layer. All of that is true. It is also only true of the **blocks**.
**A file-system snapshot of a live embedded database is not a restore point.** Taken in the middle of a write it captures a half-updated file, and Garage says so about its own metadata database in as many words — which is why it recommends *its own* snapshot mechanism "in all cases", not as an extra for the careful. So the half of the store that can be rebuilt from elsewhere was protected, and the index that makes the blocks addressable was not. Nothing reports that asymmetry: the snapshot exists, the backup job is green, the volume is restorable, and the restored database may simply not open.
Two things made it worse in that specific chart, and both generalise:
- **The durability setting was off by default.** `metadata_fsync` ships disabled, and a replicated block layer cannot help with a write that was never handed to the kernel — it is in no replica and in no snapshot. A storage layer with three replicas reads like it covers this. It does not.
- **Replication replicates corruption.** At `replication_factor = 1` the metadata has no peer to be repaired from, and the volume layer copies a corrupted file as faithfully as a good one. Redundancy against device failure is not redundancy against file corruption, and the two are easy to file under the same heading.
**Now:** a chart that puts an embedded database on a volume answers three questions before it is called backed up — does the engine have a consistent-copy mechanism of its own, is it switched **on**, and does its output live on a **different** volume than the database? The last one is the cheap part and the easy one to skip: a recovery point stored on the volume whose corruption it exists to repair is not a recovery point. `garage` keeps metadata and data on one PVC (so a volume snapshot catches index and blocks atomically) and the database snapshots on a second one.
And know the **window**, because "snapshots are on" is not a retention policy: Garage keeps exactly two, the first after half the interval, so the oldest recovery point is between one and two and a half intervals old and never older. A corruption found the next working day is older than everything that still exists.
## A permission check that assumes it owns the filesystem
[garage](garage) offers `rpc_secret_file`, which is the shape this repo prefers for a secret — a mounted file instead of an environment variable, as [valkey](valkey) argues at length. The plan specified it. It cannot work, and the reason is a collision of two correct-looking checks:
- Garage refuses to read a secret file whose mode carries a group **or** other bit: `metadata.mode() & 0o077 != 0`, with an error that says *world-readable* while the mask also tests the group.
- The kubelet ORs the group-read bit onto **every** file of a secret volume as soon as `fsGroup` is set (`Chmod(info.Mode() | 0440)` for a read-only volume). Without `fsGroup` the file stays owned by root — and a non-root container then cannot read it at all.
Kubernetes can set the **group** of such a file; it can never set its **owner**. So for a hardened, non-root pod there is no combination of `defaultMode` and `fsGroup` that satisfies both sides: one of the two always refuses. The application's check is not wrong — it is written for a host where the operator owns `chmod 0600` and the file's uid — it is simply unsatisfiable on this platform.
What makes this an entry rather than a quirk of one chart is how it presents and what the tempting repairs cost. It presents as a pod that will not start, with a message about a **file mode** nobody can change from inside Kubernetes, which sends the reader looking for the wrong knob. And the two repairs that suggest themselves are both worse than the environment variable: `allow_world_readable_secrets = true` switches the check off for **all** of that application's secret files, including ones nobody has configured yet, and an init container that copies the file with mode 0600 needs a **second container image** when the main one is `FROM scratch` — plus an emptyDir with a secret in it.
**Now:** before specifying `*_secret_file` for an application, check what its permission test actually masks. A `0o077` test (or any test that includes the group bits) means the file form is unavailable to a non-root pod, and the honest route is `secretKeyRef` with the price written at the value: the secret is in the process environment, readable by anyone who can `exec` into the pod. Write the **mechanism** next to it, not the conclusion — "we use the env var here" invites the next round to tidy it back into the file form, and then the pod stops starting.
## The recurring shape
- **Green is a claim about the assertions that ran, not about the system.**
@@ -358,6 +386,9 @@ A parse error, in somebody else's template, naming neither the value that caused
- **Document a deviation with its mechanism, not its name.** A grant whose comment does not say *why it cannot be removed* reads as dead weight to the next cleanup, and "needed for binding" is worse than nothing when the real reason is `execve`.
- **Inside a cluster, every per-client identity is a per-node identity.** Masqueraded source addresses and probe traffic both collapse onto node IPs, so any application logic keyed on the client address — bans, limits, geo, audit — is being fed one address, silently and correctly-looking.
- **Ask what an application does when its configuration is missing, not only when it is wrong.** The lenient fallback is almost always on the "absent" path, and it is the path a broken mount takes.
- **A snapshot of a live database is a backup of a file, not of a database.** Replication, encryption and an off-site copy all operate below that distinction and none of them notices it.
- **A safety check written for a host can be unsatisfiable in a pod.** Before adopting an application's preferred mechanism, check what it verifies — the platform may be unable to produce the state it demands, and the symptom will name the wrong knob.
- **Read the flags of the version you are shipping, not the ones the documentation pages describe.** Two decisions in the garage plan were argued from constraints that an existing `--single-node` had already removed; the plan was not wrong about the old way, it was wrong about which way existed.
- **A dependency on your own artifact tests the published copy, not the one you are editing.** Two changes in one PR means a green check on the wrong input — and nothing in the output says which copy was used.
- **A convention is not a rule, and the difference shows up in somebody else's validator.** `condition: <name>.enabled` is fine until the subchart has a strict schema; the error then belongs to the subchart and appears only once the feature is switched on.
- **Validation that is skipped in the default path is not validation.** A check that only runs for the non-default branch passes CI forever and fails for the first person who deviates.
_Sidebar.md
+1
@@ -25,6 +25,7 @@
- [convertx](convertx)
- [cyberchef](cyberchef)
- [freesailarr](freesailarr)
- [garage](garage)
- [gotenberg](gotenberg)
- [homeassistant](homeassistant)
- [matrix-synapse](matrix-synapse)
garage.md
+365
@@ -0,0 +1,365 @@
# garage
Deploys [Garage](https://garagehq.deuxfleurs.fr/) — an S3-compatible object store written in Rust — as **one** node with `replication_factor = 1`. The first consumer is [stalwart](stalwart)'s blob store, i.e. the bodies of the mail server's messages, and the deployment target is an encrypted, three-times-replicated Longhorn volume: Garage replication *on top of* that would be nine times the raw data across two layers that know nothing about each other.
That trade is deliberate, and the price is the first thing to understand about this chart: **the replication factor applies to the metadata as well.** Upstream writes of factor 1 "Do not use this for anything else than test deployments" and calls a corrupted metadata database "generally not recoverable" — and the volume layer does not help there, because it replicates a corrupted file exactly as faithfully as a good one. Redundancy against device failure, yes; against file corruption, no. Everything in [Traps](#traps) follows from that sentence.
| | |
|---|---|
| Image | `docker.io/dxflrs/garage` — the multi-arch image. Upstream's own in-tree chart pins `dxflrs/amd64_garage`, which would tie the deployment to one architecture |
| Versions | not listed on purpose — read the current pair off `garage/Chart.yaml`, see [Chart index](Charts) |
| Workloads | StatefulSet, 1 replica, **no init container** (the image has no shell — see [The image is `FROM scratch`](#the-image-is-from-scratch-and-that-shapes-the-whole-chart)) |
| Storage | **two** RWO PVCs: one for metadata **and** data, one for the metadata snapshots |
| Ingress | **off by default**, built to the convention so it can be switched on |
| External dependencies | none — no database, no cache, no other service |
| Secrets | 2 × `OnePasswordItem`, four fields in total |
| Network | ClusterIP Service on 3900 (S3) and 3903 (admin). No RPC port in the Service, and no NetworkPolicy yet |
| Install-test | `ci/garage/` (fixtures + test values) |
## Prerequisites
- **1Password operator** in the cluster, plus **two** items — see [What the 1Password items must contain](#what-the-1password-items-must-contain). Four fields, each of them validated by Garage at start: a malformed one is a container that exits, not a warning.
- **A storage class for both PVCs.** The intended value is `longhorn-encrypted-ns`, which resolves `${pvc.namespace}-pvc-encryption-secret` — without that entry in the namespace the PVCs stay `Pending`. The `volume-encryption` bundle belongs to `cluster-config`, not to this chart.
- **The Longhorn backup labels, also from `cluster-config`**: `recurring-job.longhorn.io/daily-snapshot` and `…/daily-backup` on the volumes. They do **not** apply automatically — as of 2026-10-02, 41 of 62 volumes in the cluster carried none. Without them the off-site half of the backup plan does not exist; it only looks like it does. And read [The metadata is not in your volume backup](#the-metadata-is-not-in-your-volume-backup) before trusting even the half that does exist.
- **Nothing else.**
## Installation
The registry URL and its credentials live in 1Password (item `jlab-registry-access`: `username`, `password`, `charts-url`) and never in a repository or a shell history file.
```bash
helm repo add jlab "$CHARTS_URL" --username "$REG_USER" --password "$REG_PASS"
helm repo update
helm install garage jlab/garage \
--namespace garage --create-namespace \
-f my-values.yaml
```
Minimal `my-values.yaml` — two required values, because everything else has a default that is safe for this deployment:
```yaml
secrets:
rpcSecretRef: op://vault/garage-rpc
s3SecretRef: op://vault/garage-s3
```
That installs a node that **configures itself**: it creates its own cluster layout, imports the access key and creates the bucket, all before it opens a listener. Sizing the volumes is the one thing worth doing in the same file:
```yaml
storage:
data:
amount: 200Gi
storageClass: longhorn-encrypted-ns
snapshots:
amount: 10Gi
storageClass: longhorn-encrypted-ns
```
`helm install` prints the chart's `NOTES.txt`, which names the four vault fields, the verification sequence for this release, and the traps in short form. Read it — it is the per-release version of this page.
### What the 1Password items must contain
The chart renders two `OnePasswordItem`s; the operator turns each into a Kubernetes Secret whose **keys are the field names of the 1Password item**. A missing or misnamed field leaves the pod in `CreateContainerConfigError`.
| Values key | Kubernetes Secret | Field | Content |
|---|---|---|---|
| `secrets.rpcSecretRef` | `<release>-garage-rpc-secret` | **`rpc-secret`** | 32 bytes of random hex (`openssl rand -hex 32`). Required even for a single node: Garage uses it as the network key of its RPC layer and refuses to start without it |
| `secrets.s3SecretRef` | `<release>-garage-s3-secret` | **`access-key`** | the S3 key id. At least 8 characters, ASCII alphanumeric plus `-`, `_`, `.` |
| | | **`secret-key`** | at least 16 printable ASCII characters (`openssl rand -hex 32`) |
| | | **`bucket`** | a valid S3 bucket name, 3–63 characters, lowercase |
**Two items rather than one, because they have different audiences.** The RPC secret belongs to this node alone. The S3 key is the credential the *consumer* uses — [stalwart](stalwart)'s `secrets.s3SecretRef` points at this same item, which is why the bucket name lives in it as a third field: endpoint, bucket, key and secret then have one source of truth instead of being kept in step by hand.
**And the direction is inwards.** With the defaults, Garage **imports** `access-key`, `secret-key` and `bucket` on start. The vault is written first and the server adopts it; nothing has to be copied out of a terminal afterwards. That is the opposite of how a Garage cluster is normally set up, and it is the reason the first install needs no operator at all.
## First install: the checklist
With `garage.singleNode: true` (the default) there is **nothing to do by hand**. What follows is verification, in order. Every command uses `/garage` directly, because the image has no shell.
1. **The rollout.**
```bash
kubectl -n garage rollout status statefulset/garage-garage-sfs
```
`Ready` is not a formality here: it means the admin API answered **200** on `/health`, which means a layout exists and the node has quorum. A running container that stays not-Ready means the layout is missing — look for `Created initial layout for single-node configuration` in the log.
2. **The node has a role.**
```bash
kubectl -n garage exec garage-garage-sfs-0 -- /garage status
```
One node, zone `dc1`, with a capacity Garage took from the detected disk size. A node **without** a role means no layout was applied.
3. **The layout.**
```bash
kubectl -n garage exec garage-garage-sfs-0 -- /garage layout show
```
4. **The bucket and the key.**
```bash
kubectl -n garage exec garage-garage-sfs-0 -- /garage bucket list
kubectl -n garage exec garage-garage-sfs-0 -- /garage bucket info <bucket>
```
The bucket exists and the imported key has read/write/owner on it.
5. **One write through the S3 API**, from a client pod. It is the only step that proves the endpoint rather than the process. In-cluster address:
```
http://<release>-garage-service.<namespace>.svc.cluster.local:3900
```
with **path-style** addressing (bucket in the path) and the region from `garage.s3.region`.
6. **Confirm the snapshot mechanism** once, roughly `metadata_auto_snapshot_interval / 2` after the start — this is the one check the pipeline cannot do for you:
```bash
kubectl -n garage exec garage-garage-sfs-0 -- /garage worker list
```
and look for the metadata snapshot worker's "Next snapshot" line; the log prints `Snapshotting metadata db to /mnt/garage-snapshots/<timestamp>` when it runs.
### The manual path (`garage.singleNode: false`)
Only for a deployment that is meant to grow into a real cluster. The release then **serves nothing** until a layout is applied, and the pod stays not-Ready — which is correct rather than broken, because `/health` answers 503 while there is no layout and nothing should route to the endpoint yet.
```bash
# 1. node id and current layout version (0 on a fresh install)
kubectl -n garage exec garage-garage-sfs-0 -- /garage status
# 2. assign a role. Capacity takes BYTE suffixes (B, KB, MB, GB, TB, PB) -
# not Kubernetes' Gi/Mi, so 10GB, never 10Gi.
kubectl -n garage exec garage-garage-sfs-0 -- \
/garage layout assign -z dc1 -c 200GB <node-id>
# 3. read the staged layout BEFORE applying it
kubectl -n garage exec garage-garage-sfs-0 -- /garage layout show
# 4. apply it. N is the version from step 1, so a fresh cluster takes 1.
kubectl -n garage exec garage-garage-sfs-0 -- /garage layout apply --version <N+1>
# 5. bucket and key - "import", not "create", so the credential still comes
# FROM the vault item instead of travelling the other way
kubectl -n garage exec garage-garage-sfs-0 -- /garage bucket create <bucket>
kubectl -n garage exec garage-garage-sfs-0 -- \
/garage key import --yes <access-key> <secret-key>
kubectl -n garage exec garage-garage-sfs-0 -- \
/garage bucket allow --read --write --owner --key <access-key> <bucket>
```
`garage.defaultAccessKey` and `garage.defaultBucket` must be `false` in this mode — the chart fails the render on the mixed combination, because Garage refuses it at start.
## Configurable values
### Global
| Key | Type | Default | Description |
|---|---|---|---|
| `scaleDown` | bool | `false` | Renders the StatefulSet with `replicas: 0` without uninstalling. Takes precedence over `garage.replicas`. Both PVCs, the Secrets, the Service and the Ingress stay. |
### `secrets`
| Key | Type | Default | Description |
|---|---|---|---|
| `secrets.rpcSecretRef` | string | `null` | **Required always.** `op://` path of the item carrying `rpc-secret`. |
| `secrets.s3SecretRef` | string | `null` | **Required** while `garage.singleNode` is on together with `defaultAccessKey` or `defaultBucket` (the default), because Garage imports those values on start. |
### `garage` — the node itself
| Key | Type | Default | Description |
|---|---|---|---|
| `garage.replicas` | int | `1` | `0` or `1` — the schema rejects anything higher. See [One node, and a second replica is not a cluster](#one-node-and-a-second-replica-is-not-a-cluster). |
| `garage.singleNode` | bool | `true` | Starts `garage server --single-node`, which creates the cluster layout in-process **before** any listener opens. Idempotent and unable to damage an existing cluster — see [The layout applies itself](#the-layout-applies-itself-and-cannot-be-applied-twice). |
| `garage.defaultAccessKey` | bool | `true` | Imports the S3 key from the vault item (`--default-access-key`). Requires `singleNode`. |
| `garage.defaultBucket` | bool | `true` | Creates the bucket from the item's `bucket` field and grants the imported key access (`--default-bucket`). Requires `singleNode`. |
| `garage.replicationFactor` | int | `1` | **Not an upgrade path**: the factor is part of the cluster layout, so raising it is a layout operation with data movement on a cluster that would first need more nodes. The value exists so the chart does not block a future multi-node setup. |
| `garage.dbEngine` | string | `lmdb` | `lmdb`, `sqlite` or the experimental `fjall`. Changing it on an existing release needs `garage convert-db`, not a values change. See [LMDB at factor 1](#lmdb-at-replication-factor-1-is-a-deviation-from-upstreams-own-advice). |
| `garage.metadataFsync` | bool | `true` | **On, against Garage's own default of off.** See [`metadata_fsync` is off upstream](#metadata_fsync-is-off-in-garages-defaults). |
| `garage.dataFsync` | bool | `false` | Garage's default. A lost block is one object and is replaceable; a lost metadata write can take the index with it. Turning it on costs an `fsync` per written block. |
| `garage.metadataAutoSnapshotInterval` | string | `"6h"` | Go-style duration. **The only consistent recovery point this deployment has.** An empty string is rejected by the schema; `null` falls back to the default, because switching the snapshots off is not a configuration this chart offers at factor 1. See [the window](#the-restore-window-is-two-snapshots-wide). |
| `garage.blockSize` | string | `"1M"` | Chunk size objects are split into. |
| `garage.compressionLevel` | int | `1` | zstd level for stored blocks; `0` disables compression. |
| `garage.bindAddress` | string | `"[::]"` | Host part of the S3, web and RPC listeners. Accepts IPv4 as well while `net.ipv6.bindv6only` is `0` — the Linux default, but a node sysctl the chart cannot see. |
| `garage.adminBindAddress` | string | `"0.0.0.0"` | Host part of the **admin** listener, separate on purpose: all three probes use this port, and a probe that cannot connect would be a pod that never becomes Ready for a reason nothing in the manifest shows. On an IPv6-only cluster set `"[::]"`. |
| `garage.ports.s3` / `.rpc` / `.web` / `.admin` | int | `3900` / `3901` / `3902` / `3903` | In **one** place: the ConfigMap renders them into `garage.toml`'s bind addresses and the StatefulSet renders them as `containerPort`s. The probes address the admin port by **name**, so they follow a change here. |
| `garage.s3.region` | string | `garage` | A label both sides must agree on, not a cloud region — a consumer configured for a different one gets a signature error, not a redirect. |
| `garage.s3.rootDomain` | string | `null` | Suffix for vhost-style bucket access (`<bucket>.<rootDomain>`). Only useful with an Ingress and a wildcard DNS entry; path-style works without it. Deliberately **not** coupled to `ingress.domain`. |
| `garage.web.enabled` | bool | `false` | Static website hosting (the `s3_web` endpoint). A second HTTP listener that serves nothing at all without a `rootDomain`, while the consumer speaks S3. |
| `garage.web.rootDomain` | string | `null` | |
| `garage.securityContext.pod` | object | non-root **20000**, `runAsGroup`/`fsGroup` **1000**, `fsGroupChangePolicy: OnRootMismatch`, `seccompProfile: RuntimeDefault` | Passed verbatim into the pod spec. `OnRootMismatch` is a documented deviation — see [Why `OnRootMismatch`](#why-onrootmismatch-and-not-always). Any uid works: the image has no `/etc/passwd`. Upstream's own chart uses 1000, which matters when migrating from it. |
| `garage.securityContext.container` | object | `readOnlyRootFilesystem: true`, all caps dropped, no privilege escalation, non-root | No capability is added and none is needed: the ports are above 1024 and the binary carries no file capabilities. |
| `garage.livenessProbe` | object | `tcpSocket` on `admin`, period 10s, timeout 5s, 3 failures | **Deliberately not `/health`** — see [the probes](#the-probes-point-at-two-different-things-on-purpose). |
| `garage.readinessProbe` | object | `httpGet /health` on `admin`, period 10s, timeout 5s, 3 failures | The honest readiness signal: 503 until a layout exists. |
| `garage.startupProbe` | object | `tcpSocket` on `admin`, period 5s, 24 failures (= 120s) | Gates liveness and readiness. The budget covers mounting two network volumes and opening (or recovering) the LMDB database, not Garage's own startup work, which is sub-second. |
| `garage.resources.limitFactor` | number | `3` | Memory and ephemeral-storage limits are computed as factor × request. |
| `garage.resources.requests` | object | `cpu: 100m`, `memory: 512Mi`, `ephemeral-storage: 32Mi` | 512Mi × 3 = 1536Mi memory limit; Garage's own guidance is "at least 1GB" per node. The ephemeral request covers container logs only — everything Garage writes goes to one of the PVCs. |
| `garage.resources.limits` | object | unset | An explicit limit always wins. A CPU limit is only rendered when set here. |
| `garage.env` | list | `[]` | Extra environment variables, rendered through `tpl`; `value`, `secretKeyRef` and `configMapKeyRef` forms. **Additive, not overriding** — see [Environment variables](Chart-Style-Guide#10-environment-variables). Do not try to set `GARAGE_RPC_SECRET` or the `GARAGE_DEFAULT_*` variables this way; change the values that produce them. `RUST_LOG` is the useful entry. |
### `storage`
| Key | Type | Default | Description |
|---|---|---|---|
| `storage.data.amount` | string | `10Gi` | `metadata_dir` **and** `data_dir`, as two subdirectories of this one volume. The value most likely to need changing before the first install. |
| `storage.data.storageClass` | string | `null` | `null` uses the cluster default — which is **not** encrypted. For a mail blob store that is a decision about data at rest, so the intended value is `longhorn-encrypted-ns`. |
| `storage.snapshots.amount` | string | `5Gi` | `metadata_snapshots_dir`. Sized for **two** compacted copies of the metadata database plus head room, not for the objects. |
| `storage.snapshots.storageClass` | string | `null` | |
Neither block is null-safe, on purpose: an erased size yields an invalid PVC that is rejected at apply time, which is loud. A chart-side fallback would silently size somebody's object store — and on an existing release it could not help anyway, because a PVC cannot shrink.
### `ingress`
| Key | Type | Default | Description |
|---|---|---|---|
| `ingress.enabled` | bool | `false` | Off, because the consumer speaks S3 inside the cluster. Switching it on publishes the **S3 API** — there is no web UI to expose. |
| `ingress.clusterIssuer` | string | `null` | Required while enabled. |
| `ingress.domain` | string | `null` | Required while enabled. Checked in both directions ([§12.1](Chart-Style-Guide#12-values-that-must-not-lie)): nothing outside the Ingress template reads it, so switching the Ingress off frees **both** values. |
| `ingress.className` | string | `nginx` | |
| `ingress.annotations` | object | `{}` | Merged with the cert-manager annotation the chart always sets. An S3 endpoint behind nginx usually wants `nginx.ingress.kubernetes.io/proxy-body-size: "0"` — without it the controller's 1m default truncates uploads. |
## Traps
### The metadata is not in your volume backup
In any usable form, that is. **A file-system snapshot of a live metadata database is not a restore point**: taken during a metadata write it is corrupt, and Garage says so itself — which is why it recommends *its own* snapshot mechanism in all cases, not as an extra. So the Longhorn snapshot and the OVH backup cover the **blocks**; the index they point into is covered by `metadata_auto_snapshot_interval` and by nothing else.
This is the correction that reversed a premise of the original plan for this chart, and it is the general shape of it that is worth carrying elsewhere: see [the snapshot that was not a restore point](Lessons-Learned#the-snapshot-that-was-not-a-restore-point) in [Lessons learned](Lessons-Learned).
### The restore window is two snapshots wide
Garage keeps exactly **two** snapshots (`KEEP_SNAPSHOTS = 2`), takes the first one after `interval / 2` and every later one after `interval` times a random factor between 1.0 and 1.2. At the default `"6h"` the first snapshot exists about three hours after a fresh start, and from then on the oldest recovery point is between one and two and a half intervals old — **never more**.
Read that as the real retention: a corruption discovered on the next working day is older than every snapshot that still exists. Shortening the interval narrows the window, at the cost of a compacted copy of the database every time; a genuinely longer retention needs the snapshot directory itself to be backed up, which is what the second PVC makes possible but which no part of this chart does.
### `metadata_fsync` is off in Garage's defaults
This chart sets it to `true`, and that is the single setting most easily lost in a round of "clean up the obvious defaults". The reason it cannot be inherited: **Longhorn's block layer cannot help with a write the engine never handed to the kernel.** A write that was never `fsync`ed is in no replica and in no snapshot, no matter how many replicas the volume has. Garage's own example configuration pairs `metadata_fsync = true` with the snapshot interval, as this chart does. If `metadata_fsync` ever disappears from the rendered `garage.toml`, that is a regression, not a tidy-up.
### LMDB at replication factor 1 is a deviation from upstream's own advice
LMDB is Garage's default since 0.9 and the faster engine, but upstream ties that recommendation to replication factor ≥ 2 — because the corruption it can suffer after an unclean shutdown is normally repaired from a peer. **There is no peer here.** `metadata_fsync` and the snapshots lower the probability; what remains is the last snapshot.
`sqlite` is the defensible alternative for anyone who weighs a single-file database's durability higher than metadata throughput — with one mail server as the only consumer, that ceiling is unlikely to be reached. Switching an existing release means `garage convert-db`, offline, not a values change.
### The layout applies itself, and cannot be applied twice
`garage server --single-node` creates the cluster layout in-process before the S3, web and admin listeners are spawned. The trap it avoids is the CLI path's: `garage layout apply --version <N+1>` demands exactly the predecessor version plus one, and upstream warns "never make multiple calls … with the same value of the `--version` flag". The flag does not exist in single-node mode, so the mistake is not available.
What makes it safe to leave on permanently is the version check:
| layout version at start | what happens |
|---|---|
| 0 | the layout is created (zone `dc1`, capacity from the detected disk size) |
| 1 | **nothing at all**, the server starts normally |
| above 1 | the start is **refused**: "layout version is already superior to 1" |
| any version, other nodes known | the start is **refused**: "more nodes are already present in the cluster" |
It also requires `replication_factor = 1` and says so loudly otherwise. So the flag cannot damage a cluster that has grown past it — it refuses instead.
### Rotating the S3 secret is not a re-save
Garage refuses to **start** when the access key already exists with a different secret: *"is associated with a secret key different than the one given in `GARAGE_DEFAULT_SECRET_KEY`"*. Changing `secret-key` in the vault item and restarting therefore produces a container that exits, not a rotated credential.
Two ways round it, both deliberate:
- put a **new** `access-key` id into the item together with the new secret. The old key stays and can be removed afterwards with `/garage key delete`, which also makes the rollover reversible for as long as both exist.
- or delete the old key first (`/garage key delete --yes <access-key>`) and then update both fields.
Either way the consuming application's item has to be updated in the same round, since it is the same item.
A **content** change inside a vault item restarts nothing by itself — Helm never sees it (there is nothing to hash), so after editing a field:
```bash
kubectl -n <namespace> rollout restart statefulset/<release>-garage-sfs
```
Repointing `secrets.rpcSecretRef` or `secrets.s3SecretRef` at a *different* item does roll the pod: that is covered by `checksum/secret-ref`.
### The RPC secret is an environment variable, and that is forced
Garage offers `rpc_secret_file`, and this chart does not use it. Not out of convenience — **the file form cannot work in a non-root pod**:
- `read_secret_file` rejects any file whose mode carries a group **or** other bit (`src/garage/secrets.rs`, mask `0o077`). The error says "world-readable"; the mask tests the group bits as well.
- The kubelet ORs the group-read bit onto every file of a secret volume as soon as `fsGroup` is set (`pkg/volume/volume_linux.go`, `Chmod(info.Mode() | 0440)` for a read-only volume). So with `fsGroup`, Garage refuses to start.
- Without `fsGroup` the file stays owned by root and the container's uid cannot read it at all.
- Kubernetes can set the **group** of such a file, never its **owner**. No combination satisfies both checks.
Rejected alternatives: `allow_world_readable_secrets = true` would switch the check off for **all** of Garage's secret files, including ones nobody has set yet; an init container copying the file with mode 0600 would need a second container image, because the main one has no shell. What remains is `GARAGE_RPC_SECRET` through `secretKeyRef` — nothing in the ConfigMap, no check disabled, and the dominant pattern in this repository anyway.
The residual price, stated rather than hidden: the secret is in the container's process environment, so anyone who can `exec` into the pod can read it. Garage declares the variable with `hide_env_values`, so it does not print it in its own output. [valkey](valkey) argues the opposite way for its password, and that argument is right **where the file form is available** — the difference is worth knowing before someone "aligns" the two charts. The general shape is in [Lessons learned](Lessons-Learned#a-permission-check-that-assumes-it-owns-the-filesystem).
### The image is `FROM scratch`, and that shapes the whole chart
The container holds exactly one file, `/garage`. No shell, no coreutils, no package manager. Three consequences that come up every time somebody changes this chart:
- **`exec` probes are impossible here, permanently.** All three probes are HTTP or TCP, and no future round can "just use a command probe".
- **An init container would need a second image** to pin, track and update. Nothing needs one: Garage creates `metadata_dir`, `data_dir` and the snapshots directory itself on start (`create_dir_all`), which is also what makes the two-subdirectories-on-one-PVC layout work.
- **Debugging is `/garage`, not a shell.** `kubectl exec -it … -- sh` fails and always will:
```bash
kubectl -n <namespace> exec <release>-garage-sfs-0 -- /garage status
kubectl -n <namespace> exec <release>-garage-sfs-0 -- /garage stats
```
### The probes point at two different things on purpose
- **Readiness** is `GET /health` on the admin port. It needs no token, and it maps the cluster health straight onto the status code: 200 for `Healthy` and `Degraded`, **503** for `Unavailable` — and an unconfigured layout is a special case that yields `Unavailable`.
- **Liveness and startup** are a TCP connect to the same port, and explicitly **not** `/health`.
The reason is the sharpest version of a lesson this repo already had from [zipline](zipline): a node waiting for its layout answers 503 indefinitely, so a liveness probe on `/health` would kill and restart it forever — and a restart is precisely what cannot fix it. A startup probe on `/health` would be worse, because the container would never leave the startup phase and could not be fixed by hand either.
And the inverse mistake is just as available: the **S3 listener binds independently of the layout state**, so a `tcpSocket` probe on 3900 answers long before the service can accept a request. A readiness probe there would report a service ready that accepts nothing.
### One node, and a second replica is not a cluster
`garage.replicas` is capped at 1 by the schema. A second pod would come up as a lone node with its own empty metadata database on its own volume, because this chart renders neither `bootstrap_peers` nor the `kubernetes_discovery` CRD. With `singleNode: true` it refuses to start once it sees the first node, which is the better of the two outcomes.
**The discovery CRD is deliberately absent.** `garagenodes.deuxfleurs.fr` is cluster-scoped, so rendering it would force the consuming bundle onto `defaultNamespace:` instead of `namespace:` — a cluster-wide object and a Fleet change for members a single node has no need to find.
### Why `OnRootMismatch` and not `Always`
`fsGroupChangePolicy: OnRootMismatch` deviates from this repo's default of `Always`, and the data is the reason: an object store's data directory holds **one file per block**, which is hundreds of thousands of files for a few hundred GB. `Always` chowns all of them recursively on **every** pod start, turning a restart into a multi-minute outage that gets worse as the store grows. `OnRootMismatch` checks the top directory's ownership and skips the walk when it already matches — which it does from the second start onwards. Same reasoning as [satisfactory](satisfactory)'s documented deviation.
### What a green pod proves here, and what it does not
`Ready` on this chart is a stronger statement than usual — it is literally an HTTP 200 from `/health`, so it proves a layout exists and the node has quorum. It still does not prove:
- that a client can **write**. Bucket permissions, the key's validity and the consumer's region all sit beyond the probe; step 5 of the checklist is the only test for that.
- that the **snapshots** are being written. The health endpoint knows nothing about the snapshot worker, and the first snapshot is due half an interval after the start. Step 6 is the check.
- anything about **durability**. `metadata_fsync` being set is visible in `garage.toml`; whether an `fsync` survived a power cut is not observable from inside a container at all.
## Upgrades and backup
- **An app-version bump is a restart, and a restart is a metadata question.** LMDB recovers from an unclean shutdown in most cases, but "most" is the whole problem at replication factor 1 — so prefer upgrading when a short outage is acceptable and the newest snapshot is recent, rather than at a moment when losing the index would be expensive.
- **Restoring the metadata** means: scale the release down (`scaleDown: true`), copy the chosen directory from `metadata_snapshots_dir` over `metadata_dir`, scale back up. The snapshot is a complete, compacted copy of the database rather than an increment, which is what makes that a file-level operation. Blocks that arrived after the snapshot are then present on disk but unknown to the index — `garage repair --yes blocks` is the command that reconciles that, and it is worth reading up on *before* needing it.
- **`helm uninstall` does not delete the PVCs.** The data survives a reinstall under the same release name; a release recreated under a *different* name starts empty while the old volumes keep occupying space.
- **Growing the store** is a PVC expansion on `storage.data.amount` plus — and this is the part that is easy to forget — a **layout change**, because the node's capacity is part of the layout. A larger volume with an unchanged capacity value is space Garage will not use.
## CI
`ci/garage/` holds `test-values.yaml` and `fixtures.yaml` (the two Secrets the operator would materialize, with values Garage actually validates). The chart is **not** excluded from the install-test, and the reason is `--single-node`: the pod reaches `Ready` by itself, so the test is an assertion rather than a formality.
A green run proves, on every PR:
- Garage accepts `metadata_dir` and `data_dir` as two **subdirectories of one PVC** and creates them itself — if that assumption were wrong the pod could not start, since the image has no `mkdir`.
- `readOnlyRootFilesystem: true` holds with **no** emptyDir anywhere.
- the layout really is applied in-process and `/health` answers 200 afterwards — `Ready` *is* that 200.
- the rendered `garage.toml` parses **as Garage reads it**, the Secret key names match what the StatefulSet references, and the imported access key and bucket pass Garage's own validation.
Three limits of that coverage, named so they are not mistaken for tested:
- **The snapshot is not observed.** The first one is due after half the interval, which is longer than the run; shortening the interval to seconds would test the timer rather than the mechanism. The CI values use `30m` so the worker is scheduled, and that is all.
- **Longhorn is not involved.** The snapshot semantics this chart's whole storage layout rests on are a property of the storage layer; the runner has `local-path`.
- **Durability is not testable.** Whether an `fsync` survives a power cut is not observable in a container.
## See also
- [stalwart](stalwart) — the first consumer; its blob store points at this release, and the two share the S3 vault item
- [Lessons learned](Lessons-Learned) — the snapshot that was not a restore point, and the permission check that assumes it owns the filesystem
- [Chart style guide](Chart-Style-Guide) — the rules this chart implements
- [Garage documentation](https://garagehq.deuxfleurs.fr/documentation/) — upstream, including the [known issues](https://garagehq.deuxfleurs.fr/documentation/reference-manual/known-issues/) page this page quotes