jlab-cluster/helm-charts

0 Fork 0

Table of Contents

garage

Prerequisites
Installation

What the 1Password items must contain

First install: the checklist

The manual path (garage.singleNode: false)

Configurable values

Global
secrets
garage — the node itself
storage
ingress
httpRoute
networkPolicy
Check your S3 consumers — this is the rule that will bite
The admin API is the probed port
The rpc bus needs a rule in both directions
The probed port is open inside the cluster by default
The policy is only a boundary where the CNI enforces one

Traps

Routing: Ingress or Gateway API
The metadata is not in your volume backup
The restore window is two snapshots wide
metadata_fsync is off in Garage's defaults
LMDB at replication factor 1 is a deviation from upstream's own advice
The layout applies itself, and cannot be applied twice
Rotating the S3 secret is not a re-save
The RPC secret is an environment variable, and that is forced
The image is FROM scratch, and that shapes the whole chart
The probes point at two different things on purpose
One node, and a second replica is not a cluster
Why OnRootMismatch and not Always
What a green pod proves here, and what it does not

Upgrades and backup
CI
See also

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

garage

Deploys Garage — an S3-compatible object store written in Rust — as one node with replication_factor = 1. The first consumer is stalwart's blob store, i.e. the bodies of the mail server's messages, and the deployment target is an encrypted, three-times-replicated Longhorn volume: Garage replication on top of that would be nine times the raw data across two layers that know nothing about each other.

That trade is deliberate, and the price is the first thing to understand about this chart: the replication factor applies to the metadata as well. Upstream writes of factor 1 "Do not use this for anything else than test deployments" and calls a corrupted metadata database "generally not recoverable" — and the volume layer does not help there, because it replicates a corrupted file exactly as faithfully as a good one. Redundancy against device failure, yes; against file corruption, no. Everything in Traps follows from that sentence.


Image	`docker.io/dxflrs/garage` — the multi-arch image. Upstream's own in-tree chart pins `dxflrs/amd64_garage`, which would tie the deployment to one architecture
Versions	not listed on purpose — read the current pair off `garage/Chart.yaml`, see Chart index
Workloads	StatefulSet, 1 replica, no init container (the image has no shell — see The image is `FROM scratch`)
Storage	two RWO PVCs: one for metadata and data, one for the metadata snapshots
Routing	both modes off by default, built to the convention so either can be switched on: Ingress (`ingress.enabled`) or Gateway API (`httpRoute.enabled`), mutually exclusive — see Routing: Ingress or Gateway API
External dependencies	none — no database, no cache, no other service
Secrets	2 × `OnePasswordItem`, four fields in total
Network	ClusterIP Service on 3900 (S3) and 3903 (admin). No RPC port in the Service, and no NetworkPolicy yet. The admin port serves two things unauthenticated — `/health` (token-free by design, the probes need it) and `/metrics` (`metrics_require_token` defaults to false) — and answers everything else with 401/403 until an admin token exists, from the config or from `garage admin-token create`
Install-test	`ci/garage/` (fixtures + test values)

Prerequisites

1Password operator in the cluster, plus two items — see What the 1Password items must contain. Four fields, each of them validated by Garage at start: a malformed one is a container that exits, not a warning.
A storage class for both PVCs. The intended value is longhorn-encrypted-ns, which resolves ${pvc.namespace}-pvc-encryption-secret — without that entry in the namespace the PVCs stay Pending. The volume-encryption bundle belongs to cluster-config, not to this chart.
The Longhorn backup labels, also from cluster-config: recurring-job.longhorn.io/daily-snapshot and …/daily-backup on the volumes. They do not apply automatically — as of 2026-10-02, 41 of 62 volumes in the cluster carried none. Without them the off-site half of the backup plan does not exist; it only looks like it does. And read The metadata is not in your volume backup before trusting even the half that does exist.
Nothing else.

Installation

The registry URL and its credentials live in 1Password (item jlab-registry-access: username, password, charts-url) and never in a repository or a shell history file.

helm repo add jlab "$CHARTS_URL" --username "$REG_USER" --password "$REG_PASS"
helm repo update

helm install garage jlab/garage \
  --namespace garage --create-namespace \
  -f my-values.yaml

Minimal my-values.yaml — two required values, because everything else has a default that is safe for this deployment:

secrets:
  rpcSecretRef: op://vault/garage-rpc
  s3SecretRef: op://vault/garage-s3

That installs a node that configures itself: it creates its own cluster layout, imports the access key and creates the bucket, all before it opens a listener. Sizing the volumes is the one thing worth doing in the same file:

storage:
  data:
    amount: 200Gi
    storageClass: longhorn-encrypted-ns
  snapshots:
    amount: 10Gi
    storageClass: longhorn-encrypted-ns

The same installation in Gateway mode instead — the Gateway, its class and its certificate belong to the cluster, not to this release:

ingress:
  enabled: false
  domain: <the public hostname>   # the HTTPRoute's hostname, see below
httpRoute:
  enabled: true
  parentRefs:
    - name: public
      namespace: gateway
      sectionName: https

This needs the Gateway-API CRDs installed and a Gateway whose listener admits routes from this namespace; ingress.clusterIssuer is not used. Both modes are off by default here, so this is an opt-in either way. See Routing: Ingress or Gateway API.

helm install prints the chart's NOTES.txt, which names the four vault fields, the verification sequence for this release, and the traps in short form. Read it — it is the per-release version of this page.

What the 1Password items must contain

The chart renders two OnePasswordItems; the operator turns each into a Kubernetes Secret whose keys are the field names of the 1Password item. A missing or misnamed field leaves the pod in CreateContainerConfigError.

Values key	Kubernetes Secret	Field	Content
`secrets.rpcSecretRef`	`<release>-garage-rpc-secret`	`rpc-secret`	32 bytes of random hex (`openssl rand -hex 32`). Required even for a single node: Garage uses it as the network key of its RPC layer and refuses to start without it
`secrets.s3SecretRef`	`<release>-garage-s3-secret`	`access-key`	the S3 key id. At least 8 characters, ASCII alphanumeric plus `-`, `_`, `.`
		`secret-key`	at least 16 printable ASCII characters (`openssl rand -hex 32`)
		`bucket`	a valid S3 bucket name, 3–63 characters, lowercase

Two items rather than one, because they have different audiences. The RPC secret belongs to this node alone. The S3 key is the credential the consumer uses — stalwart's secrets.s3SecretRef points at this same item, which is why the bucket name lives in it as a third field: endpoint, bucket, key and secret then have one source of truth instead of being kept in step by hand.

And the direction is inwards. With the defaults, Garage imports access-key, secret-key and bucket on start. The vault is written first and the server adopts it; nothing has to be copied out of a terminal afterwards. That is the opposite of how a Garage cluster is normally set up, and it is the reason the first install needs no operator at all.

First install: the checklist

With garage.singleNode: true (the default) there is nothing to do by hand. What follows is verification, in order. Every command uses /garage directly, because the image has no shell.

The rollout.
```
kubectl -n garage rollout status statefulset/garage-garage-sfs
```
Ready is not a formality here: it means the admin API answered 200 on /health, which means a layout exists and the node has quorum. A running container that stays not-Ready means the layout is missing — look for Created initial layout for single-node configuration in the log.
The node has a role.
```
kubectl -n garage exec garage-garage-sfs-0 -- /garage status
```
One node, zone dc1, with a capacity Garage took from the detected disk size. A node without a role means no layout was applied.

The layout.

kubectl -n garage exec garage-garage-sfs-0 -- /garage layout show

The bucket and the key.

kubectl -n garage exec garage-garage-sfs-0 -- /garage bucket list
kubectl -n garage exec garage-garage-sfs-0 -- /garage bucket info <bucket>

The bucket exists and the imported key has read/write/owner on it.

One write through the S3 API, from a client pod. It is the only step that proves the endpoint rather than the process. In-cluster address:
```
http://<release>-garage-service.<namespace>.svc.cluster.local:3900
```
with path-style addressing (bucket in the path) and the region from garage.s3.region.
Confirm the snapshot mechanism once, roughly metadata_auto_snapshot_interval / 2 after the start — this is the one check the pipeline cannot do for you:
```
kubectl -n garage exec garage-garage-sfs-0 -- /garage worker list
```
and look for the metadata snapshot worker's "Next snapshot" line; the log prints Snapshotting metadata db to /mnt/garage-snapshots/<timestamp> when it runs.

The manual path (`garage.singleNode: false`)

Only for a deployment that is meant to grow into a real cluster. The release then serves nothing until a layout is applied, and the pod stays not-Ready — which is correct rather than broken, because /health answers 503 while there is no layout and nothing should route to the endpoint yet.

# 1. node id and current layout version (0 on a fresh install)
kubectl -n garage exec garage-garage-sfs-0 -- /garage status

# 2. assign a role. Capacity takes BYTE suffixes (B, KB, MB, GB, TB, PB) -
#    not Kubernetes' Gi/Mi, so 10GB, never 10Gi.
kubectl -n garage exec garage-garage-sfs-0 -- \
  /garage layout assign -z dc1 -c 200GB <node-id>

# 3. read the staged layout BEFORE applying it
kubectl -n garage exec garage-garage-sfs-0 -- /garage layout show

# 4. apply it. N is the version from step 1, so a fresh cluster takes 1.
kubectl -n garage exec garage-garage-sfs-0 -- /garage layout apply --version <N+1>

# 5. bucket and key - "import", not "create", so the credential still comes
#    FROM the vault item instead of travelling the other way
kubectl -n garage exec garage-garage-sfs-0 -- /garage bucket create <bucket>
kubectl -n garage exec garage-garage-sfs-0 -- \
  /garage key import --yes <access-key> <secret-key>
kubectl -n garage exec garage-garage-sfs-0 -- \
  /garage bucket allow --read --write --owner --key <access-key> <bucket>

garage.defaultAccessKey and garage.defaultBucket must be false in this mode — the chart fails the render on the mixed combination, because Garage refuses it at start.

Configurable values

Global

Key	Type	Default	Description
`scaleDown`	bool	`false`	Renders the StatefulSet with `replicas: 0` without uninstalling. Takes precedence over `garage.replicas`. Both PVCs, the Secrets, the Service and the Ingress stay.

`secrets`

Key	Type	Default	Description
`secrets.rpcSecretRef`	string	`null`	Required always. `op://` path of the item carrying `rpc-secret`.
`secrets.s3SecretRef`	string	`null`	Required while `garage.singleNode` is on together with `defaultAccessKey` or `defaultBucket` (the default), because Garage imports those values on start.

`garage` — the node itself

Key	Type	Default	Description
`garage.replicas`	int	`1`	`0` or `1` — the schema rejects anything higher. See One node, and a second replica is not a cluster.
`garage.singleNode`	bool	`true`	Starts `garage server --single-node`, which creates the cluster layout in-process before any listener opens. Idempotent and unable to damage an existing cluster — see The layout applies itself.
`garage.defaultAccessKey`	bool	`true`	Imports the S3 key from the vault item (`--default-access-key`). Requires `singleNode`.
`garage.defaultBucket`	bool	`true`	Creates the bucket from the item's `bucket` field and grants the imported key access (`--default-bucket`). Requires `singleNode`.
`garage.replicationFactor`	int	`1`	Not an upgrade path: the factor is part of the cluster layout, so raising it is a layout operation with data movement on a cluster that would first need more nodes. The value exists so the chart does not block a future multi-node setup.
`garage.dbEngine`	string	`lmdb`	`lmdb`, `sqlite` or the experimental `fjall`. Changing it on an existing release needs `garage convert-db`, not a values change. See LMDB at factor 1.
`garage.metadataFsync`	bool	`true`	On, against Garage's own default of off. See `metadata_fsync` is off upstream.
`garage.dataFsync`	bool	`false`	Garage's default. A lost block is one object and is replaceable; a lost metadata write can take the index with it. Turning it on costs an `fsync` per written block.
`garage.metadataAutoSnapshotInterval`	string	`"6h"`	Go-style duration. The only consistent recovery point this deployment has. An empty string is rejected by the schema; `null` falls back to the default, because switching the snapshots off is not a configuration this chart offers at factor 1. See the window.
`garage.blockSize`	string	`"1M"`	Chunk size objects are split into.
`garage.compressionLevel`	int or `"none"`	`1`	zstd level for stored blocks; negative values are zstd's fast modes. To switch compression off, write the string `"none"` — not `0`. Garage accepts an integer or the literal `"none"`, and `0` is a valid integer that zstd reads as its default level (3), i.e. more compression than `1`, not none. The schema rejects any other string, and the ConfigMap quotes the `"none"` form so the rendered TOML stays valid.
`garage.bindAddress`	string	`"[::]"`	Host part of the S3, web and RPC listeners. Accepts IPv4 as well while `net.ipv6.bindv6only` is `0` — the Linux default, but a node sysctl the chart cannot see.
`garage.adminBindAddress`	string	`"0.0.0.0"`	Host part of the admin listener, separate on purpose: all three probes use this port, and a probe that cannot connect would be a pod that never becomes Ready for a reason nothing in the manifest shows. On an IPv6-only cluster set `"[::]"`.
`garage.ports.s3` / `.rpc` / `.web` / `.admin`	int	`3900` / `3901` / `3902` / `3903`	In one place: the ConfigMap renders them into `garage.toml`'s bind addresses and the StatefulSet renders them as `containerPort`s. The probes address the admin port by name, so they follow a change here.
`garage.s3.region`	string	`garage`	A label both sides must agree on, not a cloud region — a consumer configured for a different one gets a signature error, not a redirect.
`garage.s3.rootDomain`	string	`null`	Suffix for vhost-style bucket access (`<bucket>.<rootDomain>`). Only useful with an Ingress and a wildcard DNS entry; path-style works without it. Deliberately not coupled to `ingress.domain`.
`garage.web.enabled`	bool	`false`	Static website hosting (the `s3_web` endpoint). A second HTTP listener that serves nothing at all without a `rootDomain`, while the consumer speaks S3.
`garage.web.rootDomain`	string	`null`
`garage.securityContext.pod`	object	non-root 20000, `runAsGroup`/`fsGroup` 1000, `fsGroupChangePolicy: OnRootMismatch`, `seccompProfile: RuntimeDefault`	Passed verbatim into the pod spec. `OnRootMismatch` is a documented deviation — see Why `OnRootMismatch`. Any uid works: the image has no `/etc/passwd`. Upstream's own chart uses 1000, which matters when migrating from it.
`garage.securityContext.container`	object	`readOnlyRootFilesystem: true`, all caps dropped, no privilege escalation, non-root	No capability is added and none is needed: the ports are above 1024 and the binary carries no file capabilities.
`garage.livenessProbe`	object	`tcpSocket` on `admin`, period 10s, timeout 5s, 3 failures	Deliberately not `/health` — see the probes.
`garage.readinessProbe`	object	`httpGet /health` on `admin`, period 10s, timeout 5s, 3 failures	The honest readiness signal: 503 until a layout exists.
`garage.startupProbe`	object	`tcpSocket` on `admin`, period 5s, 24 failures (= 120s)	Gates liveness and readiness. The budget covers mounting two network volumes and opening (or recovering) the LMDB database, not Garage's own startup work, which is sub-second.
`garage.resources.limitFactor`	number	`3`	Memory and ephemeral-storage limits are computed as factor × request.
`garage.resources.requests`	object	`cpu: 100m`, `memory: 512Mi`, `ephemeral-storage: 32Mi`	512Mi × 3 = 1536Mi memory limit; Garage's own guidance is "at least 1GB" per node. The ephemeral request covers container logs only — everything Garage writes goes to one of the PVCs.
`garage.resources.limits`	object	unset	An explicit limit always wins. A CPU limit is only rendered when set here.
`garage.env`	list	`[]`	Extra environment variables, rendered through `tpl`; `value`, `secretKeyRef` and `configMapKeyRef` forms. Additive, not overriding — see Environment variables. Do not try to set `GARAGE_RPC_SECRET` or the `GARAGE_DEFAULT_*` variables this way; change the values that produce them. `RUST_LOG` is the useful entry.

`storage`

Key	Type	Default	Description
`storage.data.amount`	string	`10Gi`	`metadata_dir` and `data_dir`, as two subdirectories of this one volume. The value most likely to need changing before the first install.
`storage.data.storageClass`	string	`null`	`null` uses the cluster default — which is not encrypted. For a mail blob store that is a decision about data at rest, so the intended value is `longhorn-encrypted-ns`.
`storage.snapshots.amount`	string	`5Gi`	`metadata_snapshots_dir`. Sized for two compacted copies of the metadata database plus head room, not for the objects.
`storage.snapshots.storageClass`	string	`null`

Neither block is null-safe, on purpose: an erased size yields an invalid PVC that is rejected at apply time, which is loud. A chart-side fallback would silently size somebody's object store — and on an existing release it could not help anyway, because a PVC cannot shrink.

`ingress`

Key	Type	Default	Description
`ingress.enabled`	bool	`false`	Off, because the consumer speaks S3 inside the cluster. Switching it on publishes the S3 API — there is no web UI to expose. Mutually exclusive with `httpRoute.enabled`.
`ingress.clusterIssuer`	string	`null`	Required while enabled.
`ingress.domain`	string	`null`	Required while either routing mode is on. Checked in both directions (§12.1), and the answer moved with #114: the `HTTPRoute` reads it as its hostname when `httpRoute.hostnames` is empty, so `domain` is free only while both modes are off. `clusterIssuer` is unchanged — read by the Ingress alone. The vhost suffix stays its own value (`garage.s3.rootDomain`).
`ingress.className`	string	`nginx`
`ingress.annotations`	object	`{}`	Merged with the cert-manager annotation the chart always sets. An S3 endpoint behind nginx usually wants `nginx.ingress.kubernetes.io/proxy-body-size: "0"` — without it the controller's 1m default truncates uploads. That one has no counterpart in Gateway mode and it matters here more than anywhere: see Routing: Ingress or Gateway API.

`httpRoute`

The Gateway-API alternative to the Ingress, off by default. Added with group 1 of the Gateway-API sweep (Gitea #114); the form is the pilot's and is binding for every chart (style guide §4).

Key	Type	Default	Description
`httpRoute.enabled`	bool	`false`	Renders `<release>-garage-httproute` instead of the Ingress. Mutually exclusive with `ingress.enabled` — with both `true` the render aborts naming both values.
`httpRoute.parentRefs`	list	`[]`	The Gateway(s) the route attaches to — Gateway API `ParentReference` entries (`name`, and optionally `namespace`, `sectionName`, `port`, `group`, `kind`), passed through unchanged. Required while `httpRoute.enabled` is true: the render aborts on an empty list. The schema requires `name` on every entry and is otherwise left open, because Kubernetes owns that schema and adds fields to it across releases.
`httpRoute.hostnames`	list	`[]`	Empty means `[ingress.domain]`, so the hostname has a single source of truth. Set it explicitly only to publish under a different or an additional name — which is how a migration off the Ingress runs.

What the chart renders in this mode is only the HTTPRoute: one rule, PathPrefix: /, one backendRefs entry pointing at <release>-garage-service on port garage.ports.s3 (default 3900).

This is the one chart in the repo whose backendRefs port is rendered from a value. garage.ingress.yaml references the Service port by name (s3) and is therefore immune to the number changing; backendRefs takes a PortNumber and cannot name a port, so a literal 3900 in the route would become a lie the moment garage.ports.s3 is changed. The route renders the number from the same garage.settings helper the Service renders its port from, so the two cannot drift. The admin port is deliberately not routed in either mode: with this chart's configuration it serves /health and /metrics unauthenticated and answers everything else 401/403, which is a monitoring surface for the cluster, not something to publish.

`networkPolicy`

Garage's four ports split three ways, which makes this the clearest illustration of the convention after keycloak:

s3 (3900) and web (3902): restricted by source — this namespace, allowedNamespaces, and the ingress controller. No probe touches either, which is what makes the restriction possible.
admin (3903): the port all three probes address (tcpSocket twice and a httpGet /health), so it stays open inside the cluster — and it is also the admin API.
rpc (3901): this release's own pods only. It has no Service at all.

Outbound: DNS and the rpc bus, nothing else. All four numbers are derived from garage.settings, so moving a port in values moves the policy with it.

Value	Type	Default	Meaning
`networkPolicy.enabled`	bool	`true`	Renders the policy. Only a boundary where the CNI enforces one — unenforced, the API server accepts it and nothing is filtered, so everything reads as configured.
`networkPolicy.allowedNamespaces`	list	`[]`	Namespaces whose pods may reach the restricted ports, additionally to the release's own. Matched on the `kubernetes.io/metadata.name` label Kubernetes sets itself — no labelling work, and no typo protection either: a name that does not exist matches nothing and is silently ineffective. Namespace-granular, so an entry opens the port to every pod in it.
`networkPolicy.probeSources`	list	`[]`	CIDRs allowed to reach the probed port. Empty means open, not blocked — the rule is then rendered with no `from` at all. Probes come from the IP of the worker node running the pod, which the chart cannot know; a list that misses it drops every probe and the pod never becomes `Ready`.
`networkPolicy.ingressController`	object	`kube-system` / `app.kubernetes.io/name: rke2-traefik`	The controller allowed to reach the routed port once `probeSources` restricts it. The two keys render as one `from` entry, i.e. an AND — which is why this is not a line in `allowedNamespaces`, where it would open the port to every pod in `kube-system`. Default measured on the cluster, not assumed.
`networkPolicy.egress.extraPorts`	list	`[]`	Additional outbound TCP ports, to any destination. The escape hatch from the egress cap, and the only alternative to switching the policy off.

Check your S3 consumers — this is the rule that will bite

Anything in another namespace that talks to this endpoint on its in-cluster Service address — a backup job, a Zipline storing uploads, a Stalwart blob store — is blocked until that namespace is in networkPolicy.allowedNamespaces. Consumers using the published Ingress name arrive through the controller and are already covered.

The failure is an application-level error on the consumer's side, with nothing pointing at this object.

The admin API is the probed port

Key management and cluster layout live on 3903, and all three probes address it — so the cap is reachable from inside the cluster, not closed. networkPolicy.probeSources closes the rest, at the usual price of needing the worker node IPs.

The rpc bus needs a rule in both directions

policyTypes lists Egress, so a node dialling its peer is subject to this policy too. With only the inbound rule the bus would be permitted one way and the layout would never converge, while every node looked healthy.

Not permitted, and the plausible surprise: outbound HTTPS. Garage does not need it to serve S3, but remote replication or an external metrics sink does — networkPolicy.egress.extraPorts.

The probed port is open inside the cluster by default

networkPolicy.probeSources is empty out of the box, so the rule for the port the probes address is rendered with no from selector at all — any pod the CNI routes to can reach it. Not from the internet (the port carries no NodePort), but this is not pod-level isolation, and reading the object as if it were is the mistake to avoid.

It is that way on purpose. The kubelet's liveness, readiness and startup probes originate from the IP of the worker node running the pod — an address no chart can know and must not guess. A CIDR that excludes the real node blocks every probe: the pod never becomes Ready, the rollout times out, and the symptom reads as a broken application rather than as a policy. Shipping open is the honest default; narrowing it means listing the worker node IPs yourself.

Note the companion trap: the moment probeSources is non-empty the rule is restricted, so everything else that needs the port has to be in the same rule. The chart adds the release's own namespace, ingressController and allowedNamespaces for exactly that reason.

The policy is only a boundary where the CNI enforces one

The API server accepts a NetworkPolicy whatever the cluster does with it. Where the CNI does not enforce policies, everything looks configured and nothing is filtered — which is worse than no policy, because it invites the assumption that something is being filtered. Verify by connecting from an unrelated namespace rather than by reading it off the manifest.

Measured for this cluster: the CNI is Canal (Flannel for the network, Calico for policy), 6/6, and it does enforce. Measured for CI: k3s brings its own policy controller and enforces too — but it exempts the kubelet's probe traffic, so a blocked probe path does not turn the install-test red. What CI does catch is anything the application itself needs.

Traps

Routing: Ingress or Gateway API

Two mutually exclusive modes, and exactly one of them may be on. ingress.enabled ships false, httpRoute.enabled ships false.

Rule	Why it is a hard failure and not a default
`ingress.enabled` and `httpRoute.enabled` cannot both be true — the render aborts naming both value paths	two data paths onto one Service under one hostname is a conflict at DNS level already: one name, two addresses, and which one a client reaches is luck. A migration runs over a second hostname (`httpRoute.hostnames` set explicitly while the Ingress keeps `ingress.domain`), never in parallel.
`httpRoute.parentRefs` must not be empty — the render aborts	a route without a parent attaches to no Gateway. The object is created, it is schema-valid, no controller rejects it, and it routes nothing: no event, no condition, and the only symptom is a hostname that does not answer.
the mode is a value, never auto-detected	`.Capabilities.APIVersions` is unreliable at template time, and an `HTTPRoute` without the CRDs fails at install time. A chart that guessed would sometimes render an object the cluster rejects and sometimes silently render nothing.

What moves to the platform. In Gateway mode the chart renders no Gateway, no gatewayClassName, no Certificate and — deliberately — no implementation-specific policy object (ClientSettingsPolicy, ClientTrafficPolicy and their equivalents). Only HTTPRoute with parentRefs looks identical on nginx-gateway-fabric, Cilium, Envoy Gateway and Istio, and keeping to that subset is what lets the cluster's implementation be chosen, or replaced, without touching the chart again. Class and certificate therefore belong to the Gateway and its listeners, which live in cluster-config. ingress.clusterIssuer is unused and unnecessary in this mode.

YOUR UPLOAD LIMIT BECOMES THE GATEWAY'S, AND THIS IS AN OBJECT STORE. In Ingress mode the advice under ingress.annotations is proxy-body-size: "0", because ingress-nginx's 1 MB default truncates uploads. The Gateway API has no counterpart — it knows no body size at all; an upload limit is implementation-specific policy — and this chart renders no policy object, which would bind it to one implementation. So whatever limit the Gateway carries is the limit S3 PUTs get, and the symptom is uploads failing by size rather than by permission. Raise it at the Gateway or through that implementation's own policy object, next to the Gateway and not here. The Gateway-mode NOTES.txt prints this.

Worth noting against the issue that planned this sweep: #114 listed five charts as losing an upload limit in Gateway mode, measured on charts that hardcode the annotation. garage sets annotations: {} and so was not among them, but it depends on the limit all the same — it is a sixth case, and the distinction ("sets no behaviour-bearing annotation" is not "needs none") is now written down in the README.

Whether the route attached at all is only visible in its status, because no Kubernetes object reports a route that attaches to nothing:

kubectl -n <namespace> get httproute <release>-garage-httproute -o yaml

Accepted: True and ResolvedRefs: True under status.parents[] is a working attachment. An empty status means no controller has looked at the route at all — usually a parentRefs pointing at a Gateway that does not exist, or a listener whose allowedRoutes does not admit this namespace.

The metadata is not in your volume backup

In any usable form, that is. A file-system snapshot of a live metadata database is not a restore point: taken during a metadata write it is corrupt, and Garage says so itself — which is why it recommends its own snapshot mechanism in all cases, not as an extra. So the Longhorn snapshot and the OVH backup cover the blocks; the index they point into is covered by metadata_auto_snapshot_interval and by nothing else.

This is the correction that reversed a premise of the original plan for this chart, and it is the general shape of it that is worth carrying elsewhere: see the snapshot that was not a restore point in Lessons learned.

The restore window is two snapshots wide

Garage keeps exactly two snapshots (KEEP_SNAPSHOTS = 2), takes the first one after interval / 2 and every later one after interval times a random factor between 1.0 and 1.2. At the default "6h" the first snapshot exists about three hours after a fresh start, and from then on the oldest recovery point is between one and two and a half intervals old — never more.

Read that as the real retention: a corruption discovered on the next working day is older than every snapshot that still exists. Shortening the interval narrows the window, at the cost of a compacted copy of the database every time; a genuinely longer retention needs the snapshot directory itself to be backed up, which is what the second PVC makes possible but which no part of this chart does.

`metadata_fsync` is off in Garage's defaults

This chart sets it to true, and that is the single setting most easily lost in a round of "clean up the obvious defaults". The reason it cannot be inherited: Longhorn's block layer cannot help with a write the engine never handed to the kernel. A write that was never fsynced is in no replica and in no snapshot, no matter how many replicas the volume has. Garage's own example configuration pairs metadata_fsync = true with the snapshot interval, as this chart does. If metadata_fsync ever disappears from the rendered garage.toml, that is a regression, not a tidy-up.

LMDB at replication factor 1 is a deviation from upstream's own advice

LMDB is Garage's default since 0.9 and the faster engine, but upstream ties that recommendation to replication factor ≥ 2 — because the corruption it can suffer after an unclean shutdown is normally repaired from a peer. There is no peer here. metadata_fsync and the snapshots lower the probability; what remains is the last snapshot.

sqlite is the defensible alternative for anyone who weighs a single-file database's durability higher than metadata throughput — with one mail server as the only consumer, that ceiling is unlikely to be reached. Switching an existing release means garage convert-db, offline, not a values change.

The layout applies itself, and cannot be applied twice

garage server --single-node creates the cluster layout in-process before the S3, web and admin listeners are spawned. The trap it avoids is the CLI path's: garage layout apply --version <N+1> demands exactly the predecessor version plus one, and upstream warns "never make multiple calls … with the same value of the --version flag". The flag does not exist in single-node mode, so the mistake is not available.

What makes it safe to leave on permanently is the version check:

layout version at start	what happens
0	the layout is created (zone `dc1`, capacity from the detected disk size)
1	nothing at all, the server starts normally
above 1	the start is refused: "layout version is already superior to 1"
any version, other nodes known	the start is refused: "more nodes are already present in the cluster"

It also requires replication_factor = 1 and says so loudly otherwise. So the flag cannot damage a cluster that has grown past it — it refuses instead.

Rotating the S3 secret is not a re-save

Garage refuses to start when the access key already exists with a different secret: "is associated with a secret key different than the one given in GARAGE_DEFAULT_SECRET_KEY". Changing secret-key in the vault item and restarting therefore produces a container that exits, not a rotated credential.

Two ways round it, both deliberate:

put a new access-key id into the item together with the new secret. The old key stays and can be removed afterwards with /garage key delete, which also makes the rollover reversible for as long as both exist.
or delete the old key first (/garage key delete --yes <access-key>) and then update both fields.

Either way the consuming application's item has to be updated in the same round, since it is the same item.

A content change inside a vault item restarts nothing by itself — Helm never sees it (there is nothing to hash), so after editing a field:

kubectl -n <namespace> rollout restart statefulset/<release>-garage-sfs

Repointing secrets.rpcSecretRef or secrets.s3SecretRef at a different item does roll the pod: that is covered by checksum/secret-ref.

The RPC secret is an environment variable, and that is forced

Garage offers rpc_secret_file, and this chart does not use it. Not out of convenience — the file form cannot work in a non-root pod:

read_secret_file rejects any file whose mode carries a group or other bit (src/garage/secrets.rs, mask 0o077). The error says "world-readable"; the mask tests the group bits as well.
The kubelet ORs the group-read bit onto every file of a secret volume as soon as fsGroup is set (pkg/volume/volume_linux.go, Chmod(info.Mode() | 0440) for a read-only volume). So with fsGroup, Garage refuses to start.
Without fsGroup the file stays owned by root and the container's uid cannot read it at all.
Kubernetes can set the group of such a file, never its owner. No combination satisfies both checks.

Rejected alternatives: allow_world_readable_secrets = true would switch the check off for all of Garage's secret files, including ones nobody has set yet; an init container copying the file with mode 0600 would need a second container image, because the main one has no shell. What remains is GARAGE_RPC_SECRET through secretKeyRef — nothing in the ConfigMap, no check disabled, and the dominant pattern in this repository anyway.

The residual price, stated rather than hidden: the secret is in the container's process environment, so anyone who can exec into the pod can read it. Garage declares the variable with hide_env_values, so it does not print it in its own output. valkey argues the opposite way for its password, and that argument is right where the file form is available — the difference is worth knowing before someone "aligns" the two charts. The general shape is in Lessons learned.

The image is `FROM scratch`, and that shapes the whole chart

The container holds exactly one file, /garage. No shell, no coreutils, no package manager. Three consequences that come up every time somebody changes this chart:

exec probes are impossible here, permanently. All three probes are HTTP or TCP, and no future round can "just use a command probe".
An init container would need a second image to pin, track and update. Nothing needs one: Garage creates metadata_dir, data_dir and the snapshots directory itself on start (create_dir_all), which is also what makes the two-subdirectories-on-one-PVC layout work.

Debugging is /garage, not a shell. kubectl exec -it … -- sh fails and always will:

kubectl -n <namespace> exec <release>-garage-sfs-0 -- /garage status
kubectl -n <namespace> exec <release>-garage-sfs-0 -- /garage stats

The probes point at two different things on purpose

Readiness is GET /health on the admin port. It needs no token, and it maps the cluster health straight onto the status code: 200 for Healthy and Degraded, 503 for Unavailable — and an unconfigured layout is a special case that yields Unavailable.
Liveness and startup are a TCP connect to the same port, and explicitly not /health.

The reason is the sharpest version of a lesson this repo already had from zipline: a node waiting for its layout answers 503 indefinitely, so a liveness probe on /health would kill and restart it forever — and a restart is precisely what cannot fix it. A startup probe on /health would be worse, because the container would never leave the startup phase and could not be fixed by hand either.

And the inverse mistake is just as available: the S3 listener binds independently of the layout state, so a tcpSocket probe on 3900 answers long before the service can accept a request. A readiness probe there would report a service ready that accepts nothing.

One node, and a second replica is not a cluster

garage.replicas is capped at 1 by the schema. A second pod would come up as a lone node with its own empty metadata database on its own volume, because this chart renders neither bootstrap_peers nor the kubernetes_discovery CRD. With singleNode: true it refuses to start once it sees the first node, which is the better of the two outcomes.

The discovery CRD is deliberately absent. garagenodes.deuxfleurs.fr is cluster-scoped, so rendering it would force the consuming bundle onto defaultNamespace: instead of namespace: — a cluster-wide object and a Fleet change for members a single node has no need to find.

Why `OnRootMismatch` and not `Always`

fsGroupChangePolicy: OnRootMismatch deviates from this repo's default of Always, and the data is the reason: an object store's data directory holds one file per block, which is hundreds of thousands of files for a few hundred GB. Always chowns all of them recursively on every pod start, turning a restart into a multi-minute outage that gets worse as the store grows. OnRootMismatch checks the top directory's ownership and skips the walk when it already matches — which it does from the second start onwards. Same reasoning as satisfactory's documented deviation.

What a green pod proves here, and what it does not

Ready on this chart is a stronger statement than usual — it is literally an HTTP 200 from /health, so it proves a layout exists and the node has quorum. It still does not prove:

that a client can write. Bucket permissions, the key's validity and the consumer's region all sit beyond the probe; step 5 of the checklist is the only test for that.
that the snapshots are being written. The health endpoint knows nothing about the snapshot worker, and the first snapshot is due half an interval after the start. Step 6 is the check.
anything about durability. metadata_fsync being set is visible in garage.toml; whether an fsync survived a power cut is not observable from inside a container at all.

Upgrades and backup

An app-version bump is a restart, and a restart is a metadata question. LMDB recovers from an unclean shutdown in most cases, but "most" is the whole problem at replication factor 1 — so prefer upgrading when a short outage is acceptable and the newest snapshot is recent, rather than at a moment when losing the index would be expensive.
Restoring the metadata means: scale the release down (scaleDown: true), copy the chosen directory from metadata_snapshots_dir over metadata_dir, scale back up. The snapshot is a complete, compacted copy of the database rather than an increment, which is what makes that a file-level operation. Blocks that arrived after the snapshot are then present on disk but unknown to the index — garage repair --yes blocks is the command that reconciles that, and it is worth reading up on before needing it.
helm uninstall does not delete the PVCs. The data survives a reinstall under the same release name; a release recreated under a different name starts empty while the old volumes keep occupying space.
Growing the store is a PVC expansion on storage.data.amount plus — and this is the part that is easy to forget — a layout change, because the node's capacity is part of the layout. A larger volume with an unchanged capacity value is space Garage will not use.

CI

ci/garage/ holds test-values.yaml and fixtures.yaml (the two Secrets the operator would materialize, with values Garage actually validates). The chart is not excluded from the install-test, and the reason is --single-node: the pod reaches Ready by itself, so the test is an assertion rather than a formality.

A green run proves, on every PR:

Garage accepts metadata_dir and data_dir as two subdirectories of one PVC and creates them itself — if that assumption were wrong the pod could not start, since the image has no mkdir.
readOnlyRootFilesystem: true holds with no emptyDir anywhere.
the layout really is applied in-process and /health answers 200 afterwards — Ready is that 200.
the rendered garage.toml parses as Garage reads it, the Secret key names match what the StatefulSet references, and the imported access key and bucket pass Garage's own validation.

Three limits of that coverage, named so they are not mistaken for tested:

The snapshot is not observed. The first one is due after half the interval, which is longer than the run; shortening the interval to seconds would test the timer rather than the mechanism. The CI values use 30m so the worker is scheduled, and that is all.
Longhorn is not involved. The snapshot semantics this chart's whole storage layout rests on are a property of the storage layer; the runner has local-path.
Durability is not testable. Whether an fsync survives a power cut is not observable in a container.

The install-test keeps the Ingress on, and the route is held at render level — for this chart that is not a toss-up. ci/garage/test-values.yaml sets ingress.enabled: true deliberately, so the install proves the Service port the Ingress names by name (s3) exists; a typo there is otherwise invisible until somebody opens the URL. An HTTPRoute cannot make that reference at all — backendRefs takes a PortNumber — so switching the fixture into Gateway mode would have traded a check that catches a renamed port for one that does not, and given up garage's only Ingress coverage, for a CRD schema cyberchef's fixture already exercises against the real Gateway-API CRDs.

Instead ci/httproute/garage.yaml is a render-only overlay that switches this chart into Gateway mode for ci/run-httproute-match-test.sh. That test asserts every backendRefs entry names a Service and a port the same render creates — which is exactly the drift this chart risks, since its route renders the port from garage.ports.s3 rather than from a literal. A chart that has an overlay and renders no route fails the test, so the assertion cannot go vacuum-green. Neither half proves that traffic flows: no Gateway controller runs in CI. See CI/CD.