From 79b7e44fdc67787e71e29ea7120ce5488b6a0cfb Mon Sep 17 00:00:00 2001 From: Justin Klein Date: Sat, 3 Oct 2026 20:29:38 +0000 Subject: [PATCH 1/4] feat(garage): new chart for the Garage object store One Garage node with replication_factor = 1, built from template-chart to the style guide rather than copied from upstream's in-tree chart (Gitea #81). The single-node trade is deliberate and its price is named at the value: the factor applies to the metadata as well, where upstream calls a corrupted database "generally not recoverable" and a replicated volume replicates the corruption faithfully. The three countermeasures that work at factor 1 are therefore mandatory, not optional - metadata_fsync = true (off in Garage's defaults), the automatic metadata snapshots, and those snapshots on a second PVC. Storage: PVC 1 carries metadata_dir AND data_dir as two subdirectories, because volume snapshots are atomic per volume and that is what keeps index and blocks consistent with each other; PVC 2 carries metadata_snapshots_dir, because a recovery point must not die with the volume whose corruption it should repair. Garage creates all three directories itself, which matters because the image is FROM scratch and has no mkdir. Two corrections to the approved plan, both verified against the source at tag v2.4.1 and decided on the issue before implementation: - The RPC secret arrives as GARAGE_RPC_SECRET via secretKeyRef, not through rpc_secret_file. The file form cannot work in a non-root pod: read_secret_file rejects any mode with a group or other bit set (src/garage/secrets.rs:144, mask 0o077) while the kubelet OR-s the group-read bit onto every file of a secret volume once fsGroup is set (pkg/volume/volume_linux.go), and Kubernetes can set the group of such a file but never its owner. The mechanism is documented at the value so the next round does not "fix" it back. - The first start is done by the server, not by hand: garage server --single-node creates the layout in-process before any listener opens, is idempotent on the layout version and has no --version argument that could be reused, and --default-access-key/--default-bucket import key and bucket FROM the vault item. Both premises of the original "manual first start" decision fell with that. Probes follow from the same source reading: readiness is GET /health on the admin port, which answers 503 while no layout exists; liveness and startup are a TCP connect, because a probe that can only fail would restart a node waiting for its layout forever (same lesson as zipline, README 12.3). The S3 listener binds independently of the layout, so a TCP probe on 3900 would have been the lie the plan warned about. CI: fixtures under ci/garage/ rather than an exclusion - with --single-node the pod reaches Ready on its own, which makes the install-test an assertion about the layout, the one-PVC subdirectory layout and readOnlyRootFilesystem instead of a formality. The Renovate block carries a regex versioning anchored to ^v..$, because the registry receives 40-character commit-SHA tags almost daily (measured 2026-10-03: the newest 22 tags are all of that kind) and the four v1.99.x-internal builds sort above the v1.3.x line. Part of #81 --- .github/workflows/change-app-version.yml | 1 + .github/workflows/publish-garage.yml | 16 + README.md | 1 + ci/garage/fixtures.yaml | 45 +++ ci/garage/test-values.yaml | 103 ++++++ garage/.helmignore | 23 ++ garage/Chart.yaml | 26 ++ garage/templates/NOTES.txt | 144 ++++++++ garage/templates/_helpers.tpl | 392 +++++++++++++++++++++ garage/templates/garage.configmap.yaml | 146 ++++++++ garage/templates/garage.ingress.yaml | 50 +++ garage/templates/garage.pvc.yaml | 63 ++++ garage/templates/garage.secret.yaml | 71 ++++ garage/templates/garage.service.yaml | 43 +++ garage/templates/garage.sfs.yaml | 208 +++++++++++ garage/values.schema.json | 306 ++++++++++++++++ garage/values.yaml | 429 +++++++++++++++++++++++ renovate.json5 | 34 ++ 18 files changed, 2101 insertions(+) create mode 100644 .github/workflows/publish-garage.yml create mode 100644 ci/garage/fixtures.yaml create mode 100644 ci/garage/test-values.yaml create mode 100644 garage/.helmignore create mode 100644 garage/Chart.yaml create mode 100644 garage/templates/NOTES.txt create mode 100644 garage/templates/_helpers.tpl create mode 100644 garage/templates/garage.configmap.yaml create mode 100644 garage/templates/garage.ingress.yaml create mode 100644 garage/templates/garage.pvc.yaml create mode 100644 garage/templates/garage.secret.yaml create mode 100644 garage/templates/garage.service.yaml create mode 100644 garage/templates/garage.sfs.yaml create mode 100644 garage/values.schema.json create mode 100644 garage/values.yaml diff --git a/.github/workflows/change-app-version.yml b/.github/workflows/change-app-version.yml index 63c4a67..57f6815 100644 --- a/.github/workflows/change-app-version.yml +++ b/.github/workflows/change-app-version.yml @@ -41,6 +41,7 @@ on: - sure - valkey - stalwart + - garage # freesailarr is deliberately absent: none of its ten images takes # its tag from appVersion (they are all pinned in its values.yaml), # so this workflow would rewrite appVersion and republish the chart diff --git a/.github/workflows/publish-garage.yml b/.github/workflows/publish-garage.yml new file mode 100644 index 0000000..6421992 --- /dev/null +++ b/.github/workflows/publish-garage.yml @@ -0,0 +1,16 @@ +name: Publish Garage Helm Chart +on: + workflow_dispatch: + push: + branches: + - main + paths: + - garage/** +jobs: + publish-garage: + name: Publish Garage Helm Chart + uses: workflow-util/pipelines/.gitea/workflows/publish-helm-chart.yml@v3 + with: + chart-name: 'garage' + registry-owner: ${{ github.repository_owner }} + secrets: inherit diff --git a/README.md b/README.md index 6f0f476..090854c 100644 --- a/README.md +++ b/README.md @@ -19,6 +19,7 @@ Every chart has a wiki page of its own — `https://git.jk-dev.de/jlab-cluster/h | `calibre-web-automated` | Calibre-Web Automated | StatefulSet, extra consume-PVC | | `cyberchef` | CyberChef | Stateless Deployment | | `freesailarr` | gluetun VPN media stack (qbittorrent, prowlarr, flaresolverr, radarr, recyclarr, sonarr, seerr, bazarr, and mediathekarr off by default) | Single-pod StatefulSet: all apps share gluetun's network namespace; config volumes via volumeClaimTemplates, media PVCs as existingClaims (optional per-category `subPath`, so one claim can serve all three categories). One ClusterIP Service per app (mediathekarr's carries both of its ports); no global `ingress.domain`. Ingress in two shapes that mix: one host per app (default, each app's host required), or one shared `ingress.host` with the apps as sub-paths (documented ingress exception below) | +| `garage` | Garage (S3-compatible object store) | **One** node, `replication_factor = 1` — a deliberate trade with its price named (Gitea #81): the volume layer underneath already replicates and encrypts, so Garage replication on top would be nine times the raw data across two layers that know nothing about each other. What that costs is that the factor applies to the **metadata** too, where a corrupted database is upstream-documented as "generally not recoverable" and a replicated volume replicates the corruption. Hence three settings that are mandatory rather than optional: `metadata_fsync = true` (**off** in Garage's defaults), the automatic metadata snapshots, and those snapshots on a **second PVC** — a file-system snapshot of a live metadata database is not a restore point, so the volume backup covers the blocks and not the index. The first PVC carries `metadata_dir` **and** `data_dir` as two subdirectories, because volume snapshots are atomic per volume: that is what keeps index and blocks consistent with each other. Two further properties shape the whole chart: the image is **`FROM scratch`** and contains exactly one file, `/garage` — no shell, so **exec probes are impossible here** and an init container would need a second image — and the RPC secret has to arrive as an **environment variable**, because `rpc_secret_file` rejects any file with a group bit set while the kubelet sets one (the reasoning sits at the value). Started with `garage server --single-node`, which creates the cluster layout in-process before any listener opens and is idempotent on the layout version, and with `--default-access-key`/`--default-bucket`, which **import** key and bucket from the 1Password item — so the vault is the source of truth instead of a copy of what the server generated. Readiness is the admin API's `/health`, liveness and startup deliberately are **not**: `/health` answers 503 until a layout exists, so a node waiting for one would be restarted forever | | `gotenberg` | Gotenberg | Deployment + Service only (no ingress) | | `homeassistant` | Home Assistant | StatefulSet; the only chart carrying Helm tests (`templates/tests/`) | | `matrix-synapse` | Matrix Synapse | StatefulSet plus an nginx delegation Deployment/Service | diff --git a/ci/garage/fixtures.yaml b/ci/garage/fixtures.yaml new file mode 100644 index 0000000..9cba3b0 --- /dev/null +++ b/ci/garage/fixtures.yaml @@ -0,0 +1,45 @@ +# Fixtures for the garage install-test: the two Secrets the 1Password operator +# would otherwise materialize from the OnePasswordItems the chart renders +# (release name: ci, stub CRD in ci/common/). +# +# NO THROWAWAY INFRASTRUCTURE: Garage depends on nothing - no database, no +# cache, no other service. The whole deployment is one process and two volumes, +# which is also why this chart is cheap enough to test on every PR. +# +# THE FIELD NAMES MATTER AS MUCH AS THE VALUES, and here the VALUES matter too, +# which is unusual for a fixture: Garage validates all four of them on start and +# refuses to run on a malformed one. So these are not placeholders - a value +# that merely looks right (a 31-byte RPC secret, a 6-character access key) fails +# the run, which is exactly the check that makes the fixture worth having. +apiVersion: v1 +kind: Secret +metadata: + name: ci-garage-rpc-secret +stringData: + # EXACTLY 32 bytes, hex-encoded = 64 hex characters. Garage decodes this and + # rejects anything else with "Invalid RPC secret key: expected 32 bytes of + # random hex" - it is the network key of the RPC layer, required even for a + # single node. + # + # A throwaway value, and a visibly throwaway one: this is not a secret, it is + # a test vector. Nothing here survives the run, the cluster is deleted with + # the job, and the node it keys has no peers. + rpc-secret: "c1c1c1c1c1c1c1c1c1c1c1c1c1c1c1c1c1c1c1c1c1c1c1c1c1c1c1c1c1c1c1c1" +--- +apiVersion: v1 +kind: Secret +metadata: + name: ci-garage-s3-secret +stringData: + # The three fields Garage IMPORTS on start with --default-access-key and + # --default-bucket. Each one is validated (src/model/key_table.rs, + # src/model/bucket_alias_table.rs): + # access-key at least 8 characters, ASCII alphanumeric plus - _ . + # secret-key at least 16 characters, printable ASCII + # bucket a valid S3 bucket name, 3-63 characters, lowercase + # Shortening any of them below its minimum is the quickest way to check that + # this test really exercises the import path: the container exits instead of + # starting. + access-key: "ciGarageTestKey01" + secret-key: "c2c2c2c2c2c2c2c2c2c2c2c2c2c2c2c2c2c2c2c2c2c2c2c2c2c2c2c2c2c2c2c2" + bucket: "ci-garage-test" diff --git a/ci/garage/test-values.yaml b/ci/garage/test-values.yaml new file mode 100644 index 0000000..5fc2112 --- /dev/null +++ b/ci/garage/test-values.yaml @@ -0,0 +1,103 @@ +# CI values for the garage install-test. +# +# WHY THIS CHART IS TESTED RATHER THAN EXCLUDED, and what the test is worth: +# with garage.singleNode (the chart default) the server creates its own cluster +# layout before it opens any listener, so the pod reaches Ready without an +# operator. That is what makes an install-test possible here at all - without +# the flag the readiness probe would honestly never go green (no layout, no +# quorum, /health answers 503) and a green run would have required a probe that +# lies. +# +# WHAT A GREEN RUN PROVES - the four measurement points the plan for Gitea #81 +# left open, three of which this closes on every PR: +# +# 1. Garage accepts metadata_dir and data_dir as two SUBDIRECTORIES OF ONE +# PVC and creates them itself. The image is FROM scratch and has no mkdir, +# so if the assumption were wrong the pod could not start at all. +# 2. readOnlyRootFilesystem: true holds with no emptyDir anywhere - the +# container writes only into the two volumes. +# 3. The layout is really applied in-process and the admin API answers 200 on +# /health afterwards: Ready is exactly that HTTP 200, so the rollout +# status IS the assertion. +# Plus the mechanical half: the Secret key names match what the StatefulSet +# references, the rendered garage.toml PARSES AS GARAGE READS IT (it opens +# LMDB, validates replication_factor against --single-node and refuses +# malformed keys), the access key and the bucket are imported from the vault +# fields, the resource quantities are valid, and the schema accepts a +# realistic non-default values file. +# +# WHAT IT DOES NOT PROVE, and no install-test here could: +# - that the metadata auto-snapshot really lands in metadata_snapshots_dir. +# The first snapshot is taken after interval/2, and the shortest interval +# that is not absurd is still minutes - longer than the run. A snapshot +# interval of seconds would test the timer, not the thing. +# - anything about Longhorn: the snapshot semantics the chart's whole storage +# layout is built on are a property of the storage layer, and the runner +# has local-path. +# - durability. metadata_fsync is set, and whether an fsync survives a power +# cut is not observable in a container. + +secrets: + # Materialized by ci/garage/fixtures.yaml as ci-garage-rpc-secret and + # ci-garage-s3-secret (the OnePasswordItem stub CRD comes from ci/common/). + rpcSecretRef: op://ci/garage-rpc + s3SecretRef: op://ci/garage-s3 + +garage: + # Both ON, i.e. the chart's defaults, and they are the reason the run means + # something: the key and the bucket are imported FROM the fixture Secret, so + # a wrong field name or a malformed value fails the start rather than passing + # unnoticed. + singleNode: true + defaultAccessKey: true + defaultBucket: true + + # ON against the chart's default of off, together with a root domain. It + # renders a second listener, a second containerPort, a Service port and the + # [s3_web] section of garage.toml - all of which Garage has to accept, and + # none of which the default install exercises. + web: + enabled: true + rootDomain: ".web.garage.ci.example.com" + + s3: + region: garage-ci + rootDomain: ".s3.garage.ci.example.com" + + # Shortened from the chart's "6h" so the snapshot WORKER is created and + # scheduled during the run. It still does not take a snapshot - the first one + # is due after interval/2, i.e. 15 minutes here - and that gap is named above + # rather than papered over with a one-second interval that would only prove + # the timer fires. + metadataAutoSnapshotInterval: "30m" + + resources: + # Small, to stay inside the runner's budget, while still exercising the + # limit derivation: 256Mi x limitFactor 3 = 768Mi. + requests: + cpu: 100m + memory: 256Mi + ephemeral-storage: 32Mi + +storage: + data: + amount: 512Mi + # k3d's dynamic provisioner. The intended production value, + # longhorn-encrypted-ns, does not exist on the runner - and a pre-bound + # hostPath PV would be worse than this one, because the kubelet does not + # apply fsGroup to a hostPath volume at all, so the test would silently + # stop exercising the ownership path that matters for a volume the + # container did not create. + storageClass: local-path + snapshots: + amount: 256Mi + storageClass: local-path + +ingress: + # Left ON although no ingress controller answers in CI: the point is that the + # object is accepted and that the Service port it names by NAME ("s3") + # actually exists - a typo there is otherwise invisible until someone opens + # the URL. + enabled: true + domain: garage.ci.example.com + clusterIssuer: ci-selfsigned diff --git a/garage/.helmignore b/garage/.helmignore new file mode 100644 index 0000000..0e8a0eb --- /dev/null +++ b/garage/.helmignore @@ -0,0 +1,23 @@ +# Patterns to ignore when building packages. +# This supports shell glob matching, relative path matching, and +# negation (prefixed with !). Only one pattern per line. +.DS_Store +# Common VCS dirs +.git/ +.gitignore +.bzr/ +.bzrignore +.hg/ +.hgignore +.svn/ +# Common backup files +*.swp +*.bak +*.tmp +*.orig +*~ +# Various IDEs +.project +.idea/ +*.tmproj +.vscode/ diff --git a/garage/Chart.yaml b/garage/Chart.yaml new file mode 100644 index 0000000..6b4daad --- /dev/null +++ b/garage/Chart.yaml @@ -0,0 +1,26 @@ +apiVersion: v2 +name: garage +description: A Helm chart to deploy Garage, an S3-compatible object store, as a single node on Kubernetes +icon: https://garagehq.deuxfleurs.fr/images/garage-logo.svg + +type: application + +# appVersion is the exact image tag, and the "v" is part of it - so it is part +# of appVersion and of the "+up" suffix. Precedent: stalwart (+upv0.16.24), +# calibre-web-automated, matrix-synapse, ytdlp-web-player. +# +# The image is docker.io/dxflrs/garage, the MULTI-ARCH one. Upstream's own chart +# pins dxflrs/amd64_garage instead; that is a per-architecture image and pinning +# it would tie the chart to amd64 for no gain. +# +# THE TAG TRAP THIS VERSION HAS TO SURVIVE: next to the release tags the +# registry continuously receives COMMIT-SHA TAGS (40 hex characters, the newest +# of them dated 2026-10-03). A Renovate rule without a version constraint would +# offer those as "updates", which is why the garage block in renovate.json5 +# carries a versioningTemplate anchored to ^v..$ - same +# shape as the stalwart block, for the same reason. +version: 1.0.0+upv2.4.1 +appVersion: "v2.4.1" + +maintainers: + - name: jklein diff --git a/garage/templates/NOTES.txt b/garage/templates/NOTES.txt new file mode 100644 index 0000000..fd62cb9 --- /dev/null +++ b/garage/templates/NOTES.txt @@ -0,0 +1,144 @@ +{{- $settings := fromYaml (include "garage.settings" .) -}} +{{- $pod := printf "%s-%s-sfs-0" .Release.Name .Chart.Name -}} +{{- $svc := printf "%s-%s-service" .Release.Name .Chart.Name -}} +Garage {{ .Chart.AppVersion }} installed as release {{ .Release.Name }} in namespace {{ .Release.Namespace }}. + +ONE node, replication_factor = {{ $settings.replicationFactor | int64 }}, metadata engine {{ $settings.dbEngine }}. + +S3 ENDPOINT (in-cluster, path-style addressing): + http://{{ $svc }}.{{ .Release.Namespace }}.svc.cluster.local:{{ $settings.ports.s3 | int64 }} + region: {{ $settings.s3.region }} +{{- if .Values.ingress.enabled }} + published at: https://{{ .Values.ingress.domain }} +{{- end }} +{{- if $settings.web.enabled }} + static website endpoint on port {{ $settings.ports.web | int64 }}{{ with $settings.web.rootDomain }} (root domain {{ . }}){{ else }} - WITHOUT a rootDomain it serves nothing; set garage.web.rootDomain{{ end }} +{{- end }} + +THE 1PASSWORD ITEMS, AND WHICH FIELDS THEY MUST CARRY +The operator materializes one Secret key per item FIELD, so a missing or +misnamed field leaves the pod in CreateContainerConfigError. + + {{ .Values.secrets.rpcSecretRef }} + rpc-secret 32 bytes of random hex: openssl rand -hex 32 +{{ if .Values.secrets.s3SecretRef }} {{ .Values.secrets.s3SecretRef }} + access-key >= 8 chars, [A-Za-z0-9-_.]: openssl rand -hex 12 + secret-key >= 16 printable ASCII chars: openssl rand -hex 32 + bucket a valid S3 bucket name (3-63 chars, lowercase) +{{ else }} (no S3 key item configured - secrets.s3SecretRef is unset) +{{ end }} +{{- if and $settings.singleNode (or $settings.defaultAccessKey $settings.defaultBucket) }} +THE DIRECTION IS INWARDS: Garage IMPORTS{{ if $settings.defaultAccessKey }} the access key{{ end }}{{ if $settings.defaultBucket }}{{ if $settings.defaultAccessKey }} and{{ end }} the bucket{{ end }} from +that item on start. The vault is the source of truth and nothing has to be +copied out of the server. +{{- end }} + +FIRST START +{{ if $settings.singleNode }} +Nothing to do by hand. The server created its own cluster layout before it +opened any listener{{ if $settings.defaultAccessKey }}, imported the access key{{ end }}{{ if $settings.defaultBucket }} and created the bucket{{ end }}. +Verify it, in this order - all commands need /garage as the command, because +the image has no shell: + + 1. kubectl -n {{ .Release.Namespace }} rollout status statefulset/{{ .Release.Name }}-{{ .Chart.Name }}-sfs + Ready means the admin API answered 200 on /health, which means a layout + exists and the node has quorum. Not-Ready with a running container means + the layout is missing - check the logs for "Created initial layout". + + 2. kubectl -n {{ .Release.Namespace }} exec {{ $pod }} -- /garage status + One node, with a role (zone dc1, a capacity taken from the detected disk + size). A node WITHOUT a role means no layout was applied. + + 3. kubectl -n {{ .Release.Namespace }} exec {{ $pod }} -- /garage layout show +{{- if $settings.defaultBucket }} + + 4. kubectl -n {{ .Release.Namespace }} exec {{ $pod }} -- /garage bucket list + kubectl -n {{ .Release.Namespace }} exec {{ $pod }} -- /garage bucket info + The bucket exists and the imported key has read/write/owner on it. +{{- end }} + + 5. One write through the S3 API from a client pod, which is the only step + that proves the endpoint rather than the process. +{{ else }} +garage.singleNode is false, so THIS RELEASE SERVES NOTHING YET and the pod +will stay not-Ready until a layout exists - that is correct, not a defect: the +readiness probe reports /health, and /health answers 503 while there is no +layout. Apply one by hand, in this order: + + 1. kubectl -n {{ .Release.Namespace }} exec {{ $pod }} -- /garage status + Note the node id and the current layout version (0 on a fresh install). + + 2. kubectl -n {{ .Release.Namespace }} exec {{ $pod }} -- \ + /garage layout assign -z dc1 -c + The capacity is this node's usable size, roughly storage.data.amount + ({{ .Values.storage.data.amount }}). It takes BYTE suffixes B, KB, MB, GB, TB, PB - not + Kubernetes' Gi/Mi, so write e.g. 10GB, not 10Gi. + + 3. kubectl -n {{ .Release.Namespace }} exec {{ $pod }} -- /garage layout show + Read it before applying. The next step is NOT reversible by repeating it. + + 4. kubectl -n {{ .Release.Namespace }} exec {{ $pod }} -- /garage layout apply --version + N is the version from step 1, so a fresh cluster takes --version 1. + NEVER call this twice with the same --version: the flag is a guard + against applying a staged change you have not seen, and reusing a number + is how a layout is applied twice. + + 5. kubectl -n {{ .Release.Namespace }} exec {{ $pod }} -- /garage bucket create + kubectl -n {{ .Release.Namespace }} exec {{ $pod }} -- \ + /garage key import --yes + kubectl -n {{ .Release.Namespace }} exec {{ $pod }} -- \ + /garage bucket allow --read --write --owner --key + + "key import" rather than "key create", so the credential still comes FROM + the vault item. "key create" generates one and then it has to travel the + other way, through a terminal, with the vault as a copy. + + 6. The pod becomes Ready within one probe period after step 4. +{{ end }} +WHAT THIS CHART DOES NOT RENDER, AND WHAT IS MISSING WITHOUT IT + + - THE LONGHORN BACKUP LABELS. recurring-job.longhorn.io/daily-snapshot and + .../daily-backup belong to the consuming bundle, not to this chart. They do + NOT apply automatically: without them the off-site half of the backup plan + does not exist, it only looks like it does. + - The namespace's volume-encryption secret, if the StorageClass is + longhorn-encrypted-ns: it resolves + ${pvc.namespace}-pvc-encryption-secret, and without that entry the PVCs + stay Pending. + - No kubernetes_discovery CRD and no bootstrap peers: a single node has no + members to find. + +TRAPS WORTH KNOWING BEFORE YOU NEED THEM + + - THE METADATA IS NOT IN YOUR VOLUME BACKUP in any usable form. A + file-system snapshot of a live metadata database is not a restore point - + taken during a write it is corrupt. What IS a restore point are Garage's + own snapshots, on the second PVC, taken every {{ $settings.metadataAutoSnapshotInterval }}. + - THE WINDOW IS SHORT: exactly TWO snapshots are kept, the first after half + the interval and every later one after the interval times a random + 1.0-1.2. With metadata_auto_snapshot_interval = {{ $settings.metadataAutoSnapshotInterval }} the oldest + recovery point is therefore between one and two and a half intervals old, + never more. A corruption found later than that is older than every + snapshot that still exists. + - metadata_fsync is ON here and OFF in Garage's defaults. If you ever see it + missing from garage.toml, that is a regression, not a cleanup. + - replication_factor 1 applies to the METADATA as well. Upstream calls a + corrupted metadata database "generally not recoverable", and the volume + layer replicates a corrupted file as faithfully as a good one. + - ROTATING THE S3 SECRET IS NOT A RE-SAVE. Garage refuses to start when the + access key already exists with a different secret ("is associated with a + secret key different than the one given in GARAGE_DEFAULT_SECRET_KEY"). To + rotate: either put a NEW access-key id into the item (the old key stays and + can be deleted with "/garage key delete"), or delete the old key first and + then update both fields. + - A CHANGE INSIDE A VAULT ITEM RESTARTS NOTHING. Helm never sees it. After + changing a field: + kubectl -n {{ .Release.Namespace }} rollout restart statefulset/{{ .Release.Name }}-{{ .Chart.Name }}-sfs + (Repointing a secrets.*SecretRef at a different item DOES roll the pod - + that is covered by checksum/secret-ref.) + - THE IMAGE HAS NO SHELL. "kubectl exec -it ... -- sh" does not work and + never will; every command above uses /garage directly. Exec probes are + impossible in this chart for the same reason. + - THE RPC SECRET IS IN THE PROCESS ENVIRONMENT, not in a file. That is + forced, not chosen: Garage rejects a secret file with any group bit set, + and the kubelet sets one. values.yaml has the mechanism. diff --git a/garage/templates/_helpers.tpl b/garage/templates/_helpers.tpl new file mode 100644 index 0000000..7c309ea --- /dev/null +++ b/garage/templates/_helpers.tpl @@ -0,0 +1,392 @@ +{{/* +Render the body of a container "resources:" block from a values dict of the form + + resources: + limitFactor: 3 # optional, default 3 + requests: { cpu, memory, ephemeral-storage / ephemeralStorage, ... } + limits: { ... } # optional, every entry optional + +Rules (variant B, see issue #19): + - requests are rendered as given (camelCase "ephemeralStorage" is normalized + to the Kubernetes key "ephemeral-storage"). + - an explicitly set limit always wins. + - a missing memory / ephemeral-storage limit is computed as + limitFactor x the corresponding request. + - a missing cpu limit is NOT set at all (the kubelet then treats CPU as + unlimited; CPU is compressible, so this cannot kill the workload). + +NOTE: this helper is duplicated into every chart's templates/_helpers.tpl with +the chart's name as define prefix, because sharing it would mean a library +chart, and a dependency is admissible but expensive (README, guideline 14) - +this chart takes none. The reference implementation is +template-chart/templates/_helpers.tpl. +*/}} +{{- define "garage.resources" -}} +{{- $resources := . | default dict -}} +{{- $factor := 3.0 -}} +{{- if not (kindIs "invalid" $resources.limitFactor) -}} +{{- $factor = float64 $resources.limitFactor -}} +{{- end -}} +{{- $requests := dict -}} +{{- range $key, $value := ($resources.requests | default dict) -}} +{{- if not (kindIs "invalid" $value) -}} +{{- $_ := set $requests (include "garage.resourceKey" $key) $value -}} +{{- end -}} +{{- end -}} +{{- $limits := dict -}} +{{- range $key, $value := ($resources.limits | default dict) -}} +{{- if not (kindIs "invalid" $value) -}} +{{- $_ := set $limits (include "garage.resourceKey" $key) $value -}} +{{- end -}} +{{- end -}} +{{- range $key, $request := $requests -}} +{{- if and (ne $key "cpu") (not (hasKey $limits $key)) -}} +{{- $_ := set $limits $key (include "garage.multiplyQuantity" (dict "quantity" $request "factor" $factor)) -}} +{{- end -}} +{{- end -}} +{{- $out := dict -}} +{{- with $requests }}{{- $_ := set $out "requests" . }}{{- end -}} +{{- with $limits }}{{- $_ := set $out "limits" . }}{{- end -}} +{{- toYaml $out -}} +{{- end -}} + +{{/* +Normalize a resource name from values to its Kubernetes key. +*/}} +{{- define "garage.resourceKey" -}} +{{- if eq . "ephemeralStorage" -}}ephemeral-storage{{- else -}}{{ . }}{{- end -}} +{{- end -}} + +{{/* +Multiply a Kubernetes quantity by a factor, keeping its suffix. +Input: dict "quantity" "factor" . +*/}} +{{- define "garage.multiplyQuantity" -}} +{{- $quantity := include "garage.formatNumber" .quantity -}} +{{- $number := regexFind "^[0-9]+(\\.[0-9]+)?" $quantity -}} +{{- if not $number -}} +{{- fail (printf "cannot parse resource quantity %q" $quantity) -}} +{{- end -}} +{{- $suffix := trimPrefix $number $quantity -}} +{{- if not (regexMatch "^(m|k|M|G|T|P|E|Ki|Mi|Gi|Ti|Pi|Ei)?$" $suffix) -}} +{{- fail (printf "unsupported resource quantity suffix %q in %q" $suffix $quantity) -}} +{{- end -}} +{{- $product := mulf (float64 $number) .factor -}} +{{- if eq $product (floor $product) -}} +{{- printf "%d%s" (int64 $product) $suffix -}} +{{- else -}} +{{- printf "%s%s" (include "garage.formatNumber" $product) $suffix -}} +{{- end -}} +{{- end -}} + +{{/* +Render a number as a plain decimal string, never in exponential notation +(issue #157): the null-safe fallback helpers pass values through fromYaml, +which parses every number as float64, and Go's "%v" switches float64 to +exponential notation from about 1e6 upwards - "1.048576e+06" is not a valid +Kubernetes quantity. Strings ("512Mi") are passed through untouched. +*/}} +{{- define "garage.formatNumber" -}} +{{- if kindIs "float64" . -}} +{{- regexReplaceAll "\\.?0+$" (printf "%.10f" .) "" -}} +{{- else -}} +{{- printf "%v" . -}} +{{- end -}} +{{- end -}} + +{{/* +Render the replica count for the StatefulSet. + +Input: dict "scaleDown" "replicas" . + + - scaleDown: true takes precedence and always yields 0. + - otherwise an explicitly set replicas value is used as given, including an + explicit 0 - Helm's "default" would swallow 0, hence the kindIs "invalid" + nil check. + - a missing replicas value defaults to 1. + +A NOTE SPECIFIC TO THIS CHART: more than one replica is not a cluster. Garage +nodes find each other through RPC and a cluster LAYOUT, and this chart renders +neither peer bootstrapping nor the kubernetes_discovery CRD (see values.yaml). +A second replica would come up as a lone node with its own empty metadata db on +its own volume, and with --single-node it refuses to start at all once it sees +the other node. The value exists because guideline 11 requires it and because +0 is the documented way to stop the release; the schema caps it at 1. +*/}} +{{- define "garage.replicas" -}} +{{- if .scaleDown -}} +0 +{{- else if not (kindIs "invalid" .replicas) -}} +{{- .replicas -}} +{{- else -}} +1 +{{- end -}} +{{- end -}} + +{{/* +Merge a user-supplied values sub-tree onto a dict of chart defaults so that a +null / empty override falls back to the defaults instead of erasing them +(issues #61, #99). Input: dict "defaults" "given" "path" +. Output: YAML of the merged dict (consume it with fromYaml). + +An override key written without children is YAML null, and Helm treats that +null as a deletion of the chart's default. This helper defines the opposite +semantics - an empty block means "chart defaults apply" - while an explicitly +set value still wins, including false and 0. +*/}} +{{- define "garage.defaultedDict" -}} +{{- $defaults := .defaults | default dict -}} +{{- $given := .given -}} +{{- $path := .path | default "value" -}} +{{- if kindIs "invalid" $given -}} +{{- $given = dict -}} +{{- end -}} +{{- if not (kindIs "map" $given) -}} +{{- fail (printf "%s must be a mapping or null, got %s (%v)" $path (kindOf $given) $given) -}} +{{- end -}} +{{- $out := deepCopy $defaults -}} +{{- range $key, $value := $given -}} +{{- if not (kindIs "invalid" $value) -}} +{{- $default := index $defaults $key -}} +{{- if and (kindIs "map" $value) (kindIs "map" $default) -}} +{{- $_ := set $out $key (fromYaml (include "garage.defaultedDict" (dict "defaults" $default "given" $value "path" (printf "%s.%s" $path $key)))) -}} +{{- else if kindIs "map" $default -}} +{{- fail (printf "%s.%s must be a mapping or null, got %s (%v)" $path $key (kindOf $value) $value) -}} +{{- else -}} +{{- $_ := set $out $key $value -}} +{{- end -}} +{{- end -}} +{{- end -}} +{{- toYaml $out -}} +{{- end -}} + +{{/* +Pick one sub-block out of a values block that may itself be null. +Input: dict "block" "key" "path" . +Returns YAML of dict "value" . +*/}} +{{- define "garage.subBlock" -}} +{{- $block := .block -}} +{{- if kindIs "invalid" $block -}} +{{- $block = dict -}} +{{- end -}} +{{- if not (kindIs "map" $block) -}} +{{- fail (printf "%s must be a mapping or null, got %s (%v)" .path (kindOf $block) $block) -}} +{{- end -}} +{{- toYaml (dict "value" (index $block .key)) -}} +{{- end -}} + +{{/* +THE EFFECTIVE GARAGE SETTINGS - everything the ConfigMap and the StatefulSet +read out of .Values.garage, merged onto this chart's defaults. + +Why the whole settings block goes through the null-safe helper and not only the +three usual suspects (guideline 12.2: "the rule is about the shape, not about +those three block kinds"): every value below is dereferenced by a template, and +an erased one would not abort the render - it would render a garage.toml with +an empty right-hand side, or a containerPort with no number. The first produces +a pod that crash-loops on a config parse error, the second a manifest the API +server rejects at apply time. Both are failures that arrive after the render +looked fine, which is exactly the class #99 is about. Satisfactory shipped the +same shape with half its blocks null-safe and half not (Gitea #16), and the +half that was not took the Service's port with it. + +The defaults here MUST stay in sync with values.yaml, which remains the +documented, user-facing place for them; this copy is only the fallback. + +Usage: {{ $settings := fromYaml (include "garage.settings" .) }} +*/}} +{{- define "garage.settings" -}} +{{- $given := (fromYaml (include "garage.subBlock" (dict "block" .Values "key" "garage" "path" ".Values"))).value -}} +{{- $defaults := dict + "singleNode" true + "defaultAccessKey" true + "defaultBucket" true + "replicationFactor" 1 + "dbEngine" "lmdb" + "metadataFsync" true + "dataFsync" false + "metadataAutoSnapshotInterval" "6h" + "compressionLevel" 1 + "blockSize" "1M" + "bindAddress" "[::]" + "adminBindAddress" "0.0.0.0" + "ports" (dict "s3" 3900 "rpc" 3901 "web" 3902 "admin" 3903) + "s3" (dict "region" "garage" "rootDomain" nil) + "web" (dict "enabled" false "rootDomain" nil) -}} +{{- include "garage.defaultedDict" (dict "defaults" $defaults "given" $given "path" "garage") -}} +{{- end -}} + +{{/* +The three directories, defined once because TWO templates have to agree on +them: garage.toml names them and the StatefulSet mounts the volumes underneath +them. They are deliberately NOT values - a path that disagrees with its mount +is a pod that writes its metadata into the container's ephemeral layer, which +looks like it works until the pod is replaced. + +The split is the point of this chart's storage layout (see values.yaml): +metadata and data are two SUBDIRECTORIES OF ONE VOLUME, the metadata snapshots +sit on a SECOND one. Garage creates all three itself on start +(src/model/garage.rs, create_dir_all for metadata_dir and data_dir; +src/model/snapshot.rs for the snapshots dir), which matters here because the +image is FROM scratch and carries no mkdir - nothing else in this pod could +create them. +*/}} +{{- define "garage.dataVolumePath" -}}/mnt/garage{{- end -}} +{{- define "garage.metadataDir" -}}{{ include "garage.dataVolumePath" . }}/meta{{- end -}} +{{- define "garage.dataDir" -}}{{ include "garage.dataVolumePath" . }}/data{{- end -}} +{{- define "garage.snapshotsDir" -}}/mnt/garage-snapshots{{- end -}} + +{{/* +The effective security contexts, probes and resources (issue #99). + +The defaults below MUST stay in sync with values.yaml. They reproduce THIS +chart's defaults, not the repo standard - see the fsGroupChangePolicy note. +*/}} +{{- define "garage.podSecurityContext" -}} +{{- $given := (fromYaml (include "garage.subBlock" (dict "block" . "key" "pod" "path" "garage.securityContext"))).value -}} +{{- $defaults := dict + "runAsNonRoot" true + "runAsUser" 20000 + "runAsGroup" 1000 + "fsGroup" 1000 + "fsGroupChangePolicy" "OnRootMismatch" + "seccompProfile" (dict "type" "RuntimeDefault") -}} +{{- include "garage.defaultedDict" (dict "defaults" $defaults "given" $given "path" "garage.securityContext.pod") -}} +{{- end -}} + +{{- define "garage.containerSecurityContext" -}} +{{- $given := (fromYaml (include "garage.subBlock" (dict "block" . "key" "container" "path" "garage.securityContext"))).value -}} +{{- $defaults := dict + "allowPrivilegeEscalation" false + "readOnlyRootFilesystem" true + "runAsNonRoot" true + "capabilities" (dict "drop" (list "ALL") "add" (list)) -}} +{{- include "garage.defaultedDict" (dict "defaults" $defaults "given" $given "path" "garage.securityContext.container") -}} +{{- end -}} + +{{/* +LIVENESS: a TCP connect to the admin port, and NOT the /health endpoint the +readiness probe uses. The reason is the one zipline wrote down (README, +guideline 12.3) in its sharpest form: /health answers 503 until the cluster +LAYOUT exists (src/rpc/system.rs, "Layout not yet configured, special case" -> +ClusterHealthStatus::Unavailable -> 503 in src/api/admin/special.rs). A node +waiting for its layout - every install with garage.singleNode: false, and any +install whose layout was lost - would therefore be killed and restarted +forever, and a restart is precisely what cannot fix it. + +What the TCP probe does prove is what a liveness probe is for: the process is +alive and its listener is open. A wedged process that stops accepting is +restarted; a healthy process that has nothing to serve yet is not. +*/}} +{{- define "garage.livenessProbe" -}} +{{- $defaults := dict + "tcpSocket" (dict "port" "admin") + "periodSeconds" 10 + "timeoutSeconds" 5 + "failureThreshold" 3 -}} +{{- include "garage.defaultedDict" (dict "defaults" $defaults "given" . "path" "garage.livenessProbe") -}} +{{- end -}} + +{{/* +READINESS: GET /health on the admin port - the one endpoint that answers the +question a readiness probe asks, namely "can this node serve requests". + +Measured against the source at tag v2.4.1 rather than chosen: + + - the S3 API server is spawned independently of the layout state + (src/garage/server.rs, the S3/web/admin servers are all spawned after + initial_config), so the S3 socket binds and ANSWERS - with errors - before + any layout exists. A tcpSocket probe on port 3900 would therefore report a + service ready that accepts nothing, which is the lie this chart is not + allowed to tell. + - /health needs no token (@special in src/api/admin/router_v2.rs) and maps + ClusterHealthStatus straight onto the status code: Healthy and Degraded -> + 200, Unavailable -> 503 with "Quorum is not available for some/all + partitions, reads and writes will fail". An unconfigured layout is a + special case in src/rpc/system.rs and yields Unavailable. + +So with garage.singleNode: true the pod becomes Ready a moment after start +(initial_config applies the layout BEFORE the servers are spawned), and with +singleNode: false it stays not-Ready until an operator applies a layout by hand +- which is the truth, not a defect: until then the S3 endpoint accepts nothing +and no Service should route to it. +*/}} +{{- define "garage.readinessProbe" -}} +{{- $defaults := dict + "httpGet" (dict "path" "/health" "port" "admin") + "periodSeconds" 10 + "timeoutSeconds" 5 + "failureThreshold" 3 -}} +{{- include "garage.defaultedDict" (dict "defaults" $defaults "given" . "path" "garage.readinessProbe") -}} +{{- end -}} + +{{/* +STARTUP: TCP again, for the same reason as liveness - a startup probe that +fails forever kills the container, so it must not depend on a layout that only +an operator can create. + +Budget = periodSeconds * failureThreshold = 5 * 24 = 120s. It covers the slow +part of a real start, which is not Garage: attaching and mounting two network +volumes, plus opening (and on an unclean shutdown recovering) the LMDB +metadata database. Garage's own startup work after that is sub-second. +*/}} +{{- define "garage.startupProbe" -}} +{{- $defaults := dict + "tcpSocket" (dict "port" "admin") + "periodSeconds" 5 + "timeoutSeconds" 5 + "failureThreshold" 24 -}} +{{- include "garage.defaultedDict" (dict "defaults" $defaults "given" . "path" "garage.startupProbe") -}} +{{- end -}} + +{{/* +The effective resources block, fed into the resources helper above so that an +erased "resources:" still yields the chart's requests and computed limits. +*/}} +{{- define "garage.effectiveResources" -}} +{{- $defaults := dict + "limitFactor" 3 + "requests" (dict "cpu" "100m" "memory" "512Mi" "ephemeral-storage" "32Mi") -}} +{{- $merged := fromYaml (include "garage.defaultedDict" (dict "defaults" $defaults "given" . "path" "garage.resources")) -}} +{{- include "garage.resources" $merged -}} +{{- end -}} + +{{/* +The additive env block, rendered through tpl with the value, secretKeyRef and +configMapKeyRef forms (guideline 10). + +ADDITIVE MEANS ADDITIVE: an entry repeating a name this chart already sets does +not replace it. At install time both reach the pod and the later one wins; on +an EXISTING release the pair collapses to the chart's own value, because +Kubernetes merges env by name - and helm upgrade still reports success. So do +not use this block to override GARAGE_RPC_SECRET or the three +GARAGE_DEFAULT_* variables; change the values that produce them. +*/}} +{{- define "garage.env" -}} +{{- $root := .root -}} +{{- range $value := .env }} +{{- if $value.value }} +- name: {{ $value.name | quote }} + # quote is required, not cosmetic: a value like "3900" or "true" renders + # unquoted as a YAML number/bool, and the API server then rejects the whole + # manifest with "cannot unmarshal number into Go struct field EnvVar.value of + # type string". + value: {{ tpl $value.value $root | quote }} +{{- else if $value.valueFrom }} +{{- if $value.valueFrom.secretKeyRef }} +- name: {{ $value.name }} + valueFrom: + secretKeyRef: + name: {{ tpl $value.valueFrom.secretKeyRef.name $root | quote }} + key: {{ tpl $value.valueFrom.secretKeyRef.key $root | quote }} +{{- else if $value.valueFrom.configMapKeyRef }} +- name: {{ $value.name }} + valueFrom: + configMapKeyRef: + name: {{ tpl $value.valueFrom.configMapKeyRef.name $root | quote }} + key: {{ tpl $value.valueFrom.configMapKeyRef.key $root | quote }} +{{- end }} +{{- end }} +{{- end }} +{{- end -}} diff --git a/garage/templates/garage.configmap.yaml b/garage/templates/garage.configmap.yaml new file mode 100644 index 0000000..4a60a95 --- /dev/null +++ b/garage/templates/garage.configmap.yaml @@ -0,0 +1,146 @@ +{{- $settings := fromYaml (include "garage.settings" .) }} +apiVersion: v1 +kind: ConfigMap +metadata: + name: {{ .Release.Name }}-{{ .Chart.Name }}-configmap + labels: + app: {{ .Release.Name }}-{{ .Chart.Name }}-sfs +data: + # The complete garage.toml. It carries NO SECRET: the RPC secret arrives as + # the environment variable GARAGE_RPC_SECRET (see the StatefulSet and + # values.yaml for why the file form cannot be used here), so this ConfigMap + # is safe to read and the checksum/config annotation on the pod template + # still means "the configuration changed". + # + # It is mounted with subPath at /etc/garage.toml, which is the path Garage + # looks at without being told (-c defaults to /etc/garage.toml). A subPath + # mount is never refreshed by the kubelet, which is exactly why the checksum + # annotation is mandatory rather than decorative. + garage.toml: | + # Managed by the garage Helm chart - changes here are overwritten on the + # next helm upgrade. Edit the chart's values instead. + + # ONE node, and the replication factor is 1 BY DECISION, with the risk + # named (Gitea #81): longhorn-encrypted-ns already keeps three replicas of + # the encrypted volume, and Garage replication 3 on top of that would be + # nine times the raw data for two layers that know nothing about each + # other. + # + # WHAT THAT COSTS, because it is not free: upstream says of + # replication_factor = 1 "Do not use this for anything else than test + # deployments", and the reason is on its known-issues page - the same + # factor applies to the METADATA, so a single-replica setup is especially + # exposed to metadata corruption, and a corrupted database file is + # "generally not recoverable". Longhorn does not help against that: it + # replicates a corrupted file three times. Redundancy against device + # failure, yes; against file corruption, no. The three settings below are + # the countermeasures that do work at factor 1, and none of them is + # optional. + replication_factor = {{ $settings.replicationFactor | int64 }} + + # LMDB is Garage's default since 0.9 and the faster engine. The price is + # named: the reported corruption after an unclean shutdown stays possible, + # the settings below only lower the probability, and at factor 1 there is + # no peer to recover from - only the last snapshot. Upstream ties its LMDB + # recommendation to replication factor >= 2; running it at 1 is this + # chart's deliberate deviation. + # + # The alternative is "sqlite", which is slower on metadata throughput. With + # a single mail server as the only consumer that ceiling is unlikely to be + # reached, so sqlite is a defensible choice for anyone who weighs the + # durability of a single-file database higher. "fjall" is experimental. + db_engine = "{{ $settings.dbEngine }}" + + # metadata_dir and data_dir are two SUBDIRECTORIES OF ONE VOLUME; the + # snapshots live on a second one. Longhorn snapshots are atomic PER VOLUME + # and never across volumes, so keeping the index and the blocks together is + # what prevents the one failure mode that matters for a mail store: a + # backup whose metadata points at objects whose blocks were not written yet + # ("the mailbox lists a message whose body is missing"). + # + # The snapshots are on the OTHER volume for the opposite reason: a recovery + # point must not die with the volume whose corruption it is there to heal. + metadata_dir = "{{ include "garage.metadataDir" . }}" + data_dir = "{{ include "garage.dataDir" . }}" + metadata_snapshots_dir = "{{ include "garage.snapshotsDir" . }}" + + # metadata_fsync IS OFF IN GARAGE'S DEFAULTS, and this is the single + # setting most easily lost in a "tidy up the obvious defaults" pass. It is + # set here on purpose: Longhorn's block layer cannot help if the engine + # never calls fsync - a write that was never handed to the kernel is not in + # any replica and not in any snapshot. Garage's own example configuration + # pairs it with metadata_auto_snapshot_interval, as here. + metadata_fsync = {{ $settings.metadataFsync }} + # data_fsync stays off, which is Garage's default and the right trade for + # the blocks: a lost block is one object, is detectable by a scrub, and is + # replaceable from the sender's copy; a lost metadata write can take the + # whole index with it. Turning it on costs an fsync per written block. + data_fsync = {{ $settings.dataFsync }} + + # THE ONLY CONSISTENT RECOVERY POINT THIS DEPLOYMENT HAS. File-system + # snapshots of metadata_dir guarantee nothing - Garage says so itself: a + # snapshot taken during a metadata write is corrupt and useless as a + # restore point, which is why Garage recommends ITS OWN snapshot mechanism + # in all cases. This is the correction that reversed a premise of the + # original plan: Longhorn backs up the blocks, not the metadata. + # + # KNOW THE WINDOW: Garage keeps exactly TWO snapshots (KEEP_SNAPSHOTS = 2), + # the first one after interval/2 and every later one after interval times a + # random 1.0-1.2. At "6h" that means the first after ~3h and a restore + # window of roughly 6-14.4h afterwards. A corruption discovered later than + # that is older than every snapshot that still exists. + metadata_auto_snapshot_interval = "{{ $settings.metadataAutoSnapshotInterval }}" + + block_size = "{{ $settings.blockSize }}" + compression_level = {{ $settings.compressionLevel | int64 }} + + # Inter-node RPC. There are no other nodes here (no bootstrap_peers, no + # kubernetes_discovery - see values.yaml), but Garage binds this listener + # regardless and refuses to start without an rpc_secret, which arrives from + # the environment. + rpc_bind_addr = "{{ $settings.bindAddress }}:{{ $settings.ports.rpc | int64 }}" + + [s3_api] + api_bind_addr = "{{ $settings.bindAddress }}:{{ $settings.ports.s3 | int64 }}" + # The region name clients must use. It is not a cloud region, it is a label + # both sides have to agree on - a consumer configured for a different + # region gets a signature error, not a redirect. + s3_region = "{{ $settings.s3.region }}" + {{- with $settings.s3.rootDomain }} + # Vhost-style bucket access (.). Only useful with an + # Ingress and a wildcard DNS entry in front of it; path-style access works + # without it and is what an in-cluster consumer uses. + root_domain = {{ . | quote }} + {{- end }} + + {{- if $settings.web.enabled }} + + [s3_web] + # Static website hosting, OFF by default (see values.yaml): it is a second + # HTTP listener and serves nothing at all without a root_domain, while the + # consumer this chart was built for speaks S3. + bind_addr = "{{ $settings.bindAddress }}:{{ $settings.ports.web | int64 }}" + {{- with $settings.web.rootDomain }} + root_domain = {{ . | quote }} + {{- end }} + {{- end }} + + [admin] + # Bound on 0.0.0.0 rather than [::] BECAUSE ALL THREE PROBES USE THIS PORT. + # A dual-stack listener on [::] only accepts the kubelet's IPv4 connection + # while net.ipv6.bindv6only is 0 - true by default on Linux, but a node + # sysctl this chart cannot see, and the failure would be a pod that never + # becomes Ready for a reason nothing in the manifest points at. The S3 and + # RPC listeners keep the upstream [::] default; set garage.bindAddress to + # "0.0.0.0" on an IPv4-only cluster, or garage.adminBindAddress to "[::]" + # on an IPv6-only one. + # + # NO admin_token IS SET, so every authenticated admin endpoint is closed. + # That is not an omission: /health, the one endpoint the probes use, is + # explicitly token-free (@special in the admin router), and nothing in this + # deployment calls the authenticated API - the CLI inside the pod talks RPC, + # not HTTP. Adding a token would mean a fifth vault field guarding nothing + # that is in use. If you need it, it is admin_token_file / the + # GARAGE_ADMIN_TOKEN environment variable - and the file form carries the + # same permission trap as rpc_secret_file (see values.yaml). + api_bind_addr = "{{ $settings.adminBindAddress }}:{{ $settings.ports.admin | int64 }}" diff --git a/garage/templates/garage.ingress.yaml b/garage/templates/garage.ingress.yaml new file mode 100644 index 0000000..0e5496d --- /dev/null +++ b/garage/templates/garage.ingress.yaml @@ -0,0 +1,50 @@ +{{- /* +OFF BY DEFAULT (ingress.enabled: false), because the consumer this chart was +built for speaks S3 inside the cluster and needs no published endpoint. The +values exist anyway, built to this repo's ingress convention, so that +publishing the endpoint later is a values change rather than a chart change. + +GUIDELINE 1, CHECKED IN BOTH DIRECTIONS (README, "Values that must not lie"): +nothing outside this template reads ingress.domain. The S3 API's vhost-style +suffix is its own value, garage.s3.rootDomain, because the two are not the same +thing - a wildcard vhost domain and the single host an Ingress answers on. So +with the Ingress switched off, BOTH domain and clusterIssuer are free, and +their required() lives here where it can actually fire. + +WHAT IT PUBLISHES: the S3 API. There is no web UI to expose - Garage's admin +API is JSON and the authenticated half is closed in this chart (no token), and +the project's web interface is a separate program that is not part of this +image. Anyone publishing this should know what the object they are putting on +the internet is: an S3 endpoint that authenticates every request with a +signature, whose credentials are the two fields of the s3 vault item. +*/ -}} +{{- if .Values.ingress.enabled }} +apiVersion: networking.k8s.io/v1 +kind: Ingress +metadata: + name: {{ .Release.Name }}-{{ .Chart.Name }}-ingress + labels: + app: {{ .Release.Name }}-{{ .Chart.Name }}-sfs + annotations: + cert-manager.io/cluster-issuer: {{ required "A Cluster-Issuer must be provided (ingress.clusterIssuer)" .Values.ingress.clusterIssuer }} + {{- with .Values.ingress.annotations }} + {{- toYaml . | nindent 4 }} + {{- end }} +spec: + ingressClassName: {{ .Values.ingress.className }} + rules: + - host: {{ required "A Domain/Host must be provided (ingress.domain)" .Values.ingress.domain }} + http: + paths: + - backend: + service: + name: {{ .Release.Name }}-{{ .Chart.Name }}-service + port: + name: s3 + path: / + pathType: Prefix + tls: + - hosts: + - {{ required "A Domain/Host must be provided (ingress.domain)" .Values.ingress.domain }} + secretName: tls-{{ .Release.Name }}-{{ .Chart.Name }}-ingress +{{- end }} diff --git a/garage/templates/garage.pvc.yaml b/garage/templates/garage.pvc.yaml new file mode 100644 index 0000000..96d0040 --- /dev/null +++ b/garage/templates/garage.pvc.yaml @@ -0,0 +1,63 @@ +{{- /* +TWO claims, and the split is the durability design of this chart rather than a +sizing convenience (Gitea #81): + + 1. -garage-pvc metadata_dir AND data_dir, as two + subdirectories of this one volume + 2. -garage-snapshots-pvc metadata_snapshots_dir + +Claim 1 keeps index and blocks on ONE volume because Longhorn snapshots are +atomic per volume and never across volumes. Split over two volumes, a backup +can capture metadata that points at objects whose blocks were not written yet - +for a mail store exactly the failure "the mailbox lists a message whose body is +missing". + +Claim 2 exists because Garage's own metadata snapshots are the only consistent +recovery point here (file-system snapshots of a live metadata database are not +one), and a recovery point stored on the volume whose corruption it should +repair is not a recovery point. + +Both blocks are DELIBERATELY NOT null-safe: an erased size yields an invalid +PVC that is rejected at apply time, which is loud. A chart-side fallback would +silently size somebody's object store - and on an existing release it could not +help anyway, because a PVC cannot shrink. Same reasoning as satisfactory's +documented exception (Gitea #39) and valkey's persistence block. + +Neither claim is ever renamed: a renamed PVC makes Kubernetes provision a new, +empty volume and orphans the data. +*/ -}} +apiVersion: v1 +kind: PersistentVolumeClaim +metadata: + name: {{ .Release.Name }}-{{ .Chart.Name }}-pvc + labels: + app: {{ .Release.Name }}-{{ .Chart.Name }}-sfs +spec: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: {{ .Values.storage.data.amount }} + {{- if .Values.storage.data.storageClass }} + storageClassName: {{ .Values.storage.data.storageClass }} + {{- end }} +--- +apiVersion: v1 +kind: PersistentVolumeClaim +metadata: + name: {{ .Release.Name }}-{{ .Chart.Name }}-snapshots-pvc + labels: + app: {{ .Release.Name }}-{{ .Chart.Name }}-sfs +spec: + accessModes: + - ReadWriteOnce + resources: + requests: + # Sized for TWO snapshots of the metadata database plus head room, not + # for the objects - Garage keeps exactly two and deletes the older ones + # itself. A snapshot is a compacted copy of the LMDB file, so this needs + # roughly twice the size metadata_dir reaches. + storage: {{ .Values.storage.snapshots.amount }} + {{- if .Values.storage.snapshots.storageClass }} + storageClassName: {{ .Values.storage.snapshots.storageClass }} + {{- end }} diff --git a/garage/templates/garage.secret.yaml b/garage/templates/garage.secret.yaml new file mode 100644 index 0000000..fe8196a --- /dev/null +++ b/garage/templates/garage.secret.yaml @@ -0,0 +1,71 @@ +{{- /* +The 1Password items this chart materializes. Two of them, because they have +different audiences: the RPC secret belongs to this node alone, while the S3 +access key is the credential the CONSUMER uses - the same item a consuming +chart (stalwart's secrets.s3SecretRef) points at, which is why it carries the +bucket name as a third field: endpoint, bucket, key and secret then have one +source of truth instead of being kept in step by hand. + +WHAT THE FIELD NAMES ARE AND WHY THEY ARE NOT FREE-FORM: the operator +materializes one Secret key per FIELD of the item, and this chart references +those keys by name in the StatefulSet's env block. An item without the +expected field leaves the container in CreateContainerConfigError. NOTES.txt +lists all four fields for the installer. + +AND THE LIMIT THAT CANNOT BE CLOSED: a change to the CONTENT of either item +restarts nothing. Helm never sees a secret the 1Password operator +materializes, so there is nothing to hash. What this chart does cover is the +op:// PATH, through checksum/secret-ref on the pod template. After rotating a +value INSIDE an item the manual step is + kubectl -n rollout restart statefulset/-garage-sfs +and for the access key that is not enough on its own - see NOTES.txt, Garage +refuses to adopt a changed secret for an access key it already knows. +The 1Password operator's auto-restart annotation is NOT set here and must not +be: verified against operator 1.12.0 it enumerates Deployments only, so on a +StatefulSet it is a silent no-op advertising a guarantee it does not provide +(README, guideline 9). +*/ -}} +{{- $settings := fromYaml (include "garage.settings" .) -}} +apiVersion: onepassword.com/v1 +kind: OnePasswordItem +metadata: + name: {{ .Release.Name }}-{{ .Chart.Name }}-rpc-secret + labels: + app: {{ .Release.Name }}-{{ .Chart.Name }}-sfs +spec: + # Must carry the field: rpc-secret + # + # 32 bytes of random hex - Garage rejects anything else with "Invalid RPC + # secret key: expected 32 bytes of random hex". Generate it with + # openssl rand -hex 32 + # It is required even for a single node: Garage uses it as the network key of + # its RPC layer and refuses to start without it ("rpc_secret value is + # missing, not present in config file or in environment"). + itemPath: {{ required "You must provide an RPC secret reference (secrets.rpcSecretRef) - the 1Password item holding this cluster's 32-byte hex 'rpc-secret' field" .Values.secrets.rpcSecretRef }} +{{- $s3Needed := and $settings.singleNode (or $settings.defaultAccessKey $settings.defaultBucket) }} +{{- if or .Values.secrets.s3SecretRef $s3Needed }} +--- +apiVersion: onepassword.com/v1 +kind: OnePasswordItem +metadata: + name: {{ .Release.Name }}-{{ .Chart.Name }}-s3-secret + labels: + app: {{ .Release.Name }}-{{ .Chart.Name }}-sfs +spec: + # Must carry the fields: access-key, secret-key, bucket + # + # THE DIRECTION IS INWARDS, and that is the part worth knowing: with + # garage.singleNode plus defaultAccessKey/defaultBucket, Garage IMPORTS these + # three values on start (--default-access-key / --default-bucket read them + # from the environment). The vault item is the source of truth, and nothing + # has to be copied out of the server afterwards. + # + # Garage validates all three on import and refuses to START on a malformed + # one, so they are not free-form labels (src/model/key_table.rs, + # src/model/bucket_alias_table.rs): + # access-key at least 8 characters, ASCII alphanumeric plus - _ . + # secret-key at least 16 characters, printable ASCII (U+0021-U+007E) + # bucket a valid S3 bucket name (3-63 characters, lowercase) + # NOTES.txt prints the generators. + itemPath: {{ required "You must provide an S3 key reference (secrets.s3SecretRef) - the 1Password item holding the fields access-key, secret-key and bucket. It is required while garage.singleNode is true together with defaultAccessKey or defaultBucket, because Garage imports those values on start" .Values.secrets.s3SecretRef }} +{{- end }} diff --git a/garage/templates/garage.service.yaml b/garage/templates/garage.service.yaml new file mode 100644 index 0000000..d08b78e --- /dev/null +++ b/garage/templates/garage.service.yaml @@ -0,0 +1,43 @@ +{{- $settings := fromYaml (include "garage.settings" .) }} +apiVersion: v1 +kind: Service +metadata: + name: {{ .Release.Name }}-{{ .Chart.Name }}-service + labels: + app: {{ .Release.Name }}-{{ .Chart.Name }}-sfs +spec: + selector: + app: {{ .Release.Name }}-{{ .Chart.Name }}-sfs + ports: + # The S3 endpoint. In-cluster consumers reach it at + # http://-garage-service..svc.cluster.local:3900 + # with path-style addressing (bucket in the path), which needs no + # root_domain and no DNS wildcard. + - protocol: TCP + port: {{ $settings.ports.s3 | int64 }} + targetPort: s3 + name: s3 + # The admin endpoint. With no admin_token configured (see the ConfigMap) + # this serves exactly two things without authentication, /health and + # /metrics, and answers everything else with 401 - so it is a monitoring + # surface, not a management one. + - protocol: TCP + port: {{ $settings.ports.admin | int64 }} + targetPort: admin + name: admin + {{- if $settings.web.enabled }} + - protocol: TCP + port: {{ $settings.ports.web | int64 }} + targetPort: web + name: web + {{- end }} + # DELIBERATELY NO RPC PORT. Port 3901 exists on the pod because Garage binds + # it either way, but nothing should reach it: a single node has no peers, and + # this chart renders neither bootstrap_peers nor the kubernetes_discovery CRD. + # A Service port would advertise a cluster interface that is not one. + # + # ClusterIP, and no value to change it. The S3 API authenticates every + # request, but the way out of the cluster is an Ingress with a certificate + # (ingress.enabled, off by default), not a NodePort that puts an object store + # on a routable address by accident. + type: ClusterIP diff --git a/garage/templates/garage.sfs.yaml b/garage/templates/garage.sfs.yaml new file mode 100644 index 0000000..146c73b --- /dev/null +++ b/garage/templates/garage.sfs.yaml @@ -0,0 +1,208 @@ +{{- $settings := fromYaml (include "garage.settings" .) }} +{{- /* +Catch the one flag combination Garage rejects at START, at render time instead: +--default-access-key and --default-bucket "can only be used in single-node +mode" (src/garage/server.rs). Without this check the chart would install +cleanly and the container would exit immediately, with the reason only in the +pod log. +*/ -}} +{{- if and (not $settings.singleNode) (or $settings.defaultAccessKey $settings.defaultBucket) }} +{{- fail "garage.defaultAccessKey and garage.defaultBucket require garage.singleNode: true - Garage refuses to start otherwise (\"Flags --default-access-key and --default-bucket can only be used in single-node mode\"). With singleNode: false, create the key and the bucket by hand after applying the cluster layout; NOTES.txt has the commands." }} +{{- end }} +{{- $importKey := and $settings.singleNode (or $settings.defaultAccessKey $settings.defaultBucket) }} +apiVersion: apps/v1 +kind: StatefulSet +metadata: + name: {{ .Release.Name }}-{{ .Chart.Name }}-sfs + labels: + app: {{ .Release.Name }}-{{ .Chart.Name }}-sfs +spec: + replicas: {{ include "garage.replicas" (dict "scaleDown" .Values.scaleDown "replicas" .Values.garage.replicas) }} + serviceName: {{ .Release.Name }}-{{ .Chart.Name }}-service + selector: + matchLabels: + app: {{ .Release.Name }}-{{ .Chart.Name }}-sfs + template: + metadata: + name: {{ .Release.Name }}-{{ .Chart.Name }}-sfs + labels: + app: {{ .Release.Name }}-{{ .Chart.Name }}-sfs + annotations: + # Hash of the rendered ConfigMap: a changed garage.toml changes the pod + # template and thus restarts the pod. Without it the change would reach + # nothing - the config is mounted with subPath (never refreshed by the + # kubelet) and Garage reads it once, at start. + checksum/config: {{ include (print $.Template.BasePath "/garage.configmap.yaml") . | sha256sum }} + # Hash of the rendered OnePasswordItems. This covers the SOURCE of the + # secrets, not their content: repointing secrets.rpcSecretRef or + # secrets.s3SecretRef at a different vault item rolls the pod instead of + # leaving the old credentials in the running process. A change INSIDE an + # item cannot be covered - Helm never sees it (see garage.secret.yaml). + checksum/secret-ref: {{ include (print $.Template.BasePath "/garage.secret.yaml") . | sha256sum }} + spec: + automountServiceAccountToken: false + securityContext: + {{- include "garage.podSecurityContext" .Values.garage.securityContext | nindent 8 }} + # NO INIT CONTAINER, and that is a property of the image rather than a + # choice: dxflrs/garage is built FROM scratch and contains exactly one + # file, /garage. There is no shell, no mkdir, no wget - so an init + # container would have to be a SECOND image to pin, track and update. + # Nothing needs one: Garage creates metadata_dir, data_dir and the + # snapshots directory itself on start (create_dir_all in + # src/model/garage.rs and src/model/snapshot.rs), and the volumes are made + # writable by fsGroup. + # + # The same property rules out exec probes in this chart for good - all + # three probes are HTTP or TCP, see the helpers. + containers: + - name: {{ .Release.Name }}-{{ .Chart.Name }}-sfs + # Fully qualified: an unqualified "dxflrs/garage" resolves to Docker + # Hub by convention rather than by rule. The multi-arch image, not + # upstream's own chart's dxflrs/amd64_garage. + image: "docker.io/dxflrs/garage:{{ .Chart.AppVersion }}" + imagePullPolicy: IfNotPresent + # command AND args, because the image's CMD is ["/garage", "server"] + # and flags have to go after the subcommand. Overriding only args + # would drop the "server" subcommand. + command: + - /garage + args: + - server + {{- if $settings.singleNode }} + # --single-node is what makes this chart's default install reach + # Ready without an operator (Gitea #81, second plan correction): + # initial_config() in src/garage/server.rs creates the cluster + # layout in-process BEFORE the S3, web and admin servers are + # spawned - zone "dc1", capacity taken from the detected disk size, + # then apply_staged_changes(1). + # + # IT IS IDEMPOTENT, and it cannot damage an existing cluster. The + # layout version decides: 0 creates the layout, 1 does nothing at + # all, anything above 1 refuses to start ("Refusing to run in + # single-node mode: layout version is already superior to 1"), and + # a node that sees other nodes refuses as well ("more nodes are + # already present in the cluster"). There is no --version argument + # that could be passed twice with the same value, which is the trap + # the CLI path (garage layout apply --version N+1) has. + # + # It requires replication_factor = 1 and says so loudly if it is + # not. + - --single-node + {{- if or $settings.defaultAccessKey $settings.defaultBucket }} + # Imports the access key from GARAGE_DEFAULT_ACCESS_KEY / + # GARAGE_DEFAULT_SECRET_KEY below, i.e. FROM the vault item INTO + # Garage. Re-running is guarded upstream: a key that already exists + # with a different secret aborts the start, and one that was deleted + # in the cluster cannot be added back. + - --default-access-key + {{- end }} + {{- if $settings.defaultBucket }} + # Creates the bucket from GARAGE_DEFAULT_BUCKET and grants the key + # above read/write/owner on it. Implies --default-access-key. + - --default-bucket + {{- end }} + {{- end }} + env: + # THE RPC SECRET AS AN ENVIRONMENT VARIABLE, NOT AS A FILE - and the + # full reason is in values.yaml under secrets.rpcSecretRef. The + # short version, because this is the line a later round will want to + # "fix": rpc_secret_file CANNOT WORK in a non-root pod. + # read_secret_file rejects any file whose mode has a group or other + # bit set (src/garage/secrets.rs: mode & 0o077), and the kubelet + # OR-s the group-read bit onto every file of a secret volume as soon + # as fsGroup is set. Without fsGroup the file stays root-owned and + # uid 20000 cannot read it. Kubernetes can set the group of such a + # file, never its owner, so no combination satisfies both. + - name: GARAGE_RPC_SECRET + valueFrom: + secretKeyRef: + name: {{ .Release.Name }}-{{ .Chart.Name }}-rpc-secret + key: rpc-secret + {{- if $importKey }} + - name: GARAGE_DEFAULT_ACCESS_KEY + valueFrom: + secretKeyRef: + name: {{ .Release.Name }}-{{ .Chart.Name }}-s3-secret + key: access-key + - name: GARAGE_DEFAULT_SECRET_KEY + valueFrom: + secretKeyRef: + name: {{ .Release.Name }}-{{ .Chart.Name }}-s3-secret + key: secret-key + {{- end }} + {{- if and $settings.singleNode $settings.defaultBucket }} + - name: GARAGE_DEFAULT_BUCKET + valueFrom: + secretKeyRef: + name: {{ .Release.Name }}-{{ .Chart.Name }}-s3-secret + key: bucket + {{- end }} + {{- with .Values.garage.env }} + {{- include "garage.env" (dict "root" $ "env" .) | nindent 12 }} + {{- end }} + ports: + - containerPort: {{ $settings.ports.s3 | int64 }} + protocol: TCP + name: s3 + - containerPort: {{ $settings.ports.rpc | int64 }} + protocol: TCP + name: rpc + {{- if $settings.web.enabled }} + - containerPort: {{ $settings.ports.web | int64 }} + protocol: TCP + name: web + {{- end }} + # The port all three probes use. Its NAME is what the probe defaults + # reference, so renaming it here breaks them. + - containerPort: {{ $settings.ports.admin | int64 }} + protocol: TCP + name: admin + # Probes, resources and both security contexts go through the + # null-safe helpers (#99): an erased block means "chart defaults + # apply" instead of a vanished probe or an unhardened container. + # Never wrap these in an "if" - a guard hides the erased case + # completely instead of rendering a visible null. + # + # Which probe points at what, and why liveness and startup must NOT + # point at /health, is argued at each helper in _helpers.tpl. + livenessProbe: + {{- include "garage.livenessProbe" .Values.garage.livenessProbe | nindent 12 }} + readinessProbe: + {{- include "garage.readinessProbe" .Values.garage.readinessProbe | nindent 12 }} + startupProbe: + {{- include "garage.startupProbe" .Values.garage.startupProbe | nindent 12 }} + resources: + {{- include "garage.effectiveResources" .Values.garage.resources | nindent 12 }} + securityContext: + {{- include "garage.containerSecurityContext" .Values.garage.securityContext | nindent 12 }} + volumeMounts: + # ONE volume carrying metadata_dir and data_dir as subdirectories, + # so a Longhorn snapshot catches index and blocks in one atomic + # operation. Garage creates both subdirectories on start. + - name: data + mountPath: {{ include "garage.dataVolumePath" . }} + # The SECOND volume, holding the only consistent recovery point this + # deployment has. Separate on purpose: a snapshot that dies with the + # volume whose corruption it should repair is not a recovery point. + - name: snapshots + mountPath: {{ include "garage.snapshotsDir" . }} + - name: config + mountPath: /etc/garage.toml + subPath: garage.toml + readOnly: true + # No emptyDir for /tmp, and that is checked rather than assumed: with + # readOnlyRootFilesystem: true the root filesystem of this container + # holds exactly one file. Garage writes under metadata_dir, data_dir + # and the snapshots directory and uses no temporary directory + # (verified in src/model, src/db and src/block at v2.4.1). + restartPolicy: Always + volumes: + - name: data + persistentVolumeClaim: + claimName: {{ .Release.Name }}-{{ .Chart.Name }}-pvc + - name: snapshots + persistentVolumeClaim: + claimName: {{ .Release.Name }}-{{ .Chart.Name }}-snapshots-pvc + - name: config + configMap: + name: {{ .Release.Name }}-{{ .Chart.Name }}-configmap diff --git a/garage/values.schema.json b/garage/values.schema.json new file mode 100644 index 0000000..c5ca314 --- /dev/null +++ b/garage/values.schema.json @@ -0,0 +1,306 @@ +{ + "$schema": "https://json-schema.org/draft-07/schema#", + "title": "garage values", + "description": "Strict values schema (issue #68). additionalProperties is false on every chart-owned object level, so a key this chart does not read is rejected at render time instead of being silently ignored; the objects that go verbatim into the Kubernetes pod spec (probes, security contexts, global, free-form annotation maps) stay open, because the chart does not interpret their contents. The blocks the chart dereferences additionally accept null, because an erased block means 'chart defaults apply' (issue #99) rather than a vanished setting. Note what this schema CANNOT enforce: secrets.rpcSecretRef ships as null and JSON Schema's 'required' tests presence, not null-ness, so it would never fire - the enforcement is required() in the template (issue #18).", + "type": "object", + "additionalProperties": false, + "properties": { + "global": { + "description": "Helm's conventional cross-chart values. Unused by this chart, kept open so an umbrella install cannot break on it.", + "type": "object" + }, + "scaleDown": { + "description": "true renders the StatefulSet with replicas: 0 and takes precedence over garage.replicas.", + "type": "boolean" + }, + "secrets": { + "type": "object", + "additionalProperties": false, + "properties": { + "rpcSecretRef": { + "description": "op:// item path. The item must carry the field 'rpc-secret' with 32 bytes of random hex. Required always - Garage refuses to start without an RPC secret even as a single node.", + "type": ["string", "null"] + }, + "s3SecretRef": { + "description": "op:// item path. The item must carry the fields 'access-key', 'secret-key' and 'bucket'. Required while garage.singleNode is true together with defaultAccessKey or defaultBucket, because Garage imports those values on start.", + "type": ["string", "null"] + } + } + }, + "garage": { + "type": "object", + "additionalProperties": false, + "properties": { + "replicas": { + "description": "0 or 1. Capped at 1 because more than one replica is not a cluster: this chart renders no peer bootstrapping and no discovery CRD, so a second pod would be a lone node with its own empty metadata database.", + "type": "integer", + "minimum": 0, + "maximum": 1 + }, + "singleNode": { + "description": "Run 'garage server --single-node', which creates the cluster layout in-process before the listeners open. Idempotent and guarded by the layout version; requires replicationFactor 1.", + "type": "boolean" + }, + "defaultAccessKey": { + "description": "Import the S3 access key from the vault item on start (--default-access-key). Requires singleNode; the chart fails the render on the mixed combination, because Garage refuses it at start.", + "type": "boolean" + }, + "defaultBucket": { + "description": "Create the bucket from the vault item's 'bucket' field on start (--default-bucket) and grant the imported key access to it. Requires singleNode.", + "type": "boolean" + }, + "replicationFactor": { + "description": "1 for this deployment, and raising it is not an upgrade path: the factor is part of the cluster layout, so increasing it is a layout operation with data movement on a cluster that would first need more nodes.", + "type": "integer", + "minimum": 1 + }, + "dbEngine": { + "description": "Metadata engine. Changing it on an existing release needs 'garage convert-db', not a values change.", + "type": "string", + "enum": ["lmdb", "sqlite", "fjall"] + }, + "metadataFsync": { + "description": "true here, against Garage's own default of false: the volume layer cannot help with a write the engine never handed to the kernel.", + "type": "boolean" + }, + "dataFsync": { + "description": "false, Garage's default: a lost block is one object and is replaceable, a lost metadata write can take the index with it.", + "type": "boolean" + }, + "metadataAutoSnapshotInterval": { + "description": "Go-style duration, e.g. '6h'. The only consistent recovery point this deployment has; an empty string is rejected because switching the snapshots off is not a configuration this chart offers at replication factor 1.", + "type": "string", + "minLength": 1 + }, + "blockSize": { + "type": "string", + "minLength": 1 + }, + "compressionLevel": { + "description": "zstd level for stored blocks; 0 disables compression.", + "type": "integer", + "minimum": 0 + }, + "bindAddress": { + "description": "Host part of the S3, web and RPC listeners. '[::]' accepts IPv4 as well while net.ipv6.bindv6only is 0.", + "type": "string", + "minLength": 1 + }, + "adminBindAddress": { + "description": "Host part of the admin listener, separate from the others because all three probes use it.", + "type": "string", + "minLength": 1 + }, + "ports": { + "description": "May be null / empty: the chart then falls back to its defaults. The ConfigMap renders these into garage.toml's bind addresses and the StatefulSet renders them as containerPorts, so both files follow one source.", + "type": ["object", "null"], + "additionalProperties": false, + "properties": { + "s3": { "$ref": "#/definitions/port" }, + "rpc": { "$ref": "#/definitions/port" }, + "web": { "$ref": "#/definitions/port" }, + "admin": { "$ref": "#/definitions/port" } + } + }, + "s3": { + "type": ["object", "null"], + "additionalProperties": false, + "properties": { + "region": { + "description": "A label both sides must agree on; a consumer configured for a different one gets a signature error.", + "type": "string", + "minLength": 1 + }, + "rootDomain": { + "description": "Optional suffix for vhost-style bucket access. Deliberately not coupled to ingress.domain - the two are different things.", + "type": ["string", "null"] + } + } + }, + "web": { + "type": ["object", "null"], + "additionalProperties": false, + "properties": { + "enabled": { + "description": "Static website hosting (the s3_web endpoint). Off: a second HTTP listener that serves nothing without a rootDomain.", + "type": "boolean" + }, + "rootDomain": { + "type": ["string", "null"] + } + } + }, + "resources": { + "$ref": "#/definitions/resources" + }, + "livenessProbe": { + "$ref": "#/definitions/probe" + }, + "readinessProbe": { + "$ref": "#/definitions/probe" + }, + "startupProbe": { + "$ref": "#/definitions/probe" + }, + "securityContext": { + "$ref": "#/definitions/securityContext" + }, + "env": { + "$ref": "#/definitions/env" + } + } + }, + "storage": { + "description": "The two PVCs. Deliberately NOT null-safe: an erased block yields an invalid PVC, which fails loudly at apply time. Issue #99 covers only the silent class, and on an existing release a chart-side fallback could not help anyway, because a PVC cannot shrink.", + "type": "object", + "additionalProperties": false, + "properties": { + "data": { + "description": "metadata_dir AND data_dir as two subdirectories of one volume, so a volume snapshot catches index and blocks atomically.", + "type": "object", + "additionalProperties": false, + "properties": { + "amount": { "type": ["string", "null"] }, + "storageClass": { "type": ["string", "null"] } + } + }, + "snapshots": { + "description": "metadata_snapshots_dir, on a second volume so the recovery point does not die with the volume whose corruption it should repair.", + "type": "object", + "additionalProperties": false, + "properties": { + "amount": { "type": ["string", "null"] }, + "storageClass": { "type": ["string", "null"] } + } + } + } + }, + "ingress": { + "type": "object", + "additionalProperties": false, + "properties": { + "enabled": { + "description": "Gates the whole Ingress template (issue #93). Checked in both directions for this chart: nothing outside that template reads ingress.domain, so switching the Ingress off frees domain as well as clusterIssuer.", + "type": "boolean" + }, + "clusterIssuer": { + "type": ["string", "null"] + }, + "domain": { + "type": ["string", "null"] + }, + "className": { + "type": ["string", "null"] + }, + "annotations": { + "description": "Extra Ingress annotations, merged with the cert-manager annotation the chart always sets. Free-form keys by nature, so this object stays open.", + "type": "object" + } + } + } + }, + "definitions": { + "port": { + "type": ["integer", "null"], + "minimum": 1, + "maximum": 65535 + }, + "resources": { + "description": "May be null / empty: the chart then falls back to its default requests and computed limits (issue #99) instead of rendering a container without resource accounting.", + "type": ["object", "null"], + "additionalProperties": false, + "properties": { + "limitFactor": { + "type": "number", + "exclusiveMinimum": 0 + }, + "requests": { + "$ref": "#/definitions/resourceList" + }, + "limits": { + "$ref": "#/definitions/resourceList" + } + } + }, + "resourceList": { + "description": "Both spellings of the ephemeral-storage key are accepted because the resources helper normalizes camelCase to the Kubernetes key.", + "type": ["object", "null"], + "additionalProperties": false, + "properties": { + "cpu": { + "type": ["string", "number", "null"] + }, + "memory": { + "type": ["string", "number", "null"] + }, + "ephemeral-storage": { + "type": ["string", "number", "null"] + }, + "ephemeralStorage": { + "type": ["string", "number", "null"] + } + } + }, + "env": { + "description": "Additive container env entries. An entry repeating a name the chart already sets does not replace it - on an existing release the pair collapses to the chart's own value while helm upgrade reports success.", + "type": ["array", "null"], + "items": { + "type": "object", + "additionalProperties": false, + "required": ["name"], + "properties": { + "name": { + "type": "string" + }, + "value": { + "type": "string" + }, + "valueFrom": { + "type": "object", + "additionalProperties": false, + "properties": { + "secretKeyRef": { + "$ref": "#/definitions/keyRef" + }, + "configMapKeyRef": { + "$ref": "#/definitions/keyRef" + } + } + } + } + } + }, + "keyRef": { + "type": "object", + "additionalProperties": false, + "required": ["name", "key"], + "properties": { + "name": { + "type": "string" + }, + "key": { + "type": "string" + } + } + }, + "securityContext": { + "description": "May be null / empty at any level: the chart then falls back to its hardening defaults (issue #99). This chart's fallback reproduces ITS defaults, deviations included - fsGroupChangePolicy: OnRootMismatch rather than the repo's Always, because an object store's data directory holds one file per block and a recursive chown on every pod start turns a restart into a multi-minute outage that grows with the store.", + "type": ["object", "null"], + "additionalProperties": false, + "properties": { + "pod": { + "description": "Merged onto the chart's pod hardening defaults and rendered as the pod's securityContext; kept open because the chart does not interpret the individual fields.", + "type": ["object", "null"] + }, + "container": { + "description": "Merged onto the chart's container hardening defaults and rendered as the container's securityContext; kept open because the chart does not interpret the individual fields.", + "type": ["object", "null"] + } + } + }, + "probe": { + "description": "Merged onto the chart's probe defaults and rendered as a container probe; kept open because the chart does not interpret the individual fields. May be null / empty, which means the defaults apply (issue #99) rather than the probe disappearing. The defaults carry the probe's CONTEXT: the port NAME 'admin' for all three, /health for readiness only, and a TCP connect for liveness and startup - because /health answers 503 until a cluster layout exists, and a probe that can only fail would restart a node forever.", + "type": ["object", "null"] + } + } +} diff --git a/garage/values.yaml b/garage/values.yaml new file mode 100644 index 0000000..e54d38a --- /dev/null +++ b/garage/values.yaml @@ -0,0 +1,429 @@ +# Garage - an S3-compatible object store, deployed as a SINGLE NODE. +# +# READ THIS FIRST, because it is the premise every value below rests on +# (Gitea #81): this chart deploys ONE Garage node with replication_factor = 1, +# on the assumption that the volume layer underneath it is replicated and +# encrypted (longhorn-encrypted-ns in this cluster). Garage replication on top +# of that would be three times the raw data again, from two layers that know +# nothing about each other. +# +# WHAT THAT TRADE REALLY COSTS - named here because the chart cannot fix it: +# upstream says of replication_factor = 1 "Do not use this for anything else +# than test deployments", and the reason is that THE SAME FACTOR APPLIES TO THE +# METADATA. A corrupted metadata database is, in upstream's words, "generally +# not recoverable", and Longhorn does not help: it replicates the corrupted +# file three times. Redundancy against device failure, yes; against file +# corruption, no. What does help is the set of settings below - +# metadata_fsync, the automatic metadata snapshots, and those snapshots living +# on a second volume - and none of them is optional in this chart. +# +# THE IMAGE IS "FROM scratch" AND CONTAINS EXACTLY ONE FILE, /garage. No shell, +# no coreutils, no package manager. Three consequences that come up every time +# someone changes this chart: +# - exec probes are impossible here, for good. All three probes are HTTP or +# TCP (see the probe blocks below). +# - an init container would need a SECOND image to pin, track and update. +# Nothing needs one: Garage creates its own directories on start. +# - "kubectl exec" works, but only with /garage as the command and without a +# shell: kubectl exec -- /garage status + +# Scale the whole chart down without uninstalling it: scaleDown: true renders +# the StatefulSet with replicas: 0 and takes precedence over garage.replicas. +# PVCs, Secrets, the Service and the Ingress stay in place; scaleDown: false +# brings the workload back up. +scaleDown: false + +secrets: + # op:// path of the 1Password item carrying the field "rpc-secret": 32 bytes + # of random hex, i.e. "openssl rand -hex 32". REQUIRED - Garage uses it as + # the network key of its RPC layer and refuses to start without it, single + # node or not. + # + # IT ARRIVES AS AN ENVIRONMENT VARIABLE (GARAGE_RPC_SECRET), NOT AS A FILE, + # and that is the one place where this chart departs from how valkey does it. + # valkey argues explicitly for the file form - the password never appears in + # the process environment - and that argument is right in general. It cannot + # be followed here, and not because the file form would be inconvenient: + # IT IS STRUCTURALLY IMPOSSIBLE in a non-root pod. + # + # - Garage's rpc_secret_file rejects any file whose mode has a group OR + # other bit set: "if metadata.mode() & 0o077 != 0" in + # src/garage/secrets.rs (v2.4.1, line 144). The error message says + # "world-readable", but the mask tests the group bits as well. + # - The kubelet OR-s the group-read bit onto every file of a secret volume + # as soon as fsGroup is set: pkg/volume/volume_linux.go, + # changeFilePermission does Chmod(info.Mode() | 0440) for a read-only + # volume. So with fsGroup, Garage refuses to start. + # - Without fsGroup the file stays owned by root, and our uid 20000 cannot + # read it at all. + # - Kubernetes can set the GROUP of such a file, never its OWNER. There is + # no combination that satisfies both checks. + # + # The two ways around it were weighed and rejected (see Gitea #81): + # allow_world_readable_secrets = true would switch the permission check off + # for ALL of Garage's secret files, including ones nobody has set yet; and an + # init container copying the file with mode 0600 would need a second image + # for a scratch-based chart. What stays is the environment variable, which is + # also the dominant pattern in this repo (paperless-ngx, stalwart). + # + # The residual price, stated rather than hidden: the secret is in the + # container's process environment, so anyone who can exec into the pod or + # read its /proc can see it. Garage itself declares the variable with + # hide_env_values, so it does not print it in its own output. + # + # DO NOT "FIX" THIS BACK TO rpc_secret_file. The pod stops starting, and the + # error ("File /etc/garage-rpc/rpc-secret is world-readable!") points at a + # file mode nobody can change from inside Kubernetes. + rpcSecretRef: null + + # op:// path of the 1Password item carrying the fields "access-key", + # "secret-key" and "bucket" - the S3 credential a CONSUMER uses, which is why + # it is a separate item from the RPC secret above: a consuming chart + # (stalwart's secrets.s3SecretRef) points at this same item, and keeping the + # bucket name in it means endpoint, bucket, key and secret have one source of + # truth instead of being kept in step by hand. + # + # Required while garage.singleNode is true together with defaultAccessKey or + # defaultBucket (the default), because Garage then IMPORTS these values on + # start. The direction is inwards: the vault item is written first, Garage + # adopts it, and nothing has to be copied out of the server afterwards. + # + # Garage validates all three and refuses to START on a malformed one: + # access-key at least 8 characters, ASCII alphanumeric plus - _ . + # secret-key at least 16 characters, printable ASCII + # bucket a valid S3 bucket name (3-63 characters, lowercase) + # NOTES.txt prints the generators for both keys. + # + # ROTATION IS NOT A RE-SAVE. Garage refuses to start when the access key + # already exists with a different secret ("is associated with a secret key + # different than the one given in GARAGE_DEFAULT_SECRET_KEY"), so rotating + # the secret means a NEW key id, or deleting the old key with the CLI first. + # NOTES.txt has the sequence. + s3SecretRef: null + +garage: + # Replica count. 1 or 0 - and 0 is the documented way to stop the release + # without uninstalling it (the chart-global scaleDown flag does the same for + # everything at once). + # + # MORE THAN ONE REPLICA IS NOT A CLUSTER, which is why the schema caps this + # at 1: a second pod would come up as a lone node with its own empty metadata + # database on its own volume, because this chart renders neither + # bootstrap_peers nor the kubernetes_discovery CRD. With singleNode: true it + # would refuse to start at all once it saw the first node, which is the + # better of the two outcomes. + replicas: 1 + + # THE FIRST START, DONE BY GARAGE ITSELF INSTEAD OF BY HAND. + # + # --single-node makes the server create its own cluster layout on start + # (zone "dc1", capacity from the detected disk size) BEFORE it opens the S3, + # web and admin listeners. Without it, a fresh install has no layout, serves + # nothing, and stays not-Ready until an operator runs three CLI commands - + # which is a defensible way to run it, and NOTES.txt documents that path, but + # it is not a sane DEFAULT: a default install that never becomes Ready is not + # a default. + # + # IT CANNOT DAMAGE AN EXISTING CLUSTER, and that is worth knowing before + # trusting it with data. The layout version decides (src/garage/server.rs, + # initial_config): version 0 creates the layout, version 1 does nothing at + # all, anything above 1 aborts the start with "layout version is already + # superior to 1", and a node that sees other nodes aborts with "more nodes + # are already present in the cluster". It also requires replication_factor = + # 1 and says so if it is not. There is no --version argument here, so the + # trap of the CLI path - "never make multiple calls with the same value of + # the --version flag" - does not exist. + # + # Set to false to run the manual path (and then set defaultAccessKey and + # defaultBucket to false as well; the chart refuses the mixed combination, + # because Garage does). + singleNode: true + + # Import the S3 access key from the vault item on start + # (--default-access-key, reading GARAGE_DEFAULT_ACCESS_KEY and + # GARAGE_DEFAULT_SECRET_KEY). Requires singleNode. + # + # This is what removes the one manual step that could not be automated in any + # other way: without it Garage GENERATES the key, and it would have to be + # copied out of the server into the vault by hand - a secret travelling in the + # wrong direction, through a terminal, with the vault as a copy rather than + # the original. + defaultAccessKey: true + + # Create the bucket from the vault item's "bucket" field on start + # (--default-bucket) and grant the imported key read/write/owner on it. + # Implies defaultAccessKey upstream; requires singleNode. + defaultBucket: true + + # ONE. Everything above is the reason, and raising it here is NOT an upgrade + # path: the factor is part of the cluster layout, so increasing it is a layout + # operation with data movement, on a cluster that would first need more nodes + # - which this chart does not deploy. The value exists so that a future + # multi-node setup is not blocked by the chart, not as a switch to flip. + replicationFactor: 1 + + # "lmdb" (Garage's default since 0.9 and the faster engine), "sqlite", or the + # experimental "fjall". + # + # The deviation to know about: upstream ties its LMDB recommendation to + # replication factor >= 2, and we run it at 1. The reported corruption after + # an unclean shutdown stays possible, metadata_fsync and the snapshots below + # only lower the probability, and at factor 1 there is no peer to recover + # from - only the last snapshot. sqlite is the defensible alternative for + # anyone who weighs a single-file database's durability higher than metadata + # throughput; with one mail server as the only consumer that ceiling is + # unlikely to be reached. Changing this on an EXISTING release needs + # "garage convert-db", not a values change. + dbEngine: lmdb + + # ON, against Garage's own default of OFF. This is the single setting here + # most likely to be lost in a "clean up the obvious defaults" pass, so: + # Longhorn's block layer cannot help if the engine never calls fsync - a + # write that was never handed to the kernel is in no replica and in no + # snapshot. Garage's own example configuration pairs it with the snapshot + # interval below, as this chart does. + metadataFsync: true + + # OFF, which is Garage's default and the right trade for the blocks: a lost + # block is one object, is detectable by a scrub, and is replaceable from the + # sender's copy; a lost metadata write can take the whole index with it. + # Turning it on costs an fsync per written block. + dataFsync: false + + # THE ONLY CONSISTENT RECOVERY POINT THIS DEPLOYMENT HAS, and the reason the + # second PVC exists. File-system snapshots of a live metadata database + # guarantee nothing - Garage says so itself: a snapshot taken during a + # metadata write is corrupt and useless as a restore point, which is why + # Garage recommends its own mechanism in all cases. Longhorn backs up the + # blocks; it does not back up the metadata in any usable way. + # + # KNOW THE WINDOW BEFORE RELYING ON IT: Garage keeps exactly TWO snapshots, + # the first after interval/2 and every later one after interval times a + # random 1.0-1.2. At "6h" that is the first after ~3h and a restore window of + # roughly 6-14.4h afterwards. A corruption discovered later than that is + # older than every snapshot that still exists. Shorten the interval to narrow + # the window - at the cost of a compacted copy of the database every time. + # + # An empty string is rejected by the schema. A null falls back to this + # default, because switching the snapshots off is not a configuration this + # chart offers at replication factor 1. + metadataAutoSnapshotInterval: "6h" + + # Garage's defaults, exposed because they are the two knobs that trade disk + # against CPU. blockSize is the chunk size objects are split into; + # compressionLevel is the zstd level for stored blocks (1 = fast, higher = + # smaller, "none" is not a value here - set 0 to disable). + blockSize: "1M" + compressionLevel: 1 + + # The host part of the S3, web and RPC listeners. "[::]" is upstream's + # default and accepts IPv4 as well, as long as net.ipv6.bindv6only is 0 - + # the Linux default, but a node sysctl this chart cannot see. Set "0.0.0.0" + # on an IPv4-only cluster that has changed it. + bindAddress: "[::]" + + # The admin listener is bound on 0.0.0.0 SEPARATELY AND ON PURPOSE, because + # all three probes use it. If the kubelet's IPv4 probe could not reach a + # dual-stack listener, the symptom would be a pod that never becomes Ready + # for a reason nothing in the manifest points at. On an IPv6-only cluster set + # this to "[::]". + adminBindAddress: "0.0.0.0" + + # The four ports, in ONE place: the ConfigMap renders them into garage.toml's + # bind addresses and the StatefulSet renders them as containerPorts, so a + # value here that disagreed with the other file would be a listener nothing + # routes to. 3900/3901/3902/3903 are upstream's defaults. + # + # The probes address the admin port by NAME ("admin"), not by number, so they + # follow a change here. + ports: + s3: 3900 + rpc: 3901 + web: 3902 + admin: 3903 + + s3: + # The region name clients must use. Not a cloud region - a label both sides + # have to agree on; a consumer configured for a different one gets a + # signature error, not a redirect. + region: garage + + # Optional suffix for vhost-style bucket access (.). + # Only useful together with an Ingress and a wildcard DNS entry; + # path-style access works without it and is what an in-cluster consumer + # uses. Deliberately NOT ingress.domain: the two are different things, and + # coupling them would make the Ingress host decide how buckets are + # addressed. + rootDomain: null + + web: + # Static website hosting (the s3_web endpoint), OFF. It is a second HTTP + # listener, it serves nothing at all without a rootDomain, and the consumer + # this chart was built for speaks S3. Switching it on renders the + # containerPort, the Service port and the [s3_web] section. + enabled: false + rootDomain: null + + resources: + # Limits are optional: memory and ephemeral-storage limits default to + # limitFactor x the request, an explicitly set limit always wins, and a CPU + # limit is only set when configured explicitly under "limits" - CPU is + # compressible, so leaving it unlimited throttles instead of OOM-killing. + # + # 512Mi x 3 = 1536Mi memory limit. Garage's own guidance is "at least 1GB" + # for a node, and LMDB's memory-mapped database makes the RESIDENT size + # follow the working set rather than the database size. The ephemeral + # request covers container logs only: everything Garage writes goes to one + # of the two PVCs. + limitFactor: 3 + requests: + cpu: 100m + memory: 512Mi + ephemeral-storage: 32Mi + + # LIVENESS: a TCP connect to the admin port, deliberately NOT the /health + # endpoint readiness uses. /health answers 503 while no cluster layout + # exists, so a liveness probe on it would kill and restart a node that is + # merely waiting for its layout - forever, and a restart is exactly what + # cannot fix that. Same lesson as zipline (README, guideline 12.3), in its + # sharpest form. + livenessProbe: + tcpSocket: + port: admin + periodSeconds: 10 + timeoutSeconds: 5 + failureThreshold: 3 + + # READINESS: GET /health on the admin port - the one endpoint that answers + # the question a readiness probe asks. + # + # MEASURED AGAINST THE SOURCE AT v2.4.1, NOT CHOSEN: the S3 API server is + # spawned independently of the layout state, so the S3 socket binds and + # answers - with errors - before any layout exists. A tcpSocket probe on 3900 + # would therefore report a service ready that accepts nothing. /health needs + # no token and maps the cluster health straight onto the status code: 200 for + # Healthy and Degraded, 503 for Unavailable, and an unconfigured layout is a + # special case that yields Unavailable. + # + # So with singleNode: true the pod becomes Ready a moment after start, and + # with singleNode: false it stays not-Ready until a layout is applied by + # hand. The second case is the truth and not a defect: until then the S3 + # endpoint accepts nothing and no Service should route to it. + readinessProbe: + httpGet: + path: /health + port: admin + periodSeconds: 10 + timeoutSeconds: 5 + failureThreshold: 3 + + # STARTUP: TCP again, for the same reason as liveness - a startup probe that + # fails forever kills the container, so it must not depend on a layout only + # an operator can create. The startup probe gates liveness and readiness, + # which is why those carry no initialDelaySeconds. + # + # Budget = periodSeconds * failureThreshold = 120s. It covers the slow part + # of a real start, which is not Garage: attaching and mounting two network + # volumes, plus opening (and after an unclean shutdown recovering) the LMDB + # database. Garage's own startup work after that is sub-second. + startupProbe: + tcpSocket: + port: admin + periodSeconds: 5 + timeoutSeconds: 5 + failureThreshold: 24 + + securityContext: + pod: + runAsNonRoot: true + # Any uid works: the image is FROM scratch, so there is no /etc/passwd to + # contradict and nothing in it is owned by a user. 20000 is this repo's + # default. Note that upstream's own chart uses 1000 - an install migrated + # from it would need chown'ed volumes or this value changed to 1000. + runAsUser: 20000 + runAsGroup: 1000 + fsGroup: 1000 + # OnRootMismatch, DEVIATING from this repo's "Always" default, and the + # reason is the data: an object store's data directory holds one file per + # block, which is hundreds of thousands of files for a few hundred GB. + # "Always" chowns all of them recursively on EVERY pod start, which turns + # a restart into a multi-minute outage that gets worse as the store grows. + # OnRootMismatch checks the top directory's ownership and skips the walk + # when it already matches - which it does, from the second start onwards. + # Same reasoning as satisfactory's documented deviation. + fsGroupChangePolicy: OnRootMismatch + seccompProfile: + type: RuntimeDefault + container: + allowPrivilegeEscalation: false + readOnlyRootFilesystem: true + runAsNonRoot: true + capabilities: + drop: + - ALL + # EMPTY, and it can stay empty: the ports are all above 1024, and the + # binary carries no file capabilities (it is a static Rust binary in a + # scratch image), so there is no execve-time capability check like + # stalwart's. Nothing here needs a capability. + add: [] + + # Additive environment entries for the Garage container, rendered through tpl + # and supporting the value, secretKeyRef and configMapKeyRef forms. + # + # ADDITIVE MEANS ADDITIVE: an entry repeating a name the chart already sets + # does not replace it. At install time both reach the pod and the later one + # wins; on an EXISTING release the pair collapses to the chart's own value, + # because Kubernetes merges env by name - and helm upgrade still reports + # success. So do not use this to override GARAGE_RPC_SECRET or the + # GARAGE_DEFAULT_* variables; change the values that produce them. + # + # Useful entries that are genuinely additive: RUST_LOG (the image sets + # garage=info) and GARAGE_ALLOW_PUNYCODE. + env: [] + +storage: + # Both blocks are deliberately NOT null-safe: an erased size yields an + # invalid PVC that is rejected at apply time, which is loud. A chart-side + # fallback would silently size somebody's object store - and on an existing + # release it could not help anyway, because a PVC cannot shrink. + data: + # metadata_dir AND data_dir, as two subdirectories of this ONE volume. + # Together on purpose: Longhorn snapshots are atomic per volume and never + # across volumes, so this is what keeps index and blocks consistent with + # each other in a snapshot. 10Gi is a starting size for a mail blob store, + # not a recommendation - it is the value most likely to need changing + # before the first install. + amount: 10Gi + # null uses the cluster's default StorageClass. In this cluster the + # intended value is longhorn-encrypted-ns, which is also where the backup + # story lives - and note what this chart does NOT render: the Longhorn + # recurring-job labels (daily-snapshot, daily-backup) belong to the + # consuming bundle, and without them the off-site half of the backup plan + # does not exist, it only looks like it does. + storageClass: null + snapshots: + # metadata_snapshots_dir. Sized for TWO compacted copies of the metadata + # database plus head room, not for the objects - Garage keeps exactly two + # and deletes older ones itself. + amount: 5Gi + storageClass: null + +ingress: + # OFF. The consumer this chart was built for speaks S3 inside the cluster, so + # there is nothing to publish. The values exist, built to this repo's + # convention, so that publishing the endpoint later is a values change. + # + # Guideline 1, checked in both directions: nothing outside the Ingress + # template reads ingress.domain (the vhost suffix is its own value, + # garage.s3.rootDomain), so switching the Ingress off frees BOTH domain and + # clusterIssuer. With it on, both are required and the required() fires in + # the template. + enabled: false + clusterIssuer: null + domain: null + className: nginx + # Extra annotations for the Ingress; merged with the cert-manager + # cluster-issuer annotation the chart always sets. An S3 endpoint behind + # nginx usually wants nginx.ingress.kubernetes.io/proxy-body-size: "0" - + # without it the controller's 1m default truncates uploads. + annotations: {} diff --git a/renovate.json5 b/renovate.json5 index 655ddbd..2007143 100644 --- a/renovate.json5 +++ b/renovate.json5 @@ -342,6 +342,40 @@ datasourceTemplate: "docker", depTypeTemplate: "app-version", }, + { + description: [ + "garage: the registry carries three kinds of tag, and the first one is the reason", + "this block cannot be the plain four-liner most charts get. Measured on 2026-10-03", + "against docker.io/dxflrs/garage (340 tags):", + "", + " 1. COMMIT-SHA TAGS, 40 hex characters, pushed almost daily - the newest 22 tags", + " in the registry are all of that kind (99923a87... on 2026-10-03), while the", + " current release v2.4.1 dates from 2026-09-08. Without a version constraint", + " Renovate offers those as 'updates', and an appVersion of 40 hex characters", + " pins a commit nobody chose and republishes the chart for it.", + " 2. release tags v.., which is what we track.", + " 3. prereleases and internal builds: v2.0.0-rc1, v2.0.0-beta1 and the four", + " v1.99.x-internal tags. Note that those sort ABOVE v1.3.1 numerically, so they", + " are not kept out by ordering - they have to be excluded by shape.", + "", + "The regex versioning below does both jobs at once, as in the stalwart block: a", + "candidate that does not match is not a version at all, so the SHA tags (no 'v', no", + "dots) and every suffixed tag (the terminating anchor) are skipped. The three", + "numeric parts additionally reject a two-part line tag, should upstream ever add one.", + "", + "The 'v' stays outside the capture groups: it is part of the tag and carries no", + "ordering, and a group is a claim about how the value is compared. Being part of the", + "tag, it is part of appVersion (+upv2.4.1) - precedent: stalwart, matrix-synapse,", + "calibre-web-automated, ytdlp-web-player.", + ], + customType: "regex", + managerFilePatterns: ["garage/Chart.yaml"], + matchStrings: ["(?m)^appVersion:\\s*\"?(?[^\"\\s]+)\"?"], + depNameTemplate: "docker.io/dxflrs/garage", + datasourceTemplate: "docker", + depTypeTemplate: "app-version", + versioningTemplate: "regex:^v(?\\d+)\\.(?\\d+)\\.(?\\d+)$", + }, { customType: "regex", managerFilePatterns: ["gotenberg/Chart.yaml"], -- 2.54.0 From 5d10b0245820c8cf8760b913e3b7f6af02aa463d Mon Sep 17 00:00:00 2001 From: Justin Klein Date: Sat, 3 Oct 2026 20:40:47 +0000 Subject: [PATCH 2/4] fix(garage): compressionLevel disables with "none", never with 0 My own values comment claimed 0 switches compression off. It does not: Garage's deserializer takes an integer OR the literal string "none" (src/util/config.rs, deserialize_compression), and 0 is a valid integer that zstd reads as its default level - 3, i.e. MORE compression than the chart's 1, not none. A comment that invites an operator to write 0 for "off" is the values-that-must-not-lie shape with the lie in the documentation instead of in the template. The schema now accepts an integer (negative included, those are zstd's fast modes) or exactly "none", and the ConfigMap quotes the string form so the rendered TOML stays valid. Verified: 1, 7, -3 render unquoted, "none" renders quoted and the result still parses as TOML, and any other string is rejected by the schema. No version bump: the chart is not published yet. --- garage/templates/garage.configmap.yaml | 9 +++++++++ garage/values.schema.json | 8 +++++--- garage/values.yaml | 10 +++++++++- 3 files changed, 23 insertions(+), 4 deletions(-) diff --git a/garage/templates/garage.configmap.yaml b/garage/templates/garage.configmap.yaml index 4a60a95..5f60c28 100644 --- a/garage/templates/garage.configmap.yaml +++ b/garage/templates/garage.configmap.yaml @@ -92,7 +92,16 @@ data: metadata_auto_snapshot_interval = "{{ $settings.metadataAutoSnapshotInterval }}" block_size = "{{ $settings.blockSize }}" + # An integer is the zstd level; the literal "none" disables compression and + # has to be QUOTED to stay valid TOML. Garage's deserializer takes exactly + # those two forms - and 0 is not the second one: it is a valid integer that + # zstd reads as its default level (3), so writing 0 to "switch compression + # off" quietly turns it up. + {{- if kindIs "string" $settings.compressionLevel }} + compression_level = {{ $settings.compressionLevel | quote }} + {{- else }} compression_level = {{ $settings.compressionLevel | int64 }} + {{- end }} # Inter-node RPC. There are no other nodes here (no bootstrap_peers, no # kubernetes_discovery - see values.yaml), but Garage binds this listener diff --git a/garage/values.schema.json b/garage/values.schema.json index c5ca314..b33211e 100644 --- a/garage/values.schema.json +++ b/garage/values.schema.json @@ -77,9 +77,11 @@ "minLength": 1 }, "compressionLevel": { - "description": "zstd level for stored blocks; 0 disables compression.", - "type": "integer", - "minimum": 0 + "description": "zstd level for stored blocks, or the literal string \"none\" to disable compression. NOT 0 for disabling: Garage accepts 0 as an integer and zstd reads it as its default level (3), i.e. more compression than 1, not none. Negative integers are zstd's fast modes.", + "oneOf": [ + { "type": "integer" }, + { "type": "string", "enum": ["none"] } + ] }, "bindAddress": { "description": "Host part of the S3, web and RPC listeners. '[::]' accepts IPv4 as well while net.ipv6.bindv6only is 0.", diff --git a/garage/values.yaml b/garage/values.yaml index e54d38a..85f1601 100644 --- a/garage/values.yaml +++ b/garage/values.yaml @@ -212,7 +212,15 @@ garage: # Garage's defaults, exposed because they are the two knobs that trade disk # against CPU. blockSize is the chunk size objects are split into; # compressionLevel is the zstd level for stored blocks (1 = fast, higher = - # smaller, "none" is not a value here - set 0 to disable). + # smaller, negative levels are zstd's "fast" modes). + # + # TO DISABLE COMPRESSION, WRITE THE STRING "none" - and not 0, which is the + # trap this comment exists for: Garage's deserializer accepts an integer or + # the literal "none" (src/util/config.rs, deserialize_compression), and 0 is + # a perfectly valid integer that zstd reads as "use my default level", i.e. + # level 3 - MORE compression than the 1 below, not less. The schema therefore + # accepts an integer or exactly "none", and the ConfigMap quotes the string + # form so the rendered TOML stays valid. blockSize: "1M" compressionLevel: 1 -- 2.54.0 From 01336af953f734a2c568da6d6b46d2f8d125fef5 Mon Sep 17 00:00:00 2001 From: Justin Klein Date: Sat, 3 Oct 2026 20:42:20 +0000 Subject: [PATCH 3/4] docs(garage): state the admin surface precisely, including open /metrics "No admin_token is set" read like "nothing is open", and two parts of that were wrong. /metrics needs no authentication either, because metrics_require_token defaults to false - internal metrics on a ClusterIP port. And a config token is not the only way to reach the authenticated API: Garage 2.x keeps admin tokens in the cluster as well ("garage admin-token create"), which work without any entry in garage.toml. Both now stated at the ConfigMap and the Service, plus the way to close /metrics. --- garage/templates/garage.configmap.yaml | 24 ++++++++++++++++++------ garage/templates/garage.service.yaml | 10 ++++++---- 2 files changed, 24 insertions(+), 10 deletions(-) diff --git a/garage/templates/garage.configmap.yaml b/garage/templates/garage.configmap.yaml index 5f60c28..362cf02 100644 --- a/garage/templates/garage.configmap.yaml +++ b/garage/templates/garage.configmap.yaml @@ -144,12 +144,24 @@ data: # "0.0.0.0" on an IPv4-only cluster, or garage.adminBindAddress to "[::]" # on an IPv6-only one. # - # NO admin_token IS SET, so every authenticated admin endpoint is closed. - # That is not an omission: /health, the one endpoint the probes use, is - # explicitly token-free (@special in the admin router), and nothing in this - # deployment calls the authenticated API - the CLI inside the pod talks RPC, - # not HTTP. Adding a token would mean a fifth vault field guarding nothing - # that is in use. If you need it, it is admin_token_file / the + # NO admin_token IS SET, and that is not an omission: /health, the one + # endpoint the probes use, is explicitly token-free (@special in the admin + # router), and nothing in this deployment calls the authenticated API - the + # CLI inside the pod talks RPC, not HTTP. A token here would be a fifth + # vault field guarding nothing that is in use. + # + # WHAT IS AND IS NOT REACHABLE AS A RESULT, stated precisely because "no + # token" reads like "nothing is open": + # /health open, by design, and the probes depend on it + # /metrics OPEN TOO - metrics_require_token defaults to false, so with + # no metrics_token the endpoint needs no authentication. That + # is internal metrics on a ClusterIP port; set + # metrics_require_token = true plus a token to close it. + # everything else 401/403 until a token exists. Note that a config + # token is not the only way to get one: Garage 2.x also keeps + # admin tokens in the cluster ("garage admin-token create"), + # and those work without any entry in this file. + # If you do want a config token, it is admin_token_file / the # GARAGE_ADMIN_TOKEN environment variable - and the file form carries the # same permission trap as rpc_secret_file (see values.yaml). api_bind_addr = "{{ $settings.adminBindAddress }}:{{ $settings.ports.admin | int64 }}" diff --git a/garage/templates/garage.service.yaml b/garage/templates/garage.service.yaml index d08b78e..2d6620b 100644 --- a/garage/templates/garage.service.yaml +++ b/garage/templates/garage.service.yaml @@ -17,10 +17,12 @@ spec: port: {{ $settings.ports.s3 | int64 }} targetPort: s3 name: s3 - # The admin endpoint. With no admin_token configured (see the ConfigMap) - # this serves exactly two things without authentication, /health and - # /metrics, and answers everything else with 401 - so it is a monitoring - # surface, not a management one. + # The admin endpoint. With this chart's configuration it serves exactly two + # things without authentication - /health (token-free by design) and + # /metrics (because metrics_require_token defaults to false) - and answers + # everything else with 401/403 until an admin token exists, whether from + # the config or from "garage admin-token create". So it is a monitoring + # surface rather than a management one, and it is a ClusterIP port. - protocol: TCP port: {{ $settings.ports.admin | int64 }} targetPort: admin -- 2.54.0 From b19da4e580a6103162cd3c04979c4757dbe71c54 Mon Sep 17 00:00:00 2001 From: Justin Klein Date: Sat, 3 Oct 2026 20:45:56 +0000 Subject: [PATCH 4/4] fix(garage): a comment in the ConfigMap tripped Trivy's KSV-0109 Run 446 reported one HIGH finding, and it was prose, not data: the line "metrics_require_token = true plus a token to close it" sits inside the ConfigMap's data block, and KSV-0109 reads a = assignment with a credential-ish key name as a stored secret - comments included. The ConfigMap holds no credential at all; the RPC secret arrives as an environment variable precisely so that it cannot. Reworded rather than excepted: an entry in ci/trivy/trivyignore.yaml would carry a permanent exception for one sentence, and the next reader of the report would have to re-establish that the ConfigMap really is secret-free. The replacement names the mechanism at the line, so the shape does not come back. --- garage/templates/garage.configmap.yaml | 14 ++++++++++---- 1 file changed, 10 insertions(+), 4 deletions(-) diff --git a/garage/templates/garage.configmap.yaml b/garage/templates/garage.configmap.yaml index 362cf02..b368519 100644 --- a/garage/templates/garage.configmap.yaml +++ b/garage/templates/garage.configmap.yaml @@ -153,10 +153,16 @@ data: # WHAT IS AND IS NOT REACHABLE AS A RESULT, stated precisely because "no # token" reads like "nothing is open": # /health open, by design, and the probes depend on it - # /metrics OPEN TOO - metrics_require_token defaults to false, so with - # no metrics_token the endpoint needs no authentication. That - # is internal metrics on a ClusterIP port; set - # metrics_require_token = true plus a token to close it. + # /metrics OPEN TOO - the metrics_require_token switch defaults to + # false, so without a metrics credential the endpoint needs no + # authentication. That is internal metrics on a ClusterIP port; + # to close it, set that switch true and add a credential. + # + # (And a note for whoever edits these lines: do NOT write them + # as " = " with a credential-ish key name. Trivy's + # KSV-0109 reads an assignment inside this ConfigMap's DATA - + # comments included - as a stored secret and reports HIGH. It + # cost one run here, 2026-10-03.) # everything else 401/403 until a token exists. Note that a config # token is not the only way to get one: Garage 2.x also keeps # admin tokens in the cluster ("garage admin-token create"), -- 2.54.0