{{- $settings := fromYaml (include "garage.settings" .) -}} {{- $pod := printf "%s-%s-sfs-0" .Release.Name .Chart.Name -}} {{- $svc := printf "%s-%s-service" .Release.Name .Chart.Name -}} Garage {{ .Chart.AppVersion }} installed as release {{ .Release.Name }} in namespace {{ .Release.Namespace }}. ONE node, replication_factor = {{ $settings.replicationFactor | int64 }}, metadata engine {{ $settings.dbEngine }}. S3 ENDPOINT (in-cluster, path-style addressing): http://{{ $svc }}.{{ .Release.Namespace }}.svc.cluster.local:{{ $settings.ports.s3 | int64 }} region: {{ $settings.s3.region }} {{- if .Values.ingress.enabled }} published at: https://{{ .Values.ingress.domain }} {{- end }} {{- if $settings.web.enabled }} static website endpoint on port {{ $settings.ports.web | int64 }}{{ with $settings.web.rootDomain }} (root domain {{ . }}){{ else }} - WITHOUT a rootDomain it serves nothing; set garage.web.rootDomain{{ end }} {{- end }} THE 1PASSWORD ITEMS, AND WHICH FIELDS THEY MUST CARRY The operator materializes one Secret key per item FIELD, so a missing or misnamed field leaves the pod in CreateContainerConfigError. {{ .Values.secrets.rpcSecretRef }} rpc-secret 32 bytes of random hex: openssl rand -hex 32 {{ if .Values.secrets.s3SecretRef }} {{ .Values.secrets.s3SecretRef }} access-key >= 8 chars, [A-Za-z0-9-_.]: openssl rand -hex 12 secret-key >= 16 printable ASCII chars: openssl rand -hex 32 bucket a valid S3 bucket name (3-63 chars, lowercase) {{ else }} (no S3 key item configured - secrets.s3SecretRef is unset) {{ end }} {{- if and $settings.singleNode (or $settings.defaultAccessKey $settings.defaultBucket) }} THE DIRECTION IS INWARDS: Garage IMPORTS{{ if $settings.defaultAccessKey }} the access key{{ end }}{{ if $settings.defaultBucket }}{{ if $settings.defaultAccessKey }} and{{ end }} the bucket{{ end }} from that item on start. The vault is the source of truth and nothing has to be copied out of the server. {{- end }} FIRST START {{ if $settings.singleNode }} Nothing to do by hand. The server created its own cluster layout before it opened any listener{{ if $settings.defaultAccessKey }}, imported the access key{{ end }}{{ if $settings.defaultBucket }} and created the bucket{{ end }}. Verify it, in this order - all commands need /garage as the command, because the image has no shell: 1. kubectl -n {{ .Release.Namespace }} rollout status statefulset/{{ .Release.Name }}-{{ .Chart.Name }}-sfs Ready means the admin API answered 200 on /health, which means a layout exists and the node has quorum. Not-Ready with a running container means the layout is missing - check the logs for "Created initial layout". 2. kubectl -n {{ .Release.Namespace }} exec {{ $pod }} -- /garage status One node, with a role (zone dc1, a capacity taken from the detected disk size). A node WITHOUT a role means no layout was applied. 3. kubectl -n {{ .Release.Namespace }} exec {{ $pod }} -- /garage layout show {{- if $settings.defaultBucket }} 4. kubectl -n {{ .Release.Namespace }} exec {{ $pod }} -- /garage bucket list kubectl -n {{ .Release.Namespace }} exec {{ $pod }} -- /garage bucket info The bucket exists and the imported key has read/write/owner on it. {{- end }} 5. One write through the S3 API from a client pod, which is the only step that proves the endpoint rather than the process. {{ else }} garage.singleNode is false, so THIS RELEASE SERVES NOTHING YET and the pod will stay not-Ready until a layout exists - that is correct, not a defect: the readiness probe reports /health, and /health answers 503 while there is no layout. Apply one by hand, in this order: 1. kubectl -n {{ .Release.Namespace }} exec {{ $pod }} -- /garage status Note the node id and the current layout version (0 on a fresh install). 2. kubectl -n {{ .Release.Namespace }} exec {{ $pod }} -- \ /garage layout assign -z dc1 -c The capacity is this node's usable size, roughly storage.data.amount ({{ .Values.storage.data.amount }}). It takes BYTE suffixes B, KB, MB, GB, TB, PB - not Kubernetes' Gi/Mi, so write e.g. 10GB, not 10Gi. 3. kubectl -n {{ .Release.Namespace }} exec {{ $pod }} -- /garage layout show Read it before applying. The next step is NOT reversible by repeating it. 4. kubectl -n {{ .Release.Namespace }} exec {{ $pod }} -- /garage layout apply --version N is the version from step 1, so a fresh cluster takes --version 1. NEVER call this twice with the same --version: the flag is a guard against applying a staged change you have not seen, and reusing a number is how a layout is applied twice. 5. kubectl -n {{ .Release.Namespace }} exec {{ $pod }} -- /garage bucket create kubectl -n {{ .Release.Namespace }} exec {{ $pod }} -- \ /garage key import --yes kubectl -n {{ .Release.Namespace }} exec {{ $pod }} -- \ /garage bucket allow --read --write --owner --key "key import" rather than "key create", so the credential still comes FROM the vault item. "key create" generates one and then it has to travel the other way, through a terminal, with the vault as a copy. 6. The pod becomes Ready within one probe period after step 4. {{ end }} WHAT THIS CHART DOES NOT RENDER, AND WHAT IS MISSING WITHOUT IT - THE LONGHORN BACKUP LABELS. recurring-job.longhorn.io/daily-snapshot and .../daily-backup belong to the consuming bundle, not to this chart. They do NOT apply automatically: without them the off-site half of the backup plan does not exist, it only looks like it does. - The namespace's volume-encryption secret, if the StorageClass is longhorn-encrypted-ns: it resolves ${pvc.namespace}-pvc-encryption-secret, and without that entry the PVCs stay Pending. - No kubernetes_discovery CRD and no bootstrap peers: a single node has no members to find. TRAPS WORTH KNOWING BEFORE YOU NEED THEM - THE METADATA IS NOT IN YOUR VOLUME BACKUP in any usable form. A file-system snapshot of a live metadata database is not a restore point - taken during a write it is corrupt. What IS a restore point are Garage's own snapshots, on the second PVC, taken every {{ $settings.metadataAutoSnapshotInterval }}. - THE WINDOW IS SHORT: exactly TWO snapshots are kept, the first after half the interval and every later one after the interval times a random 1.0-1.2. With metadata_auto_snapshot_interval = {{ $settings.metadataAutoSnapshotInterval }} the oldest recovery point is therefore between one and two and a half intervals old, never more. A corruption found later than that is older than every snapshot that still exists. - metadata_fsync is ON here and OFF in Garage's defaults. If you ever see it missing from garage.toml, that is a regression, not a cleanup. - replication_factor 1 applies to the METADATA as well. Upstream calls a corrupted metadata database "generally not recoverable", and the volume layer replicates a corrupted file as faithfully as a good one. - ROTATING THE S3 SECRET IS NOT A RE-SAVE. Garage refuses to start when the access key already exists with a different secret ("is associated with a secret key different than the one given in GARAGE_DEFAULT_SECRET_KEY"). To rotate: either put a NEW access-key id into the item (the old key stays and can be deleted with "/garage key delete"), or delete the old key first and then update both fields. - A CHANGE INSIDE A VAULT ITEM RESTARTS NOTHING. Helm never sees it. After changing a field: kubectl -n {{ .Release.Namespace }} rollout restart statefulset/{{ .Release.Name }}-{{ .Chart.Name }}-sfs (Repointing a secrets.*SecretRef at a different item DOES roll the pod - that is covered by checksum/secret-ref.) - THE IMAGE HAS NO SHELL. "kubectl exec -it ... -- sh" does not work and never will; every command above uses /garage directly. Exec probes are impossible in this chart for the same reason. - THE RPC SECRET IS IN THE PROCESS ENVIRONMENT, not in a file. That is forced, not chosen: Garage rejects a secret file with any group bit set, and the kubelet sets one. values.yaml has the mechanism.