helm-charts/garage/templates/NOTES.txt

{{- $settings := fromYaml (include "garage.settings" .) -}}
{{- $pod := printf "%s-%s-sfs-0" .Release.Name .Chart.Name -}}
{{- $svc := printf "%s-%s-service" .Release.Name .Chart.Name -}}
Garage {{ .Chart.AppVersion }} installed as release {{ .Release.Name }} in namespace {{ .Release.Namespace }}.

ONE node, replication_factor = {{ $settings.replicationFactor | int64 }}, metadata engine {{ $settings.dbEngine }}.

S3 ENDPOINT (in-cluster, path-style addressing):
  http://{{ $svc }}.{{ .Release.Namespace }}.svc.cluster.local:{{ $settings.ports.s3 | int64 }}
  region: {{ $settings.s3.region }}
{{- if .Values.ingress.enabled }}
  published at: https://{{ .Values.ingress.domain }}
{{- end }}
{{- if $settings.web.enabled }}
  static website endpoint on port {{ $settings.ports.web | int64 }}{{ with $settings.web.rootDomain }} (root domain {{ . }}){{ else }} - WITHOUT a rootDomain it serves nothing; set garage.web.rootDomain{{ end }}
{{- end }}

THE 1PASSWORD ITEMS, AND WHICH FIELDS THEY MUST CARRY
The operator materializes one Secret key per item FIELD, so a missing or
misnamed field leaves the pod in CreateContainerConfigError.

  {{ .Values.secrets.rpcSecretRef }}
    rpc-secret    32 bytes of random hex:  openssl rand -hex 32
{{ if .Values.secrets.s3SecretRef }}  {{ .Values.secrets.s3SecretRef }}
    access-key    >= 8 chars, [A-Za-z0-9-_.]:  openssl rand -hex 12
    secret-key    >= 16 printable ASCII chars: openssl rand -hex 32
    bucket        a valid S3 bucket name (3-63 chars, lowercase)
{{ else }}  (no S3 key item configured - secrets.s3SecretRef is unset)
{{ end }}
{{- if and $settings.singleNode (or $settings.defaultAccessKey $settings.defaultBucket) }}
THE DIRECTION IS INWARDS: Garage IMPORTS{{ if $settings.defaultAccessKey }} the access key{{ end }}{{ if $settings.defaultBucket }}{{ if $settings.defaultAccessKey }} and{{ end }} the bucket{{ end }} from
that item on start. The vault is the source of truth and nothing has to be
copied out of the server.
{{- end }}

FIRST START
{{ if $settings.singleNode }}
Nothing to do by hand. The server created its own cluster layout before it
opened any listener{{ if $settings.defaultAccessKey }}, imported the access key{{ end }}{{ if $settings.defaultBucket }} and created the bucket{{ end }}.
Verify it, in this order - all commands need /garage as the command, because
the image has no shell:

  1. kubectl -n {{ .Release.Namespace }} rollout status statefulset/{{ .Release.Name }}-{{ .Chart.Name }}-sfs
     Ready means the admin API answered 200 on /health, which means a layout
     exists and the node has quorum. Not-Ready with a running container means
     the layout is missing - check the logs for "Created initial layout".

  2. kubectl -n {{ .Release.Namespace }} exec {{ $pod }} -- /garage status
     One node, with a role (zone dc1, a capacity taken from the detected disk
     size). A node WITHOUT a role means no layout was applied.

  3. kubectl -n {{ .Release.Namespace }} exec {{ $pod }} -- /garage layout show
{{- if $settings.defaultBucket }}

  4. kubectl -n {{ .Release.Namespace }} exec {{ $pod }} -- /garage bucket list
     kubectl -n {{ .Release.Namespace }} exec {{ $pod }} -- /garage bucket info <bucket>
     The bucket exists and the imported key has read/write/owner on it.
{{- end }}

  5. One write through the S3 API from a client pod, which is the only step
     that proves the endpoint rather than the process.
{{ else }}
garage.singleNode is false, so THIS RELEASE SERVES NOTHING YET and the pod
will stay not-Ready until a layout exists - that is correct, not a defect: the
readiness probe reports /health, and /health answers 503 while there is no
layout. Apply one by hand, in this order:

  1. kubectl -n {{ .Release.Namespace }} exec {{ $pod }} -- /garage status
     Note the node id and the current layout version (0 on a fresh install).

  2. kubectl -n {{ .Release.Namespace }} exec {{ $pod }} -- \
       /garage layout assign -z dc1 -c <capacity> <node-id>
     The capacity is this node's usable size, roughly storage.data.amount
     ({{ .Values.storage.data.amount }}). It takes BYTE suffixes B, KB, MB, GB, TB, PB - not
     Kubernetes' Gi/Mi, so write e.g. 10GB, not 10Gi.

  3. kubectl -n {{ .Release.Namespace }} exec {{ $pod }} -- /garage layout show
     Read it before applying. The next step is NOT reversible by repeating it.

  4. kubectl -n {{ .Release.Namespace }} exec {{ $pod }} -- /garage layout apply --version <N+1>
     N is the version from step 1, so a fresh cluster takes --version 1.
     NEVER call this twice with the same --version: the flag is a guard
     against applying a staged change you have not seen, and reusing a number
     is how a layout is applied twice.

  5. kubectl -n {{ .Release.Namespace }} exec {{ $pod }} -- /garage bucket create <bucket>
     kubectl -n {{ .Release.Namespace }} exec {{ $pod }} -- \
       /garage key import --yes <access-key> <secret-key>
     kubectl -n {{ .Release.Namespace }} exec {{ $pod }} -- \
       /garage bucket allow --read --write --owner --key <access-key> <bucket>

     "key import" rather than "key create", so the credential still comes FROM
     the vault item. "key create" generates one and then it has to travel the
     other way, through a terminal, with the vault as a copy.

  6. The pod becomes Ready within one probe period after step 4.
{{ end }}
WHAT THIS CHART DOES NOT RENDER, AND WHAT IS MISSING WITHOUT IT

  - THE LONGHORN BACKUP LABELS. recurring-job.longhorn.io/daily-snapshot and
    .../daily-backup belong to the consuming bundle, not to this chart. They do
    NOT apply automatically: without them the off-site half of the backup plan
    does not exist, it only looks like it does.
  - The namespace's volume-encryption secret, if the StorageClass is
    longhorn-encrypted-ns: it resolves
    ${pvc.namespace}-pvc-encryption-secret, and without that entry the PVCs
    stay Pending.
  - No kubernetes_discovery CRD and no bootstrap peers: a single node has no
    members to find.

TRAPS WORTH KNOWING BEFORE YOU NEED THEM

  - THE METADATA IS NOT IN YOUR VOLUME BACKUP in any usable form. A
    file-system snapshot of a live metadata database is not a restore point -
    taken during a write it is corrupt. What IS a restore point are Garage's
    own snapshots, on the second PVC, taken every {{ $settings.metadataAutoSnapshotInterval }}.
  - THE WINDOW IS SHORT: exactly TWO snapshots are kept, the first after half
    the interval and every later one after the interval times a random
    1.0-1.2. With metadata_auto_snapshot_interval = {{ $settings.metadataAutoSnapshotInterval }} the oldest
    recovery point is therefore between one and two and a half intervals old,
    never more. A corruption found later than that is older than every
    snapshot that still exists.
  - metadata_fsync is ON here and OFF in Garage's defaults. If you ever see it
    missing from garage.toml, that is a regression, not a cleanup.
  - replication_factor 1 applies to the METADATA as well. Upstream calls a
    corrupted metadata database "generally not recoverable", and the volume
    layer replicates a corrupted file as faithfully as a good one.
  - ROTATING THE S3 SECRET IS NOT A RE-SAVE. Garage refuses to start when the
    access key already exists with a different secret ("is associated with a
    secret key different than the one given in GARAGE_DEFAULT_SECRET_KEY"). To
    rotate: either put a NEW access-key id into the item (the old key stays and
    can be deleted with "/garage key delete"), or delete the old key first and
    then update both fields.
  - A CHANGE INSIDE A VAULT ITEM RESTARTS NOTHING. Helm never sees it. After
    changing a field:
      kubectl -n {{ .Release.Namespace }} rollout restart statefulset/{{ .Release.Name }}-{{ .Chart.Name }}-sfs
    (Repointing a secrets.*SecretRef at a different item DOES roll the pod -
    that is covered by checksum/secret-ref.)
  - THE IMAGE HAS NO SHELL. "kubectl exec -it ... -- sh" does not work and
    never will; every command above uses /garage directly. Exec probes are
    impossible in this chart for the same reason.
  - THE RPC SECRET IS IN THE PROCESS ENVIRONMENT, not in a file. That is
    forced, not chosen: Garage rejects a secret file with any group bit set,
    and the kubelet sets one. values.yaml has the mechanism.