helm-charts/garage/templates/NOTES.txt

{{- $settings := fromYaml (include "garage.settings" .) -}}
{{- $pod := printf "%s-%s-sfs-0" .Release.Name .Chart.Name -}}
{{- $svc := printf "%s-%s-service" .Release.Name .Chart.Name -}}
{{- $httpRoute := .Values.httpRoute | default dict -}}
{{- $hostnames := $httpRoute.hostnames | default list -}}
{{- if not $hostnames -}}
{{- $hostnames = list .Values.ingress.domain -}}
{{- end -}}
Garage {{ .Chart.AppVersion }} installed as release {{ .Release.Name }} in namespace {{ .Release.Namespace }}.

ONE node, replication_factor = {{ $settings.replicationFactor | int64 }}, metadata engine {{ $settings.dbEngine }}.

S3 ENDPOINT (in-cluster, path-style addressing):
  http://{{ $svc }}.{{ .Release.Namespace }}.svc.cluster.local:{{ $settings.ports.s3 | int64 }}
  region: {{ $settings.s3.region }}
{{- if .Values.ingress.enabled }}
  published at: https://{{ .Values.ingress.domain }}   (INGRESS MODE)
{{- else if $httpRoute.enabled }}
{{- range $hostnames }}
  published at: https://{{ . }}   (GATEWAY MODE)
{{- end }}
{{- end }}

{{- if .Values.ingress.enabled }}

PUBLISHED THROUGH AN INGRESS (ingress.enabled: true, httpRoute.enabled: false)
  The certificate is requested by cert-manager through the "{{ .Values.ingress.clusterIssuer }}"
  cluster issuer:

    kubectl -n {{ .Release.Namespace }} get certificate tls-{{ .Release.Name }}-{{ .Chart.Name }}-ingress

  The Ingress references the Service's s3 port BY NAME, so the port number
  never has to be kept in step by hand here.
{{- else if $httpRoute.enabled }}

PUBLISHED THROUGH AN HTTPROUTE (httpRoute.enabled: true, ingress.enabled: false)
  This release renders an HTTPRoute ({{ .Release.Name }}-{{ .Chart.Name }}-httproute) pointing at the
  Service's s3 port ({{ $settings.ports.s3 | int64 }}) and NOTHING else of the routing path — no
  Gateway, no class, no Certificate. So three things belong to the cluster and
  not to this chart, and if any is missing the hostname above does not answer:

    1. The Gateway-API CRDs. Without them this install would have failed.
    2. The Gateway named in httpRoute.parentRefs
{{- range $httpRoute.parentRefs }}
         {{ .name }}{{ if .namespace }} in namespace {{ .namespace }}{{ end }}{{ if .sectionName }}, listener {{ .sectionName }}{{ end }}
{{- end }}
       must exist AND its listener must allow routes from this namespace
       ({{ .Release.Namespace }}) in allowedRoutes. A rejected attachment shows
       up only in the route's status:

         kubectl -n {{ .Release.Namespace }} get httproute {{ .Release.Name }}-{{ .Chart.Name }}-httproute -o yaml

       Read status.parents[].conditions — "Accepted: True" and
       "ResolvedRefs: True" is a working attachment. An empty status means no
       controller has looked at the route at all.
    3. The TLS certificate for the hostname, on the Gateway's listener.
       ingress.clusterIssuer is NOT used in this mode and is not required: the
       chart renders no Certificate, so there is no
       tls-{{ .Release.Name }}-{{ .Chart.Name }}-ingress secret and nothing for cert-manager to do.

  YOUR UPLOAD LIMIT IS NOW THE GATEWAY'S, AND THIS IS AN OBJECT STORE.
  In Ingress mode the advice is nginx.ingress.kubernetes.io/proxy-body-size: "0"
  via ingress.annotations, because the controller's 1m default truncates
  uploads. The Gateway API has NO counterpart — it knows no body size at all,
  that is implementation-specific policy — and this chart deliberately renders
  no policy object, which would bind it to one implementation. So whatever
  limit the Gateway has is the limit your S3 PUTs get, and the symptom is
  uploads failing by size rather than by permission. Raise it at the Gateway
  or at its implementation's own policy object, next to the Gateway and not
  here.

  Mind also that the route carries the port NUMBER ({{ $settings.ports.s3 | int64 }}), not the name
  "s3": an HTTPRoute backendRef takes a PortNumber. The chart renders it from
  the same garage.ports.s3 value the Service uses, so the two cannot drift.
{{- end }}
{{- if $settings.web.enabled }}
  static website endpoint on port {{ $settings.ports.web | int64 }}{{ with $settings.web.rootDomain }} (root domain {{ . }}){{ else }} - WITHOUT a rootDomain it serves nothing; set garage.web.rootDomain{{ end }}
{{- end }}

THE 1PASSWORD ITEMS, AND WHICH FIELDS THEY MUST CARRY
The operator materializes one Secret key per item FIELD, so a missing or
misnamed field leaves the pod in CreateContainerConfigError.

  {{ .Values.secrets.rpcSecretRef }}
    rpc-secret    32 bytes of random hex:  openssl rand -hex 32
{{ if .Values.secrets.s3SecretRef }}  {{ .Values.secrets.s3SecretRef }}
    access-key    >= 8 chars, [A-Za-z0-9-_.]:  openssl rand -hex 12
    secret-key    >= 16 printable ASCII chars: openssl rand -hex 32
    bucket        a valid S3 bucket name (3-63 chars, lowercase)
{{ else }}  (no S3 key item configured - secrets.s3SecretRef is unset)
{{ end }}
{{- if and $settings.singleNode (or $settings.defaultAccessKey $settings.defaultBucket) }}
THE DIRECTION IS INWARDS: Garage IMPORTS{{ if $settings.defaultAccessKey }} the access key{{ end }}{{ if $settings.defaultBucket }}{{ if $settings.defaultAccessKey }} and{{ end }} the bucket{{ end }} from
that item on start. The vault is the source of truth and nothing has to be
copied out of the server.
{{- end }}

FIRST START
{{ if $settings.singleNode }}
Nothing to do by hand. The server created its own cluster layout before it
opened any listener{{ if $settings.defaultAccessKey }}, imported the access key{{ end }}{{ if $settings.defaultBucket }} and created the bucket{{ end }}.
Verify it, in this order - all commands need /garage as the command, because
the image has no shell:

  1. kubectl -n {{ .Release.Namespace }} rollout status statefulset/{{ .Release.Name }}-{{ .Chart.Name }}-sfs
     Ready means the admin API answered 200 on /health, which means a layout
     exists and the node has quorum. Not-Ready with a running container means
     the layout is missing - check the logs for "Created initial layout".

  2. kubectl -n {{ .Release.Namespace }} exec {{ $pod }} -- /garage status
     One node, with a role (zone dc1, a capacity taken from the detected disk
     size). A node WITHOUT a role means no layout was applied.

  3. kubectl -n {{ .Release.Namespace }} exec {{ $pod }} -- /garage layout show
{{- if $settings.defaultBucket }}

  4. kubectl -n {{ .Release.Namespace }} exec {{ $pod }} -- /garage bucket list
     kubectl -n {{ .Release.Namespace }} exec {{ $pod }} -- /garage bucket info <bucket>
     The bucket exists and the imported key has read/write/owner on it.
{{- end }}

  5. One write through the S3 API from a client pod, which is the only step
     that proves the endpoint rather than the process.
{{ else }}
garage.singleNode is false, so THIS RELEASE SERVES NOTHING YET and the pod
will stay not-Ready until a layout exists - that is correct, not a defect: the
readiness probe reports /health, and /health answers 503 while there is no
layout. Apply one by hand, in this order:

  1. kubectl -n {{ .Release.Namespace }} exec {{ $pod }} -- /garage status
     Note the node id and the current layout version (0 on a fresh install).

  2. kubectl -n {{ .Release.Namespace }} exec {{ $pod }} -- \
       /garage layout assign -z dc1 -c <capacity> <node-id>
     The capacity is this node's usable size, roughly storage.data.amount
     ({{ .Values.storage.data.amount }}). It takes BYTE suffixes B, KB, MB, GB, TB, PB - not
     Kubernetes' Gi/Mi, so write e.g. 10GB, not 10Gi.

  3. kubectl -n {{ .Release.Namespace }} exec {{ $pod }} -- /garage layout show
     Read it before applying. The next step is NOT reversible by repeating it.

  4. kubectl -n {{ .Release.Namespace }} exec {{ $pod }} -- /garage layout apply --version <N+1>
     N is the version from step 1, so a fresh cluster takes --version 1.
     NEVER call this twice with the same --version: the flag is a guard
     against applying a staged change you have not seen, and reusing a number
     is how a layout is applied twice.

  5. kubectl -n {{ .Release.Namespace }} exec {{ $pod }} -- /garage bucket create <bucket>
     kubectl -n {{ .Release.Namespace }} exec {{ $pod }} -- \
       /garage key import --yes <access-key> <secret-key>
     kubectl -n {{ .Release.Namespace }} exec {{ $pod }} -- \
       /garage bucket allow --read --write --owner --key <access-key> <bucket>

     "key import" rather than "key create", so the credential still comes FROM
     the vault item. "key create" generates one and then it has to travel the
     other way, through a terminal, with the vault as a copy.

  6. The pod becomes Ready within one probe period after step 4.
{{ end }}
WHAT THIS CHART DOES NOT RENDER, AND WHAT IS MISSING WITHOUT IT

  - THE LONGHORN BACKUP LABELS. recurring-job.longhorn.io/daily-snapshot and
    .../daily-backup belong to the consuming bundle, not to this chart. They do
    NOT apply automatically: without them the off-site half of the backup plan
    does not exist, it only looks like it does.
  - The namespace's volume-encryption secret, if the StorageClass is
    longhorn-encrypted-ns: it resolves
    ${pvc.namespace}-pvc-encryption-secret, and without that entry the PVCs
    stay Pending.
  - No kubernetes_discovery CRD and no bootstrap peers: a single node has no
    members to find.

TRAPS WORTH KNOWING BEFORE YOU NEED THEM

  - THE METADATA IS NOT IN YOUR VOLUME BACKUP in any usable form. A
    file-system snapshot of a live metadata database is not a restore point -
    taken during a write it is corrupt. What IS a restore point are Garage's
    own snapshots, on the second PVC, taken every {{ $settings.metadataAutoSnapshotInterval }}.
  - THE WINDOW IS SHORT: exactly TWO snapshots are kept, the first after half
    the interval and every later one after the interval times a random
    1.0-1.2. With metadata_auto_snapshot_interval = {{ $settings.metadataAutoSnapshotInterval }} the oldest
    recovery point is therefore between one and two and a half intervals old,
    never more. A corruption found later than that is older than every
    snapshot that still exists.
  - metadata_fsync is ON here and OFF in Garage's defaults. If you ever see it
    missing from garage.toml, that is a regression, not a cleanup.
  - replication_factor 1 applies to the METADATA as well. Upstream calls a
    corrupted metadata database "generally not recoverable", and the volume
    layer replicates a corrupted file as faithfully as a good one.
  - ROTATING THE S3 SECRET IS NOT A RE-SAVE. Garage refuses to start when the
    access key already exists with a different secret ("is associated with a
    secret key different than the one given in GARAGE_DEFAULT_SECRET_KEY"). To
    rotate: either put a NEW access-key id into the item (the old key stays and
    can be deleted with "/garage key delete"), or delete the old key first and
    then update both fields.
  - A CHANGE INSIDE A VAULT ITEM RESTARTS NOTHING. Helm never sees it. After
    changing a field:
      kubectl -n {{ .Release.Namespace }} rollout restart statefulset/{{ .Release.Name }}-{{ .Chart.Name }}-sfs
    (Repointing a secrets.*SecretRef at a different item DOES roll the pod -
    that is covered by checksum/secret-ref.)
  - THE IMAGE HAS NO SHELL. "kubectl exec -it ... -- sh" does not work and
    never will; every command above uses /garage directly. Exec probes are
    impossible in this chart for the same reason.
  - THE RPC SECRET IS IN THE PROCESS ENVIRONMENT, not in a file. That is
    forced, not chosen: Garage rejects a secret file with any group bit set,
    and the kubelet sets one. values.yaml has the mechanism.
{{- if .Values.networkPolicy.enabled }}

NETWORKPOLICY IS ACTIVE — WHAT IT CAPS, AND WHAT IT DOES NOT
  {{ .Release.Name }}-{{ .Chart.Name }}-networkpolicy selects this workload's pods.
  Inbound, and the four ports split three ways:
    - s3 ({{ $settings.ports.s3 | int64 }}) and web ({{ $settings.ports.web | int64 }}): RESTRICTED BY SOURCE — this namespace,
      networkPolicy.allowedNamespaces, and the ingress controller.
    - admin ({{ $settings.ports.admin | int64 }}): open inside the cluster, because all three probes address
      it. This is the ADMIN API, so read that as a smaller ceiling rather than
      a closed door; networkPolicy.probeSources closes the rest.
    - rpc ({{ $settings.ports.rpc | int64 }}): this release's own pods only. It has no Service.
  Outbound: DNS and TCP {{ $settings.ports.rpc | int64 }} (the cluster bus, outbound as well as in){{ with .Values.networkPolicy.egress }}{{ with .extraPorts }},
  plus TCP {{ join ", " . }}{{ end }}{{ end }}.

  CHECK YOUR S3 CONSUMERS. Anything in another namespace that talks to this
  endpoint on its in-cluster Service address — a backup job, a Zipline storing
  uploads, a Stalwart blob store — is blocked until that namespace is in
  networkPolicy.allowedNamespaces. Consumers using the published Ingress name
  arrive through the controller and are already covered. The failure is an
  application-level error with nothing pointing at this object.

  Three limits belong in the same breath, because none of them is visible in
  the object itself:

  - EGRESS IS BY PORT, NEVER BY DESTINATION. A NetworkPolicy matches CIDRs and
    selectors and cannot express a hostname, so every egress rule permits its
    port to ANY address. That is a limit of the object, not a shortcut — this
    cluster's CNI (Canal: Flannel plus Calico) has no FQDN rule either. An
    outbound port the chart does not know about is BLOCKED: add it to
    networkPolicy.egress.extraPorts, and expect to find out through an
    application-level connection error rather than through this object.
{{- if .Values.networkPolicy.probeSources }}
  - THE PROBE PATH IS RESTRICTED in this release, because
    networkPolicy.probeSources is set to
      {{ join ", " .Values.networkPolicy.probeSources }}
    Probes come from the IP of the WORKER NODE running the pod. If none of those
    CIDRs covers the node this pod lands on, the kubelet's probes are dropped
    and THE POD NEVER BECOMES READY — the rollout then times out and the symptom
    looks like a broken application. Compare

      kubectl -n {{ .Release.Namespace }} describe pod -l app={{ .Release.Name }}-{{ .Chart.Name }}-sfs

    against `kubectl get node -o wide`.
{{- else }}
  - THE PROBED PORT IS OPEN INSIDE THE CLUSTER. networkPolicy.probeSources is
    empty, so that rule carries no source selector at all and any pod the CNI
    routes to can reach it. Not from the internet — but this is NOT pod-level
    isolation. Narrowing it means listing the worker node IPs in
    networkPolicy.probeSources, because the kubelet probes from the node's own
    address and the chart cannot know it; a wrong guess there leaves the pod
    permanently un-Ready.
{{- end }}
  - IT IS ONLY A BOUNDARY WHERE THE CNI ENFORCES POLICIES. Where it does not,
    this object is accepted and filters nothing, so everything reads as
    configured while nothing is filtered. `kubectl -n {{ .Release.Namespace }} get networkpolicy`
    shows the object exists; only a connection attempt from an unrelated
    namespace shows that it bites.

  Switch it off with networkPolicy.enabled: false.
{{- end }}