helm-charts/garage/values.yaml

# Garage - an S3-compatible object store, deployed as a SINGLE NODE.
#
# READ THIS FIRST, because it is the premise every value below rests on
# (Gitea #81): this chart deploys ONE Garage node with replication_factor = 1,
# on the assumption that the volume layer underneath it is replicated and
# encrypted (longhorn-encrypted-ns in this cluster). Garage replication on top
# of that would be three times the raw data again, from two layers that know
# nothing about each other.
#
# WHAT THAT TRADE REALLY COSTS - named here because the chart cannot fix it:
# upstream says of replication_factor = 1 "Do not use this for anything else
# than test deployments", and the reason is that THE SAME FACTOR APPLIES TO THE
# METADATA. A corrupted metadata database is, in upstream's words, "generally
# not recoverable", and Longhorn does not help: it replicates the corrupted
# file three times. Redundancy against device failure, yes; against file
# corruption, no. What does help is the set of settings below -
# metadata_fsync, the automatic metadata snapshots, and those snapshots living
# on a second volume - and none of them is optional in this chart.
#
# THE IMAGE IS "FROM scratch" AND CONTAINS EXACTLY ONE FILE, /garage. No shell,
# no coreutils, no package manager. Three consequences that come up every time
# someone changes this chart:
#   - exec probes are impossible here, for good. All three probes are HTTP or
#     TCP (see the probe blocks below).
#   - an init container would need a SECOND image to pin, track and update.
#     Nothing needs one: Garage creates its own directories on start.
#   - "kubectl exec" works, but only with /garage as the command and without a
#     shell: kubectl exec <pod> -- /garage status

# Scale the whole chart down without uninstalling it: scaleDown: true renders
# the StatefulSet with replicas: 0 and takes precedence over garage.replicas.
# PVCs, Secrets, the Service and the Ingress stay in place; scaleDown: false
# brings the workload back up.
scaleDown: false

secrets:
  # op:// path of the 1Password item carrying the field "rpc-secret": 32 bytes
  # of random hex, i.e. "openssl rand -hex 32". REQUIRED - Garage uses it as
  # the network key of its RPC layer and refuses to start without it, single
  # node or not.
  #
  # IT ARRIVES AS AN ENVIRONMENT VARIABLE (GARAGE_RPC_SECRET), NOT AS A FILE,
  # and that is the one place where this chart departs from how valkey does it.
  # valkey argues explicitly for the file form - the password never appears in
  # the process environment - and that argument is right in general. It cannot
  # be followed here, and not because the file form would be inconvenient:
  # IT IS STRUCTURALLY IMPOSSIBLE in a non-root pod.
  #
  #   - Garage's rpc_secret_file rejects any file whose mode has a group OR
  #     other bit set: "if metadata.mode() & 0o077 != 0" in
  #     src/garage/secrets.rs (v2.4.1, line 144). The error message says
  #     "world-readable", but the mask tests the group bits as well.
  #   - The kubelet OR-s the group-read bit onto every file of a secret volume
  #     as soon as fsGroup is set: pkg/volume/volume_linux.go,
  #     changeFilePermission does Chmod(info.Mode() | 0440) for a read-only
  #     volume. So with fsGroup, Garage refuses to start.
  #   - Without fsGroup the file stays owned by root, and our uid 20000 cannot
  #     read it at all.
  #   - Kubernetes can set the GROUP of such a file, never its OWNER. There is
  #     no combination that satisfies both checks.
  #
  # The two ways around it were weighed and rejected (see Gitea #81):
  # allow_world_readable_secrets = true would switch the permission check off
  # for ALL of Garage's secret files, including ones nobody has set yet; and an
  # init container copying the file with mode 0600 would need a second image
  # for a scratch-based chart. What stays is the environment variable, which is
  # also the dominant pattern in this repo (paperless-ngx, stalwart).
  #
  # The residual price, stated rather than hidden: the secret is in the
  # container's process environment, so anyone who can exec into the pod or
  # read its /proc can see it. Garage itself declares the variable with
  # hide_env_values, so it does not print it in its own output.
  #
  # DO NOT "FIX" THIS BACK TO rpc_secret_file. The pod stops starting, and the
  # error ("File /etc/garage-rpc/rpc-secret is world-readable!") points at a
  # file mode nobody can change from inside Kubernetes.
  rpcSecretRef: null

  # op:// path of the 1Password item carrying the fields "access-key",
  # "secret-key" and "bucket" - the S3 credential a CONSUMER uses, which is why
  # it is a separate item from the RPC secret above: a consuming chart
  # (stalwart's secrets.s3SecretRef) points at this same item, and keeping the
  # bucket name in it means endpoint, bucket, key and secret have one source of
  # truth instead of being kept in step by hand.
  #
  # Required while garage.singleNode is true together with defaultAccessKey or
  # defaultBucket (the default), because Garage then IMPORTS these values on
  # start. The direction is inwards: the vault item is written first, Garage
  # adopts it, and nothing has to be copied out of the server afterwards.
  #
  # Garage validates all three and refuses to START on a malformed one:
  #   access-key  at least 8 characters, ASCII alphanumeric plus - _ .
  #   secret-key  at least 16 characters, printable ASCII
  #   bucket      a valid S3 bucket name (3-63 characters, lowercase)
  # NOTES.txt prints the generators for both keys.
  #
  # ROTATION IS NOT A RE-SAVE. Garage refuses to start when the access key
  # already exists with a different secret ("is associated with a secret key
  # different than the one given in GARAGE_DEFAULT_SECRET_KEY"), so rotating
  # the secret means a NEW key id, or deleting the old key with the CLI first.
  # NOTES.txt has the sequence.
  s3SecretRef: null

garage:
  # Replica count. 1 or 0 - and 0 is the documented way to stop the release
  # without uninstalling it (the chart-global scaleDown flag does the same for
  # everything at once).
  #
  # MORE THAN ONE REPLICA IS NOT A CLUSTER, which is why the schema caps this
  # at 1: a second pod would come up as a lone node with its own empty metadata
  # database on its own volume, because this chart renders neither
  # bootstrap_peers nor the kubernetes_discovery CRD. With singleNode: true it
  # would refuse to start at all once it saw the first node, which is the
  # better of the two outcomes.
  replicas: 1

  # THE FIRST START, DONE BY GARAGE ITSELF INSTEAD OF BY HAND.
  #
  # --single-node makes the server create its own cluster layout on start
  # (zone "dc1", capacity from the detected disk size) BEFORE it opens the S3,
  # web and admin listeners. Without it, a fresh install has no layout, serves
  # nothing, and stays not-Ready until an operator runs three CLI commands -
  # which is a defensible way to run it, and NOTES.txt documents that path, but
  # it is not a sane DEFAULT: a default install that never becomes Ready is not
  # a default.
  #
  # IT CANNOT DAMAGE AN EXISTING CLUSTER, and that is worth knowing before
  # trusting it with data. The layout version decides (src/garage/server.rs,
  # initial_config): version 0 creates the layout, version 1 does nothing at
  # all, anything above 1 aborts the start with "layout version is already
  # superior to 1", and a node that sees other nodes aborts with "more nodes
  # are already present in the cluster". It also requires replication_factor =
  # 1 and says so if it is not. There is no --version argument here, so the
  # trap of the CLI path - "never make multiple calls with the same value of
  # the --version flag" - does not exist.
  #
  # Set to false to run the manual path (and then set defaultAccessKey and
  # defaultBucket to false as well; the chart refuses the mixed combination,
  # because Garage does).
  singleNode: true

  # Import the S3 access key from the vault item on start
  # (--default-access-key, reading GARAGE_DEFAULT_ACCESS_KEY and
  # GARAGE_DEFAULT_SECRET_KEY). Requires singleNode.
  #
  # This is what removes the one manual step that could not be automated in any
  # other way: without it Garage GENERATES the key, and it would have to be
  # copied out of the server into the vault by hand - a secret travelling in the
  # wrong direction, through a terminal, with the vault as a copy rather than
  # the original.
  defaultAccessKey: true

  # Create the bucket from the vault item's "bucket" field on start
  # (--default-bucket) and grant the imported key read/write/owner on it.
  # Implies defaultAccessKey upstream; requires singleNode.
  defaultBucket: true

  # ONE. Everything above is the reason, and raising it here is NOT an upgrade
  # path: the factor is part of the cluster layout, so increasing it is a layout
  # operation with data movement, on a cluster that would first need more nodes
  # - which this chart does not deploy. The value exists so that a future
  # multi-node setup is not blocked by the chart, not as a switch to flip.
  replicationFactor: 1

  # "lmdb" (Garage's default since 0.9 and the faster engine), "sqlite", or the
  # experimental "fjall".
  #
  # The deviation to know about: upstream ties its LMDB recommendation to
  # replication factor >= 2, and we run it at 1. The reported corruption after
  # an unclean shutdown stays possible, metadata_fsync and the snapshots below
  # only lower the probability, and at factor 1 there is no peer to recover
  # from - only the last snapshot. sqlite is the defensible alternative for
  # anyone who weighs a single-file database's durability higher than metadata
  # throughput; with one mail server as the only consumer that ceiling is
  # unlikely to be reached. Changing this on an EXISTING release needs
  # "garage convert-db", not a values change.
  dbEngine: lmdb

  # ON, against Garage's own default of OFF. This is the single setting here
  # most likely to be lost in a "clean up the obvious defaults" pass, so:
  # Longhorn's block layer cannot help if the engine never calls fsync - a
  # write that was never handed to the kernel is in no replica and in no
  # snapshot. Garage's own example configuration pairs it with the snapshot
  # interval below, as this chart does.
  metadataFsync: true

  # OFF, which is Garage's default and the right trade for the blocks: a lost
  # block is one object, is detectable by a scrub, and is replaceable from the
  # sender's copy; a lost metadata write can take the whole index with it.
  # Turning it on costs an fsync per written block.
  dataFsync: false

  # THE ONLY CONSISTENT RECOVERY POINT THIS DEPLOYMENT HAS, and the reason the
  # second PVC exists. File-system snapshots of a live metadata database
  # guarantee nothing - Garage says so itself: a snapshot taken during a
  # metadata write is corrupt and useless as a restore point, which is why
  # Garage recommends its own mechanism in all cases. Longhorn backs up the
  # blocks; it does not back up the metadata in any usable way.
  #
  # KNOW THE WINDOW BEFORE RELYING ON IT: Garage keeps exactly TWO snapshots,
  # the first after interval/2 and every later one after interval times a
  # random 1.0-1.2. At "6h" that is the first after ~3h and a restore window of
  # roughly 6-14.4h afterwards. A corruption discovered later than that is
  # older than every snapshot that still exists. Shorten the interval to narrow
  # the window - at the cost of a compacted copy of the database every time.
  #
  # An empty string is rejected by the schema. A null falls back to this
  # default, because switching the snapshots off is not a configuration this
  # chart offers at replication factor 1.
  metadataAutoSnapshotInterval: "6h"

  # Garage's defaults, exposed because they are the two knobs that trade disk
  # against CPU. blockSize is the chunk size objects are split into;
  # compressionLevel is the zstd level for stored blocks (1 = fast, higher =
  # smaller, negative levels are zstd's "fast" modes).
  #
  # TO DISABLE COMPRESSION, WRITE THE STRING "none" - and not 0, which is the
  # trap this comment exists for: Garage's deserializer accepts an integer or
  # the literal "none" (src/util/config.rs, deserialize_compression), and 0 is
  # a perfectly valid integer that zstd reads as "use my default level", i.e.
  # level 3 - MORE compression than the 1 below, not less. The schema therefore
  # accepts an integer or exactly "none", and the ConfigMap quotes the string
  # form so the rendered TOML stays valid.
  blockSize: "1M"
  compressionLevel: 1

  # The host part of the S3, web and RPC listeners. "[::]" is upstream's
  # default and accepts IPv4 as well, as long as net.ipv6.bindv6only is 0 -
  # the Linux default, but a node sysctl this chart cannot see. Set "0.0.0.0"
  # on an IPv4-only cluster that has changed it.
  bindAddress: "[::]"

  # The admin listener is bound on 0.0.0.0 SEPARATELY AND ON PURPOSE, because
  # all three probes use it. If the kubelet's IPv4 probe could not reach a
  # dual-stack listener, the symptom would be a pod that never becomes Ready
  # for a reason nothing in the manifest points at. On an IPv6-only cluster set
  # this to "[::]".
  adminBindAddress: "0.0.0.0"

  # The four ports, in ONE place: the ConfigMap renders them into garage.toml's
  # bind addresses and the StatefulSet renders them as containerPorts, so a
  # value here that disagreed with the other file would be a listener nothing
  # routes to. 3900/3901/3902/3903 are upstream's defaults.
  #
  # The probes address the admin port by NAME ("admin"), not by number, so they
  # follow a change here.
  ports:
    s3: 3900
    rpc: 3901
    web: 3902
    admin: 3903

  s3:
    # The region name clients must use. Not a cloud region - a label both sides
    # have to agree on; a consumer configured for a different one gets a
    # signature error, not a redirect.
    region: garage

    # Optional suffix for vhost-style bucket access (<bucket>.<rootDomain>).
    # Only useful together with an Ingress and a wildcard DNS entry;
    # path-style access works without it and is what an in-cluster consumer
    # uses. Deliberately NOT ingress.domain: the two are different things, and
    # coupling them would make the Ingress host decide how buckets are
    # addressed.
    rootDomain: null

  web:
    # Static website hosting (the s3_web endpoint), OFF. It is a second HTTP
    # listener, it serves nothing at all without a rootDomain, and the consumer
    # this chart was built for speaks S3. Switching it on renders the
    # containerPort, the Service port and the [s3_web] section.
    enabled: false
    rootDomain: null

  resources:
    # Limits are optional: memory and ephemeral-storage limits default to
    # limitFactor x the request, an explicitly set limit always wins, and a CPU
    # limit is only set when configured explicitly under "limits" - CPU is
    # compressible, so leaving it unlimited throttles instead of OOM-killing.
    #
    # 512Mi x 3 = 1536Mi memory limit. Garage's own guidance is "at least 1GB"
    # for a node, and LMDB's memory-mapped database makes the RESIDENT size
    # follow the working set rather than the database size. The ephemeral
    # request covers container logs only: everything Garage writes goes to one
    # of the two PVCs.
    limitFactor: 3
    requests:
      cpu: 100m
      memory: 512Mi
      ephemeral-storage: 32Mi

  # LIVENESS: a TCP connect to the admin port, deliberately NOT the /health
  # endpoint readiness uses. /health answers 503 while no cluster layout
  # exists, so a liveness probe on it would kill and restart a node that is
  # merely waiting for its layout - forever, and a restart is exactly what
  # cannot fix that. Same lesson as zipline (README, guideline 12.3), in its
  # sharpest form.
  livenessProbe:
    tcpSocket:
      port: admin
    periodSeconds: 10
    timeoutSeconds: 5
    failureThreshold: 3

  # READINESS: GET /health on the admin port - the one endpoint that answers
  # the question a readiness probe asks.
  #
  # MEASURED AGAINST THE SOURCE AT v2.4.1, NOT CHOSEN: the S3 API server is
  # spawned independently of the layout state, so the S3 socket binds and
  # answers - with errors - before any layout exists. A tcpSocket probe on 3900
  # would therefore report a service ready that accepts nothing. /health needs
  # no token and maps the cluster health straight onto the status code: 200 for
  # Healthy and Degraded, 503 for Unavailable, and an unconfigured layout is a
  # special case that yields Unavailable.
  #
  # So with singleNode: true the pod becomes Ready a moment after start, and
  # with singleNode: false it stays not-Ready until a layout is applied by
  # hand. The second case is the truth and not a defect: until then the S3
  # endpoint accepts nothing and no Service should route to it.
  readinessProbe:
    httpGet:
      path: /health
      port: admin
    periodSeconds: 10
    timeoutSeconds: 5
    failureThreshold: 3

  # STARTUP: TCP again, for the same reason as liveness - a startup probe that
  # fails forever kills the container, so it must not depend on a layout only
  # an operator can create. The startup probe gates liveness and readiness,
  # which is why those carry no initialDelaySeconds.
  #
  # Budget = periodSeconds * failureThreshold = 120s. It covers the slow part
  # of a real start, which is not Garage: attaching and mounting two network
  # volumes, plus opening (and after an unclean shutdown recovering) the LMDB
  # database. Garage's own startup work after that is sub-second.
  startupProbe:
    tcpSocket:
      port: admin
    periodSeconds: 5
    timeoutSeconds: 5
    failureThreshold: 24

  securityContext:
    pod:
      runAsNonRoot: true
      # Any uid works: the image is FROM scratch, so there is no /etc/passwd to
      # contradict and nothing in it is owned by a user. 20000 is this repo's
      # default. Note that upstream's own chart uses 1000 - an install migrated
      # from it would need chown'ed volumes or this value changed to 1000.
      runAsUser: 20000
      runAsGroup: 1000
      fsGroup: 1000
      # OnRootMismatch, DEVIATING from this repo's "Always" default, and the
      # reason is the data: an object store's data directory holds one file per
      # block, which is hundreds of thousands of files for a few hundred GB.
      # "Always" chowns all of them recursively on EVERY pod start, which turns
      # a restart into a multi-minute outage that gets worse as the store grows.
      # OnRootMismatch checks the top directory's ownership and skips the walk
      # when it already matches - which it does, from the second start onwards.
      # Same reasoning as satisfactory's documented deviation.
      fsGroupChangePolicy: OnRootMismatch
      seccompProfile:
        type: RuntimeDefault
    container:
      allowPrivilegeEscalation: false
      readOnlyRootFilesystem: true
      runAsNonRoot: true
      capabilities:
        drop:
          - ALL
        # EMPTY, and it can stay empty: the ports are all above 1024, and the
        # binary carries no file capabilities (it is a static Rust binary in a
        # scratch image), so there is no execve-time capability check like
        # stalwart's. Nothing here needs a capability.
        add: []

  # Additive environment entries for the Garage container, rendered through tpl
  # and supporting the value, secretKeyRef and configMapKeyRef forms.
  #
  # ADDITIVE MEANS ADDITIVE: an entry repeating a name the chart already sets
  # does not replace it. At install time both reach the pod and the later one
  # wins; on an EXISTING release the pair collapses to the chart's own value,
  # because Kubernetes merges env by name - and helm upgrade still reports
  # success. So do not use this to override GARAGE_RPC_SECRET or the
  # GARAGE_DEFAULT_* variables; change the values that produce them.
  #
  # Useful entries that are genuinely additive: RUST_LOG (the image sets
  # garage=info) and GARAGE_ALLOW_PUNYCODE.
  env: []

storage:
  # Both blocks are deliberately NOT null-safe: an erased size yields an
  # invalid PVC that is rejected at apply time, which is loud. A chart-side
  # fallback would silently size somebody's object store - and on an existing
  # release it could not help anyway, because a PVC cannot shrink.
  data:
    # metadata_dir AND data_dir, as two subdirectories of this ONE volume.
    # Together on purpose: Longhorn snapshots are atomic per volume and never
    # across volumes, so this is what keeps index and blocks consistent with
    # each other in a snapshot. 10Gi is a starting size for a mail blob store,
    # not a recommendation - it is the value most likely to need changing
    # before the first install.
    amount: 10Gi
    # null uses the cluster's default StorageClass. In this cluster the
    # intended value is longhorn-encrypted-ns, which is also where the backup
    # story lives - and note what this chart does NOT render: the Longhorn
    # recurring-job labels (daily-snapshot, daily-backup) belong to the
    # consuming bundle, and without them the off-site half of the backup plan
    # does not exist, it only looks like it does.
    storageClass: null
  snapshots:
    # metadata_snapshots_dir. Sized for TWO compacted copies of the metadata
    # database plus head room, not for the objects - Garage keeps exactly two
    # and deletes older ones itself.
    amount: 5Gi
    storageClass: null

ingress:
  # OFF. The consumer this chart was built for speaks S3 inside the cluster, so
  # there is nothing to publish. The values exist, built to this repo's
  # convention, so that publishing the endpoint later is a values change.
  #
  # Mutually exclusive with httpRoute.enabled below (Gitea #114): with both
  # true the render aborts, naming both values. Both ship off, so this chart
  # publishes nothing until asked - in either mode.
  #
  # Guideline 1, checked in both directions, and the answer MOVED with #114:
  # the Ingress template is no longer the only reader of ingress.domain - the
  # HTTPRoute reads it too, as its single hostname, when httpRoute.hostnames is
  # empty. So domain is free only while BOTH routing modes are off; with either
  # one on it is required, and the required() fires in whichever template is
  # rendering. clusterIssuer is unchanged: it is read only by the Ingress, so
  # ingress.enabled: false frees it in every case - Gateway mode renders no
  # Certificate at all.
  #
  # Unchanged as well: the vhost suffix is its own value, garage.s3.rootDomain,
  # and is still deliberately not coupled to ingress.domain.
  enabled: false
  clusterIssuer: null
  domain: null
  className: nginx
  # Extra annotations for the Ingress; merged with the cert-manager
  # cluster-issuer annotation the chart always sets. An S3 endpoint behind
  # nginx usually wants nginx.ingress.kubernetes.io/proxy-body-size: "0" -
  # without it the controller's 1m default truncates uploads.
  #
  # THAT ONE HAS NO COUNTERPART IN GATEWAY MODE. The Gateway API has no
  # annotation channel and no notion of a body size at all - an upload limit is
  # implementation-specific policy - so an S3 endpoint published through an
  # HTTPRoute silently inherits the Gateway's limit, whatever it is. For an
  # object store that is not a detail: it is where PUTs start failing by size.
  # The chart does not rebuild it (a policy object would bind it to one
  # implementation); NOTES.txt prints the warning in that mode and the limit
  # has to be raised at the Gateway.
  annotations: {}

# Gateway-API alternative to the Ingress above (Gitea #114). Off by default -
# like the Ingress here, so this chart still publishes nothing unless asked.
# Exactly one of ingress.enabled and httpRoute.enabled may be true.
#
# The chart renders ONLY an HTTPRoute with parentRefs - no Gateway, no
# gatewayClassName, no Certificate and no implementation-specific policy
# object. Class and certificate belong to the Gateway and therefore to the
# platform, which is what keeps this chart independent of whether the cluster
# runs nginx-gateway-fabric, Cilium, Envoy Gateway or Istio.
#
# The Gateway-API CRDs must be installed in the cluster: the chart does not and
# cannot detect them (.Capabilities.APIVersions is unreliable at template
# time), so with httpRoute.enabled: true and no CRDs the INSTALL fails.
#
# WHAT GETS PUBLISHED IS THE S3 API, in this mode exactly as in Ingress mode:
# the route points at the Service's s3 port and nothing else. The admin port is
# deliberately not routed. Read the Ingress comment above before turning either
# mode on - and read the upload-limit note, which is the one real regression of
# this mode for an object store.
httpRoute:
  enabled: false
  # The Gateway(s) this route attaches to - a platform object, usually in
  # another namespace, which must allow this namespace in its listener's
  # allowedRoutes. One entry per parent, the fields are the Gateway API's
  # ParentReference (name, and optionally namespace, sectionName, port, group,
  # kind), passed through unchanged:
  #
  #   parentRefs:
  #     - name: public
  #       namespace: gateway
  #       sectionName: https
  #
  # REQUIRED while httpRoute.enabled is true - the render aborts on an empty
  # list, because a route without a parent is created, is valid and routes
  # nothing, with no event to say so.
  parentRefs: []
  # Empty means [ingress.domain], so the hostname has one source of truth. Set
  # explicitly only to publish under a different or an additional name - which
  # is how a migration from the Ingress runs: the HTTPRoute takes a second
  # hostname while the Ingress keeps ingress.domain.
  #
  # For vhost-style bucket access this is where the wildcard goes
  # ("*.s3.example.com" alongside the plain host), together with
  # garage.s3.rootDomain - the two still have to be set in step, as they do for
  # the Ingress.
  hostnames: []