Files
jklein 28a0230ef0
Validate Helm Charts / Detect changed charts (pull_request) Successful in 3s
Validate Helm Charts / Resource quantities (pull_request) Successful in 23s
Validate Helm Charts / Bump level (pull_request) Successful in 4s
Validate Helm Charts / Selector match (pull_request) Successful in 12s
Validate Helm Charts / HTTPRoute match (pull_request) Successful in 10s
Validate Helm Charts / YAML scalars (pull_request) Successful in 9s
Validate Helm Charts / NetworkPolicy (pull_request) Failing after 10s
Validate Helm Charts / Trivy config apache-tika (pull_request) Successful in 23s
Validate Helm Charts / Build apache-tika (pull_request) Successful in 14s
Validate Helm Charts / Build bentopdf (pull_request) Successful in 6s
Validate Helm Charts / Build bulwark (pull_request) Successful in 6s
Validate Helm Charts / Build calibre-web-automated (pull_request) Successful in 5s
Validate Helm Charts / Build cyberchef (pull_request) Successful in 5s
Validate Helm Charts / Build homeassistant (pull_request) Successful in 10s
Validate Helm Charts / Build garage (pull_request) Successful in 6s
Validate Helm Charts / Build gotenberg (pull_request) Successful in 11s
Validate Helm Charts / Build keycloak (pull_request) Successful in 5s
Validate Helm Charts / Build minecraft-router (pull_request) Successful in 8s
Validate Helm Charts / Build patchmon (pull_request) Successful in 8s
Validate Helm Charts / Build rallly (pull_request) Successful in 5s
Validate Helm Charts / Build satisfactory (pull_request) Successful in 5s
Validate Helm Charts / Build minecraft-server (pull_request) Successful in 7s
Validate Helm Charts / Build outline (pull_request) Successful in 5s
Validate Helm Charts / Build rss-bridge (pull_request) Successful in 5s
Validate Helm Charts / Build samba (pull_request) Successful in 5s
Validate Helm Charts / Build searxng (pull_request) Successful in 5s
Validate Helm Charts / Build sure (pull_request) Successful in 11s
Validate Helm Charts / Build teamspeak-server (pull_request) Successful in 18s
Validate Helm Charts / Build template-chart (pull_request) Successful in 12s
Validate Helm Charts / Build urlaubsverwaltung (pull_request) Successful in 4s
Validate Helm Charts / Build ytdlp-web-player (pull_request) Successful in 5s
Validate Helm Charts / Build voucher-vault (pull_request) Successful in 5s
Validate Helm Charts / Build zipline (pull_request) Successful in 4s
Validate Helm Charts / Trivy config bentopdf (pull_request) Successful in 13s
Validate Helm Charts / Trivy config bulwark (pull_request) Successful in 15s
Validate Helm Charts / Trivy config keycloak (pull_request) Successful in 12s
Validate Helm Charts / Trivy config minecraft-router (pull_request) Successful in 17s
Validate Helm Charts / Trivy config calibre-web-automated (pull_request) Successful in 11s
Validate Helm Charts / Trivy config cyberchef (pull_request) Successful in 13s
Validate Helm Charts / Trivy config garage (pull_request) Successful in 20s
Validate Helm Charts / Trivy config gotenberg (pull_request) Successful in 14s
Validate Helm Charts / Trivy config homeassistant (pull_request) Successful in 11s
Validate Helm Charts / Trivy config outline (pull_request) Successful in 14s
Validate Helm Charts / Trivy config rallly (pull_request) Successful in 15s
Validate Helm Charts / Trivy config minecraft-server (pull_request) Successful in 27s
Validate Helm Charts / Trivy config patchmon (pull_request) Successful in 12s
Validate Helm Charts / Trivy config rss-bridge (pull_request) Successful in 13s
Validate Helm Charts / Trivy config teamspeak-server (pull_request) Successful in 11s
Validate Helm Charts / Trivy config samba (pull_request) Successful in 16s
Validate Helm Charts / Trivy config satisfactory (pull_request) Successful in 22s
Validate Helm Charts / Trivy config searxng (pull_request) Successful in 22s
Validate Helm Charts / Trivy config sure (pull_request) Successful in 17s
Validate Helm Charts / Trivy config template-chart (pull_request) Successful in 14s
Validate Helm Charts / Trivy config urlaubsverwaltung (pull_request) Successful in 11s
Validate Helm Charts / Trivy config voucher-vault (pull_request) Successful in 12s
Validate Helm Charts / Trivy config ytdlp-web-player (pull_request) Successful in 17s
Validate Helm Charts / Trivy config zipline (pull_request) Successful in 18s
Install-test / Install-test bulwark (pull_request) Successful in 1m18s
Install-test / Install-test homeassistant (pull_request) Successful in 2m3s
Install-test / Install-test outline (pull_request) Successful in 2m9s
Install-test / Install-test rss-bridge (pull_request) Successful in 51s
Install-test / Install-test samba (pull_request) Successful in 1m14s
Install-test / Install-test apache-tika (pull_request) Successful in 1m7s
Install-test / Install-test bentopdf (pull_request) Successful in 1m5s
Install-test / Install-test calibre-web-automated (pull_request) Successful in 1m47s
Install-test / Install-test cyberchef (pull_request) Successful in 49s
Install-test / Install-test garage (pull_request) Successful in 1m10s
Install-test / Install-test gotenberg (pull_request) Successful in 1m35s
Install-test / Install-test keycloak (pull_request) Successful in 2m23s
Install-test / Install-test minecraft-router (pull_request) Successful in 37s
Install-test / Install-test minecraft-server (pull_request) Successful in 1m34s
Install-test / Install-test patchmon (pull_request) Successful in 1m17s
Install-test / Install-test rallly (pull_request) Successful in 1m30s
Install-test / Install-test searxng (pull_request) Successful in 1m9s
Install-test / Install-test sure (pull_request) Successful in 2m10s
Install-test / Install-test voucher-vault (pull_request) Successful in 1m48s
Install-test / Install-test ytdlp-web-player (pull_request) Successful in 59s
Install-test / Install-test zipline (pull_request) Successful in 1m59s
Validate Helm Charts / Install-test (pull_request) Successful in 33m44s
Install-test / Install-test teamspeak-server (pull_request) Successful in 1m15s
Install-test / Install-test urlaubsverwaltung (pull_request) Successful in 1m51s
feat!: NetworkPolicy in the NodePort and game-server charts, on by default
garage, minecraft-router, minecraft-server, samba, teamspeak-server,
satisfactory. One major each. These are the charts the plan said to leave for
last, and they earned it - three of them changed the convention rather than
just consuming it.

GARAGE is the clearest illustration after keycloak. Its four ports split three
ways: s3 and web are RESTRICTED by source (no probe touches them), admin is the
probe path and stays open inside the cluster - which is uncomfortable, because
admin is also the admin API, and that is said out loud instead of glossed - and
rpc gets the narrowest rule the object can express, a podSelector with no
namespaceSelector. All four numbers come from garage.settings.

MINECRAFT-SERVER is the tightest policy in the repository, and the reason is
worth knowing: all three of its probes are EXEC probes, which put no packet on
the network, so no port has to stay open for the kubelet and no probeSources
value is needed at all. RCON and metrics are restricted outright. The game port
has two branches rendered from the same value the Service reads -
server.service.type NodePort means players arrive from the internet and the
rule must be open; anything else means the only path is in-cluster, so the rule
is restricted and the router's namespace has to be in allowedNamespaces.

MINECRAFT-ROUTER derives its egress from the routing table instead of guessing:
server.backendPort plus the "port" override of every mapping that sets one. Add
a backend on a non-standard port and the policy follows. What it cannot express
is the destination - mappings name hosts, and this object matches no hostname -
so the rule is wider than the routing table, and that is written down.

SATISFACTORY gets policyTypes: [Ingress] and no egress section, and the second
half of the reason is the deciding one. Steam's content delivery is not a
pinnable port set (SteamCMD over TCP 80/443/27015-27050, master servers over
UDP 27000-27100 and 4380, changed by Valve, re-downloaded on every update) -
and this is the ONE chart with no install-test at all, because the image is
multiple GB and the first start pulls over 10GB. An egress cap here would be
the only one in the sweep with no empirical check behind it, on the chart whose
correct port set is least knowable.

SAMBA and TEAMSPEAK-SERVER carry the NodePort lesson explicitly: the rules name
445/139 and the pod's voice and file-transfer ports, never the NodePorts, and
teamspeak's file is explicit that its pod port and NodePort happening to share a
number is a coincidence of its values rather than a pattern to copy. Teamspeak's
voice rule is UDP, and the file says why that matters: a TCP rule on the same
number is accepted, looks complete, and drops every voice packet.

THREE THINGS THE CHECKS CAUGHT WHILE THIS BATCH WAS WRITTEN, all of them real:

  - minecraft-server's RCON rule named a port the pod does not declare. The
    containerPort is gated on server.rcon.enabled and the rule was not, so
    assertion 2 rejected it. The rule now carries the same gate.
  - samba and satisfactory would have shipped allowedNamespaces and
    probeSources values that reach NO rule - every one of their inbound rules
    is open to any source. A value that silently does nothing is worse than one
    that is absent, because it reads as a control, so both are gone from those
    two charts and from their schemas, with the reason in values.yaml.
    teamspeak-server loses probeSources for the same reason and keeps
    allowedNamespaces, which its web-query rule uses.
  - the policy templates no longer declare template variables they do not use.
    samba keeps one, satisfactory none.

Part of #5
2026-10-14 14:37:51 +00:00

593 lines
31 KiB
YAML

# Garage - an S3-compatible object store, deployed as a SINGLE NODE.
#
# READ THIS FIRST, because it is the premise every value below rests on
# (Gitea #81): this chart deploys ONE Garage node with replication_factor = 1,
# on the assumption that the volume layer underneath it is replicated and
# encrypted (longhorn-encrypted-ns in this cluster). Garage replication on top
# of that would be three times the raw data again, from two layers that know
# nothing about each other.
#
# WHAT THAT TRADE REALLY COSTS - named here because the chart cannot fix it:
# upstream says of replication_factor = 1 "Do not use this for anything else
# than test deployments", and the reason is that THE SAME FACTOR APPLIES TO THE
# METADATA. A corrupted metadata database is, in upstream's words, "generally
# not recoverable", and Longhorn does not help: it replicates the corrupted
# file three times. Redundancy against device failure, yes; against file
# corruption, no. What does help is the set of settings below -
# metadata_fsync, the automatic metadata snapshots, and those snapshots living
# on a second volume - and none of them is optional in this chart.
#
# THE IMAGE IS "FROM scratch" AND CONTAINS EXACTLY ONE FILE, /garage. No shell,
# no coreutils, no package manager. Three consequences that come up every time
# someone changes this chart:
# - exec probes are impossible here, for good. All three probes are HTTP or
# TCP (see the probe blocks below).
# - an init container would need a SECOND image to pin, track and update.
# Nothing needs one: Garage creates its own directories on start.
# - "kubectl exec" works, but only with /garage as the command and without a
# shell: kubectl exec <pod> -- /garage status
# Scale the whole chart down without uninstalling it: scaleDown: true renders
# the StatefulSet with replicas: 0 and takes precedence over garage.replicas.
# PVCs, Secrets, the Service and the Ingress stay in place; scaleDown: false
# brings the workload back up.
scaleDown: false
secrets:
# op:// path of the 1Password item carrying the field "rpc-secret": 32 bytes
# of random hex, i.e. "openssl rand -hex 32". REQUIRED - Garage uses it as
# the network key of its RPC layer and refuses to start without it, single
# node or not.
#
# IT ARRIVES AS AN ENVIRONMENT VARIABLE (GARAGE_RPC_SECRET), NOT AS A FILE,
# and that is the one place where this chart departs from how valkey does it.
# valkey argues explicitly for the file form - the password never appears in
# the process environment - and that argument is right in general. It cannot
# be followed here, and not because the file form would be inconvenient:
# IT IS STRUCTURALLY IMPOSSIBLE in a non-root pod.
#
# - Garage's rpc_secret_file rejects any file whose mode has a group OR
# other bit set: "if metadata.mode() & 0o077 != 0" in
# src/garage/secrets.rs (v2.4.1, line 144). The error message says
# "world-readable", but the mask tests the group bits as well.
# - The kubelet OR-s the group-read bit onto every file of a secret volume
# as soon as fsGroup is set: pkg/volume/volume_linux.go,
# changeFilePermission does Chmod(info.Mode() | 0440) for a read-only
# volume. So with fsGroup, Garage refuses to start.
# - Without fsGroup the file stays owned by root, and our uid 20000 cannot
# read it at all.
# - Kubernetes can set the GROUP of such a file, never its OWNER. There is
# no combination that satisfies both checks.
#
# The two ways around it were weighed and rejected (see Gitea #81):
# allow_world_readable_secrets = true would switch the permission check off
# for ALL of Garage's secret files, including ones nobody has set yet; and an
# init container copying the file with mode 0600 would need a second image
# for a scratch-based chart. What stays is the environment variable, which is
# also the dominant pattern in this repo (paperless-ngx, stalwart).
#
# The residual price, stated rather than hidden: the secret is in the
# container's process environment, so anyone who can exec into the pod or
# read its /proc can see it. Garage itself declares the variable with
# hide_env_values, so it does not print it in its own output.
#
# DO NOT "FIX" THIS BACK TO rpc_secret_file. The pod stops starting, and the
# error ("File /etc/garage-rpc/rpc-secret is world-readable!") points at a
# file mode nobody can change from inside Kubernetes.
rpcSecretRef: null
# op:// path of the 1Password item carrying the fields "access-key",
# "secret-key" and "bucket" - the S3 credential a CONSUMER uses, which is why
# it is a separate item from the RPC secret above: a consuming chart
# (stalwart's secrets.s3SecretRef) points at this same item, and keeping the
# bucket name in it means endpoint, bucket, key and secret have one source of
# truth instead of being kept in step by hand.
#
# Required while garage.singleNode is true together with defaultAccessKey or
# defaultBucket (the default), because Garage then IMPORTS these values on
# start. The direction is inwards: the vault item is written first, Garage
# adopts it, and nothing has to be copied out of the server afterwards.
#
# Garage validates all three and refuses to START on a malformed one:
# access-key at least 8 characters, ASCII alphanumeric plus - _ .
# secret-key at least 16 characters, printable ASCII
# bucket a valid S3 bucket name (3-63 characters, lowercase)
# NOTES.txt prints the generators for both keys.
#
# ROTATION IS NOT A RE-SAVE. Garage refuses to start when the access key
# already exists with a different secret ("is associated with a secret key
# different than the one given in GARAGE_DEFAULT_SECRET_KEY"), so rotating
# the secret means a NEW key id, or deleting the old key with the CLI first.
# NOTES.txt has the sequence.
s3SecretRef: null
garage:
# Replica count. 1 or 0 - and 0 is the documented way to stop the release
# without uninstalling it (the chart-global scaleDown flag does the same for
# everything at once).
#
# MORE THAN ONE REPLICA IS NOT A CLUSTER, which is why the schema caps this
# at 1: a second pod would come up as a lone node with its own empty metadata
# database on its own volume, because this chart renders neither
# bootstrap_peers nor the kubernetes_discovery CRD. With singleNode: true it
# would refuse to start at all once it saw the first node, which is the
# better of the two outcomes.
replicas: 1
# THE FIRST START, DONE BY GARAGE ITSELF INSTEAD OF BY HAND.
#
# --single-node makes the server create its own cluster layout on start
# (zone "dc1", capacity from the detected disk size) BEFORE it opens the S3,
# web and admin listeners. Without it, a fresh install has no layout, serves
# nothing, and stays not-Ready until an operator runs three CLI commands -
# which is a defensible way to run it, and NOTES.txt documents that path, but
# it is not a sane DEFAULT: a default install that never becomes Ready is not
# a default.
#
# IT CANNOT DAMAGE AN EXISTING CLUSTER, and that is worth knowing before
# trusting it with data. The layout version decides (src/garage/server.rs,
# initial_config): version 0 creates the layout, version 1 does nothing at
# all, anything above 1 aborts the start with "layout version is already
# superior to 1", and a node that sees other nodes aborts with "more nodes
# are already present in the cluster". It also requires replication_factor =
# 1 and says so if it is not. There is no --version argument here, so the
# trap of the CLI path - "never make multiple calls with the same value of
# the --version flag" - does not exist.
#
# Set to false to run the manual path (and then set defaultAccessKey and
# defaultBucket to false as well; the chart refuses the mixed combination,
# because Garage does).
singleNode: true
# Import the S3 access key from the vault item on start
# (--default-access-key, reading GARAGE_DEFAULT_ACCESS_KEY and
# GARAGE_DEFAULT_SECRET_KEY). Requires singleNode.
#
# This is what removes the one manual step that could not be automated in any
# other way: without it Garage GENERATES the key, and it would have to be
# copied out of the server into the vault by hand - a secret travelling in the
# wrong direction, through a terminal, with the vault as a copy rather than
# the original.
defaultAccessKey: true
# Create the bucket from the vault item's "bucket" field on start
# (--default-bucket) and grant the imported key read/write/owner on it.
# Implies defaultAccessKey upstream; requires singleNode.
defaultBucket: true
# ONE. Everything above is the reason, and raising it here is NOT an upgrade
# path: the factor is part of the cluster layout, so increasing it is a layout
# operation with data movement, on a cluster that would first need more nodes
# - which this chart does not deploy. The value exists so that a future
# multi-node setup is not blocked by the chart, not as a switch to flip.
replicationFactor: 1
# "lmdb" (Garage's default since 0.9 and the faster engine), "sqlite", or the
# experimental "fjall".
#
# The deviation to know about: upstream ties its LMDB recommendation to
# replication factor >= 2, and we run it at 1. The reported corruption after
# an unclean shutdown stays possible, metadata_fsync and the snapshots below
# only lower the probability, and at factor 1 there is no peer to recover
# from - only the last snapshot. sqlite is the defensible alternative for
# anyone who weighs a single-file database's durability higher than metadata
# throughput; with one mail server as the only consumer that ceiling is
# unlikely to be reached. Changing this on an EXISTING release needs
# "garage convert-db", not a values change.
dbEngine: lmdb
# ON, against Garage's own default of OFF. This is the single setting here
# most likely to be lost in a "clean up the obvious defaults" pass, so:
# Longhorn's block layer cannot help if the engine never calls fsync - a
# write that was never handed to the kernel is in no replica and in no
# snapshot. Garage's own example configuration pairs it with the snapshot
# interval below, as this chart does.
metadataFsync: true
# OFF, which is Garage's default and the right trade for the blocks: a lost
# block is one object, is detectable by a scrub, and is replaceable from the
# sender's copy; a lost metadata write can take the whole index with it.
# Turning it on costs an fsync per written block.
dataFsync: false
# THE ONLY CONSISTENT RECOVERY POINT THIS DEPLOYMENT HAS, and the reason the
# second PVC exists. File-system snapshots of a live metadata database
# guarantee nothing - Garage says so itself: a snapshot taken during a
# metadata write is corrupt and useless as a restore point, which is why
# Garage recommends its own mechanism in all cases. Longhorn backs up the
# blocks; it does not back up the metadata in any usable way.
#
# KNOW THE WINDOW BEFORE RELYING ON IT: Garage keeps exactly TWO snapshots,
# the first after interval/2 and every later one after interval times a
# random 1.0-1.2. At "6h" that is the first after ~3h and a restore window of
# roughly 6-14.4h afterwards. A corruption discovered later than that is
# older than every snapshot that still exists. Shorten the interval to narrow
# the window - at the cost of a compacted copy of the database every time.
#
# An empty string is rejected by the schema. A null falls back to this
# default, because switching the snapshots off is not a configuration this
# chart offers at replication factor 1.
metadataAutoSnapshotInterval: "6h"
# Garage's defaults, exposed because they are the two knobs that trade disk
# against CPU. blockSize is the chunk size objects are split into;
# compressionLevel is the zstd level for stored blocks (1 = fast, higher =
# smaller, negative levels are zstd's "fast" modes).
#
# TO DISABLE COMPRESSION, WRITE THE STRING "none" - and not 0, which is the
# trap this comment exists for: Garage's deserializer accepts an integer or
# the literal "none" (src/util/config.rs, deserialize_compression), and 0 is
# a perfectly valid integer that zstd reads as "use my default level", i.e.
# level 3 - MORE compression than the 1 below, not less. The schema therefore
# accepts an integer or exactly "none", and the ConfigMap quotes the string
# form so the rendered TOML stays valid.
blockSize: "1M"
compressionLevel: 1
# The host part of the S3, web and RPC listeners. "[::]" is upstream's
# default and accepts IPv4 as well, as long as net.ipv6.bindv6only is 0 -
# the Linux default, but a node sysctl this chart cannot see. Set "0.0.0.0"
# on an IPv4-only cluster that has changed it.
bindAddress: "[::]"
# The admin listener is bound on 0.0.0.0 SEPARATELY AND ON PURPOSE, because
# all three probes use it. If the kubelet's IPv4 probe could not reach a
# dual-stack listener, the symptom would be a pod that never becomes Ready
# for a reason nothing in the manifest points at. On an IPv6-only cluster set
# this to "[::]".
adminBindAddress: "0.0.0.0"
# The four ports, in ONE place: the ConfigMap renders them into garage.toml's
# bind addresses and the StatefulSet renders them as containerPorts, so a
# value here that disagreed with the other file would be a listener nothing
# routes to. 3900/3901/3902/3903 are upstream's defaults.
#
# The probes address the admin port by NAME ("admin"), not by number, so they
# follow a change here.
ports:
s3: 3900
rpc: 3901
web: 3902
admin: 3903
s3:
# The region name clients must use. Not a cloud region - a label both sides
# have to agree on; a consumer configured for a different one gets a
# signature error, not a redirect.
region: garage
# Optional suffix for vhost-style bucket access (<bucket>.<rootDomain>).
# Only useful together with an Ingress and a wildcard DNS entry;
# path-style access works without it and is what an in-cluster consumer
# uses. Deliberately NOT ingress.domain: the two are different things, and
# coupling them would make the Ingress host decide how buckets are
# addressed.
rootDomain: null
web:
# Static website hosting (the s3_web endpoint), OFF. It is a second HTTP
# listener, it serves nothing at all without a rootDomain, and the consumer
# this chart was built for speaks S3. Switching it on renders the
# containerPort, the Service port and the [s3_web] section.
enabled: false
rootDomain: null
resources:
# Limits are optional: memory and ephemeral-storage limits default to
# limitFactor x the request, an explicitly set limit always wins, and a CPU
# limit is only set when configured explicitly under "limits" - CPU is
# compressible, so leaving it unlimited throttles instead of OOM-killing.
#
# 512Mi x 3 = 1536Mi memory limit. Garage's own guidance is "at least 1GB"
# for a node, and LMDB's memory-mapped database makes the RESIDENT size
# follow the working set rather than the database size. The ephemeral
# request covers container logs only: everything Garage writes goes to one
# of the two PVCs.
limitFactor: 3
requests:
cpu: 100m
memory: 512Mi
ephemeral-storage: 32Mi
# LIVENESS: a TCP connect to the admin port, deliberately NOT the /health
# endpoint readiness uses. /health answers 503 while no cluster layout
# exists, so a liveness probe on it would kill and restart a node that is
# merely waiting for its layout - forever, and a restart is exactly what
# cannot fix that. Same lesson as zipline (README, guideline 12.3), in its
# sharpest form.
livenessProbe:
tcpSocket:
port: admin
periodSeconds: 10
timeoutSeconds: 5
failureThreshold: 3
# READINESS: GET /health on the admin port - the one endpoint that answers
# the question a readiness probe asks.
#
# MEASURED AGAINST THE SOURCE AT v2.4.1, NOT CHOSEN: the S3 API server is
# spawned independently of the layout state, so the S3 socket binds and
# answers - with errors - before any layout exists. A tcpSocket probe on 3900
# would therefore report a service ready that accepts nothing. /health needs
# no token and maps the cluster health straight onto the status code: 200 for
# Healthy and Degraded, 503 for Unavailable, and an unconfigured layout is a
# special case that yields Unavailable.
#
# So with singleNode: true the pod becomes Ready a moment after start, and
# with singleNode: false it stays not-Ready until a layout is applied by
# hand. The second case is the truth and not a defect: until then the S3
# endpoint accepts nothing and no Service should route to it.
readinessProbe:
httpGet:
path: /health
port: admin
periodSeconds: 10
timeoutSeconds: 5
failureThreshold: 3
# STARTUP: TCP again, for the same reason as liveness - a startup probe that
# fails forever kills the container, so it must not depend on a layout only
# an operator can create. The startup probe gates liveness and readiness,
# which is why those carry no initialDelaySeconds.
#
# Budget = periodSeconds * failureThreshold = 120s. It covers the slow part
# of a real start, which is not Garage: attaching and mounting two network
# volumes, plus opening (and after an unclean shutdown recovering) the LMDB
# database. Garage's own startup work after that is sub-second.
startupProbe:
tcpSocket:
port: admin
periodSeconds: 5
timeoutSeconds: 5
failureThreshold: 24
securityContext:
pod:
runAsNonRoot: true
# Any uid works: the image is FROM scratch, so there is no /etc/passwd to
# contradict and nothing in it is owned by a user. 20000 is this repo's
# default. Note that upstream's own chart uses 1000 - an install migrated
# from it would need chown'ed volumes or this value changed to 1000.
runAsUser: 20000
runAsGroup: 1000
fsGroup: 1000
# OnRootMismatch, DEVIATING from this repo's "Always" default, and the
# reason is the data: an object store's data directory holds one file per
# block, which is hundreds of thousands of files for a few hundred GB.
# "Always" chowns all of them recursively on EVERY pod start, which turns
# a restart into a multi-minute outage that gets worse as the store grows.
# OnRootMismatch checks the top directory's ownership and skips the walk
# when it already matches - which it does, from the second start onwards.
# Same reasoning as satisfactory's documented deviation.
fsGroupChangePolicy: OnRootMismatch
seccompProfile:
type: RuntimeDefault
container:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
runAsNonRoot: true
capabilities:
drop:
- ALL
# EMPTY, and it can stay empty: the ports are all above 1024, and the
# binary carries no file capabilities (it is a static Rust binary in a
# scratch image), so there is no execve-time capability check like
# stalwart's. Nothing here needs a capability.
add: []
# Additive environment entries for the Garage container, rendered through tpl
# and supporting the value, secretKeyRef and configMapKeyRef forms.
#
# ADDITIVE MEANS ADDITIVE: an entry repeating a name the chart already sets
# does not replace it. At install time both reach the pod and the later one
# wins; on an EXISTING release the pair collapses to the chart's own value,
# because Kubernetes merges env by name - and helm upgrade still reports
# success. So do not use this to override GARAGE_RPC_SECRET or the
# GARAGE_DEFAULT_* variables; change the values that produce them.
#
# Useful entries that are genuinely additive: RUST_LOG (the image sets
# garage=info) and GARAGE_ALLOW_PUNYCODE.
env: []
storage:
# Both blocks are deliberately NOT null-safe: an erased size yields an
# invalid PVC that is rejected at apply time, which is loud. A chart-side
# fallback would silently size somebody's object store - and on an existing
# release it could not help anyway, because a PVC cannot shrink.
data:
# metadata_dir AND data_dir, as two subdirectories of this ONE volume.
# Together on purpose: Longhorn snapshots are atomic per volume and never
# across volumes, so this is what keeps index and blocks consistent with
# each other in a snapshot. 10Gi is a starting size for a mail blob store,
# not a recommendation - it is the value most likely to need changing
# before the first install.
amount: 10Gi
# null uses the cluster's default StorageClass. In this cluster the
# intended value is longhorn-encrypted-ns, which is also where the backup
# story lives - and note what this chart does NOT render: the Longhorn
# recurring-job labels (daily-snapshot, daily-backup) belong to the
# consuming bundle, and without them the off-site half of the backup plan
# does not exist, it only looks like it does.
storageClass: null
snapshots:
# metadata_snapshots_dir. Sized for TWO compacted copies of the metadata
# database plus head room, not for the objects - Garage keeps exactly two
# and deletes older ones itself.
amount: 5Gi
storageClass: null
ingress:
# OFF. The consumer this chart was built for speaks S3 inside the cluster, so
# there is nothing to publish. The values exist, built to this repo's
# convention, so that publishing the endpoint later is a values change.
#
# Mutually exclusive with httpRoute.enabled below (Gitea #114): with both
# true the render aborts, naming both values. Both ship off, so this chart
# publishes nothing until asked - in either mode.
#
# Guideline 1, checked in both directions, and the answer MOVED with #114:
# the Ingress template is no longer the only reader of ingress.domain - the
# HTTPRoute reads it too, as its single hostname, when httpRoute.hostnames is
# empty. So domain is free only while BOTH routing modes are off; with either
# one on it is required, and the required() fires in whichever template is
# rendering. clusterIssuer is unchanged: it is read only by the Ingress, so
# ingress.enabled: false frees it in every case - Gateway mode renders no
# Certificate at all.
#
# Unchanged as well: the vhost suffix is its own value, garage.s3.rootDomain,
# and is still deliberately not coupled to ingress.domain.
enabled: false
clusterIssuer: null
domain: null
className: nginx
# Extra annotations for the Ingress; merged with the cert-manager
# cluster-issuer annotation the chart always sets. An S3 endpoint behind
# nginx usually wants nginx.ingress.kubernetes.io/proxy-body-size: "0" -
# without it the controller's 1m default truncates uploads.
#
# THAT ONE HAS NO COUNTERPART IN GATEWAY MODE. The Gateway API has no
# annotation channel and no notion of a body size at all - an upload limit is
# implementation-specific policy - so an S3 endpoint published through an
# HTTPRoute silently inherits the Gateway's limit, whatever it is. For an
# object store that is not a detail: it is where PUTs start failing by size.
# The chart does not rebuild it (a policy object would bind it to one
# implementation); NOTES.txt prints the warning in that mode and the limit
# has to be raised at the Gateway.
annotations: {}
# Gateway-API alternative to the Ingress above (Gitea #114). Off by default -
# like the Ingress here, so this chart still publishes nothing unless asked.
# Exactly one of ingress.enabled and httpRoute.enabled may be true.
#
# The chart renders ONLY an HTTPRoute with parentRefs - no Gateway, no
# gatewayClassName, no Certificate and no implementation-specific policy
# object. Class and certificate belong to the Gateway and therefore to the
# platform, which is what keeps this chart independent of whether the cluster
# runs nginx-gateway-fabric, Cilium, Envoy Gateway or Istio.
#
# The Gateway-API CRDs must be installed in the cluster: the chart does not and
# cannot detect them (.Capabilities.APIVersions is unreliable at template
# time), so with httpRoute.enabled: true and no CRDs the INSTALL fails.
#
# WHAT GETS PUBLISHED IS THE S3 API, in this mode exactly as in Ingress mode:
# the route points at the Service's s3 port and nothing else. The admin port is
# deliberately not routed. Read the Ingress comment above before turning either
# mode on - and read the upload-limit note, which is the one real regression of
# this mode for an object store.
httpRoute:
enabled: false
# The Gateway(s) this route attaches to - a platform object, usually in
# another namespace, which must allow this namespace in its listener's
# allowedRoutes. One entry per parent, the fields are the Gateway API's
# ParentReference (name, and optionally namespace, sectionName, port, group,
# kind), passed through unchanged:
#
# parentRefs:
# - name: public
# namespace: gateway
# sectionName: https
#
# REQUIRED while httpRoute.enabled is true - the render aborts on an empty
# list, because a route without a parent is created, is valid and routes
# nothing, with no event to say so.
parentRefs: []
# Empty means [ingress.domain], so the hostname has one source of truth. Set
# explicitly only to publish under a different or an additional name - which
# is how a migration from the Ingress runs: the HTTPRoute takes a second
# hostname while the Ingress keeps ingress.domain.
#
# For vhost-style bucket access this is where the wildcard goes
# ("*.s3.example.com" alongside the plain host), together with
# garage.s3.rootDomain - the two still have to be set in step, as they do for
# the Ingress.
hostnames: []
# ---------------------------------------------------------------------------
# NetworkPolicy (Gitea #5). On by default. The reference implementation with
# the full reasoning is template-chart; the convention is in README, "Chart
# style guidelines", and the operator's view on this chart's wiki page.
# ---------------------------------------------------------------------------
networkPolicy:
# WHAT THIS POLICY ALLOWS, and Garage's four ports split three ways:
# - s3 (3900) and web (3902): RESTRICTED BY SOURCE to this namespace,
# allowedNamespaces and the ingress controller. No probe touches them,
# which is what makes the restriction possible.
# - admin (3903): open inside the cluster, because all three probes address
# it. This is the ADMIN API - key management and cluster layout - so the
# cap is "reachable from inside the cluster", not "closed". probeSources
# closes the rest.
# - rpc (3901): this release's own pods only. It has no Service at all.
# Outbound: DNS and the rpc bus, nothing else.
#
# THE SOURCE LIST FOR s3 IS THE THING TO GET RIGHT. Every S3 consumer in
# ANOTHER namespace - a backup job, a Zipline storing uploads, a Stalwart
# blob store - is blocked unless that namespace is in allowedNamespaces.
# Consumers that use the published Ingress name arrive through the controller
# and are already covered. A blocked consumer fails at the application level,
# with nothing naming this object.
#
# NOT allowed, and the plausible surprise: outbound HTTPS. Garage does not
# need it to serve S3, but remote replication or an external metrics sink
# does - see egress.extraPorts.
enabled: true
# Namespaces whose pods may reach this workload, in addition to the release's
# own namespace, which is always allowed. Empty means: own namespace only.
#
# Matched on the "kubernetes.io/metadata.name" label Kubernetes sets on every
# namespace itself, so no labelling work is needed on a namespace this chart
# does not own - and no typo protection either: a name that does not exist
# matches nothing and is silently ineffective. Note the granularity: an entry
# opens the port to EVERY pod in that namespace.
allowedNamespaces: []
# Source CIDRs allowed to reach the admin port (garage.ports.admin, 3903 by default), which is the port all three probes address.
#
# EMPTY MEANS OPEN, NOT BLOCKED. With no entry that rule is rendered WITHOUT a
# "from" selector, so the port is reachable from anywhere the CNI routes to -
# inside the cluster, since it carries no NodePort.
#
# That is the deliberate choice between two bad defaults. The kubelet's probes
# originate from the IP of the WORKER NODE running the pod, an address this
# chart cannot know and must not guess: a CIDR that excludes the real node
# blocks every probe, the pod never becomes Ready, the rollout times out, and
# the symptom reads as a broken application rather than as a policy.
#
# Narrow it by listing the worker node IPs (e.g. ["10.0.1.11/32"]) - and mind
# the companion trap: once this list is non-empty the rule IS restricted, so
# everything else that needs the port has to be in it. The chart adds the own
# namespace, the ingressController below and allowedNamespaces to the same rule for exactly that reason.
probeSources: []
# The ingress controller allowed to reach the routed port once probeSources
# restricts it. Rendered only while ingress.enabled or httpRoute.enabled is true, and only in
# the restricted branch - with the default (open) rule the controller reaches
# the port anyway. The two keys become ONE "from" entry and are therefore an
# AND: pods carrying podLabels IN namespace. That is the point of having them
# at all - the controller's namespace in allowedNamespaces would open the port
# to every pod in kube-system.
#
# MEASURED ON THIS CLUSTER, NOT ASSUMED (2026-10-14): ingress-nginx is gone,
# the only IngressClass is "traefik" and it is the default, and the controller
# is the DaemonSet rke2-traefik in kube-system. The same Traefik serves the
# Gateway API here, so this one source covers the Ingress and the HTTPRoute
# path alike. A policy written against the wrong controller is accepted, looks
# strict, and silently blocks all web traffic - and ingress.className says
# nothing about it, because a class names an implementation, not the pods the
# traffic comes from.
ingressController:
namespace: kube-system
podLabels:
app.kubernetes.io/name: rke2-traefik
egress:
# Additional outbound TCP ports. DNS (UDP and TCP 53) is always allowed and
# has no switch - a knob whose only other setting breaks every name lookup
# in the pod is a trap, not a feature.
#
# PORTS, NOT DESTINATIONS, and that is a limit of the object rather than
# laziness: a NetworkPolicy matches CIDRs and selectors and never hostnames,
# so "only to the database host" cannot be written at all. Each entry
# permits its port to ANY destination. What the cap still buys is real: a
# compromised pod cannot open arbitrary outbound connections.
#
# 443 is the entry to add for remote replication or an external metrics
# sink. Nothing is needed for plain S3 serving.
extraPorts: []