jklein
d9267b2180
Second install-test run, three findings, all measured: - Readiness on "/" failed with "http: server gave HTTP response to HTTPS client" on a pod that was answering correctly. The kubelet's HTTP prober follows redirects to the same host, and with RAILS_ASSUME_SSL=true the app builds its redirect as https://<pod-ip>:3000/registration/new, which Puma does not speak. The probe now runs curl without -L and accepts 2xx-3xx itself. The same log also proves what the probe is for: against an empty database the redirect target was /registration/new, the branch User.count decides, so the path really does reach Postgres. - The pod was evicted within two minutes of every start: "ephemeral local storage usage exceeds the total limit of containers 60Mi". emptyDirs and the stdout log both count there, and the image rebuilds its Bootsnap cache under /rails/tmp on every start because the emptyDir hides the one baked into the image. The request is now 256Mi (limit 768Mi) per app container - headroom, not a measurement. - /home/rails is read-only, so Bundler created a throwaway home under TMPDIR on every invocation, worker probe included. HOME now points at a directory the init container creates. The CI postgres fixture gets -d sure on its readiness probe, so the server stops logging a missing database that nothing asked for.
313 lines
16 KiB
YAML
313 lines
16 KiB
YAML
# Scale the whole chart down without uninstalling it: scaleDown: true renders
|
|
# every Deployment/StatefulSet with replicas: 0 (and suspends CronJobs) and
|
|
# takes precedence over any per-workload replicas value. PVCs, Secrets and
|
|
# Services stay in place; scaleDown: false brings the workloads back up.
|
|
scaleDown: false
|
|
|
|
# 1Password item paths (op://<vault>/<item>). The 1Password operator turns each
|
|
# OnePasswordItem into a Kubernetes Secret whose keys are the FIELD NAMES of the
|
|
# item; the chart references those field names, so a renamed field is a pod in
|
|
# CreateContainerConfigError, not a render error. NOTES.txt lists every field.
|
|
#
|
|
# Helm never sees the CONTENT of these secrets, so no checksum/config-style
|
|
# annotation can cover them (README, guideline 9). The pod template does carry a
|
|
# checksum/secret-ref over the rendered OnePasswordItems, so REPOINTING one of
|
|
# the *SecretRef values at a different item restarts the pod; editing a value
|
|
# INSIDE the same item does not, and needs a manual
|
|
# kubectl rollout restart statefulset/<release>-sure-sfs
|
|
# The operator's operator.1password.io/auto-restart annotation is deliberately
|
|
# NOT set: it only ever restarts Deployments, so on this StatefulSet it would
|
|
# advertise a guarantee that does not exist (README, guideline 9).
|
|
secrets:
|
|
# SECRET_KEY_BASE (field: secret-key-base)
|
|
appSecretRef: null
|
|
# PostgreSQL connection (fields: host, port, database, username, password)
|
|
dbSecretRef: null
|
|
# Redis connection URL (field: connection-url)
|
|
redisSecretRef: null
|
|
|
|
sure:
|
|
# Replica count for the StatefulSet; 0 is allowed and scales it down. The
|
|
# chart-global scaleDown flag takes precedence.
|
|
#
|
|
# Both containers live in the SAME pod, so this value scales web and worker
|
|
# together. That is a consequence of the single ReadWriteOnce PVC they share
|
|
# at /rails/storage (Active Storage): splitting them into two workloads would
|
|
# need a ReadWriteMany volume. Above 1 the extra pods normally stay Pending on
|
|
# "Multi-Attach error for volume" - see NOTES.txt.
|
|
replicas: 1
|
|
# Pod-level hardening, shared by both containers and by the init container.
|
|
# The upstream image declares USER 1000:1000 (group and user "rails", created
|
|
# in the Dockerfile), so the chart needs no deviation from the repo hardening
|
|
# standard; fsGroup 1000 makes the storage PVC writable for that uid.
|
|
#
|
|
# readOnlyRootFilesystem: true holds, and the writable paths are emptyDirs:
|
|
# /rails/tmp, /rails/log and /tmp, one set per container. /tmp needs one extra
|
|
# step that is easy to mistake for decoration - a tiny init container creates
|
|
# two private subdirectories in it, and the containers run with TMPDIR and
|
|
# HOME pointing there. Ruby refuses a world-writable directory without a
|
|
# sticky bit as its temporary directory and an emptyDir is exactly that, so
|
|
# without this the app aborts during boot; HOME follows because the image's
|
|
# /home/rails is read-only and Bundler otherwise builds a throwaway home per
|
|
# invocation. The StatefulSet carries the measured evidence for both.
|
|
securityContext:
|
|
pod:
|
|
runAsNonRoot: true
|
|
runAsUser: 1000
|
|
runAsGroup: 1000
|
|
fsGroup: 1000
|
|
fsGroupChangePolicy: Always
|
|
# RuntimeDefault applies the container runtime's default seccomp filter,
|
|
# which blocks the syscalls a normal workload never makes. It sits in the
|
|
# pod context on purpose: from here it covers every container in the pod
|
|
# - web and worker alike - including init containers, while a
|
|
# container-level seccompProfile would have to be repeated for each one
|
|
# (and silently missed on the next one added). A container that genuinely
|
|
# needs a different filter can still override it under
|
|
# <container>.securityContext.container, which wins over this value.
|
|
seccompProfile:
|
|
type: RuntimeDefault
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# web - Puma, the container that also runs the database migrations
|
|
# ---------------------------------------------------------------------------
|
|
web:
|
|
securityContext:
|
|
container:
|
|
allowPrivilegeEscalation: false
|
|
readOnlyRootFilesystem: true
|
|
runAsNonRoot: true
|
|
capabilities:
|
|
drop:
|
|
- ALL
|
|
resources:
|
|
# Limits are optional: memory and ephemeral-storage limits default to
|
|
# limitFactor x the request, an explicitly set limit always wins, and a
|
|
# CPU limit is only set when configured explicitly under "limits".
|
|
#
|
|
# The request follows upstream's own sizing note (docs/hosting/docker.md
|
|
# for 0.7.4): steady-state memory "around 352 MB", comfortable under a
|
|
# 512 MB limit, while "if you enable AI, run at least 1 GB". 512Mi request
|
|
# x limitFactor 3 = 1536Mi limit, which holds both the everyday figure and
|
|
# the AI recommendation without reserving the larger number on the node.
|
|
# Read from upstream's documentation, not measured here.
|
|
#
|
|
# ephemeral-storage is NOT the template default of 10Mi, and the reason is
|
|
# measured: with 10Mi (limit 30Mi) per container the kubelet evicted the
|
|
# pod within two minutes of every start - "Pod ephemeral local storage
|
|
# usage exceeds the total limit of containers 60Mi". Both emptyDirs and
|
|
# the container's stdout log count against that limit, and this image
|
|
# rebuilds its Bootsnap cache under /rails/tmp on every pod start because
|
|
# the emptyDir hides the one baked into the image. 256Mi request (768Mi
|
|
# limit) is headroom for that, not a measurement of it - the steady-state
|
|
# figure is still unknown.
|
|
limitFactor: 3
|
|
requests:
|
|
cpu: 0.1
|
|
memory: 512Mi
|
|
ephemeral-storage: 256Mi
|
|
# PROBES - what each one actually proves (README, guideline 12.3)
|
|
#
|
|
# /up is Rails' own health route (config/routes.rb: get "up" =>
|
|
# "rails/health#show"). READ FROM THE RAILS SOURCE, NOT MEASURED HERE: that
|
|
# controller renders 200 as soon as the app has booted without raising and
|
|
# touches NO database and NO Redis. It is a boot check, and the widespread
|
|
# belief that it tests the database is exactly what guideline 12.3 exists to
|
|
# catch - upstream's own Helm chart (charts/sure in the app repository) uses
|
|
# /up for readiness, which is the mistake this chart does not copy.
|
|
#
|
|
# Liveness therefore uses /up: a Postgres outage must not restart the pod.
|
|
livenessProbe:
|
|
httpGet:
|
|
path: /up
|
|
port: http
|
|
periodSeconds: 10
|
|
timeoutSeconds: 5
|
|
failureThreshold: 3
|
|
# Readiness asks for "/" instead, because that path provably reaches the
|
|
# database - and it does so through curl rather than an httpGet probe, for
|
|
# a reason that was measured rather than guessed (see below).
|
|
#
|
|
# Why "/" reaches the database: every request runs ApplicationController's
|
|
# authenticate_user! before_action (app/controllers/concerns/
|
|
# authentication.rb); without a session cookie - which a probe never has -
|
|
# it falls into self_hosted_first_login?, i.e.
|
|
# "config.app_mode.self_hosted? && User.count.zero?". User.count is an SQL
|
|
# query. Measured in the install-test: against an empty database the app
|
|
# answered 302 to /registration/new, which is the branch that query decides,
|
|
# so the path demonstrably touches Postgres. With the database down the
|
|
# query raises and Rails answers 500.
|
|
#
|
|
# Why exec + curl and not httpGet: the kubelet's HTTP prober FOLLOWS
|
|
# redirects to the same host, and with RAILS_ASSUME_SSL=true (which the
|
|
# chart sets whenever the Ingress is on) Rails builds that redirect as
|
|
# https://<pod-ip>:3000/... - a scheme Puma does not speak. The install-test
|
|
# failed with "http: server gave HTTP response to HTTPS client" on a pod
|
|
# that was answering correctly. curl without -L asks once and reports the
|
|
# status code; the test accepts 2xx and 3xx exactly as the kubelet would,
|
|
# and a 500 from a missing database still fails. curl is part of the image
|
|
# (the Dockerfile installs it), so nothing extra is pulled in for this.
|
|
#
|
|
# One condition this depends on: SELF_HOSTED=true, which the chart always
|
|
# sets. In managed mode the same code short-circuits before User.count and
|
|
# "/" would redirect without touching Postgres at all.
|
|
readinessProbe:
|
|
exec:
|
|
command:
|
|
- /bin/bash
|
|
- -c
|
|
- "code=\"$(curl -sS -o /dev/null -w '%{http_code}' --max-time 5 http://127.0.0.1:3000/)\"; test \"$code\" -ge 200 -a \"$code\" -lt 400"
|
|
periodSeconds: 10
|
|
timeoutSeconds: 10
|
|
failureThreshold: 3
|
|
# The startup probe gates liveness and readiness, so neither needs an
|
|
# initialDelaySeconds. Its budget has to cover the first start, which is the
|
|
# expensive one: the image entrypoint (/rails/bin/docker-entrypoint) runs
|
|
# "./bin/rails db:prepare" whenever the command is "./bin/rails server", so
|
|
# the web container creates and migrates the schema before Puma binds the
|
|
# port. Budget 10s x 60 = 600s. Raise it before a large app-version jump;
|
|
# a restart in the middle of a migration is the failure mode it prevents.
|
|
startupProbe:
|
|
httpGet:
|
|
path: /up
|
|
port: http
|
|
periodSeconds: 10
|
|
timeoutSeconds: 5
|
|
failureThreshold: 60
|
|
# Extra environment variables, appended to the ones the chart always sets.
|
|
# Values are rendered through tpl; value, secretKeyRef and configMapKeyRef
|
|
# forms are supported.
|
|
#
|
|
# This is also where the AI features go - the chart models none of them as
|
|
# values on purpose. OPENAI_ACCESS_TOKEN (secretKeyRef form), OPENAI_MODEL,
|
|
# OPENAI_URI_BASE, LLM_CONTEXT_WINDOW, OPENAI_REQUEST_TIMEOUT,
|
|
# AI_RESPONSE_TIMEOUT, ASSISTANT_MAX_TOOL_CALL_ITERATIONS and the SMTP_*
|
|
# block are plain environment variables to this image. Set them on BOTH
|
|
# containers where a background job needs them as well (the assistant runs
|
|
# in the worker).
|
|
#
|
|
# Additive means additive: an entry repeating a name the chart already sets
|
|
# reaches the pod twice at install time (the later one, yours, wins) but is
|
|
# LOST on a later helm upgrade, because Kubernetes merges env by name and
|
|
# the pair collapses to the chart's own value - with a green upgrade either
|
|
# way (README, guideline 10).
|
|
env: []
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# worker - Sidekiq: the same image, started with "bundle exec sidekiq" as
|
|
# its args (the image entrypoint stays in place; see the StatefulSet)
|
|
# ---------------------------------------------------------------------------
|
|
worker:
|
|
securityContext:
|
|
container:
|
|
allowPrivilegeEscalation: false
|
|
readOnlyRootFilesystem: true
|
|
runAsNonRoot: true
|
|
capabilities:
|
|
drop:
|
|
- ALL
|
|
resources:
|
|
# Same sizing as the web container, for a documented reason: upstream
|
|
# notes that the sample/demo-data generator runs in the WORKER and is the
|
|
# one operation that deterministically exceeds 512 MB (observed flat at
|
|
# ~680 MB on a 2 GB container). The computed limit of 3 x 512Mi = 1536Mi
|
|
# covers that; the request stays at the everyday figure.
|
|
#
|
|
# ephemeral-storage: same 256Mi as the web container and for the same
|
|
# measured reason - see the comment there.
|
|
limitFactor: 3
|
|
requests:
|
|
cpu: 0.1
|
|
memory: 512Mi
|
|
ephemeral-storage: 256Mi
|
|
# PROBES - the worker speaks no HTTP at all, so all three are exec probes.
|
|
#
|
|
# The command asks Sidekiq's own monitor whether THIS process is in the
|
|
# process set: sidekiqmon prints one line per registered process, each
|
|
# starting with the identity "<hostname>:<pid>:<nonce>" (sidekiq 8.0.5,
|
|
# lib/sidekiq/component.rb), and in a pod the hostname is the pod name that
|
|
# bash exposes as $HOSTNAME. Sidekiq keeps that registration alive with a
|
|
# heartbeat whose Redis key expires after about 60s, so a process that hangs
|
|
# disappears from the set while its PID still exists - which is precisely
|
|
# what a process check like "pgrep sidekiq" could not tell you (and would
|
|
# not tell you anything at all: Sidekiq is PID 1 here, so if it dies the
|
|
# container dies with it and the check can never fail - a check a broken
|
|
# system passes, README).
|
|
#
|
|
# READ FROM THE SIDEKIQ 8.0.5 SOURCE, NOT MEASURED: sidekiqmon is a plain
|
|
# Ruby entry point that requires sidekiq/api only - it does NOT boot Rails -
|
|
# and it reads REDIS_URL from the environment, which the chart sets on this
|
|
# container. Cost per check is therefore one Ruby start, not an application
|
|
# boot; the periods below are chosen on the assumption that this takes a few
|
|
# seconds, and the PR install-test is the first place it actually runs.
|
|
#
|
|
# Liveness hangs on Redis, which guideline 12.3 allows only where no
|
|
# dependency-free path proves anything - that is the case here (see above) -
|
|
# and it is given a deliberately generous budget of 60s x 5 = 300s of
|
|
# missing heartbeat, so a short Redis blip does not restart the pod.
|
|
livenessProbe:
|
|
exec:
|
|
command:
|
|
- /bin/bash
|
|
- -c
|
|
- bundle exec sidekiqmon processes | grep -q "^${HOSTNAME}:"
|
|
periodSeconds: 60
|
|
timeoutSeconds: 15
|
|
failureThreshold: 5
|
|
# Readiness uses the same check with a tighter budget: a worker that is not
|
|
# registered in Redis is not doing any work, whatever its process state.
|
|
readinessProbe:
|
|
exec:
|
|
command:
|
|
- /bin/bash
|
|
- -c
|
|
- bundle exec sidekiqmon processes | grep -q "^${HOSTNAME}:"
|
|
periodSeconds: 30
|
|
timeoutSeconds: 15
|
|
failureThreshold: 3
|
|
# The startup probe gates the other two. Sidekiq boots the full Rails app
|
|
# before it registers, and on a first install it waits for the web container
|
|
# to finish db:prepare, so the budget matches the web container's: 20s x 30
|
|
# = 600s.
|
|
startupProbe:
|
|
exec:
|
|
command:
|
|
- /bin/bash
|
|
- -c
|
|
- bundle exec sidekiqmon processes | grep -q "^${HOSTNAME}:"
|
|
periodSeconds: 20
|
|
timeoutSeconds: 15
|
|
failureThreshold: 30
|
|
# Extra environment variables for the worker, appended to the ones the chart
|
|
# always sets. Same rules as sure.web.env above.
|
|
env: []
|
|
|
|
# The Active Storage volume, mounted at /rails/storage in BOTH containers.
|
|
# Deliberately not null-safe: an erased block yields an invalid PVC, which fails
|
|
# loudly (README, guideline 12).
|
|
storage:
|
|
amount: 10Gi
|
|
storageClass: null
|
|
|
|
ingress:
|
|
# Gates the whole Ingress template. Note that ingress.domain is read three
|
|
# more times outside it - APP_DOMAIN (the host Action Mailer builds link URLs
|
|
# from), WEBAUTHN_RP_ID and WEBAUTHN_ALLOWED_ORIGINS (the relying party a
|
|
# passkey is bound to) - so it keeps its own required() there and stays
|
|
# mandatory with the Ingress switched off; only clusterIssuer becomes
|
|
# unnecessary (README, guideline 12.1). Without that guard the app would come
|
|
# up with WEBAUTHN_RP_ID="" and register passkeys against the wrong relying
|
|
# party - rendered cleanly, pod green.
|
|
#
|
|
# ingress.enabled also decides how the app is told about TLS: enabled renders
|
|
# RAILS_ASSUME_SSL=true and RAILS_FORCE_SSL=true (the Ingress terminates TLS,
|
|
# so Rails must trust that and may set secure cookies and HSTS), disabled
|
|
# renders both as "false" so a plain-HTTP port-forward can log in at all.
|
|
enabled: true
|
|
clusterIssuer: null
|
|
domain: null
|
|
className: nginx
|
|
# Extra annotations for the Ingress; merged with the cert-manager
|
|
# cluster-issuer annotation the chart always sets.
|
|
annotations: {}
|