sure: fix the readiness probe, the ephemeral-storage budget and HOME
Publish Helm Chart with patched Application Version / publish-helm-chart (push) Successful in 6s
Publish Sure Helm Chart / Publish Sure Helm Chart (push) Successful in 6s

Second install-test run, three findings, all measured:

- Readiness on "/" failed with "http: server gave HTTP response to HTTPS
  client" on a pod that was answering correctly. The kubelet's HTTP prober
  follows redirects to the same host, and with RAILS_ASSUME_SSL=true the
  app builds its redirect as https://<pod-ip>:3000/registration/new, which
  Puma does not speak. The probe now runs curl without -L and accepts
  2xx-3xx itself. The same log also proves what the probe is for: against
  an empty database the redirect target was /registration/new, the branch
  User.count decides, so the path really does reach Postgres.
- The pod was evicted within two minutes of every start: "ephemeral local
  storage usage exceeds the total limit of containers 60Mi". emptyDirs and
  the stdout log both count there, and the image rebuilds its Bootsnap
  cache under /rails/tmp on every start because the emptyDir hides the one
  baked into the image. The request is now 256Mi (limit 768Mi) per app
  container - headroom, not a measurement.
- /home/rails is read-only, so Bundler created a throwaway home under
  TMPDIR on every invocation, worker probe included. HOME now points at a
  directory the init container creates.

The CI postgres fixture gets -d sure on its readiness probe, so the server
stops logging a missing database that nothing asked for.
This commit was merged in pull request #36.
This commit is contained in:
jklein authored and Justin Klein committed 2026-09-26 21:08:55 +00:00
1 parent a414af089d
commit d9267b2180
4 files changed
+83 -39

No files matched your search

+53 -29
View File
@@ -44,10 +44,12 @@ sure:
# readOnlyRootFilesystem: true holds, and the writable paths are emptyDirs:
# /rails/tmp, /rails/log and /tmp, one set per container. /tmp needs one extra
# step that is easy to mistake for decoration - a tiny init container creates
# a private subdirectory in it and the containers run with TMPDIR pointing
# there, because Ruby refuses a world-writable directory without a sticky bit
# as its temporary directory and an emptyDir is exactly that. Without it the
# app aborts during boot; the StatefulSet carries the measured evidence.
# two private subdirectories in it, and the containers run with TMPDIR and
# HOME pointing there. Ruby refuses a world-writable directory without a
# sticky bit as its temporary directory and an emptyDir is exactly that, so
# without this the app aborts during boot; HOME follows because the image's
# /home/rails is read-only and Bundler otherwise builds a throwaway home per
# invocation. The StatefulSet carries the measured evidence for both.
securityContext:
pod:
runAsNonRoot: true
@@ -89,11 +91,21 @@ sure:
# x limitFactor 3 = 1536Mi limit, which holds both the everyday figure and
# the AI recommendation without reserving the larger number on the node.
# Read from upstream's documentation, not measured here.
#
# ephemeral-storage is NOT the template default of 10Mi, and the reason is
# measured: with 10Mi (limit 30Mi) per container the kubelet evicted the
# pod within two minutes of every start - "Pod ephemeral local storage
# usage exceeds the total limit of containers 60Mi". Both emptyDirs and
# the container's stdout log count against that limit, and this image
# rebuilds its Bootsnap cache under /rails/tmp on every pod start because
# the emptyDir hides the one baked into the image. 256Mi request (768Mi
# limit) is headroom for that, not a measurement of it - the steady-state
# figure is still unknown.
limitFactor: 3
requests:
cpu: 0.1
memory: 512Mi
ephemeral-storage: 10Mi
ephemeral-storage: 256Mi
# PROBES - what each one actually proves (README, guideline 12.3)
#
# /up is Rails' own health route (config/routes.rb: get "up" =>
@@ -112,32 +124,41 @@ sure:
periodSeconds: 10
timeoutSeconds: 5
failureThreshold: 3
# Readiness uses "/" instead, because that path provably reaches the
# database. Read from the source of 0.7.4, not measured: every request runs
# ApplicationController's authenticate_user! before_action
# (app/controllers/concerns/authentication.rb); without a session cookie -
# which a kubelet probe never has - it falls into self_hosted_first_login?,
# which is "config.app_mode.self_hosted? && User.count.zero?". User.count is
# an SQL query. With the database up the request ends in a 302 to the login
# or registration page, which the kubelet counts as success (2xx-3xx); with
# the database down the query raises and Rails answers 500.
# Readiness asks for "/" instead, because that path provably reaches the
# database - and it does so through curl rather than an httpGet probe, for
# a reason that was measured rather than guessed (see below).
#
# Two conditions this depends on, both of them chart-set:
# - SELF_HOSTED=true. In managed mode the same code short-circuits before
# User.count and "/" would redirect without touching Postgres.
# - the probe must not be answered by a redirect BEFORE the controller.
# Sure's production.rb defaults RAILS_FORCE_SSL to true, and a forced
# redirect to https is a 301 - i.e. a "success" that proves nothing.
# The chart sets RAILS_ASSUME_SSL alongside it (see the env block that
# the templates always render), so requests are treated as SSL and no
# redirect happens. Overriding RAILS_ASSUME_SSL to "false" while
# RAILS_FORCE_SSL stays "true" turns this probe back into a lie.
# Why "/" reaches the database: every request runs ApplicationController's
# authenticate_user! before_action (app/controllers/concerns/
# authentication.rb); without a session cookie - which a probe never has -
# it falls into self_hosted_first_login?, i.e.
# "config.app_mode.self_hosted? && User.count.zero?". User.count is an SQL
# query. Measured in the install-test: against an empty database the app
# answered 302 to /registration/new, which is the branch that query decides,
# so the path demonstrably touches Postgres. With the database down the
# query raises and Rails answers 500.
#
# Why exec + curl and not httpGet: the kubelet's HTTP prober FOLLOWS
# redirects to the same host, and with RAILS_ASSUME_SSL=true (which the
# chart sets whenever the Ingress is on) Rails builds that redirect as
# https://<pod-ip>:3000/... - a scheme Puma does not speak. The install-test
# failed with "http: server gave HTTP response to HTTPS client" on a pod
# that was answering correctly. curl without -L asks once and reports the
# status code; the test accepts 2xx and 3xx exactly as the kubelet would,
# and a 500 from a missing database still fails. curl is part of the image
# (the Dockerfile installs it), so nothing extra is pulled in for this.
#
# One condition this depends on: SELF_HOSTED=true, which the chart always
# sets. In managed mode the same code short-circuits before User.count and
# "/" would redirect without touching Postgres at all.
readinessProbe:
httpGet:
path: /
port: http
exec:
command:
- /bin/bash
- -c
- "code=\"$(curl -sS -o /dev/null -w '%{http_code}' --max-time 5 http://127.0.0.1:3000/)\"; test \"$code\" -ge 200 -a \"$code\" -lt 400"
periodSeconds: 10
timeoutSeconds: 5
timeoutSeconds: 10
failureThreshold: 3
# The startup probe gates liveness and readiness, so neither needs an
# initialDelaySeconds. Its budget has to cover the first start, which is the
@@ -191,11 +212,14 @@ sure:
# one operation that deterministically exceeds 512 MB (observed flat at
# ~680 MB on a 2 GB container). The computed limit of 3 x 512Mi = 1536Mi
# covers that; the request stays at the everyday figure.
#
# ephemeral-storage: same 256Mi as the web container and for the same
# measured reason - see the comment there.
limitFactor: 3
requests:
cpu: 0.1
memory: 512Mi
ephemeral-storage: 10Mi
ephemeral-storage: 256Mi
# PROBES - the worker speaks no HTTP at all, so all three are exec probes.
#
# The command asks Sidekiq's own monitor whether THIS process is in the