calibre-web-automated 7.0.0, minecraft-server 6.0.0 and urlaubsverwaltung
10.0.0 each majors: an installation that renders today with an empty itemPath
stops rendering. searxng 1.0.6 is documentation only - its valkey reference is
correctly optional and stays gated by {{- if }}.
Every message names the values path. values.yaml now says per key why it is
required, reusing the wording the charts' NOTES.txt already carry. The
hand-rolled "cwaSecretRef IS UNSET" block in calibre-web-automated's NOTES.txt
is gone: after the guard the render aborts first and that block is
unreachable. minecraft-server's note stays true and now points at the guard.
No "required" array in the schemas, and README guideline 12 now says why. The
plan called for one; measuring it showed it could never fire: the key ships as
null, so it is present at validation time, and JSON Schema's required tests
presence, not null-ness. Making it fire means either dropping the null default
or narrowing the type to "string", and both turn helm lint --strict red on the
chart's own defaults. The schema therefore describes the key as required and
required() enforces it. The README bullet on how null reaches the schema was
also wrong in the case it was being applied to - a null chart default is not
coalesced away - and now distinguishes the three cases.
searxng: the claim "charts in this repo have no dependencies (README,
guideline 13)" in the valkey template, in values.yaml and twice in _helpers.tpl
has been false since #214 and cited the wrong guideline; dependencies are
admissible and are guideline 14. The same sentence survives in other charts'
helpers and needs its own round.
Second install-test run, three findings, all measured:
- Readiness on "/" failed with "http: server gave HTTP response to HTTPS
client" on a pod that was answering correctly. The kubelet's HTTP prober
follows redirects to the same host, and with RAILS_ASSUME_SSL=true the
app builds its redirect as https://<pod-ip>:3000/registration/new, which
Puma does not speak. The probe now runs curl without -L and accepts
2xx-3xx itself. The same log also proves what the probe is for: against
an empty database the redirect target was /registration/new, the branch
User.count decides, so the path really does reach Postgres.
- The pod was evicted within two minutes of every start: "ephemeral local
storage usage exceeds the total limit of containers 60Mi". emptyDirs and
the stdout log both count there, and the image rebuilds its Bootsnap
cache under /rails/tmp on every start because the emptyDir hides the one
baked into the image. The request is now 256Mi (limit 768Mi) per app
container - headroom, not a measurement.
- /home/rails is read-only, so Bundler created a throwaway home under
TMPDIR on every invocation, worker probe included. HOME now points at a
directory the init container creates.
The CI postgres fixture gets -d sure on its readiness probe, so the server
stops logging a missing database that nothing asked for.
The first install-test run failed with both containers crash-looping
during boot:
system temporary path is world-writable: /tmp
/tmp is world-writable: /tmp
. is not writable: /rails
ArgumentError: could not find a temporary directory
raised from MiniMagick, which Active Storage loads while Rails boots.
Ruby's Dir.tmpdir skips a candidate that is world-writable without a
sticky bit, and a Kubernetes emptyDir is created mode 0777 with no sticky
bit - so with a read-only root filesystem there was no candidate left.
A tiny init container, the app's own image running as the same non-root
uid and with no capability added, now creates one private subdirectory
(mode 0700) inside each tmp emptyDir, and both containers run with
TMPDIR pointing at it. Creating an entry in a 0777 directory and chmod'ing
what one just created needs no privilege, so the hardening standard holds
unchanged.
One StatefulSet with two containers - the Puma web process and a Sidekiq
worker - sharing one ReadWriteOnce PVC at /rails/storage. That shared
volume is the reason they live in one pod: splitting them would need an
RWX volume. Only the web container migrates, because the image entrypoint
runs db:prepare solely for the './bin/rails server' command; the worker is
started through args: so the entrypoint (and its jemalloc preload) stays
in place and the two can never race.
Probes are chosen for what they prove: liveness and startup use Rails'
/up, which is a boot check and touches no database, while readiness uses
'/', which runs User.count through authenticate_user! in self-hosted mode
and therefore fails while Postgres is unreachable. The worker speaks no
HTTP, so all three of its probes ask sidekiqmon whether this pod still
holds a Sidekiq heartbeat in Redis.
Three OnePasswordItem CRs (app, database, redis), ingress.domain kept
required outside the Ingress because APP_DOMAIN and the WebAuthn relying
party read it, strict values.schema.json, CI fixtures with throwaway
Postgres 16 and Redis, and a Renovate manager on semver so the
-alpha/-rc/-hotfix channel and the moving tags are never selected.
Walks back the one part of the previous commit that changed rendered output:
storage was pulled into the server fallback, so an erased size fell back to
20Gi instead of rendering a PVC without one.
values.schema.json had documented the opposite decision - "an erased value
yields an invalid PVC rather than a silently degraded workload" - and that is
someone's reasoned trade, not collateral of the nil-pointer bug next to it.
Overturning it inside a patch would have made this the only release where
valid values render differently, and it would have settled an open design
question in a PR nobody opened for that. It goes to #39 instead.
So storage is out of the defaults again and the schema note is restored
verbatim, extended only by where the exception is argued. values.yaml marks
it as the one key of the block that is deliberately not null-safe. The
README rule that had leaned on this example now says the thing that actually
holds: an exception is fine, an unmarked one is not.
Everything else stands - dev and server null-safe, the single-key findings on
the ports, formatNumber against exponential notation, 7.0.1.
Re-measured after the walk-back: rendered output against the chart defaults
and against a full override set is byte-identical to 7.0.0, NOTES.txt
included, and "--set server=null" now renders exactly the sizeless PVC that
"--set server.storage=null" rendered on 7.0.0.
--set dev=null aborted the render at .Values.dev.debug and --set server=null
at .Values.server.replicas, while the third block, game, has fallen back to
the chart defaults since 7.0.0. Half-safe is the worse state of the two:
whoever sees game=null work assumes server=null does too.
Both blocks now run through satisfactory.defaultedDict like game, via two new
helpers next to it. server.resources is deliberately NOT re-defaulted there -
it already runs through satisfactory.effectiveResources, which carries those
defaults and is null-safe on its own, and a second copy of limitFactor: 2 /
4Gi / 1 / 5Gi would be a third place for them to drift.
server is read in four files, not one, and the fourth is NOTES.txt, which
helm template never renders - a fix confined to the manifests would have
looked green and still aborted on a real helm install. All four go through
the helpers.
The single erased key was the quieter half of the bug and is fixed with it:
a nulled server.gamePort rendered "port:" and "nodePort:" empty into the
Service and an empty containerPort into the StatefulSet, and a nulled
server.storage a PVC asking for no size - manifests rejected by nothing until
the API server saw them. server.storage was documented in values.schema.json
as deliberately not null-safe; that reasoning does not survive the block-level
case, because it made "server: null" render a sizeless PVC instead of falling
back like everything else. The schema note is replaced rather than left lying.
Rendered output for valid values is unchanged: helm template against the
chart defaults and against a full override set is byte-identical to 7.0.0,
and so is NOTES.txt. Hence a patch bump, appVersion untouched.
Fixes#16
The shared jlab-renovate-bot instance serves this repo and cluster-config,
and two repos behaving differently under one bot is a source of surprises
rather than of safety. This adopts cluster-config's configuration:
config:best-practices in extends, labels ["renovate"], no PR caps
(prHourlyLimit 0, prConcurrentLimit 100), separateMinorPatch, an explicit
dependencyDashboard, and global automerge with platformAutomerge.
Two of those choices were previously wrong here and are now right, because
the branch protection on main has caught up: a review requirement and
validate-charts as a required check are both active. platformAutomerge
therefore arms the PR on the forge and Gitea merges it when the check
passes AND the review exists - the review is the gate, which is what makes
automerging majors something other than unattended merging.
Comments that asserted the opposite are rewritten rather than left to rot:
the platformAutomerge: false comment ("not (yet) a required check on either
side"), the "Majors stay manual" rule, the README PR policy bullets and the
required-check paragraph, and the CLAUDE.md forge notes.
config:best-practices pulls in docker:pinDigests, which is switched off
again for custom.regex and helm-values: an appVersion: is a tag by
construction and has nowhere to put a digest. Kept deliberately:
minimumReleaseAge of three days on the routine updates (every merge here
publishes a chart), the ignorePaths and ci/** opt-outs, every per-chart
manager, the nginx and qbittorrent constraints, and postUpgradeTasks.
Fixes#34
Work on this repo happens in a Coder workspace that has no 1Password CLI.
The rule that tea may only be called as 'op plugin run -- tea' therefore
described an environment that no longer exists here, and following it meant
either waiting for a tool that is not coming or ending with a pushed branch
and no PR.
tea is now called directly. It carries the workspace's stored login, so what
it opens is authored by the bot account - accepted, and named in the report
when it happens. The older instruction is kept for machines that do have op.
Signing needed the same correction: it runs through the workspace's own SSH
signer, not op-ssh-sign, and the check is 'git log --format=%G?' printing G.
The wiki now carries one page per chart (prerequisites, installation, the
required 1Password fields, the complete values reference and the chart's
traps) plus the workflow, the pipelines, versioning, Renovate, the style
guide and the incidents behind each rule.
README points at it and keeps its own role: the overview for someone reading
the code, while anything an operator needs lives in the wiki. CLAUDE.md gains
the wiki edit as step 7 of the mandatory workflow, with a table of what
obliges an edit - and the one thing that does not, a plain app-version bump,
since Renovate automerges those and never touches the wiki.
Also corrects the patchmon row: the 2.x server serves frontend and /api/* from
one process, so the chart renders one Deployment plus an optional guacd
sidecar, not two Deployments.
The documentation still described a migration in progress, and the worst
line claimed the install-test "cannot run on Gitea at all" - true two
days ago, the opposite of true now, and exactly the sort of sentence
someone builds a wrong assumption on.
README and CLAUDE.md now state what holds: everything runs on Gitea,
github.com is a mirror with Actions and Renovate off, branch protection
with a signature requirement is active, and the tool is tea. The move
itself is not retold - that is Gitea #14.
"Where this repo lives" keeps its place but turns around: it is no
longer a list of what blocks us, it is what a Gitea runner does
differently from a GitHub one, with the measurements. Newly written
down because it holds permanently and stood nowhere: a matrix runs
concurrently in ONE job under ONE status context, so a required check
per chart is impossible and the context name carries the unexpanded
matrix expression; expected runtimes, so a later deviation is visible
as one, with the note that the difference to GitHub is structural - one
runner for both legs against a VM per leg - and not performance.
The six assumptions a GitHub runner had been meeting quietly are split
rather than listed. Five are environment (MTU, fs.inotify, missing
kubectl and envsubst, kubeconfig on 0.0.0.0): they fail loudly, say
nothing about our code, and are fixed on the node side. The sixth gets
its own block because it is the one that does say something about our
code - the hard-coded k3d cluster name was wrong on GitHub too and only
never showed there, because a VM per matrix leg meant two legs never saw
each other. A generous environment is no friend when you want to know
whether something is correct.
Two rules generalised out of this round, as rules rather than as a
story: a check that a broken system passes is worse than no check (with
the two vacuous forms that caused it), and green does not mean checked,
pointing at Gitea #24 as an open case of the same kind - explicitly not
claimed as solved.
Also fixes a hazard the move created: most #n in these files name closed
GitHub issues that were never migrated, while Gitea resolves a bare #n
against its own counter - #22 there is an unrelated throwaway pull
request, not the install-test issue the text means. Stated once at the
top instead of rewriting sixteen references into numbers that do not
exist.
The branch predates the Gitea move (#253) by 37 commits and both sides
touched the same two files. Merged rather than rebased, so the signed
commit of the original work stays intact, and resolved by combining
rather than choosing - both statements are true at once: the chart has
ten images now, and the repo lives on Gitea.
README.md, the one real conflict: this branch rewrote the freesailarr
image bullet from nine upstreams to ten, while main had grown a
minecraft-server bullet next to it that this branch never had. Kept
both - our "ten" wording and main's new bullet.
renovate.json5 merged cleanly, but "cleanly" is not "correctly", so the
result was read rather than trusted: the freesailarr exception now says
ten and gained the pointer to the mediathekarr digest block, while the
two things main added right beside it survived unchanged - the note that
the post-upgrade hook is allow-listed on the shared Gitea instance, and
the forge-neutral wording of the automerge rule. Nothing from before the
move brought "GitHub" back into a sentence that had been made neutral.
ci/run-install-test.sh renders ci/<chart>/fixtures.env.yaml with
envsubst. GitHub's hosted image ships gettext-base, the Gitea runner
image does not, and without it the fixtures collapse to 'error: no
objects passed to apply' - gluetun never gets its Wireguard key and
nothing behind its kill switch becomes Ready. The script is left
untouched; it does nothing wrong, a standard tool is simply absent.
Checked for rather than pinned, and the difference from kubectl is
written next to it: kubectl's version changes what the test MEANS,
because its distance to the API server is a real source of failure, so
it is pinned, tracked and guarded. envsubst substitutes text and the
release performing it does not change the result, so a pin would be
upkeep without value. Present means do nothing, which keeps the GitHub
path free of any package install; missing means fetch it, so the
difference between the two runners is stated instead of hidden behind
an unconditional apt-get.
Also rewrites the reachability comment into one block. It had grown into
two stacked halves whose first half still described the behaviour of the
version before last - both reasons (the loop against a discovery 503,
counting Ready nodes against an empty list that exits 0) now stand
together with the evidence that the second one is what made the inotify
failure visible at all.
Second defect in the same check, and the more serious one: kubectl get
nodes exits 0 on an EMPTY list. The cluster on Gitea answered while its
node had not registered, the step logged 'No resources found' and went
green, and the failure surfaced five minutes later as a pod stuck in
Pending with no node and no scheduling event - the cause nowhere near
the symptom. kubectl wait --for=condition=Ready nodes --all has the
same hole; with zero nodes it succeeds vacuously. It now counts Ready
nodes, and on timeout dumps the node conditions and the k3s server log
so the error picture is in the job that caused it.
The reachability check added with the containerisation fix was itself
racy and went red on one of two legs of the same GitHub run: k3d's
--wait returns when the node container reports ready, but the API
server can still answer discovery with 503 for a moment after that, and
kubectl cluster-info asked exactly once. A check that produces sporadic
red is worse than no check - that is the one property #227 rules out
explicitly. It now polls kubectl get nodes, which is the thing that has
to work and exercises discovery on the way.
Closes the last gap from #227. k3d publishes the API server port on the
Docker HOST. On GitHub's hosted runners the job is that host, so the
https://0.0.0.0:<port> k3d writes into the kubeconfig works. Every Gitea
Actions job is itself a container, and there the host is the dind
container: 0.0.0.0 points at the job container, where nothing listens.
The cluster comes up cleanly and the first kubectl call is refused - as
measured, surfacing two steps later as a validation error on the
fixtures, which named nothing useful.
The case is decided on whether THIS job runs in a container, not on
which forge it runs on. github.server_url would pick the right branch
today and still be wrong: a self-hosted GitHub runner in a container has
the identical problem, and a condition that says GitHub while meaning
not-containerised is the kind of lie this repo exists to remove. The
detection is the same question the fix needs answered anyway - ask the
daemon about ourselves. In a container it returns our network, k3d joins
it, and Docker's DNS resolves the node names; on a host it returns
nothing and the previous path is taken unchanged.
kubectl cluster-info and get nodes now run inside the create step, so a
cluster that is up but unreachable fails where it is caused instead of
two steps downstream.
Two assumptions about the runner environment, both found by the Gitea
probe of #225 and both wrong on GitHub as well - they simply could not
show there.
CLUSTER NAME. Every leg created a cluster called "install-test". On
GitHub each matrix leg gets its own VM with its own Docker daemon, so
two legs never saw each other and the fixed name looked fine for two
years. Gitea runs all legs of a matrix CONCURRENTLY in ONE job on ONE
daemon, and there it collides immediately: one leg logs "Re-using
existing network 'k3d-install-test'", the other dies on "The container
name /k3d-install-test-server-0 is already in use", and then the first
leg's rollback deletes the shared network under the second. Two red
legs, and neither message names the cause. The cluster is now named
after the chart, in the create step and in the delete step - a blanket
cleanup would tear down the neighbour's cluster, which was the half of
the collision that turned one failure into two. ci/run-install-test.sh
never touches k3d, so nothing else had to move.
The bare chart name rather than a prefixed one, measured not assumed:
k3d v5.9.0 accepts a cluster name of 32 characters and rejects 33
("Cluster name must be <= 32 characters"), and the longest chart here is
calibre-web-automated at 21. "install-test-<chart>" would be 34 - it
would break that one chart and no other, which is worse than the fixed
name because it only sometimes breaks. A guard states the limit instead
of leaving it to be rediscovered.
KUBECTL. It was inherited from the runner image. The Gitea image has
none ("kubectl: command not found", 0 seconds into the chart install),
but that is how it was noticed, not why it is being changed: a client
version that arrives with the image drifts with every image rebuild, and
an install-test should run against a known one. GitHub's image carrying
one only makes the drift invisible there. Pinned to v1.35.5 to match the
K3S_IMAGE it talks to, tracked by Renovate the same way TRIVY_VERSION
is, and guarded against skew: kubectl supports one minor away from the
server, Renovate moves KUBECTL_VERSION on its own, and K3S_IMAGE is
tracked by nothing. Dry run proves both directions - the rule extracts
(kubernetes/kubernetes, github-releases) and it fires (from v1.34.0 the
lookup offers v1.37.0, minor). That same measurement shows the guard is
not theoretical: Renovate jumps several minors in one PR, so it will go
red on purpose, naming which pin to move.
The migration from GitHub issues to Gitea runs exactly once, by hand over
the API. Checking a script in for it would leave permanently maintained
code behind for a task that stops existing after its single run - and
worse, it would read to every later visitor as a tool meant to be reused.
A one-off move is carried out once and is history afterwards; it is not a
part of the repository.
The one reference in CLAUDE.md is rewritten rather than deleted: that the
issue numbers do not survive the move is still true and still worth
knowing, so the bullet now says a `#n` in this repo means the GitHub
issue, and that what to do about the old references is open in #225.
A measurement overtook this part of the change. A shared `jlab-renovate-bot`
instance already runs on the Gitea side and already has this repository
enrolled - its first run there opened PRs for searxng and gotenberg plus the
dependency dashboard. A second runner in this repo would produce duplicate
PRs and two schedulers rewriting each other's branches, so
.github/workflows/renovate.yml is removed again.
Two things fall out of that run, and both were open risks:
- The instance reads our renovate.json5. searxng and gotenberg are
customManagers targets of ours - no built-in manager finds an appVersion
in a Chart.yaml - so the config is demonstrably in effect.
- allowedCommands is already granted there. The searxng PR's diff carries
`version: 1.0.4+up2026.9.11.ffe96f8a6 -> 1.0.5+up2026.9.13.96479fff6`
next to the appVersion, and nothing but ci/renovate-bump-chart.sh writes
`version:`. Had that allow-list been missing, postUpgradeTasks would have
been skipped SILENTLY and every chart PR on Gitea would have been red on
the version check with nothing naming the cause.
With the workflow gone there is no container image to pin either; verified
by dry run that the ghcr.io/renovatebot/renovate dependency is no longer
extracted, while TRIVY_VERSION still is - that one belongs to the Trivy
replacement and stays.
README and CLAUDE.md now say Renovate runs on Gitea as an existing shared
instance and that no workflow belongs here, so the next reader who finds
none knows why instead of building one.
Groundwork for #225, built on a throwaway pull request against the Gitea
instance rather than on assumptions. The move itself is not part of this
change - no Actions are switched off, no mirror is enabled, no branch
protection is touched.
What the probe run measured, and what follows from it:
- Which jobs need a Docker socket: only the install-test. Measured per
job - detect-changed-charts, build-chart, the quantity render and the
Trivy config scan all pass on `ubuntu-latest`, which on that instance
has no socket. The install-test dies there at "Create k3d cluster"
with "Cannot connect to the Docker daemon", an error naming k3d and
never the runner label. Its `runs-on` is now an expression: the label
`ubuntu-docker` on Gitea, `ubuntu-latest` on GitHub, because the
former does not exist there and a chart PR would otherwise queue
forever during the handover. Gitea evaluates it - the probe job was
taken by the docker runner.
- The install-test still cannot pass on Gitea. That daemon is rootless
with "Cgroup Driver: none"; the k3s node restarts immediately with an
empty log because k3s needs cgroup delegation. The label is necessary
and not sufficient, and the fix is the maintainer's.
- aquasecurity/setup-trivy cannot be used on Gitea: it checks out
aquasecurity/trivy from github.com with `github.token`, which there is
a Gitea token, and ends on "could not read Username for
https://github.com". Replaced by the release tarball - one request,
same behaviour on both forges, green on the probe run.
Also in this change:
- .github/workflows/renovate.yml, Renovate in self-operation: a daily
scheduled run in the official container, inert on github.com via a
`github.server_url` guard. Every rule proved by dry run instead of
assumed - the pinned Renovate image is tracked by the built-in
github-actions manager (44.80.0 -> 44.87.1, minor), TRIVY_VERSION is
still tracked after setup-trivy is gone (v0.72.0 -> v0.74.0, minor),
and `ubuntu-docker` is not extracted by the github-runners manager at
all, so nothing will quietly "correct" it.
- ci/migrate-issues-to-gitea.sh, runnable, with --dry-run, and NOT run.
Its header names what it cannot do: comment authorship, the #<n>
references between issues, and attachments.
- CLAUDE.md and README.md on the new flow: which forge and which tool,
auto-merge on green checks in place of the merge queue Gitea does not
have, and the measured differences of the two CIs.
Refs #225 - deliberately no "Fixes": the move is not finished with this.
jklein
jlab-renovate-bot
jlab-renovate[bot]