PROBE: Install-Test auf dem umgebauten Docker-Runner (Wegwerf, nicht mergen) #22

Closed

jlab-bot wants to merge 4 commits from ci/gitea-install-test-probe into main

pull from: ci/gitea-install-test-probe

merge into: jlab-cluster:main

jlab-cluster:main

jlab-cluster:renovate/valkey-2.x

jlab-cluster:renovate/kubernetes-kubernetes-1.x

jlab-cluster:feat/convertx-chart

jlab-cluster:feat/install-test-all-charts

Conversation 1 Commits 4 Files Changed 2

+305 -20

jlab-bot commented 2026-09-16 16:08:58 +02:00

Member

Copy link

Copy source

Wegwerf-PR zu #14. Misst, ob der Install-Test nach der Umstellung auf rootful DinD durchlaeuft: cyberchef als schnelle Vergleichsbasis, freesailarr als haertester Fall. Wird samt Branch wieder entfernt.

Wegwerf-PR zu #14. Misst, ob der Install-Test nach der Umstellung auf rootful DinD durchlaeuft: cyberchef als schnelle Vergleichsbasis, freesailarr als haertester Fall. Wird samt Branch wieder entfernt.

jlab-bot added 3 commits 2026-09-16 16:08:58 +02:00

Prepare the Gitea move: measured CI deviations, self-hosted Renovate, issue migration script ... a6c27ebef3

Groundwork for #225, built on a throwaway pull request against the Gitea
instance rather than on assumptions. The move itself is not part of this
change - no Actions are switched off, no mirror is enabled, no branch
protection is touched.

What the probe run measured, and what follows from it:

- Which jobs need a Docker socket: only the install-test. Measured per
  job - detect-changed-charts, build-chart, the quantity render and the
  Trivy config scan all pass on `ubuntu-latest`, which on that instance
  has no socket. The install-test dies there at "Create k3d cluster"
  with "Cannot connect to the Docker daemon", an error naming k3d and
  never the runner label. Its `runs-on` is now an expression: the label
  `ubuntu-docker` on Gitea, `ubuntu-latest` on GitHub, because the
  former does not exist there and a chart PR would otherwise queue
  forever during the handover. Gitea evaluates it - the probe job was
  taken by the docker runner.

- The install-test still cannot pass on Gitea. That daemon is rootless
  with "Cgroup Driver: none"; the k3s node restarts immediately with an
  empty log because k3s needs cgroup delegation. The label is necessary
  and not sufficient, and the fix is the maintainer's.

- aquasecurity/setup-trivy cannot be used on Gitea: it checks out
  aquasecurity/trivy from github.com with `github.token`, which there is
  a Gitea token, and ends on "could not read Username for
  https://github.com". Replaced by the release tarball - one request,
  same behaviour on both forges, green on the probe run.

Also in this change:

- .github/workflows/renovate.yml, Renovate in self-operation: a daily
  scheduled run in the official container, inert on github.com via a
  `github.server_url` guard. Every rule proved by dry run instead of
  assumed - the pinned Renovate image is tracked by the built-in
  github-actions manager (44.80.0 -> 44.87.1, minor), TRIVY_VERSION is
  still tracked after setup-trivy is gone (v0.72.0 -> v0.74.0, minor),
  and `ubuntu-docker` is not extracted by the github-runners manager at
  all, so nothing will quietly "correct" it.

- ci/migrate-issues-to-gitea.sh, runnable, with --dry-run, and NOT run.
  Its header names what it cannot do: comment authorship, the #<n>
  references between issues, and attachments.

- CLAUDE.md and README.md on the new flow: which forge and which tool,
  auto-merge on green checks in place of the merge queue Gitea does not
  have, and the measured differences of the two CIs.

Refs #225 - deliberately no "Fixes": the move is not finished with this.

Drop the Renovate workflow: the Gitea side already has a shared instance ... c3889a66b6

A measurement overtook this part of the change. A shared `jlab-renovate-bot`
instance already runs on the Gitea side and already has this repository
enrolled - its first run there opened PRs for searxng and gotenberg plus the
dependency dashboard. A second runner in this repo would produce duplicate
PRs and two schedulers rewriting each other's branches, so
.github/workflows/renovate.yml is removed again.

Two things fall out of that run, and both were open risks:

- The instance reads our renovate.json5. searxng and gotenberg are
  customManagers targets of ours - no built-in manager finds an appVersion
  in a Chart.yaml - so the config is demonstrably in effect.

- allowedCommands is already granted there. The searxng PR's diff carries
  `version: 1.0.4+up2026.9.11.ffe96f8a6 -> 1.0.5+up2026.9.13.96479fff6`
  next to the appVersion, and nothing but ci/renovate-bump-chart.sh writes
  `version:`. Had that allow-list been missing, postUpgradeTasks would have
  been skipped SILENTLY and every chart PR on Gitea would have been red on
  the version check with nothing naming the cause.

With the workflow gone there is no container image to pin either; verified
by dry run that the ghcr.io/renovatebot/renovate dependency is no longer
extracted, while TRIVY_VERSION still is - that one belongs to the Trivy
replacement and stays.

README and CLAUDE.md now say Renovate runs on Gitea as an existing shared
instance and that no workflow belongs here, so the next reader who finds
none knows why instead of building one.

Drop the issue-migration script: a one-off job is not repo content ... b9917505cd

The migration from GitHub issues to Gitea runs exactly once, by hand over
the API. Checking a script in for it would leave permanently maintained
code behind for a task that stops existing after its single run - and
worse, it would read to every later visitor as a tool meant to be reused.
A one-off move is carried out once and is history afterwards; it is not a
part of the repository.

The one reference in CLAUDE.md is rewritten rather than deleted: that the
issue numbers do not survive the move is still true and still worth
knowing, so the bullet now says a `#n` in this repo means the GitHub
issue, and that what to do about the old references is open in #225.

jklein force-pushed ci/gitea-install-test-probe from f8f524b94f to 9a676c1700 2026-09-17 00:10:38 +02:00 Compare

jklein added 1 commit 2026-09-17 00:22:35 +02:00

install-test: one cluster per matrix leg, and a pinned kubectl ... 301fc0d4e5

Two assumptions about the runner environment, both found by the Gitea
probe of #225 and both wrong on GitHub as well - they simply could not
show there.

CLUSTER NAME. Every leg created a cluster called "install-test". On
GitHub each matrix leg gets its own VM with its own Docker daemon, so
two legs never saw each other and the fixed name looked fine for two
years. Gitea runs all legs of a matrix CONCURRENTLY in ONE job on ONE
daemon, and there it collides immediately: one leg logs "Re-using
existing network 'k3d-install-test'", the other dies on "The container
name /k3d-install-test-server-0 is already in use", and then the first
leg's rollback deletes the shared network under the second. Two red
legs, and neither message names the cause. The cluster is now named
after the chart, in the create step and in the delete step - a blanket
cleanup would tear down the neighbour's cluster, which was the half of
the collision that turned one failure into two. ci/run-install-test.sh
never touches k3d, so nothing else had to move.

The bare chart name rather than a prefixed one, measured not assumed:
k3d v5.9.0 accepts a cluster name of 32 characters and rejects 33
("Cluster name must be <= 32 characters"), and the longest chart here is
calibre-web-automated at 21. "install-test-<chart>" would be 34 - it
would break that one chart and no other, which is worse than the fixed
name because it only sometimes breaks. A guard states the limit instead
of leaving it to be rediscovered.

KUBECTL. It was inherited from the runner image. The Gitea image has
none ("kubectl: command not found", 0 seconds into the chart install),
but that is how it was noticed, not why it is being changed: a client
version that arrives with the image drifts with every image rebuild, and
an install-test should run against a known one. GitHub's image carrying
one only makes the drift invisible there. Pinned to v1.35.5 to match the
K3S_IMAGE it talks to, tracked by Renovate the same way TRIVY_VERSION
is, and guarded against skew: kubectl supports one minor away from the
server, Renovate moves KUBECTL_VERSION on its own, and K3S_IMAGE is
tracked by nothing. Dry run proves both directions - the rule extracts
(kubernetes/kubernetes, github-releases) and it fires (from v1.34.0 the
lookup offers v1.37.0, minor). That same measurement shows the guard is
not theoretical: Renovate jumps several minors in one PR, so it will go
red on purpose, naming which pin to move.

jklein force-pushed ci/gitea-install-test-probe from 2775e5d850 to ffcf3f344f 2026-09-17 00:22:35 +02:00 Compare

jklein added 1 commit 2026-09-17 00:29:22 +02:00

install-test: reach the k3d cluster when the job itself is a container ... 31e4260580

Closes the last gap from #227. k3d publishes the API server port on the
Docker HOST. On GitHub's hosted runners the job is that host, so the
https://0.0.0.0:<port> k3d writes into the kubeconfig works. Every Gitea
Actions job is itself a container, and there the host is the dind
container: 0.0.0.0 points at the job container, where nothing listens.
The cluster comes up cleanly and the first kubectl call is refused - as
measured, surfacing two steps later as a validation error on the
fixtures, which named nothing useful.

The case is decided on whether THIS job runs in a container, not on
which forge it runs on. github.server_url would pick the right branch
today and still be wrong: a self-hosted GitHub runner in a container has
the identical problem, and a condition that says GitHub while meaning
not-containerised is the kind of lie this repo exists to remove. The
detection is the same question the fix needs answered anyway - ask the
daemon about ourselves. In a container it returns our network, k3d joins
it, and Docker's DNS resolves the node names; on a host it returns
nothing and the previous path is taken unchanged.

kubectl cluster-info and get nodes now run inside the create step, so a
cluster that is up but unreachable fails where it is caused instead of
two steps downstream.

jklein force-pushed ci/gitea-install-test-probe from ffcf3f344f to b0eca80b2a 2026-09-17 00:29:22 +02:00 Compare

jklein added 1 commit 2026-09-17 00:39:07 +02:00

install-test: poll for the cluster instead of asking once ... 4766126618

The reachability check added with the containerisation fix was itself
racy and went red on one of two legs of the same GitHub run: k3d's
--wait returns when the node container reports ready, but the API
server can still answer discovery with 503 for a moment after that, and
kubectl cluster-info asked exactly once. A check that produces sporadic
red is worse than no check - that is the one property #227 rules out
explicitly. It now polls kubectl get nodes, which is the thing that has
to work and exercises discovery on the way.

jklein force-pushed ci/gitea-install-test-probe from b0eca80b2a to ad573406a5 2026-09-17 00:39:07 +02:00 Compare

jklein added 1 commit 2026-09-17 00:47:00 +02:00

install-test: wait for a Ready node, not for an answer ... 417dff0f80

Second defect in the same check, and the more serious one: kubectl get
nodes exits 0 on an EMPTY list. The cluster on Gitea answered while its
node had not registered, the step logged 'No resources found' and went
green, and the failure surfaced five minutes later as a pod stuck in
Pending with no node and no scheduling event - the cause nowhere near
the symptom. kubectl wait --for=condition=Ready nodes --all has the
same hole; with zero nodes it succeeds vacuously. It now counts Ready
nodes, and on timeout dumps the node conditions and the k3s server log
so the error picture is in the job that caused it.

jklein force-pushed ci/gitea-install-test-probe from ad573406a5 to eab82e7e10 2026-09-17 00:47:00 +02:00 Compare

jklein added 2 commits 2026-09-17 01:00:57 +02:00

install-test: provide envsubst, and say why it is not pinned like kubectl ... fdd798e1d9

ci/run-install-test.sh renders ci/<chart>/fixtures.env.yaml with
envsubst. GitHub's hosted image ships gettext-base, the Gitea runner
image does not, and without it the fixtures collapse to 'error: no
objects passed to apply' - gluetun never gets its Wireguard key and
nothing behind its kill switch becomes Ready. The script is left
untouched; it does nothing wrong, a standard tool is simply absent.

Checked for rather than pinned, and the difference from kubectl is
written next to it: kubectl's version changes what the test MEANS,
because its distance to the API server is a real source of failure, so
it is pinned, tracked and guarded. envsubst substitutes text and the
release performing it does not change the result, so a pin would be
upkeep without value. Present means do nothing, which keeps the GitHub
path free of any package install; missing means fetch it, so the
difference between the two runners is stated instead of hidden behind
an unconditional apt-get.

Also rewrites the reachability comment into one block. It had grown into
two stacked halves whose first half still described the behaviour of the
version before last - both reasons (the loop against a discovery 503,
counting Ready nodes against an empty list that exits 0) now stand
together with the evidence that the second one is what made the inotify
failure visible at all.

probe: both charts, complete state 7efda51bc7

jklein force-pushed ci/gitea-install-test-probe from eab82e7e10 to 7efda51bc7 2026-09-17 01:00:57 +02:00 Compare

jklein added 1 commit 2026-09-17 01:17:05 +02:00

probe run 1/3 (post inotify rollout) 221e9d4d43

jklein added 1 commit 2026-09-17 01:21:53 +02:00

probe run 2/3 (post inotify rollout) c8bf5b3e17

jklein added 1 commit 2026-09-17 01:26:38 +02:00

probe run 3/3 (post inotify rollout) 48bfc02029

jlab-bot commented 2026-09-17 09:45:22 +02:00

Author

Member

Copy link

Copy source

Zweck erfuellt: drei vollstaendige Laeufe, drei mal gruen, beide Charts. Job gesamt 216 / 212 / 210 s, Cluster-Anlegen 19-25 s. Ergebnis in #14 festgehalten, #227 auf GitHub geschlossen. PR wird geschlossen, Branch geloescht.

Zweck erfuellt: drei vollstaendige Laeufe, drei mal gruen, beide Charts. Job gesamt 216 / 212 / 210 s, Cluster-Anlegen 19-25 s. Ergebnis in #14 festgehalten, #227 auf GitHub geschlossen. PR wird geschlossen, Branch geloescht.

jlab-bot closed this pull request 2026-09-17 09:45:23 +02:00

jklein referenced this issue from a commit 2026-09-17 17:13:47 +02:00

docs: describe what is, now that the move is done

jlab-bot referenced this pull request 2026-09-17 17:14:28 +02:00

docs: describe what is, now that the move is done #26

Pull request closed

This pull request cannot be reopened because the branch was deleted.

Sign in to join this conversation.

Reviewers

No Reviewers

Labels

Clear labels

inhouse-chart

Inhouse-Chart aus der JLab-Git-Paketregistry

renovate

Von Renovate angelegt

No labels

Milestone

No items

No Milestone

Assignees

Clear assignees

jklein (Justin Klein) jlab-bot (jlab-git-bot) jlab-renovate-bot (jlab-renovate-bot)

No Assignees

2 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: jlab-cluster/helm-charts#22