Install-test over all charts, on demand, through one shared job #54

Merged

jklein merged 1 commits from feat/install-test-all-charts into main 2026-09-27 21:37:18 +02:00

Conversation 2 Commits 1 Files Changed 6

+686 -318

jlab-bot commented 2026-09-27 16:33:27 +02:00

Member

Copy link

Copy source

The install-test matrix follows the charts a pull request changes, so there is no moment at which the charts are checked as a whole against the current environment — and a chart nobody has touched since the forge move carries a green history from a different runner. This adds a manually dispatched all-charts run, and it does so by turning the existing install-test job into one shared job with two callers rather than a second copy of it.

Fixes #52

Neither path is verifiable in this PR

Neither path is verifiable in this PR. A workflow_dispatch workflow is only triggerable once it is on the default branch, so the first real all-charts run happens after the merge. And the reworked PR path only runs when a PR changes a chart — this one changes none, so the first proof is the next chart PR. Both are acceptable because the failure mode is loud: if the rework breaks the PR pipeline, the next chart PR goes red rather than silently green.

No chart was bumped to force a run, on purpose.

What moved, and what stayed

New .github/workflows/install-test.yml (workflow_call, one input: a JSON array of chart names, one optional secret: OP_CONNECT_TOKEN) holds the job: the runner-label expression, the pinned kubectl with its skew check, envsubst, the containerised-job detection, the per-leg k3d cluster name, the freesailarr VPN credentials, the install and the cleanup. Its callers decide which charts are tested and whether the job runs at all — never how.

validate-charts.yml keeps detect-changed-charts unchanged, including the exclusion filter and the excluded-charts summary, and its install-test job is now four lines of uses:/with:/secrets: inherit plus the comment explaining why the job is not copied.

How I convinced myself the extraction is behaviour-identical

By diffing the moved block against HEAD, not by reading it twice. Every executable line is byte-identical except for three intended changes:

- chart: ${{ fromJSON(needs.detect-changed-charts.outputs.install-charts) }}
+ chart: ${{ fromJSON(inputs.charts) }}
+ id: install            # on the existing install step, read by the new record step
+ RUNNER_LABEL: …        # new job env, for the record only

Everything else in the diff is comments, plus the added Record the result step at the end. The needs/if moved to the caller verbatim, and no needs.* reference survives in the called workflow. The filtered diff below prints exactly those four items and nothing else — reproduce it with:

git show origin/main:.github/workflows/validate-charts.yml | sed -n '230,541p' > /tmp/base-job
sed -n '43,415p' .github/workflows/install-test.yml > /tmp/new-job
# every difference that is not a comment:
diff /tmp/base-job /tmp/new-job | grep -E '^[<>]' | grep -v '^[<>][[:space:]]*#'

Three points that are deliberate rather than accidental:

• OP_CONNECT_TOKEN is declared required: false, unlike in publish-helm-chart.yml. A missing secret here does not fail the job, it fails job creation (README) — with required: true a PR from a fork would lose the whole install-test, every chart, with nothing naming the cause. Optional instead: only the freesailarr leg fails, loudly, at the :? guards.
• The runner-label expression is written twice (in runs-on and as RUNNER_LABEL), because the env context is not available in runs-on. Both lines sit on one screen, with the reason between them.
• One comment was not copied verbatim. The old KNOWN … this job cannot pass on Gitea note about rootless cgroups is no longer true — PR #48 installed 19 of 20 charts on ubuntu-docker. It now says what was measured and names #48 and #51 as the evidence.

The dispatch workflow

install-test-all-charts.yml, workflow_dispatch only (no schedule: it would spend compute daily on a question that changes rarely), three jobs:

1. select-charts — builds the matrix and writes the header of the record.
2. all-charts — calls install-test.yml. One throwaway cluster per chart, no reuse.
3. record — the verdict over the whole matrix, if: always().

Input charts: space- or comma-separated names, empty = all (24 charts today). The input is read as an environment variable and never interpolated into the shell, and names are validated against [a-z0-9-] plus an existing Chart.yaml. A name that is not a chart, a name listed in ci/excluded-charts.txt, and an empty matrix are all loud errors — a green run over zero charts would be the worst answer this workflow could give. Requesting a subset is what lets a single chart be tested against the runner, which is what #51 needs.

What the summary records

• Per chart, written by the leg itself (so the PR pipeline gets it too): outcome, date, runner label, runner name, K3S_IMAGE, kubectl version. It reads steps.install.outcome, so "the chart did not come up" and "the cluster never came up" are distinguishable — they mean opposite things here. Per leg rather than one table, because a matrix leg only ever sees its own GITHUB_STEP_SUMMARY.
• From select-charts: date, what was requested, the matrix and its size, the expected runtime, the one-cluster-per-chart note, and the excluded charts with their justification, from the one parser (ci/run-install-test.sh --list-excluded), exactly as in the PR pipeline.
• From record: the verdict over the matrix, including an explicit "no verdict, nothing was tested" for a skipped or cancelled run.

12m41s for 20 charts (PR #48) is documented as the expectation in the workflow, the README and the wiki, next to the note that the count is 24 today and printed by each run.

Verified

• All three workflows parse; inputs/secrets/with:/needs wiring asserted programmatically.
• The select-charts script run locally against this checkout: empty input → 24 charts; rss-bridge → one; "rss-bridge, cyberchef zipline" → three; nosuchchart → error, exit 1; satisfactory and template-chart → error naming the documented reason, exit 1; cyberchef; touch … → rejected by the name check, nothing executed.
• Both record scripts run for every outcome (success, failure, empty, and skipped/cancelled/unexpected for the verdict).
• ci/run-install-test.sh --list-excluded unchanged in behaviour; the script needed no change beyond three stale cross-references (validate-charts.yml → install-test.yml, and the second matrix builder).
• ci/run-quantity-render-test.sh: 24 exercised, 0 failed (gotenberg, rss-bridge as always "helper not reached").
• No chart directory in the diff. Rebased on main after #53.

Not verified, and why

• Anything requiring a Docker daemon or a runner — there is none here.
• That Gitea calls a local reusable workflow from a pull_request event, with needs/if on the calling job. The 24 publish hulls prove secrets: inherit and uses: ./… on push; this combination is new. Loud if wrong: the next chart PR goes red.
• needs.all-charts.result for a called workflow in the record job — first exercised by the first dispatched run.
• The status-context name of the install-test changes with this move (the matrix expression it carries is now inputs.charts, and the context may gain the calling job's name). What is required on main is validate-charts, which is untouched, but the documented example name in README and the wiki now says to read the current one off a real run.

The install-test matrix follows the charts a pull request **changes**, so there is no moment at which the charts are checked as a whole against the current environment — and a chart nobody has touched since the forge move carries a green history from a *different runner*. This adds a manually dispatched all-charts run, and it does so by turning the existing install-test job into **one shared job with two callers** rather than a second copy of it. Fixes #52 ## Neither path is verifiable in this PR **Neither path is verifiable in this PR.** A `workflow_dispatch` workflow is only triggerable once it is on the default branch, so the first real all-charts run happens after the merge. And the reworked PR path only runs when a PR changes a chart — this one changes none, so the first proof is the next chart PR. Both are acceptable because the failure mode is loud: if the rework breaks the PR pipeline, the next chart PR goes red rather than silently green. No chart was bumped to force a run, on purpose. ## What moved, and what stayed New `.github/workflows/install-test.yml` (`workflow_call`, one input: a JSON array of chart names, one optional secret: `OP_CONNECT_TOKEN`) holds the **job**: the runner-label expression, the pinned `kubectl` with its skew check, `envsubst`, the containerised-job detection, the per-leg k3d cluster name, the freesailarr VPN credentials, the install and the cleanup. Its callers decide *which* charts are tested and *whether* the job runs at all — never *how*. `validate-charts.yml` keeps `detect-changed-charts` unchanged, including the exclusion filter and the excluded-charts summary, and its `install-test` job is now four lines of `uses:`/`with:`/`secrets: inherit` plus the comment explaining why the job is not copied. ## How I convinced myself the extraction is behaviour-identical By diffing the moved block against `HEAD`, not by reading it twice. Every executable line is byte-identical except for three intended changes: ``` - chart: ${{ fromJSON(needs.detect-changed-charts.outputs.install-charts) }} + chart: ${{ fromJSON(inputs.charts) }} + id: install # on the existing install step, read by the new record step + RUNNER_LABEL: … # new job env, for the record only ``` Everything else in the diff is comments, plus the added `Record the result` step at the end. The `needs`/`if` moved to the caller verbatim, and no `needs.*` reference survives in the called workflow. The filtered diff below prints exactly those four items and nothing else — reproduce it with: ``` git show origin/main:.github/workflows/validate-charts.yml | sed -n '230,541p' > /tmp/base-job sed -n '43,415p' .github/workflows/install-test.yml > /tmp/new-job # every difference that is not a comment: diff /tmp/base-job /tmp/new-job | grep -E '^[<>]' | grep -v '^[<>][[:space:]]*#' ``` Three points that are deliberate rather than accidental: - **`OP_CONNECT_TOKEN` is declared `required: false`**, unlike in `publish-helm-chart.yml`. A missing secret here does not fail the job, it fails job *creation* (README) — with `required: true` a PR from a fork would lose the whole install-test, every chart, with nothing naming the cause. Optional instead: only the freesailarr leg fails, loudly, at the `:?` guards. - **The runner-label expression is written twice** (in `runs-on` and as `RUNNER_LABEL`), because the `env` context is not available in `runs-on`. Both lines sit on one screen, with the reason between them. - **One comment was not copied verbatim.** The old `KNOWN … this job cannot pass on Gitea` note about rootless cgroups is no longer true — PR #48 installed 19 of 20 charts on `ubuntu-docker`. It now says what was measured and names #48 and #51 as the evidence. ## The dispatch workflow `install-test-all-charts.yml`, `workflow_dispatch` only (no schedule: it would spend compute daily on a question that changes rarely), three jobs: 1. `select-charts` — builds the matrix and writes the header of the record. 2. `all-charts` — calls `install-test.yml`. One throwaway cluster per chart, no reuse. 3. `record` — the verdict over the whole matrix, `if: always()`. Input `charts`: space- or comma-separated names, empty = all (24 charts today). The input is read as an environment variable and never interpolated into the shell, and names are validated against `[a-z0-9-]` plus an existing `Chart.yaml`. A name that is not a chart, a name listed in `ci/excluded-charts.txt`, and an empty matrix are all **loud errors** — a green run over zero charts would be the worst answer this workflow could give. Requesting a subset is what lets a single chart be tested against the runner, which is what #51 needs. ## What the summary records - **Per chart, written by the leg itself** (so the PR pipeline gets it too): outcome, date, runner label, runner name, `K3S_IMAGE`, `kubectl` version. It reads `steps.install.outcome`, so "the chart did not come up" and "the cluster never came up" are distinguishable — they mean opposite things here. Per leg rather than one table, because a matrix leg only ever sees its own `GITHUB_STEP_SUMMARY`. - **From `select-charts`**: date, what was requested, the matrix and its size, the expected runtime, the one-cluster-per-chart note, and the excluded charts **with their justification**, from the one parser (`ci/run-install-test.sh --list-excluded`), exactly as in the PR pipeline. - **From `record`**: the verdict over the matrix, including an explicit "no verdict, nothing was tested" for a skipped or cancelled run. **12m41s for 20 charts** (PR #48) is documented as the expectation in the workflow, the README and the wiki, next to the note that the count is 24 today and printed by each run. ## Verified - All three workflows parse; inputs/secrets/`with:`/`needs` wiring asserted programmatically. - The `select-charts` script run locally against this checkout: empty input → 24 charts; `rss-bridge` → one; `"rss-bridge, cyberchef zipline"` → three; `nosuchchart` → error, exit 1; `satisfactory` and `template-chart` → error naming the documented reason, exit 1; `cyberchef; touch …` → rejected by the name check, nothing executed. - Both record scripts run for every outcome (`success`, `failure`, empty, and `skipped`/`cancelled`/unexpected for the verdict). - `ci/run-install-test.sh --list-excluded` unchanged in behaviour; the script needed no change beyond three stale cross-references (`validate-charts.yml` → `install-test.yml`, and the second matrix builder). - `ci/run-quantity-render-test.sh`: 24 exercised, 0 failed (`gotenberg`, `rss-bridge` as always "helper not reached"). - No chart directory in the diff. Rebased on `main` after #53. ## Not verified, and why - Anything requiring a Docker daemon or a runner — there is none here. - **That Gitea calls a local reusable workflow from a `pull_request` event, with `needs`/`if` on the calling job.** The 24 publish hulls prove `secrets: inherit` and `uses: ./…` on `push`; this combination is new. Loud if wrong: the next chart PR goes red. - **`needs.all-charts.result` for a called workflow** in the record job — first exercised by the first dispatched run. - **The status-context name of the install-test changes** with this move (the matrix expression it carries is now `inputs.charts`, and the context may gain the calling job's name). What is required on `main` is `validate-charts`, which is untouched, but the documented example name in README and the wiki now says to read the current one off a real run.

jlab-bot added 1 commit 2026-09-27 16:33:27 +02:00

Make the all-charts install-test repeatable, as one shared job ... 6a3b05503e

The install-test matrix follows the charts a pull request changes, so a
chart nobody has touched since the forge move carries a green history from
a different runner - and that green says nothing about today. Gitea PR #48
showed what the answer is worth: correcting a comment in 21 charts made
every one of them "changed" and turned the run into the first all-charts
install-test after the move, 19 of 20 matrix charts Ready in 12m41s and
exactly one not (rss-bridge, Gitea #51). By accident.

The install-test job moves into the reusable .github/workflows/
install-test.yml and is called twice: by validate-charts.yml with the
charts a pull request changed, and by the new install-test-all-charts.yml
with all of them. Not copied for the second caller - a second copy is a
second chance to test differently, unnoticed, which is the argument that
gives ci/excluded-charts.txt exactly one parser. The parts that would
drift are the ones that were paid for once already: the pinned kubectl
with its skew check, envsubst, the containerised-job detection, the
per-leg cluster name.

Nothing about how a chart is installed changed. The moved job differs from
its predecessor in the matrix source (inputs.charts instead of the detect
job's output), an id on the install step, and a step that records the
result; every other step is byte-identical.

The all-charts run is workflow_dispatch only - a schedule would spend
compute every day on a question that changes rarely - with an optional
subset input, which is what lets a single chart be tested against the
runner as Gitea #51 needs. One throwaway cluster per chart, no reuse:
reuse saves time and changes what the test proves. An unknown or excluded
chart name, and an empty matrix, are loud errors rather than a green run
over nothing.

Every leg writes its own outcome into the job summary next to the four
facts that decide what the outcome means - date, runner label, K3S_IMAGE,
kubectl version - and a final job records the verdict over the matrix. A
green run nobody notes is unknown again next time. The excluded charts
appear with their justification, from the one parser, as in the pull
request pipeline. 20 charts in 12m41s is written down as the expectation,
so a later deviation is visible as one.

Fixes #52

jlab-bot commented 2026-09-27 16:35:44 +02:00

Author

Member

Copy link

Copy source

One thing this run does settle, ahead of the next chart PR.

The pipeline on this PR came up green (detect 3s, resource quantities 10s, Trivy sample 21s), and two of the unverified points are now partly measured:

• The forge accepted both new workflow files and created the run, so neither is a parse error waiting for the merge.
• needs + if on a calling uses: job is evaluated here. The install-test reports as Skipped, which is the correct outcome for a PR that changes no chart - it did not fail job creation and it did not run.
• The status context reads Validate Helm Charts / Install-test (pull_request) - the calling job name, with no matrix expression in it, next to the still-unexpanded Build ${{ fromJSON(...) }} context of the sibling job. Measured on a skipped job whose matrix was never expanded, so the name a real leg reports is still to be read off the next chart PR, as the README and the wiki now say.

What stays unverifiable here is unchanged and named in the description: the first real all-charts run happens after the merge, and the reworked PR path is first proven by the next chart PR.

One thing this run does settle, ahead of the next chart PR. The pipeline on this PR came up green (detect 3s, resource quantities 10s, Trivy sample 21s), and two of the unverified points are now partly measured: - **The forge accepted both new workflow files** and created the run, so neither is a parse error waiting for the merge. - **`needs` + `if` on a calling `uses:` job is evaluated here.** The install-test reports as `Skipped`, which is the correct outcome for a PR that changes no chart - it did not fail job creation and it did not run. - **The status context reads `Validate Helm Charts / Install-test (pull_request)`** - the calling job name, with no matrix expression in it, next to the still-unexpanded `Build ${{ fromJSON(...) }}` context of the sibling job. Measured on a *skipped* job whose matrix was never expanded, so the name a real leg reports is still to be read off the next chart PR, as the README and the wiki now say. What stays unverifiable here is unchanged and named in the description: the first real all-charts run happens after the merge, and the reworked PR path is first proven by the next chart PR.

jklein approved these changes 2026-09-27 21:37:10 +02:00

jklein merged commit 6a3b05503e into main 2026-09-27 21:37:18 +02:00

jlab-bot referenced this pull request 2026-09-27 23:07:56 +02:00

rss-bridge: Install-Test scheitert auf dem Gitea-Runner (uid 33 kann /app/index.php nicht lesen) #51

Sign in to join this conversation.

Reviewers

No Reviewers

jklein

Labels

Clear labels

inhouse-chart

Inhouse-Chart aus der JLab-Git-Paketregistry

renovate

Von Renovate angelegt

No labels

Milestone

No items

No Milestone

Assignees

Clear assignees

jklein (Justin Klein) jlab-bot (jlab-git-bot) jlab-renovate-bot (jlab-renovate-bot)

No Assignees

2 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: jlab-cluster/helm-charts#54