Make the all-charts install-test repeatable, as one shared job
Validate Helm Charts / Detect changed charts (pull_request) Successful in 3s
Validate Helm Charts / Build ${{ fromJSON(needs.detect-changed-charts.outputs.charts) }} (pull_request) Skipped
Validate Helm Charts / Install-test (pull_request) Skipped
Validate Helm Charts / Resource quantities (pull_request) Successful in 10s
Validate Helm Charts / Trivy config ${{ fromJSON(needs.detect-changed-charts.outputs.scan-charts) }} (pull_request) Successful in 21s

The install-test matrix follows the charts a pull request changes, so a
chart nobody has touched since the forge move carries a green history from
a different runner - and that green says nothing about today. Gitea PR #48
showed what the answer is worth: correcting a comment in 21 charts made
every one of them "changed" and turned the run into the first all-charts
install-test after the move, 19 of 20 matrix charts Ready in 12m41s and
exactly one not (rss-bridge, Gitea #51). By accident.

The install-test job moves into the reusable .github/workflows/
install-test.yml and is called twice: by validate-charts.yml with the
charts a pull request changed, and by the new install-test-all-charts.yml
with all of them. Not copied for the second caller - a second copy is a
second chance to test differently, unnoticed, which is the argument that
gives ci/excluded-charts.txt exactly one parser. The parts that would
drift are the ones that were paid for once already: the pinned kubectl
with its skew check, envsubst, the containerised-job detection, the
per-leg cluster name.

Nothing about how a chart is installed changed. The moved job differs from
its predecessor in the matrix source (inputs.charts instead of the detect
job's output), an id on the install step, and a step that records the
result; every other step is byte-identical.

The all-charts run is workflow_dispatch only - a schedule would spend
compute every day on a question that changes rarely - with an optional
subset input, which is what lets a single chart be tested against the
runner as Gitea #51 needs. One throwaway cluster per chart, no reuse:
reuse saves time and changes what the test proves. An unknown or excluded
chart name, and an empty matrix, are loud errors rather than a green run
over nothing.

Every leg writes its own outcome into the job summary next to the four
facts that decide what the outcome means - date, runner label, K3S_IMAGE,
kubectl version - and a final job records the verdict over the matrix. A
green run nobody notes is unknown again next time. The excluded charts
appear with their justification, from the one parser, as in the pull
request pipeline. 20 charts in 12m41s is written down as the expectation,
so a later deviation is visible as one.

Fixes #52
This commit was merged in pull request #54.
This commit is contained in:
jklein committed 2026-09-27 14:30:39 +00:00
1 parent 813c796a7b
commit 6a3b05503e
6 files changed
+686 -318

No files matched your search

+1 -1
View File
@@ -5,7 +5,7 @@
# (REQUIRED_ENV), and pipes the result straight into kubectl.
#
# The values come from a 1Password item holding a CI-only Mullvad key, loaded in
# the install-test job of .github/workflows/validate-charts.yml. A CI-only key
# the install-test job of .github/workflows/install-test.yml. A CI-only key
# on purpose: Mullvad ties an address to a key and sends the tunnel to the
# endpoint it saw last, so a shared key would let every pull request pull the
# tunnel out from under the running stack.
+10 -7
View File
@@ -21,9 +21,11 @@ ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
CI_DIR="$ROOT/ci"
# --- The exclusion list, parsed in exactly one place ------------------------
# Two consumers read ci/excluded-charts.txt: this script, and the install-test
# matrix in .github/workflows/validate-charts.yml, which drops the excluded
# charts before the matrix is built (issue #231). Two parsers would be two
# The consumers of ci/excluded-charts.txt are this script and the two jobs that
# build an install-test matrix - detect-changed-charts in
# .github/workflows/validate-charts.yml and select-charts in
# .github/workflows/install-test-all-charts.yml - both of which drop the
# excluded charts before the matrix exists (issue #231). Two parsers would be two
# chances to select different charts - and the failure would be silent in the
# worst direction, a chart the workflow believes excluded and the script
# believes testable, or the reverse. So there is one parser, here, and the
@@ -47,9 +49,10 @@ excluded_reason() {
list_excluded | awk -F'\t' -v chart="$1" '$1 == chart { print $2; exit }'
}
# Emits "<chart><TAB><reason>" for every excluded chart. Read by the detect job
# of validate-charts.yml before any chart is picked, which is why this runs
# before the argument is required.
# Emits "<chart><TAB><reason>" for every excluded chart. Read by both jobs that
# assemble an install-test matrix (detect-changed-charts in validate-charts.yml,
# select-charts in install-test-all-charts.yml) before any chart is picked,
# which is why this runs before the argument is required.
if [ "${1:-}" = "--list-excluded" ]; then
list_excluded
exit 0
@@ -100,7 +103,7 @@ fi
# REQUIRED_ENV="FREESAILARR_WG_KEY FREESAILARR_WG_ADDRESSES"
#
# The values come from the environment - in CI from 1Password Connect, see the
# install-test job in .github/workflows/validate-charts.yml.
# install-test job in .github/workflows/install-test.yml.
#
# A missing or empty value is a hard error on purpose. Skipping the chart
# instead would restore exactly the green-but-meaningless signal this mechanism