From cd49460ed9afdfefbd2a0a544e5d2e106d1d12e6 Mon Sep 17 00:00:00 2001 From: Justin Klein Date: Sun, 27 Sep 2026 14:29:00 +0000 Subject: [PATCH] CI/CD: the all-charts install-test, and where the install-test job now lives The install-test job moved into the reusable install-test.yml with two callers - the PR pipeline with the charts a pull request changed, the new install-test-all-charts.yml with every installable chart, on demand (Gitea #52). CI-CD: a section on the all-charts run - why it exists (the accidental first one, Gitea PR #48: 19 of 20 charts Ready in 12m41s, rss-bridge not), when to run it, what it records, the 20-charts-in-12m41s expectation, and that it runs the same job with a different matrix. Plus the runtime row, the shared-job note under the install-test stage, and select-charts next to detect-changed-charts where the exclusion list is read. Forge-and-Tooling: the quoted required-check name predates the move of the job into a called workflow - read the current one off a real run. Lessons-Learned: the green history from another machine, and one line in the recurring shape about making an accidental answer repeatable. Home: a row for "does every chart still come up on the current runner". --- CI-CD.md | 46 +++++++++++++++++++++++++++++++++++++++----- Forge-and-Tooling.md | 2 ++ Home.md | 1 + Lessons-Learned.md | 9 +++++++++ 4 files changed, 53 insertions(+), 5 deletions(-) diff --git a/CI-CD.md b/CI-CD.md index 7ebc52f..14f9d4b 100644 --- a/CI-CD.md +++ b/CI-CD.md @@ -1,6 +1,6 @@ # CI/CD -Two pipelines: one that **validates** a pull request, one that **publishes** a chart after the merge. Both run on Gitea Actions and read the files under `.github/workflows/` unchanged. +Two pipelines: one that **validates** a pull request, one that **publishes** a chart after the merge. Both run on Gitea Actions and read the files under `.github/workflows/` unchanged. Next to them one manually dispatched run, the [all-charts install-test](#the-all-charts-install-test-on-demand), which asks a question no pull request asks: does the environment under an *unchanged* chart still hold. ## Publishing @@ -70,6 +70,10 @@ Trivy is installed from its release tarball, not via `aquasecurity/setup-trivy` `ci/run-install-test.sh ` against a throwaway [k3d](https://k3d.io) cluster: apply the stub `OnePasswordItem` CRD plus the chart's fixtures, `helm install` with the CI values, and wait for every Deployment and StatefulSet to roll out. Since probes and hardening are active, "Ready" means the app actually answers. On failure the job dumps pod descriptions, events and container logs. +The job does not live in `validate-charts.yml`. It is the reusable workflow **`install-test.yml`** (`workflow_call`, one input: a JSON array of chart names), called from here with the changed charts and from `install-test-all-charts.yml` with all of them. Everything that decides *how* a chart is installed is in that one file — the runner-label expression, the pinned `kubectl` with its skew check, `envsubst`, the containerised-job detection, the per-leg cluster name, the `freesailarr` credentials; the callers decide only *which* charts and *whether* the job runs at all. It is **not copied** for the second caller: a second copy is a second chance to test differently, unnoticed, which is the same argument that gives `ci/excluded-charts.txt` exactly one parser. + +Every leg records its own outcome in the job summary next to the four facts that decide what that outcome means — date, runner label, `K3S_IMAGE`, `kubectl` version — and it distinguishes "the chart did not come up" from "the cluster never came up", because the two mean opposite things. + **The job blocks the PR.** That stopped being negotiable when Renovate began automerging patch and minor updates: a chart that no longer starts must not reach `main` and publish unattended. It counts double now that automerge covers **every** update type — `validate-charts` is a required check on `main`, so a red install-test keeps the armed PR where it is. ## Install-test fixtures @@ -93,9 +97,9 @@ Fixture images avoid Docker Hub where possible (`public.ecr.aws`, `ghcr.io`) to ### Excluded charts are dropped from the matrix, not skipped inside it -An excluded chart used to get a job that ran the script, printed `SKIP` and exited 0 — and the forge knows only the exit code, so the check list showed `Install-test satisfactory ✓` for a chart whose image was never pulled, indistinguishable from one that really came up. `detect-changed-charts` now removes the excluded charts **before** the matrix is built, so no job exists and nothing claims they passed. +An excluded chart used to get a job that ran the script, printed `SKIP` and exited 0 — and the forge knows only the exit code, so the check list showed `Install-test satisfactory ✓` for a chart whose image was never pulled, indistinguishable from one that really came up. `detect-changed-charts` now removes the excluded charts **before** the matrix is built, so no job exists and nothing claims they passed. The all-charts run does the same in its `select-charts` job. -Because an absence on its own says nothing, the job writes every excluded chart of the PR **with its documented justification** into the job summary. The list is parsed in exactly one place: `ci/run-install-test.sh --list-excluded` emits `` and the workflow calls it rather than re-implementing the format in YAML — a second parser would be a second chance to select different charts, silently and in either direction. The script keeps its `SKIP` branch for local runs. +Because an absence on its own says nothing, the job that builds the matrix writes every excluded chart **with its documented justification** into the job summary. The list is parsed in exactly one place: `ci/run-install-test.sh --list-excluded` emits `` and both workflows call it rather than re-implementing the format in YAML — a second parser would be a second chance to select different charts, silently and in either direction. The script keeps its `SKIP` branch for local runs. **What cannot be done instead:** deciding per chart in an `if` on the job, to get a visible `Skipped` rather than an absent entry. It is rejected with `context "matrix" is not allowed here` — `jobs..if` is evaluated *before* the matrix is expanded, so the `matrix` context does not exist there, and the whole workflow ends in failure with zero jobs created. @@ -103,6 +107,37 @@ Because an absence on its own says nothing, the job writes every excluded chart A line in `ci/excluded-charts.txt` carrying only a chart name is deliberately **not** an exclusion: a bare name would drop a chart out of CI with nothing to review. +## The all-charts install-test (on demand) + +`install-test-all-charts.yml` runs **the same job** as the PR pipeline — the same `install-test.yml`, the same pinned `kubectl`, the same fixtures — with a matrix over **every installable chart** instead of the ones a pull request changed. `workflow_dispatch` only, **no schedule**. + +### Why it exists + +The install-test matrix follows the charts a pull request **changes**. So there is no moment at which the charts are checked as a whole against the current environment — and a chart nobody has touched since the forge move carries a green history from a *different runner*. That green says nothing about today, which is exactly the signal [rule 2](#two-rules-for-every-check-added-here) is about. + +The value of the answer was demonstrated by accident. Gitea PR #48 corrected a **comment** in 21 charts, which made every one of them "changed" and turned the run into the first all-charts install-test after the move: **19 of 20 matrix charts installed and became Ready in 12m41s**, in one job, on the Gitea runner with the pinned `kubectl` inside the job container — and exactly one did not. That one was `rss-bridge`, whose last green run was from **2026-08-29 and a GitHub runner** (Gitea #51). Without the comment round, nobody would have learned either half. + +### When to run it + +- after a change to the **runner** +- after a change to **`K3S_IMAGE`** or the pinned **`KUBECTL_VERSION`** +- after a change to the **fixtures** under `ci/` +- after a **forge move** +- and for a pull request that only changes the harness and therefore tests nothing by itself — run it once the change is on `main` + +Not on a schedule: it would spend compute every day on a question that changes rarely, and a red run nobody asked for is a red run nobody reads. + +### How it behaves + +| | | +|---|---| +| **Input `charts`** | space- or comma-separated chart names, empty = all. This is what lets a single chart be tested against the runner, which is what Gitea #51 needs. | +| **Unknown or excluded name** | a **loud error**, not a silent omission — someone typed that name, and a green run over zero charts would be the worst possible answer. An empty matrix fails the run for the same reason. | +| **One cluster per chart** | no reuse. Reuse saves time and changes what the test proves: leftovers of the previous install, a warm image cache, fixtures already applied. "Came up from nothing" is the statement. | +| **Excluded charts** | dropped through the one parser, `ci/run-install-test.sh --list-excluded`, and listed in the summary **with their justification**, as in the PR pipeline. | +| **The record** | per chart: outcome, date, runner label, `K3S_IMAGE`, `kubectl` version. Plus a final job with the verdict over the whole matrix. A green run nobody notes is unknown again next time. | +| **Expected runtime** | **20 charts in 12m41s** (Gitea PR #48, one job, all legs concurrent on one Docker daemon), against 23 charts in the matrix today. Written down so a later deviation is visible as one. The timeout is 30 minutes per leg, in `install-test.yml`, hence identical for both callers. | + ## `ci/helm-dependencies.sh` — the step they all share Not a workflow but the prerequisite of every helm command in this repo: it fetches a chart's declared dependencies into `charts/`, and returns silently for the charts that declare none (all but one). @@ -113,7 +148,7 @@ It runs `helm dependency update`, not `build`: `Chart.lock` is not committed, be ## A PR that touches only `ci/` -Both matrices follow the changed **chart directories**, so a PR that only changes the harness exercises nothing by itself. The Trivy scan has a fixed-sample fallback for that case; the install-test has none — bump the affected chart's patch version to pull it into the matrix. The quantity render test is the exception to the rule: it always runs over everything. +Both matrices follow the changed **chart directories**, so a PR that only changes the harness exercises nothing by itself. The Trivy scan has a fixed-sample fallback for that case; the install-test has none — bump the affected chart's patch version to pull it into the matrix, or run the [all-charts install-test](#the-all-charts-install-test-on-demand) once the change is on `main`. The quantity render test is the exception to the rule: it always runs over everything. ## What a Gitea runner does differently @@ -123,7 +158,7 @@ All measured on this instance, not taken from documentation, and the reason for **Runner labels mean different machines.** `ubuntu-latest` has **no Docker socket** here; `ubuntu-docker` has one. Exactly one job cares — the install-test, because k3d is a Docker client — so its `runs-on` is an expression rather than a fixed label, and the same file still works on a GitHub runner where `ubuntu-docker` does not exist. -**A matrix runs concurrently inside ONE job, under ONE status context.** Operationally: all legs share one Docker daemon, which makes a per-leg resource name mandatory. For branch protection: a required check per chart is not possible, and the context name carries the *unexpanded* matrix expression, so required-check names must be read off a real run. What `main` does require is **`validate-charts`**, as that single context, alongside a review — the two conditions Renovate's auto-merge waits for (see [Renovate → PR policy](Renovate#pr-policy)). +**A matrix runs concurrently inside ONE job, under ONE status context.** Operationally: all legs share one Docker daemon, which makes a per-leg resource name mandatory. For branch protection: a required check per chart is not possible, and the context name carries the *unexpanded* matrix expression, so required-check names must be read off a real run — the more so since the install-test job moved into a called workflow, which may add the calling job's name to the context. What `main` does require is **`validate-charts`**, as that single context, alongside a review — the two conditions Renovate's auto-merge waits for (see [Renovate → PR policy](Renovate#pr-policy)). **Expected runtimes**, so a later deviation is visible as one: @@ -132,6 +167,7 @@ All measured on this instance, not taken from documentation, and the reason for | `Create k3d cluster` | 19–25 s | 22–23 s | | install-test, two charts in the matrix (cyberchef + freesailarr) | 210–216 s over three runs, spread 6 s | ~145 s wall, two parallel VMs | | install-test, freesailarr alone, ten containers and a real tunnel | 225 s | not comparable | +| install-test, **20 charts** in the matrix — the accidental all-charts run of Gitea PR #48 | **12m41s**, one job, all legs concurrent; 19 of 20 Ready | never ran there | The runner is **not slower** — cluster creation is a touch faster. The difference in total time is **structural**: two matrix legs share one runner here, while GitHub gives each leg its own VM. Reading it as "Gitea is slower" leads to the wrong conclusion. diff --git a/Forge-and-Tooling.md b/Forge-and-Tooling.md index fa7160c..5afcc1b 100644 --- a/Forge-and-Tooling.md +++ b/Forge-and-Tooling.md @@ -66,6 +66,8 @@ Validate Helm Charts / Install-test ${{ fromJSON(needs.detect-changed-charts.out Required-check names have to be read off a real run; they cannot be copied from GitHub. Relevant whenever branch protection is touched. +The name above is from **before** the install-test job moved into the reusable `install-test.yml` (Gitea #52): the matrix expression now reads `${{ fromJSON(inputs.charts) }}`, and a called workflow's job may carry the calling job's name in the context as well. `validate-charts` itself is unaffected — but read the current name off a real run before putting it into branch protection. + ## Issue numbers changed with the move - **`#n` inside this repo's files means the GitHub tracker.** Most of those numbers name *closed* issues, which were not migrated and exist only there. diff --git a/Home.md b/Home.md index ab12a84..97b961b 100644 --- a/Home.md +++ b/Home.md @@ -12,6 +12,7 @@ The repo lives on the Gitea instance **JLab-Git**. Issues, pull requests, CI and | know how work gets done here (issues → plan → agent → PR → merge) | [Workflow](Workflow) | | know which forge, which CLI, which remote, signed commits | [Forge and tooling](Forge-and-Tooling) | | understand the PR pipeline, the install-test, the render test | [CI/CD](CI-CD) | +| check whether every chart still comes up on the current runner | [CI/CD → all-charts install-test](CI-CD#the-all-charts-install-test-on-demand) | | understand how chart versions are numbered and bumped | [Versioning](Versioning) | | write or review a chart | [Chart style guide](Chart-Style-Guide) | | add a new chart | [Creating a new chart](New-Chart) | diff --git a/Lessons-Learned.md b/Lessons-Learned.md index 642096f..b52e97e 100644 --- a/Lessons-Learned.md +++ b/Lessons-Learned.md @@ -178,6 +178,14 @@ Same chart, the other half of the same review. [valkey](valkey) takes its passwo **Now:** an init container greps the mounted file for a `requirepass ` line and refuses to let the pod start without it, naming the 1Password field and never printing its contents. Generally: **when a secret's *format* becomes load-bearing, something has to validate the format** — a value that is merely *present* satisfies Kubernetes, and the failure it produces can be a working server with its protection off. +## The green history from another machine + +The move to Gitea changed the execution environment. The install-test matrix follows the charts a pull request **changes**, so a chart nobody had touched since the move still carried its last green tick — earned on a GitHub runner, months earlier — and it was read as "this chart comes up". It said nothing about the current runner. + +It surfaced by accident. A pull request correcting a *comment* in 21 charts made every one of them "changed" and thereby became the first all-charts install-test after the move: 19 of 20 matrix charts came up in 12m41s, and `rss-bridge` did not. Its last green run was from **2026-08-29 and a GitHub runner** (Gitea #51). + +**Now:** the install-test job is one reusable workflow with two callers, and the second runs it over **every** chart on demand (Gitea #52) — after a runner change, a `K3S_IMAGE` or `kubectl` change, a fixture change, a forge move. Every leg records date, runner label, `K3S_IMAGE` and `kubectl` version next to its outcome, because a green run nobody notes is unknown again next time. See [CI/CD](CI-CD#the-all-charts-install-test-on-demand). Generalized: **a check whose scope follows the diff never re-checks what the diff does not touch** — and what it quietly stops covering is the environment itself. + ## The recurring shape - **Green is a claim about the assertions that ran, not about the system.** @@ -186,3 +194,4 @@ Same chart, the other half of the same review. [valkey](valkey) takes its passwo - **When a rename and a silent behaviour change are both available, take the rename.** Schema validation failing loudly beats a value that quietly means the opposite. - **A safeguard is only a safeguard if it can fail.** Both the missing `required()` and the `required` array that replaced it read as protection; neither could ever have fired. - **Write down what was measured, and say when something was not.** Half of this page exists because someone recorded a number instead of an impression. +- **An answer that arrived by accident is worth making repeatable.** The all-charts install-test exists because one unplanned run produced a result nobody could have asked for; the fix is a way to ask the question on purpose, not a note to remember the accident.