ci: run the freesailarr install-test for real instead of skipping it

freesailarr was excluded from the install-test because gluetun never becomes
healthy without a real Wireguard key and no app container behind its kill
switch can become Ready. The green "Install-test freesailarr pass" was a
25-second SKIP on the most complex chart in the repo - a signal that proved
nothing.

With a CI-only Mullvad key in 1Password the reason is gone:

- ci/run-install-test.sh learns a general rule: ci/<chart>/fixtures.env.yaml is
  rendered with envsubst before it is applied, restricted to the variable names
  a chart declares in REQUIRED_ENV. A missing or empty declared variable fails
  the run loudly and names the variable - it never skips the chart.
- The install-test job loads the key from 1Password Connect for freesailarr
  only, following the pattern publish-helm-chart.yml documents: values via
  steps.<id>.outputs.*, mapped into the step's own env, quoted, with ":?"
  guards that name the missing op:// field before a cluster is created.
- Fixtures for freesailarr: three media claims, the gluetun secret template
  with variable names only, firewallOutboundSubnets so the kill switch does not
  block cluster DNS, and INSTALL_TIMEOUT=900 for nine containers.

freesailarr/Chart.yaml goes 3.0.0+up1.0.0 -> 3.0.1+up1.0.0. That bump is
deliberately empty of content: the install-test matrix follows changed chart
directories, and without it this PR could not run the test it adds.
This commit is contained in:
jklein committed 2026-08-26 14:30:42 +02:00
1 parent 5280554329
commit 69a280399c
9 files changed
+215 -7

No files matched your search

+6 -3
View File
@@ -40,7 +40,7 @@ Collection of some self-made helm-charts. Each top-level directory is a standalo
- `publish-<chart>.yml` — one thin workflow per chart; triggers on pushes to `main` touching `<chart>/**` (or manually) and calls the reusable workflow below. **A push to a chart directory on `main` publishes that chart**, so the chart version must be bumped in the same change.
- `publish-helm-chart.yml` — reusable workflow: `helm package`, then `helm push` to the private JK-Registry. Registry credentials come from 1Password Connect (`OP_CONNECT_TOKEN` secret, items under `op://GitHub-Actions/`). The loaded values are read via **`steps.<id>.outputs.*`**, mapped into each step's own `env:` block and used as quoted shell variables — never interpolated into the command line. `load-secrets-action` **v5 no longer exports them as environment variables** for later steps the way v4 did, so after the v5 bump every `${{ env.JK_CR_* }}` reference resolved to an empty string while the loading step still logged `Populating variable` and stayed green. Each value is additionally guarded with `:?`, because an empty one used to collapse into a missing argument rather than an error: `-u` consumed the following `--password-stdin`, helm fell back to an interactive prompt and the job died with `inappropriate ioctl for device` — a terminal error for what was really an unset variable.
- `change-app-version.yml` — manually dispatched app-version bump; the chart must be listed in its `chart-name` choice list.
- `validate-charts.yml` — PR pipeline: detects which charts a PR touches and, for exactly those, runs `helm lint --strict`, `helm package`, and a chart-version bump check (`template-chart` is exempt from the version check). PRs without chart changes skip the build matrix. After the build stage, an **install-test** job (same changed-charts matrix) installs each changed chart into an ephemeral [k3d](https://k3d.io) cluster on the runner and requires every workload to become Ready — see below.
- `validate-charts.yml` — PR pipeline: detects which charts a PR touches and, for exactly those, runs `helm lint --strict`, `helm package`, and a chart-version bump check (`template-chart` is exempt from the version check). PRs without chart changes skip the build matrix. After the build stage, an **install-test** job (same changed-charts matrix) installs each changed chart into an ephemeral [k3d](https://k3d.io) cluster on the runner and requires every workload to become Ready — see below. Both matrices follow the changed chart directories, so a PR that only changes the harness under `ci/` exercises nothing by itself; the Trivy scan has a fixed-sample fallback for that case, the install-test has none — bump the affected chart's patch version to pull it into the matrix.
### Install-test (PR pipeline stage 2)
@@ -51,9 +51,12 @@ Fixtures live at the **repo root** under `ci/` (deliberately *not* inside the ch
- `ci/common/` — fixtures applied for every chart (schema-less stub CRD for `OnePasswordItem`)
- `ci/<chart>/test-values.yaml` — values for the CI install (required for every installable chart)
- `ci/<chart>/fixtures.yaml` — dummy secrets with in-cluster connection data, throwaway infrastructure (postgres/redis/dex), static hostPath PVs/StorageClasses where the chart pins a storage class or needs RWX, and seed jobs (e.g. `synapse generate`)
- `ci/<chart>/settings.env` — optional overrides (e.g. `INSTALL_TIMEOUT` for slow starters)
- `ci/<chart>/fixtures.env.yaml` — fixtures that need a value which must not be committed. The file holds `${VARIABLE}` placeholders and nothing else; `ci/run-install-test.sh` renders it with `envsubst` and pipes the result straight into `kubectl` (never printed). Substitution is restricted to the names from `REQUIRED_ENV`, so an undeclared variable is never filled in and no unrelated `$…` in the manifest is touched
- `ci/<chart>/settings.env` — optional overrides: `INSTALL_TIMEOUT` for slow starters, and `REQUIRED_ENV="VAR_A VAR_B"` to declare the variables `fixtures.env.yaml` needs. **A missing or empty declared variable fails the install-test loudly, naming the variable — it never skips the chart.** Skipping would restore exactly the green-but-meaningless signal this mechanism exists to remove; the one legitimate case of absent secrets, a PR from a fork, belongs in an `if` on the workflow job rather than in the script
- `ci/excluded-charts.txt` — charts that cannot run in CI, each with a justification. Currently: `satisfactory` (multi-GB image + 10GB+ Steam download), `template-chart` (scaffold with placeholder image)
`freesailarr` is the only chart using `fixtures.env.yaml` today: gluetun needs a real Wireguard key, or it never completes a handshake and no app container behind its kill switch becomes Ready. The install-test job loads the key from 1Password Connect for that chart only, using the same pattern as `publish-helm-chart.yml` — `steps.<id>.outputs.*` into the step's own `env:`, quoted, with `:?` guards that name the missing `op://` field before the cluster is even created. The key in that item is a **CI-only Mullvad key**, deliberately not the production one: Mullvad ties an address to a key and routes the tunnel to the endpoint it saw last, so a shared key would make every PR run pull the tunnel out from under the running stack — with the symptom in the cluster and the cause in a pull request.
Fixture images avoid Docker Hub where possible (`public.ecr.aws`/`ghcr.io`) to dodge runner rate limits. The job **blocks the PR** on failure — the stabilization phase of #22 ended when Renovate started automerging patch/minor updates, since a chart that no longer starts must not reach `main` and publish unattended. When adding a new chart, add its `ci/<chart>/` fixtures (or an entry in `ci/excluded-charts.txt`) — the install-test fails if both are missing. The script also runs locally against any existing cluster: `k3d cluster create t && ci/run-install-test.sh <chart> && k3d cluster delete t`.
## Dependency updates (Renovate)
@@ -251,5 +254,5 @@ Reference implementation: `template-chart/values.schema.json` and `template-char
- `icon:` is mandatory in every `Chart.yaml`. **Documented exception — freesailarr, the only chart without one**: the rule everywhere else is "the icon is this chart's one upstream's logo", and a chart bundling nine independently versioned projects has no such upstream — picking one of the nine would imply a "main app" the stack deliberately does not have. That is the whole criterion, and it applies to no other chart in the repo: a single-upstream chart without an icon is a missing icon, not an exception. The omission costs an INFO line in `helm lint --strict` (`icon is recommended`), which still exits 0.
- `helm lint --strict` must pass without findings.
- `NOTES.txt` for every chart is planned (#43); seven charts ship one so far (`apache-tika`, `cyberchef`, `freesailarr`, `gotenberg`, `homeassistant`, `patchmon`, `rss-bridge`).
- CI: every installable chart has fixtures under `ci/<chart>/` (`test-values.yaml`, optionally `fixtures.yaml`/`settings.env`) or a justified entry in `ci/excluded-charts.txt` — the install-test fails if both are missing.
- CI: every installable chart has fixtures under `ci/<chart>/` (`test-values.yaml`, optionally `fixtures.yaml`/`fixtures.env.yaml`/`settings.env`) or a justified entry in `ci/excluded-charts.txt` — the install-test fails if both are missing.
- Charts have no dependencies (`charts/` directories are empty).