Files
helm-charts/.github/workflows/install-test.yml
T
jlab-renovate-bot d7707bdd54
renovate/stability-days Updates have met minimum release age requirement
Validate Helm Charts / Detect changed charts (pull_request) Successful in 4s
Validate Helm Charts / Build ${{ matrix.chart }} (pull_request) Skipped
Validate Helm Charts / Trivy config ${{ matrix.chart }} (pull_request) Skipped
Validate Helm Charts / Install-test (pull_request) Skipped
Validate Helm Charts / Resource quantities (pull_request) Successful in 16s
Validate Helm Charts / Selector match (pull_request) Successful in 15s
Validate Helm Charts / HTTPRoute match (pull_request) Successful in 15s
Update dependency kubernetes/kubernetes to v1.37.1
2026-10-12 19:35:23 +00:00

418 lines
25 KiB
YAML

# The install-test, as one job with two callers.
#
# Both callers install a chart the SAME way; they differ only in which charts
# they name. validate-charts.yml passes the charts a pull request changed;
# install-test-all-charts.yml passes every installable chart on demand, to ask
# whether the environment underneath an UNCHANGED chart still holds (Gitea
# #52) - a chart untouched since the forge move carries a green history from a
# different runner, and that says nothing about today.
#
# It is a called workflow and not a second job, for the same reason
# ci/excluded-charts.txt has exactly one parser: a second copy is a second
# chance to test DIFFERENTLY, unnoticed. The parts that would drift are worth
# naming, because each was paid for once already - the pinned kubectl with its
# skew check, envsubst, the detection of a containerised job, the per-leg
# cluster name. A copy that slowly disagrees about any of them does not go red;
# it turns a green run into a statement about something other than what was
# assumed.
#
# What stays with the callers: WHICH charts are tested, and whether the job
# runs at all (needs/if). Everything that decides HOW one is installed is here.
name: Install-test
on:
workflow_call:
inputs:
charts:
required: true
type: string
description: 'JSON array of chart names, e.g. ["cyberchef","zipline"]. The entries of ci/excluded-charts.txt must already be gone - see the matrix comment below.'
secrets:
OP_CONNECT_TOKEN:
# Needed by exactly one leg - freesailarr, for the 1Password Connect
# lookup of its CI Wireguard key. Declared OPTIONAL on purpose, where
# the shared publish block in workflow-util/pipelines declares its token
# required: on this instance a missing secret does not fail the job, it
# fails job CREATION, so the run shows no steps at all (README, "Where
# this repo lives"; the long form is on that repository's wiki). With
# required: true a pull request from a fork - which carries no secrets -
# would lose the entire install-test that way, every chart at once, with
# nothing naming the cause. Optional instead: only the freesailarr leg
# fails, loudly, at the ":?" guards that name the missing op:// field.
required: false
description: "1Password Connect token; only the freesailarr leg reads it"
jobs:
install-test:
name: "Install-test ${{ matrix.chart }}"
# The one job in this repo that does not simply say ubuntu-latest, and the
# expression is not decoration - the same label means different machines on
# the two forges this repo currently lives on (#225).
#
# This is the only job that needs a Docker socket: it creates a k3d
# cluster, and k3d is a Docker client. On GitHub's hosted runners every
# label carries Docker, so the question never came up. On the Gitea
# instance it was MEASURED on a probe pull request, per job:
# detect-changed-charts, build-chart, the quantity render and the Trivy
# scan all pass on `ubuntu-latest` - none of them needs Docker - while the
# install-test dies there at "Create k3d cluster" with "Cannot connect to
# the Docker daemon at unix:///var/run/docker.sock", an error that points
# at k3d and never at the runner label. `ubuntu-docker` is the label that
# carries a socket, and Gitea does evaluate this expression: the probe job
# was picked up by the docker runner with labels ["ubuntu-docker"].
#
# Why an expression and not a plain `ubuntu-docker`: that label does not
# exist on GitHub, so every chart PR would sit queued forever for as long
# as this repo still builds on github.com - blocking the merge with no
# error anywhere. github.server_url is the one context value that separates
# the two without naming our instance.
#
# The label is necessary and was, for a while, not sufficient:
# `ubuntu-docker`'s daemon reported "Cgroup Driver: none" plus "Running in
# rootless-mode without cgroups", and k3d got as far as starting the k3s
# node, which restarted immediately with an empty log because k3s needs
# cgroup delegation (#225). Nothing in this file addressed that - it was
# dealt with on the instance side. It no longer blocks, and the evidence is
# Gitea PR #48: 19 of 20 matrix charts installed and became Ready on this
# label. The twentieth, rss-bridge, failed and is Gitea #51.
runs-on: ${{ github.server_url == 'https://github.com' && 'ubuntu-latest' || 'ubuntu-docker' }}
# Blocking since Renovate automerges patch/minor updates: a chart that no
# longer starts must stop the PR before it reaches main and publishes. This
# ends the stabilization phase of issue #22 - the job reported failures
# without blocking while it proved itself across several chart PRs.
#
# The 30 minutes are per leg and hold for the all-charts caller too, with
# the measured figure to compare against: Gitea PR #48 ran 20 charts in
# 12m41s in one job, all legs concurrently on one Docker daemon. If a run
# ever comes near this timeout, it is raised HERE, once, for both callers -
# a value that differs per caller would mean the two tests give up at
# different points while claiming to be the same test.
timeout-minutes: 30
strategy:
fail-fast: false
# The charts arrive from the caller ALREADY filtered against
# ci/excluded-charts.txt: validate-charts.yml does it in
# detect-changed-charts, install-test-all-charts.yml in select-charts, and
# both go through ci/run-install-test.sh --list-excluded, the one parser
# of that file. Both also name every chart they dropped WITH its
# documented reason in their own job summary, because an absent matrix
# entry says "not run" and never why (issue #231).
#
# DO NOT move that decision in here, e.g. as an "if" on this job
# if: ... && !contains(fromJSON(inputs.excluded-charts), matrix.chart)
# to get a visible "Skipped" per chart instead of an absent entry.
# GitHub rejects it - jobs.<id>.if is evaluated BEFORE the matrix is
# expanded, so the matrix context does not exist there:
# context "matrix" is not allowed here.
# available contexts are "github", "inputs", "needs", "vars".
# Measured on run 33650724109 (PR #233, closed): the whole workflow ended
# in failure with zero jobs created. The per-chart value has to be gone
# from the matrix before this workflow is called.
matrix:
chart: ${{ fromJSON(inputs.charts) }}
env:
K3D_VERSION: v5.9.0
K3S_IMAGE: docker.io/rancher/k3s:v1.35.5-k3s1
# Pinned rather than taken from the runner image. The argument is not
# "Gitea's image has no kubectl" - that is only how it was noticed. A
# kubectl inherited from whatever the image happens to ship drifts
# unannounced with every image rebuild, and an install-test is supposed
# to run against a KNOWN client. That GitHub's hosted image already
# carries one does not make the pin unnecessary; it only makes the drift
# invisible there. The value mirrors K3S_IMAGE above (v1.35.5), which is
# the cluster this client talks to.
# renovate: datasource=github-releases depName=kubernetes/kubernetes
KUBECTL_VERSION: v1.37.1
# The same expression as runs-on above, written a second time because the
# env context is not available in runs-on - so it cannot be the other way
# round, and reading the label back off the runner is not possible either
# (RUNNER_NAME below names the machine, not the label that picked it).
# It exists for the run record at the end of this job: "on which runner
# was this green tick measured" is the whole question of Gitea #52. Kept
# right under runs-on, so a change to one that forgets the other is
# visible on a single screen.
RUNNER_LABEL: ${{ github.server_url == 'https://github.com' && 'ubuntu-latest' || 'ubuntu-docker' }}
steps:
- name: "Checkout Repository"
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
- name: "Install Helm"
uses: azure/setup-helm@9bc31f4ebc9c6b171d7bfbaa5d006ae7abdb4310 # v5
- name: "Install kubectl"
run: |
# kubectl and the cluster are a pair: the client talks to the server
# K3S_IMAGE starts, and Kubernetes supports at most one minor of
# skew between them. Renovate updates KUBECTL_VERSION on its own
# while K3S_IMAGE is tracked by nothing, so without this check the
# two drift apart until the client is talking to a server it no
# longer supports - silently, because kubectl only warns. And it is
# not theoretical: measured by dry run, a lookup from v1.34.0
# offers v1.37.0 in ONE step, because Renovate fills the non-major
# bucket with the newest candidate rather than walking the minors.
# So this check WILL fire one day, on purpose, with a message that
# says which of the two pins to move.
#
# Deliberately not solved with an allowedVersions constraint pinning
# kubectl to the k3s minor: that is the nginx trap the README
# describes - a filter that silently stops offering updates the day
# the other side moves. A loud failure beats a quiet freeze.
read -r KMAJ KMIN <<<"$(printf '%s' "$KUBECTL_VERSION" | sed -E 's/^v([0-9]+)\.([0-9]+).*/\1 \2/')"
read -r SMAJ SMIN <<<"$(printf '%s' "$K3S_IMAGE" | sed -E 's/.*:v([0-9]+)\.([0-9]+).*/\1 \2/')"
SKEW=$(( (KMAJ * 100 + KMIN) - (SMAJ * 100 + SMIN) ))
if [ "${SKEW#-}" -gt 1 ]; then
echo "::error::kubectl $KUBECTL_VERSION and k3s $K3S_IMAGE are more than one minor apart. Bump K3S_IMAGE to match, or pin KUBECTL_VERSION back."
exit 1
fi
curl -sSfL -o /tmp/kubectl "https://dl.k8s.io/release/${KUBECTL_VERSION}/bin/linux/amd64/kubectl"
mkdir -p "$HOME/.local/bin"
install -m 0755 /tmp/kubectl "$HOME/.local/bin/kubectl"
echo "$HOME/.local/bin" >> "$GITHUB_PATH"
"$HOME/.local/bin/kubectl" version --client
- name: "Provide envsubst"
run: |
# ci/run-install-test.sh renders ci/<chart>/fixtures.env.yaml with
# envsubst - today only freesailarr has one, and without the tool
# the fixtures collapse to "error: no objects passed to apply",
# gluetun never gets its Wireguard key, and nothing behind its kill
# switch becomes Ready. GitHub's hosted image ships gettext-base,
# the Gitea runner image does not. The script is fine as it is; a
# standard tool is simply missing.
#
# CHECKED FOR, NOT PINNED - and that is a different decision from
# kubectl above, on purpose. kubectl's VERSION changes what the test
# means: its distance to the API server is a real source of failure,
# which is why it is pinned, tracked and guarded. envsubst
# substitutes text; which release performs the substitution does not
# change the result. Pinning it would buy nothing and add a line for
# Renovate to carry. So: present - do nothing, which keeps the
# GitHub path free of any package install; missing - fetch it, and
# the difference between the two runners is stated rather than
# hidden behind an unconditional apt-get.
if command -v envsubst >/dev/null 2>&1; then
echo "envsubst present at $(command -v envsubst) - nothing to do"
exit 0
fi
echo "envsubst missing on this runner image - installing gettext-base"
if [ "$(id -u)" -eq 0 ]; then SUDO=""; else SUDO="sudo"; fi
$SUDO apt-get update -qq
$SUDO apt-get install -y -qq --no-install-recommends gettext-base
command -v envsubst
# freesailarr is the one chart whose install-test needs credentials:
# without a real Wireguard key gluetun never completes a handshake, and
# behind its kill switch not a single app container becomes Ready. The key
# comes from 1Password Connect like the registry credentials in the shared
# publish block, and it is a CI-ONLY Mullvad key: Mullvad ties an
# address to a key and sends the tunnel to the endpoint it saw last, so
# reusing the production key would let every PR run pull the tunnel out
# from under the running stack - with the symptom in the cluster and the
# cause in a pull request. Loaded only for this chart; no other chart run
# touches the VPN key.
- name: "Configure OP-Connect"
if: matrix.chart == 'freesailarr'
uses: 1password/load-secrets-action/configure@70062d7a876d3eb6334754fa26efd2fbd90c32f2 # v5
with:
connect-host: https://op-connect.jk-dev.de
- name: "Load freesailarr VPN credentials"
id: load-freesailarr-vpn
if: matrix.chart == 'freesailarr'
uses: 1password/load-secrets-action@70062d7a876d3eb6334754fa26efd2fbd90c32f2 # v5
env:
OP_CONNECT_TOKEN: ${{ secrets.OP_CONNECT_TOKEN }}
FREESAILARR_WG_KEY: op://GitHub-Actions/freesailarr-mullvad-test-config/credential
FREESAILARR_WG_ADDRESSES: op://GitHub-Actions/freesailarr-mullvad-test-config/ip-address
# The same trap the shared publish block documents at length - since
# Gitea #66 that long form lives on the wiki of workflow-util/pipelines,
# page "publish-helm-chart", not in this repository: v5 of
# load-secrets-action does NOT export the loaded values as environment
# variables for later steps, only its step outputs carry them, and the
# loading step logs "Populating variable" and stays green either way. So
# the values are mapped into each step's own env block via
# steps.<id>.outputs.* and read as quoted shell variables. The ":?" guards
# run before the cluster is even created and name the op:// field that is
# missing - an empty key would otherwise reach gluetun and look like a
# broken chart instead of a missing secret. They print the variable name,
# never its value.
- name: "Verify the VPN credentials arrived"
if: matrix.chart == 'freesailarr'
env:
FREESAILARR_WG_KEY: ${{ steps.load-freesailarr-vpn.outputs.FREESAILARR_WG_KEY }}
FREESAILARR_WG_ADDRESSES: ${{ steps.load-freesailarr-vpn.outputs.FREESAILARR_WG_ADDRESSES }}
run: |-
: "${FREESAILARR_WG_KEY:?empty - check op://GitHub-Actions/freesailarr-mullvad-test-config/credential}"
: "${FREESAILARR_WG_ADDRESSES:?empty - check op://GitHub-Actions/freesailarr-mullvad-test-config/ip-address}"
echo "Both VPN values arrived non-empty."
- name: "Install k3d"
run: curl -sSfL https://raw.githubusercontent.com/k3d-io/k3d/main/install.sh | TAG="$K3D_VERSION" bash
# One cluster per matrix leg, named after the chart. It used to be the
# fixed name "install-test", and that was wrong on GitHub too - it just
# could not show there, because GitHub gives every matrix leg its own VM
# with its own Docker daemon, so two legs never saw each other's
# cluster. The environment hid the bug for two years rather than
# preventing it. Gitea runs all legs of a matrix CONCURRENTLY in ONE job
# on ONE daemon, and there the fixed name collides at once: measured on
# the probe PR of #225, one leg logs "Re-using existing network
# 'k3d-install-test'", the other dies on "The container name
# /k3d-install-test-server-0 is already in use" - and then the first
# leg's rollback deletes the shared network out from under the second,
# which fails with "network k3d-install-test not found". Both legs red,
# neither cause in either message. DO NOT put the fixed name back.
#
# Why the bare chart name and not a prefixed one: k3d rejects a cluster
# name over 32 characters ("Cluster name must be <= 32 characters",
# measured against k3d v5.9.0 - 32 passes, 33 fails), and every
# container is named k3d-<cluster>-server-0 from it. The longest chart
# here is calibre-web-automated at 21, so "install-test-<chart>" would
# be 34 and break exactly that one chart and no other - the kind of
# prefix that is worse than the fixed name because it only sometimes
# breaks. The guard below states the limit instead of leaving it to be
# rediscovered.
- name: "Create k3d cluster"
run: |
CLUSTER="${{ matrix.chart }}"
if [ "${#CLUSTER}" -gt 32 ]; then
echo "::error::chart name '$CLUSTER' is ${#CLUSTER} characters and k3d allows at most 32 for a cluster name."
exit 1
fi
# Does THIS job run inside a container? That, and not which forge we
# are on, is what decides whether k3d's cluster is reachable (#227).
#
# k3d publishes the API server port on the Docker *host*. When the
# job runs straight on the host - GitHub's hosted runners - that is
# the same machine, and the "https://0.0.0.0:<port>" k3d writes into
# the kubeconfig works. When the job itself is a container, as every
# Gitea Actions job is, the host is the dind container and 0.0.0.0
# points at the job container itself, where nothing listens. The
# cluster comes up fine and the first kubectl call is refused.
#
# Deliberately NOT keyed on github.server_url, although that would
# pick the right branch today: a self-hosted GitHub runner in a
# container has exactly this problem, and a condition that says
# "GitHub" while meaning "not containerised" is the kind of lie this
# repo exists to remove. So the state is detected instead of
# inferred, and the detection is the same question the fix needs
# answered anyway: ask the daemon about ourselves. Inside a
# container that the daemon knows, this returns our network and k3d
# joins it, which puts k3s and this job on one bridge where Docker's
# own DNS resolves the container names. On a host it returns nothing,
# because the hostname is not a container, and the old path is taken
# unchanged.
# The "|| true" is load-bearing: this step runs under `bash -e -o
# pipefail`, and on a host `docker inspect` exits non-zero, which
# would abort the step instead of taking the else branch.
JOB_NET="$(docker inspect -f '{{range $k, $v := .NetworkSettings.Networks}}{{$k}} {{end}}' "$(hostname)" 2>/dev/null | awk '{print $1}' || true)"
if [ -n "$JOB_NET" ]; then
echo "Containerised job: $(hostname) on docker network '$JOB_NET' - k3d joins it."
k3d cluster create "$CLUSTER" --image "$K3S_IMAGE" --network "$JOB_NET" --wait --timeout 180s
# Same reason: the published port is useless from in here, so the
# kubeconfig is pointed at the load balancer container by name,
# over the shared network. If the API server certificate did not
# carry that name this would fail on the TLS handshake and the
# cluster-info below would say so - it holds without an extra
# --tls-san, measured on the probe PR of #225.
kubectl config set-cluster "k3d-$CLUSTER" --server="https://k3d-$CLUSTER-serverlb:6443"
else
echo "Job runs on the docker host - k3d's published port is reachable as written."
k3d cluster create "$CLUSTER" --image "$K3S_IMAGE" --wait --timeout 180s
fi
# Reachability is proven HERE, and the shape of this check is the
# product of getting it wrong twice. Both mistakes are written down
# because both look like something to simplify away.
#
# It is a LOOP and not a single call. k3d's --wait returns when the
# node container reports ready, but the API server can still answer
# discovery with 503 for a moment after that. Measured: a single
# `kubectl cluster-info` right here failed with "ServiceUnavailable:
# the server is currently unable to handle the request" on one of
# two legs of the SAME GitHub run while the other passed. A check
# that is itself racy produces exactly the sporadic red this job
# must never have (#227) - and a test people have learned to re-run
# is one they have stopped believing.
#
# It counts READY NODES and not "did kubectl answer". `kubectl get
# nodes` exits 0 on an EMPTY list, so the first version passed
# against a cluster whose API server was up and whose node had never
# registered: the log said "No resources found", the step went
# green, and the failure surfaced five minutes later as a pod stuck
# in Pending with no node and no scheduling event - nowhere near its
# cause. A check that a broken cluster passes is worse than no
# check. `kubectl wait --for=condition=Ready nodes --all` is not the
# shorter way to write this: it has the identical hole, because with
# zero nodes the condition is vacuously satisfied.
#
# The evidence that this version earns its keep: only after it
# counted nodes did the real cause become visible in the dump below
# - k3s could not start containerd ("error creating fsnotify
# watcher: too many open files", an inotify limit on the host), and
# nothing above this line had ever said so.
for ATTEMPT in $(seq 1 90); do
READY_NODES="$(kubectl get nodes --no-headers 2>/dev/null | awk '$2 == "Ready"' | wc -l | tr -d ' ')"
[ "$READY_NODES" -ge 1 ] && break
if [ "$ATTEMPT" -eq 90 ]; then
echo "::error::the k3d cluster '$CLUSTER' answers but has no Ready node after 180s"
kubectl get nodes -o wide || true
kubectl describe nodes || true
echo "--- k3s server log ---"
docker logs "k3d-$CLUSTER-server-0" 2>&1 | tail -80 || true
exit 1
fi
sleep 2
done
kubectl get nodes -o wide
# Every chart reaching this step is one the exclusion filter let through,
# so the script's SKIP branch is never taken here - it exists for local
# runs. The variables below are empty for every chart but the one that
# declares them in ci/<chart>/settings.env (REQUIRED_ENV); the script
# fails loudly if a declared one is missing, it never skips over it.
- name: "Install chart and wait for readiness"
# The id is read by the run record at the end of this job: its outcome
# is what separates "the chart did not come up" from "the cluster never
# came up", and the two mean opposite things for Gitea #52.
id: install
env:
FREESAILARR_WG_KEY: ${{ steps.load-freesailarr-vpn.outputs.FREESAILARR_WG_KEY }}
FREESAILARR_WG_ADDRESSES: ${{ steps.load-freesailarr-vpn.outputs.FREESAILARR_WG_ADDRESSES }}
run: ci/run-install-test.sh "${{ matrix.chart }}"
# Must carry the same per-leg name as the create step above: a blanket
# cleanup, or the old fixed name, would tear down the neighbouring leg's
# cluster on a runner that shares one Docker daemon - which is precisely
# the half of the collision that turned one red leg into two.
- name: "Delete k3d cluster"
if: always()
run: k3d cluster delete "${{ matrix.chart }}" || true
# The run record (Gitea #52). A green install-test says "this chart came
# up" and nothing about WHERE, and where is exactly what the question
# needs: a chart that has not been touched since the forge move carries a
# green history from another environment. So every leg writes down the
# four facts that decide what its result means - date, runner label,
# K3S_IMAGE, kubectl - next to its own outcome. A green run nobody notes
# is unknown again next time.
#
# Per leg, and not one table for the whole matrix, because there is no
# place to write one: a matrix leg only ever sees its own
# GITHUB_STEP_SUMMARY, on either forge. The aggregate verdict is written
# by the calling workflow, which does see the matrix as a whole.
#
# It reports steps.install.outcome and not the job status: a leg that died
# in "Create k3d cluster" never installed anything, and recording that as
# a failed chart would blame the chart for the environment - precisely the
# confusion this issue is about.
- name: "Record the result"
if: always()
env:
CHART: ${{ matrix.chart }}
INSTALL_OUTCOME: ${{ steps.install.outcome }}
run: |
case "$INSTALL_OUTCOME" in
success) RESULT="installed, every workload Ready" ;;
failure) RESULT="**FAILED** - the chart did not come up" ;;
*) RESULT="**FAILED before the install** - cluster or tooling, not the chart (install step outcome: '${INSTALL_OUTCOME:-none}')" ;;
esac
{
echo "### Install-test \`$CHART\`: $RESULT"
echo
echo "| Date (UTC) | Runner label | Runner name | K3S_IMAGE | kubectl |"
echo "| --- | --- | --- | --- | --- |"
echo "| $(date -u '+%Y-%m-%d %H:%M') | \`$RUNNER_LABEL\` | \`${RUNNER_NAME:-unknown}\` | \`$K3S_IMAGE\` | \`$KUBECTL_VERSION\` |"
echo
} >> "$GITHUB_STEP_SUMMARY"