# The install-test, as one job with two callers. # # Both callers install a chart the SAME way; they differ only in which charts # they name. validate-charts.yml passes the charts a pull request changed; # install-test-all-charts.yml passes every installable chart on demand, to ask # whether the environment underneath an UNCHANGED chart still holds (Gitea # #52) - a chart untouched since the forge move carries a green history from a # different runner, and that says nothing about today. # # It is a called workflow and not a second job, for the same reason # ci/excluded-charts.txt has exactly one parser: a second copy is a second # chance to test DIFFERENTLY, unnoticed. The parts that would drift are worth # naming, because each was paid for once already - the pinned kubectl with its # skew check, envsubst, the detection of a containerised job, the per-leg # cluster name. A copy that slowly disagrees about any of them does not go red; # it turns a green run into a statement about something other than what was # assumed. # # What stays with the callers: WHICH charts are tested, and whether the job # runs at all (needs/if). Everything that decides HOW one is installed is here. name: Install-test on: workflow_call: inputs: charts: required: true type: string description: 'JSON array of chart names, e.g. ["cyberchef","zipline"]. The entries of ci/excluded-charts.txt must already be gone - see the matrix comment below.' secrets: OP_CONNECT_TOKEN: # Needed by exactly one leg - freesailarr, for the 1Password Connect # lookup of its CI Wireguard key. Declared OPTIONAL on purpose, where # the shared publish block in workflow-util/pipelines declares its token # required: on this instance a missing secret does not fail the job, it # fails job CREATION, so the run shows no steps at all (README, "Where # this repo lives"; the long form is on that repository's wiki). With # required: true a pull request from a fork - which carries no secrets - # would lose the entire install-test that way, every chart at once, with # nothing naming the cause. Optional instead: only the freesailarr leg # fails, loudly, at the ":?" guards that name the missing op:// field. required: false description: "1Password Connect token; only the freesailarr leg reads it" jobs: install-test: name: "Install-test ${{ matrix.chart }}" # The one job in this repo that does not simply say ubuntu-latest, and the # expression is not decoration - the same label means different machines on # the two forges this repo currently lives on (#225). # # This is the only job that needs a Docker socket: it creates a k3d # cluster, and k3d is a Docker client. On GitHub's hosted runners every # label carries Docker, so the question never came up. On the Gitea # instance it was MEASURED on a probe pull request, per job: # detect-changed-charts, build-chart, the quantity render and the Trivy # scan all pass on `ubuntu-latest` - none of them needs Docker - while the # install-test dies there at "Create k3d cluster" with "Cannot connect to # the Docker daemon at unix:///var/run/docker.sock", an error that points # at k3d and never at the runner label. `ubuntu-docker` is the label that # carries a socket, and Gitea does evaluate this expression: the probe job # was picked up by the docker runner with labels ["ubuntu-docker"]. # # Why an expression and not a plain `ubuntu-docker`: that label does not # exist on GitHub, so every chart PR would sit queued forever for as long # as this repo still builds on github.com - blocking the merge with no # error anywhere. github.server_url is the one context value that separates # the two without naming our instance. # # The label is necessary and was, for a while, not sufficient: # `ubuntu-docker`'s daemon reported "Cgroup Driver: none" plus "Running in # rootless-mode without cgroups", and k3d got as far as starting the k3s # node, which restarted immediately with an empty log because k3s needs # cgroup delegation (#225). Nothing in this file addressed that - it was # dealt with on the instance side. It no longer blocks, and the evidence is # Gitea PR #48: 19 of 20 matrix charts installed and became Ready on this # label. The twentieth, rss-bridge, failed and is Gitea #51. runs-on: ${{ github.server_url == 'https://github.com' && 'ubuntu-latest' || 'ubuntu-docker' }} # Blocking since Renovate automerges patch/minor updates: a chart that no # longer starts must stop the PR before it reaches main and publishes. This # ends the stabilization phase of issue #22 - the job reported failures # without blocking while it proved itself across several chart PRs. # # The 30 minutes are per leg and hold for the all-charts caller too, with # the measured figure to compare against: Gitea PR #48 ran 20 charts in # 12m41s in one job, all legs concurrently on one Docker daemon. If a run # ever comes near this timeout, it is raised HERE, once, for both callers - # a value that differs per caller would mean the two tests give up at # different points while claiming to be the same test. timeout-minutes: 30 strategy: fail-fast: false # The charts arrive from the caller ALREADY filtered against # ci/excluded-charts.txt: validate-charts.yml does it in # detect-changed-charts, install-test-all-charts.yml in select-charts, and # both go through ci/run-install-test.sh --list-excluded, the one parser # of that file. Both also name every chart they dropped WITH its # documented reason in their own job summary, because an absent matrix # entry says "not run" and never why (issue #231). # # DO NOT move that decision in here, e.g. as an "if" on this job # if: ... && !contains(fromJSON(inputs.excluded-charts), matrix.chart) # to get a visible "Skipped" per chart instead of an absent entry. # GitHub rejects it - jobs..if is evaluated BEFORE the matrix is # expanded, so the matrix context does not exist there: # context "matrix" is not allowed here. # available contexts are "github", "inputs", "needs", "vars". # Measured on run 33650724109 (PR #233, closed): the whole workflow ended # in failure with zero jobs created. The per-chart value has to be gone # from the matrix before this workflow is called. matrix: chart: ${{ fromJSON(inputs.charts) }} env: K3D_VERSION: v5.9.0 K3S_IMAGE: docker.io/rancher/k3s:v1.35.5-k3s1 # Pinned rather than taken from the runner image. The argument is not # "Gitea's image has no kubectl" - that is only how it was noticed. A # kubectl inherited from whatever the image happens to ship drifts # unannounced with every image rebuild, and an install-test is supposed # to run against a KNOWN client. That GitHub's hosted image already # carries one does not make the pin unnecessary; it only makes the drift # invisible there. The value mirrors K3S_IMAGE above (v1.35.5), which is # the cluster this client talks to. # renovate: datasource=github-releases depName=kubernetes/kubernetes KUBECTL_VERSION: v1.37.1 # The same expression as runs-on above, written a second time because the # env context is not available in runs-on - so it cannot be the other way # round, and reading the label back off the runner is not possible either # (RUNNER_NAME below names the machine, not the label that picked it). # It exists for the run record at the end of this job: "on which runner # was this green tick measured" is the whole question of Gitea #52. Kept # right under runs-on, so a change to one that forgets the other is # visible on a single screen. RUNNER_LABEL: ${{ github.server_url == 'https://github.com' && 'ubuntu-latest' || 'ubuntu-docker' }} steps: - name: "Checkout Repository" uses: actions/checkout@v7 - name: "Install Helm" uses: azure/setup-helm@v5 - name: "Install kubectl" run: | # kubectl and the cluster are a pair: the client talks to the server # K3S_IMAGE starts, and Kubernetes supports at most one minor of # skew between them. Renovate updates KUBECTL_VERSION on its own # while K3S_IMAGE is tracked by nothing, so without this check the # two drift apart until the client is talking to a server it no # longer supports - silently, because kubectl only warns. And it is # not theoretical: measured by dry run, a lookup from v1.34.0 # offers v1.37.0 in ONE step, because Renovate fills the non-major # bucket with the newest candidate rather than walking the minors. # So this check WILL fire one day, on purpose, with a message that # says which of the two pins to move. # # Deliberately not solved with an allowedVersions constraint pinning # kubectl to the k3s minor: that is the nginx trap the README # describes - a filter that silently stops offering updates the day # the other side moves. A loud failure beats a quiet freeze. read -r KMAJ KMIN <<<"$(printf '%s' "$KUBECTL_VERSION" | sed -E 's/^v([0-9]+)\.([0-9]+).*/\1 \2/')" read -r SMAJ SMIN <<<"$(printf '%s' "$K3S_IMAGE" | sed -E 's/.*:v([0-9]+)\.([0-9]+).*/\1 \2/')" SKEW=$(( (KMAJ * 100 + KMIN) - (SMAJ * 100 + SMIN) )) if [ "${SKEW#-}" -gt 1 ]; then echo "::error::kubectl $KUBECTL_VERSION and k3s $K3S_IMAGE are more than one minor apart. Bump K3S_IMAGE to match, or pin KUBECTL_VERSION back." exit 1 fi curl -sSfL -o /tmp/kubectl "https://dl.k8s.io/release/${KUBECTL_VERSION}/bin/linux/amd64/kubectl" mkdir -p "$HOME/.local/bin" install -m 0755 /tmp/kubectl "$HOME/.local/bin/kubectl" echo "$HOME/.local/bin" >> "$GITHUB_PATH" "$HOME/.local/bin/kubectl" version --client - name: "Provide envsubst" run: | # ci/run-install-test.sh renders ci//fixtures.env.yaml with # envsubst - today only freesailarr has one, and without the tool # the fixtures collapse to "error: no objects passed to apply", # gluetun never gets its Wireguard key, and nothing behind its kill # switch becomes Ready. GitHub's hosted image ships gettext-base, # the Gitea runner image does not. The script is fine as it is; a # standard tool is simply missing. # # CHECKED FOR, NOT PINNED - and that is a different decision from # kubectl above, on purpose. kubectl's VERSION changes what the test # means: its distance to the API server is a real source of failure, # which is why it is pinned, tracked and guarded. envsubst # substitutes text; which release performs the substitution does not # change the result. Pinning it would buy nothing and add a line for # Renovate to carry. So: present - do nothing, which keeps the # GitHub path free of any package install; missing - fetch it, and # the difference between the two runners is stated rather than # hidden behind an unconditional apt-get. if command -v envsubst >/dev/null 2>&1; then echo "envsubst present at $(command -v envsubst) - nothing to do" exit 0 fi echo "envsubst missing on this runner image - installing gettext-base" if [ "$(id -u)" -eq 0 ]; then SUDO=""; else SUDO="sudo"; fi $SUDO apt-get update -qq $SUDO apt-get install -y -qq --no-install-recommends gettext-base command -v envsubst # freesailarr is the one chart whose install-test needs credentials: # without a real Wireguard key gluetun never completes a handshake, and # behind its kill switch not a single app container becomes Ready. The key # comes from 1Password Connect like the registry credentials in the shared # publish block, and it is a CI-ONLY Mullvad key: Mullvad ties an # address to a key and sends the tunnel to the endpoint it saw last, so # reusing the production key would let every PR run pull the tunnel out # from under the running stack - with the symptom in the cluster and the # cause in a pull request. Loaded only for this chart; no other chart run # touches the VPN key. - name: "Configure OP-Connect" if: matrix.chart == 'freesailarr' uses: 1password/load-secrets-action/configure@v5 with: connect-host: https://op-connect.jk-dev.de - name: "Load freesailarr VPN credentials" id: load-freesailarr-vpn if: matrix.chart == 'freesailarr' uses: 1password/load-secrets-action@v5 env: OP_CONNECT_TOKEN: ${{ secrets.OP_CONNECT_TOKEN }} FREESAILARR_WG_KEY: op://GitHub-Actions/freesailarr-mullvad-test-config/credential FREESAILARR_WG_ADDRESSES: op://GitHub-Actions/freesailarr-mullvad-test-config/ip-address # The same trap the shared publish block documents at length - since # Gitea #66 that long form lives on the wiki of workflow-util/pipelines, # page "publish-helm-chart", not in this repository: v5 of # load-secrets-action does NOT export the loaded values as environment # variables for later steps, only its step outputs carry them, and the # loading step logs "Populating variable" and stays green either way. So # the values are mapped into each step's own env block via # steps..outputs.* and read as quoted shell variables. The ":?" guards # run before the cluster is even created and name the op:// field that is # missing - an empty key would otherwise reach gluetun and look like a # broken chart instead of a missing secret. They print the variable name, # never its value. - name: "Verify the VPN credentials arrived" if: matrix.chart == 'freesailarr' env: FREESAILARR_WG_KEY: ${{ steps.load-freesailarr-vpn.outputs.FREESAILARR_WG_KEY }} FREESAILARR_WG_ADDRESSES: ${{ steps.load-freesailarr-vpn.outputs.FREESAILARR_WG_ADDRESSES }} run: |- : "${FREESAILARR_WG_KEY:?empty - check op://GitHub-Actions/freesailarr-mullvad-test-config/credential}" : "${FREESAILARR_WG_ADDRESSES:?empty - check op://GitHub-Actions/freesailarr-mullvad-test-config/ip-address}" echo "Both VPN values arrived non-empty." - name: "Install k3d" run: curl -sSfL https://raw.githubusercontent.com/k3d-io/k3d/main/install.sh | TAG="$K3D_VERSION" bash # One cluster per matrix leg, named after the chart. It used to be the # fixed name "install-test", and that was wrong on GitHub too - it just # could not show there, because GitHub gives every matrix leg its own VM # with its own Docker daemon, so two legs never saw each other's # cluster. The environment hid the bug for two years rather than # preventing it. Gitea runs all legs of a matrix CONCURRENTLY in ONE job # on ONE daemon, and there the fixed name collides at once: measured on # the probe PR of #225, one leg logs "Re-using existing network # 'k3d-install-test'", the other dies on "The container name # /k3d-install-test-server-0 is already in use" - and then the first # leg's rollback deletes the shared network out from under the second, # which fails with "network k3d-install-test not found". Both legs red, # neither cause in either message. DO NOT put the fixed name back. # # Why the bare chart name and not a prefixed one: k3d rejects a cluster # name over 32 characters ("Cluster name must be <= 32 characters", # measured against k3d v5.9.0 - 32 passes, 33 fails), and every # container is named k3d--server-0 from it. The longest chart # here is calibre-web-automated at 21, so "install-test-" would # be 34 and break exactly that one chart and no other - the kind of # prefix that is worse than the fixed name because it only sometimes # breaks. The guard below states the limit instead of leaving it to be # rediscovered. - name: "Create k3d cluster" run: | CLUSTER="${{ matrix.chart }}" if [ "${#CLUSTER}" -gt 32 ]; then echo "::error::chart name '$CLUSTER' is ${#CLUSTER} characters and k3d allows at most 32 for a cluster name." exit 1 fi # Does THIS job run inside a container? That, and not which forge we # are on, is what decides whether k3d's cluster is reachable (#227). # # k3d publishes the API server port on the Docker *host*. When the # job runs straight on the host - GitHub's hosted runners - that is # the same machine, and the "https://0.0.0.0:" k3d writes into # the kubeconfig works. When the job itself is a container, as every # Gitea Actions job is, the host is the dind container and 0.0.0.0 # points at the job container itself, where nothing listens. The # cluster comes up fine and the first kubectl call is refused. # # Deliberately NOT keyed on github.server_url, although that would # pick the right branch today: a self-hosted GitHub runner in a # container has exactly this problem, and a condition that says # "GitHub" while meaning "not containerised" is the kind of lie this # repo exists to remove. So the state is detected instead of # inferred, and the detection is the same question the fix needs # answered anyway: ask the daemon about ourselves. Inside a # container that the daemon knows, this returns our network and k3d # joins it, which puts k3s and this job on one bridge where Docker's # own DNS resolves the container names. On a host it returns nothing, # because the hostname is not a container, and the old path is taken # unchanged. # The "|| true" is load-bearing: this step runs under `bash -e -o # pipefail`, and on a host `docker inspect` exits non-zero, which # would abort the step instead of taking the else branch. JOB_NET="$(docker inspect -f '{{range $k, $v := .NetworkSettings.Networks}}{{$k}} {{end}}' "$(hostname)" 2>/dev/null | awk '{print $1}' || true)" if [ -n "$JOB_NET" ]; then echo "Containerised job: $(hostname) on docker network '$JOB_NET' - k3d joins it." k3d cluster create "$CLUSTER" --image "$K3S_IMAGE" --network "$JOB_NET" --wait --timeout 180s # Same reason: the published port is useless from in here, so the # kubeconfig is pointed at the load balancer container by name, # over the shared network. If the API server certificate did not # carry that name this would fail on the TLS handshake and the # cluster-info below would say so - it holds without an extra # --tls-san, measured on the probe PR of #225. kubectl config set-cluster "k3d-$CLUSTER" --server="https://k3d-$CLUSTER-serverlb:6443" else echo "Job runs on the docker host - k3d's published port is reachable as written." k3d cluster create "$CLUSTER" --image "$K3S_IMAGE" --wait --timeout 180s fi # Reachability is proven HERE, and the shape of this check is the # product of getting it wrong twice. Both mistakes are written down # because both look like something to simplify away. # # It is a LOOP and not a single call. k3d's --wait returns when the # node container reports ready, but the API server can still answer # discovery with 503 for a moment after that. Measured: a single # `kubectl cluster-info` right here failed with "ServiceUnavailable: # the server is currently unable to handle the request" on one of # two legs of the SAME GitHub run while the other passed. A check # that is itself racy produces exactly the sporadic red this job # must never have (#227) - and a test people have learned to re-run # is one they have stopped believing. # # It counts READY NODES and not "did kubectl answer". `kubectl get # nodes` exits 0 on an EMPTY list, so the first version passed # against a cluster whose API server was up and whose node had never # registered: the log said "No resources found", the step went # green, and the failure surfaced five minutes later as a pod stuck # in Pending with no node and no scheduling event - nowhere near its # cause. A check that a broken cluster passes is worse than no # check. `kubectl wait --for=condition=Ready nodes --all` is not the # shorter way to write this: it has the identical hole, because with # zero nodes the condition is vacuously satisfied. # # The evidence that this version earns its keep: only after it # counted nodes did the real cause become visible in the dump below # - k3s could not start containerd ("error creating fsnotify # watcher: too many open files", an inotify limit on the host), and # nothing above this line had ever said so. for ATTEMPT in $(seq 1 90); do READY_NODES="$(kubectl get nodes --no-headers 2>/dev/null | awk '$2 == "Ready"' | wc -l | tr -d ' ')" [ "$READY_NODES" -ge 1 ] && break if [ "$ATTEMPT" -eq 90 ]; then echo "::error::the k3d cluster '$CLUSTER' answers but has no Ready node after 180s" kubectl get nodes -o wide || true kubectl describe nodes || true echo "--- k3s server log ---" docker logs "k3d-$CLUSTER-server-0" 2>&1 | tail -80 || true exit 1 fi sleep 2 done kubectl get nodes -o wide # Every chart reaching this step is one the exclusion filter let through, # so the script's SKIP branch is never taken here - it exists for local # runs. The variables below are empty for every chart but the one that # declares them in ci//settings.env (REQUIRED_ENV); the script # fails loudly if a declared one is missing, it never skips over it. - name: "Install chart and wait for readiness" # The id is read by the run record at the end of this job: its outcome # is what separates "the chart did not come up" from "the cluster never # came up", and the two mean opposite things for Gitea #52. id: install env: FREESAILARR_WG_KEY: ${{ steps.load-freesailarr-vpn.outputs.FREESAILARR_WG_KEY }} FREESAILARR_WG_ADDRESSES: ${{ steps.load-freesailarr-vpn.outputs.FREESAILARR_WG_ADDRESSES }} run: ci/run-install-test.sh "${{ matrix.chart }}" # Must carry the same per-leg name as the create step above: a blanket # cleanup, or the old fixed name, would tear down the neighbouring leg's # cluster on a runner that shares one Docker daemon - which is precisely # the half of the collision that turned one red leg into two. - name: "Delete k3d cluster" if: always() run: k3d cluster delete "${{ matrix.chart }}" || true # The run record (Gitea #52). A green install-test says "this chart came # up" and nothing about WHERE, and where is exactly what the question # needs: a chart that has not been touched since the forge move carries a # green history from another environment. So every leg writes down the # four facts that decide what its result means - date, runner label, # K3S_IMAGE, kubectl - next to its own outcome. A green run nobody notes # is unknown again next time. # # Per leg, and not one table for the whole matrix, because there is no # place to write one: a matrix leg only ever sees its own # GITHUB_STEP_SUMMARY, on either forge. The aggregate verdict is written # by the calling workflow, which does see the matrix as a whole. # # It reports steps.install.outcome and not the job status: a leg that died # in "Create k3d cluster" never installed anything, and recording that as # a failed chart would blame the chart for the environment - precisely the # confusion this issue is about. - name: "Record the result" if: always() env: CHART: ${{ matrix.chart }} INSTALL_OUTCOME: ${{ steps.install.outcome }} run: | case "$INSTALL_OUTCOME" in success) RESULT="installed, every workload Ready" ;; failure) RESULT="**FAILED** - the chart did not come up" ;; *) RESULT="**FAILED before the install** - cluster or tooling, not the chart (install step outcome: '${INSTALL_OUTCOME:-none}')" ;; esac { echo "### Install-test \`$CHART\`: $RESULT" echo echo "| Date (UTC) | Runner label | Runner name | K3S_IMAGE | kubectl |" echo "| --- | --- | --- | --- | --- |" echo "| $(date -u '+%Y-%m-%d %H:%M') | \`$RUNNER_LABEL\` | \`${RUNNER_NAME:-unknown}\` | \`$K3S_IMAGE\` | \`$KUBECTL_VERSION\` |" echo } >> "$GITHUB_STEP_SUMMARY"