ci: drop excluded charts from the install-test matrix

A chart in ci/excluded-charts.txt got an install-test job that ran the script, printed SKIP and exited 0. GitHub knows only the exit code, so the check list showed "Install-test satisfactory OK" for a chart whose image was never pulled - indistinguishable from one that really came up in the cluster (#231). detect-changed-charts now removes those charts before the matrix is built, so no job exists for them and nothing claims they passed.

An absence says nothing by itself, so the detect job writes every excluded chart of the PR with its documented justification into the job summary. A "Skipped" would only say "not run"; the reason is the question a review actually asks.

The list is not parsed in YAML. The workflow calls ci/run-install-test.sh --list-excluded, the one parser, and compares whole strings with jq, so a chart name is never read as a pattern. The script's own exclusion branch now goes through the same excluded_reason() helper instead of a grep that used the chart name as a regex; it stays for local runs, where the CI filter does not apply.

Not viable, and recorded at the matrix as a comment: deciding per chart in an "if" on the install-test job, which would give a visible Skipped entry instead of an absent one. GitHub rejects it - jobs.<id>.if is evaluated before the matrix is expanded, so the matrix context does not exist there ("context \"matrix\" is not allowed here. available contexts are \"github\", \"inputs\", \"needs\", \"vars\"."). Run 33650724109 on the closed PR #233 ended in failure with zero jobs created.

.github/workflows/validate-charts.yml

@@ -11,6 +11,8 @@ jobs:
       any-chart-changed: ${{ steps.detect.outputs.any-chart-changed }}
       scan-charts: ${{ steps.detect.outputs.scan-charts }}
       any-scan: ${{ steps.detect.outputs.any-scan }}
+      install-charts: ${{ steps.detect.outputs.install-charts }}
+      any-install: ${{ steps.detect.outputs.any-install }}
     steps:
       - name: "Checkout Repository"
         uses: actions/checkout@v7
@@ -54,6 +56,55 @@ jobs:
             echo "any-scan=true" >> "$GITHUB_OUTPUT"
           fi
 
+          # --- The install-test matrix: changed charts minus the excluded ones
+          # (issue #231). A chart in ci/excluded-charts.txt used to get a job
+          # that ran the script, printed SKIP and exited 0 - and GitHub knows
+          # only the exit code, so the check list showed a green tick for a
+          # chart nothing was ever attempted on. Dropping it here means no job
+          # exists for it and nothing claims it passed.
+          #
+          # The list is NOT parsed here. ci/run-install-test.sh --list-excluded
+          # emits "<chart><TAB><reason>", and that script is the one parser;
+          # re-implementing the format in YAML would be a second chance to
+          # select different charts, silently. The comparison below is jq's
+          # "!=" on whole strings, so a chart name is never read as a pattern.
+          INSTALL_CHARTS="$CHARTS"
+          SUMMARY_ROWS=""
+          while IFS=$'\t' read -r EX_NAME EX_REASON; do
+            [ -n "$EX_NAME" ] || continue
+            if [ "$(echo "$INSTALL_CHARTS" | jq -r --arg c "$EX_NAME" 'index($c) != null')" = "true" ]; then
+              # A "|" in the justification would break out of the table cell.
+              SUMMARY_ROWS="${SUMMARY_ROWS}| \`$EX_NAME\` | ${EX_REASON//|/\\|} |"$'\n'
+              echo "Excluded from the install-test: $EX_NAME - $EX_REASON"
+            fi
+            INSTALL_CHARTS="$(echo "$INSTALL_CHARTS" | jq -c --arg c "$EX_NAME" 'map(select(. != $c))')"
+          done <<< "$(ci/run-install-test.sh --list-excluded)"
+          echo "install-charts=$INSTALL_CHARTS" >> "$GITHUB_OUTPUT"
+          if [ "$INSTALL_CHARTS" = "[]" ]; then
+            echo "any-install=false" >> "$GITHUB_OUTPUT"
+          else
+            echo "any-install=true" >> "$GITHUB_OUTPUT"
+          fi
+
+          # An absent job says "not run" but never why, and an absence is easy
+          # to miss entirely. So every excluded chart of this PR is named here
+          # WITH its documented justification - which is the question a review
+          # actually asks. This summary is the visible half of the change; the
+          # dropped matrix entry is only the half that stops the lie.
+          if [ -n "$SUMMARY_ROWS" ]; then
+            {
+              echo "### Charts excluded from the install-test"
+              echo
+              echo "These charts are changed by this PR but listed in \`ci/excluded-charts.txt\`, so they get **no** install-test job. There is no green tick for them, because nothing was attempted (issue #231)."
+              echo
+              echo "| Chart | Documented reason |"
+              echo "| --- | --- |"
+              printf '%s' "$SUMMARY_ROWS"
+              echo
+            } >> "$GITHUB_STEP_SUMMARY"
+          fi
+          echo "Install-test matrix: $INSTALL_CHARTS"
+
   build-chart:
     name: "Build ${{ matrix.chart }}"
     needs: detect-changed-charts
@@ -162,7 +213,7 @@ jobs:
   install-test:
     name: "Install-test ${{ matrix.chart }}"
     needs: [detect-changed-charts, build-chart]
-    if: needs.detect-changed-charts.outputs.any-chart-changed == 'true'
+    if: needs.detect-changed-charts.outputs.any-install == 'true'
     runs-on: ubuntu-latest
     # Blocking since Renovate automerges patch/minor updates: a chart that no
     # longer starts must stop the PR before it reaches main and publishes. This
@@ -171,8 +222,22 @@ jobs:
     timeout-minutes: 30
     strategy:
       fail-fast: false
+      # "install-charts", not "charts": the changed charts minus the ones in
+      # ci/excluded-charts.txt, filtered in detect-changed-charts. Excluded
+      # charts are named with their reason in that job's summary (issue #231).
+      #
+      # DO NOT try to move that decision into an "if" on this job, e.g.
+      #   if: ... && !contains(fromJSON(needs.detect-changed-charts.outputs.excluded-charts), matrix.chart)
+      # to get a visible "Skipped" per chart instead of an absent entry.
+      # GitHub rejects it - jobs.<id>.if is evaluated BEFORE the matrix is
+      # expanded, so the matrix context does not exist there:
+      #   context "matrix" is not allowed here.
+      #   available contexts are "github", "inputs", "needs", "vars".
+      # Measured on run 33650724109 (PR #233, closed): the whole workflow ended
+      # in failure with zero jobs created. The per-chart value has to be gone
+      # from the matrix by the time this job is reached.
       matrix:
-        chart: ${{ fromJSON(needs.detect-changed-charts.outputs.charts) }}
+        chart: ${{ fromJSON(needs.detect-changed-charts.outputs.install-charts) }}
     env:
       K3D_VERSION: v5.9.0
       K3S_IMAGE: docker.io/rancher/k3s:v1.35.5-k3s1
@@ -227,11 +292,11 @@ jobs:
         run: curl -sSfL https://raw.githubusercontent.com/k3d-io/k3d/main/install.sh | TAG="$K3D_VERSION" bash
       - name: "Create k3d cluster"
         run: k3d cluster create install-test --image "$K3S_IMAGE" --wait --timeout 180s
-      # Excluded charts (see ci/excluded-charts.txt) are skipped inside the
+      # Every chart reaching this step is one the exclusion filter let through,
-      # script with their documented reason. The variables below are empty for
+      # so the script's SKIP branch is never taken here - it exists for local
-      # every other chart and are read only by the chart that declares them in
+      # runs. The variables below are empty for every chart but the one that
-      # ci/<chart>/settings.env (REQUIRED_ENV); the script fails loudly if a
+      # declares them in ci/<chart>/settings.env (REQUIRED_ENV); the script
-      # declared one is missing, it never skips over it.
+      # fails loudly if a declared one is missing, it never skips over it.
       - name: "Install chart and wait for readiness"
         env:
           FREESAILARR_WG_KEY: ${{ steps.load-freesailarr-vpn.outputs.FREESAILARR_WG_KEY }}

ci/run-install-test.sh

@@ -69,12 +69,16 @@ if [ -f "$CI_DIR/$CHART/settings.env" ]; then
 fi
 
 # --- Exclusion list ---------------------------------------------------------
-if [ -f "$CI_DIR/excluded-charts.txt" ]; then
+# Through excluded_reason(), i.e. the same parser --list-excluded uses, so a
-  REASON="$(grep -E "^$CHART([[:space:]]|$)" "$CI_DIR/excluded-charts.txt" | sed -E "s/^$CHART[[:space:]]*//" || true)"
+# local run and the workflow matrix can never disagree about which charts are
-  if [ -n "$REASON" ]; then
+# excluded. In CI this branch is no longer reached: the detect job drops the
-    echo "SKIP: chart '$CHART' is excluded from the install-test: $REASON"
+# excluded charts before the matrix is built, so no install-test job exists for
-    exit 0
+# them (#231). It stays for local runs, where "ci/run-install-test.sh
-  fi
+# satisfactory" should say why it does nothing instead of trying.
+REASON="$(excluded_reason "$CHART")"
+if [ -n "$REASON" ]; then
+  echo "SKIP: chart '$CHART' is excluded from the install-test: $REASON"
+  exit 0
 fi
 
 if [ ! -d "$ROOT/$CHART" ] || [ ! -f "$ROOT/$CHART/Chart.yaml" ]; then