feat(template-chart): networkPolicy.ingress.ambientMesh, plus its checked CI exception

Gitea #143. A pod cannot be reached inside Istio's ambient mesh unless its
NetworkPolicy lets ztunnel's HBONE tunnel in - ambient explicitly does not
bypass policy enforcement - and 15008 is deliberately no containerPort of the
application, so assertion 2 of ci/run-networkpolicy-test.sh rejected it.

The switch is a named intention rather than a free port list, and it renders
two things that cannot be had separately: 15008/TCP inbound, and - where
probeSources is non-empty - 169.254.7.127/32 as an additional peer, because
inside the mesh the kubelet's probes are redirected away from the worker
node's address. Whoever opens 15008 by hand has not added the probe address,
and the first narrowed probeSources then drops every probe: the pod never
becomes Ready and the symptom reads as a broken application. That is
assertion 4 of the same script, one value further along.

Assertion 2 keeps its edge. It gets ONE named exception for ONE port number,
gated on one value, and it is not granted in the main loop at all - which
renders every chart with the value at its default, so 15008 stays an ordinary
failure there. The free-port variant was design A of #143 and would have
opened the assertion for all 30 charts.

Assertion 7 holds the exception to five things on two renders per chart:
the port and the ipBlock are absent while the switch is off, 15008 is present
while it is on, 15008 is still the ONLY ingress port that is not a
containerPort, and wherever the rule carrying it restricts sources it also
carries the ipBlock. The last one is reachable because these renders apply
ci/networkpolicy/<chart>.yaml, where probeSources is non-empty.

Built into the existing script, as for #136: the exception weakens assertion 2
and belongs beside it - a separate script could go green while the exception
it is paired with is wrong, with nothing connecting the two. The job already
runs over every chart on every pull request.

Found by running it: the policy's name comes from Chart.yaml's name, not from
the directory - template-chart is called "xxx".

Not proven, and said so in values.yaml, the template header and the script's
header: that 15008 alone suffices comes from Istio's documentation, and the
probe-source half reaches no release, because probeSources is empty in every
chart here. No namespace of this cluster carries istio.io/dataplane-mode, so
there is no path on which either could be measured. What is proven is the
render and the assertions.

ci/run-networkpolicy-test.sh

@@ -83,6 +83,15 @@
 #     because a green tick for a code path nobody entered is worse than an
 #     honest gap (README, "Two rules for every check added here").
 #   - PORT RANGES (endPort). Same treatment: reported, never counted.
+#   - THAT 15008 IS ENOUGH TO REACH A POD IN THE AMBIENT MESH. Assertion 7
+#     asserts the RENDER of networkPolicy.ingress.ambientMesh in both
+#     directions, and nothing beyond it. That the HBONE port alone suffices
+#     comes from Istio's documentation; measured on this cluster is only which
+#     listeners ztunnel opens, and no namespace carries istio.io/dataplane-mode
+#     yet, so there is no path on which it could be measured end to end. The
+#     probe-source half (169.254.7.127/32) is in the same position: its render is
+#     asserted, its effect is not, because probeSources is empty in every chart
+#     of this repository.
 #
 # AND THE ANTI-VACUUM GUARDS, because this test could be green over nothing:
 #
@@ -492,6 +501,20 @@ for policy in policies:
     # port is a port on the DESTINATION and has nothing to do with this pod's
     # containerPorts, so holding egress against them would fail every chart that
     # reaches a database.
+    #
+    # THE ONE NAMED EXCEPTION, AND IT IS NOT GRANTED HERE (Gitea #143). Port
+    # 15008 - ztunnel's HBONE tunnel - is terminated by the mesh and not by the
+    # application, so it is legitimately no containerPort while
+    # networkPolicy.ingress.ambientMesh is true. This loop renders every chart
+    # with that value at its DEFAULT, which is false, so for this assertion
+    # 15008 is an ordinary number and stays an ordinary failure - that is the
+    # negative half of the exception and it costs nothing. The exception itself
+    # lives in assertion 7, on a SECOND render with the value on, and it permits
+    # exactly that one number. It is deliberately not implemented as "skip
+    # operator-supplied ports when a gate is set": that was design A in #143 and
+    # would have opened this assertion for all 30 charts, letting every mistyped
+    # ingress port through - including the NodePort mistake this assertion
+    # exists for.
     for i, rule in enumerate(spec.get("ingress") or []):
         if not isinstance(rule, dict):
             fail("%s ingress[%d] is not a mapping" % (tag, i))
@@ -870,6 +893,11 @@ EGRESS_PEER_ROWS="paperless-ngx|apache-tika,gotenberg|paperless.files.tikaGotenb
 # leaves the parsed JSON in $2. The render/split/parse steps are repeated here
 # instead of refactored out of the loop on purpose: the five assertions above
 # are load-bearing and were not worth disturbing for this.
+# RCJ_WITH_OVERLAY=1 additionally applies ci/networkpolicy/<chart>.yaml, last,
+# exactly as the main loop does. Off by default so the cross-chart rows below
+# keep rendering the consumer the way its release does; assertion 7 switches it
+# on, because the overlay is the ONLY way the source-restricted branch - and
+# with it the probe-source half of the ambient switch - is reachable at all.
 render_chart_json() {
   RCJ_CHART="$1"
   RCJ_OUT="$2"
@@ -883,6 +911,9 @@ render_chart_json() {
     RCJ_VALUES=(-f "$CI_DIR/trivy/values/$RCJ_CHART.yaml")
     RCJ_LABEL="ci/trivy/values/$RCJ_CHART.yaml"
   fi
+  if [ "${RCJ_WITH_OVERLAY:-}" = "1" ] && [ -f "$OVERLAY_DIR/$RCJ_CHART.yaml" ]; then
+    RCJ_VALUES+=(-f "$OVERLAY_DIR/$RCJ_CHART.yaml")
+  fi
   "$ROOT/ci/helm-dependencies.sh" "$ROOT/$RCJ_CHART" >/dev/null || return 1
   RCJ_MANIFEST="$RCJ_OUT.manifest.yaml"
   helm template "$RELEASE" "$ROOT/$RCJ_CHART" \
@@ -1210,6 +1241,385 @@ PY
   fi
 done
 
+# --- Assertion 7: the ambient-mesh switch (Gitea #143) -----------------------
+# networkPolicy.ingress.ambientMesh renders the two things a pod needs before it
+# can be reached inside Istio's ambient mesh, and the whole design question of
+# #143 was how to allow the first of them without blunting assertion 2:
+#
+#   15008/TCP inbound           - ztunnel's HBONE tunnel. Terminated by the
+#                                 mesh, so it is NOT a containerPort of the
+#                                 application. Assertion 2 would reject it, and
+#                                 it is right to reject it by default.
+#   169.254.7.127/32 as a peer  - where probeSources is non-empty. Inside the
+#                                 mesh the kubelet's probes are redirected and
+#                                 no longer arrive from the worker node's
+#                                 address.
+#
+# WHAT THIS ASSERTION IS FOR. The exception assertion 2 grants is NAMED and
+# single-valued: one port number, gated on one value. The danger of any
+# exception is that it stops being checked and becomes the hole it was carved
+# around, so the exception is held to five things at once, on TWO renders of
+# every chart:
+#
+#   switch OFF (the main loop's render, reused)
+#     - 15008 must be ABSENT. Without this half the exception is a hole: a
+#       chart could render the port unconditionally and nothing would notice,
+#       because assertion 7 would find it present and assertion 2 is where it
+#       would have failed.
+#     - 169.254.7.127/32 must be absent too, for the same reason.
+#   switch ON
+#     - 15008/TCP must be PRESENT, or the value is decorative.
+#     - 15008 must be the ONLY ingress port that is not a containerPort. This is
+#       the line between a named exception and design A of #143: every other
+#       mistyped or NodePort number still fails here, with the switch on.
+#     - wherever the rule carrying 15008 restricts sources, that rule's "from"
+#       must contain 169.254.7.127/32. This is the inseparability of the two
+#       halves, asserted rather than trusted - and it is REACHABLE, because
+#       these renders apply ci/networkpolicy/<chart>.yaml, which is where
+#       probeSources is non-empty (template-chart, gotenberg).
+#
+# And the policy is found by the NAME CONVENTION, <release>-<chart>-*, the same
+# way assertion 6 finds it: an exception must not be granted to an object this
+# test did not identify. A chart whose policy stops following the convention
+# fails here instead of silently losing its coverage.
+#
+# BUILT INTO THIS SCRIPT rather than added as a new test with its own pipeline
+# job - the same decision as assertion 6 for #136, and here it is stronger than
+# convenience. The exception WEAKENS assertion 2, so it belongs next to it: a
+# separate script can be green while the exception it is paired with is wrong,
+# and nothing would connect the two. Everything else is already here as well -
+# the job runs over every chart on every pull request and so cannot be skipped
+# by a path filter, helm is already the YAML parser through ci/yaml2json, and
+# the verdict, grouping and summary plumbing is shared.
+#
+# EVERY CHART MUST OFFER THE VALUE, and a chart that does not fails. Same
+# mechanism as assertion 0: the mistake arrives by copy into a NEW chart whose
+# own diff nobody reads for it, which is exactly how garage (#81) and keycloak
+# (#83) arrived with no policy at all after #5 had been filed.
+AMBIENT_VALUE="networkPolicy.ingress.ambientMesh"
+for CHART in "${CHARTS[@]}"; do
+  # A chart whose main-loop render already failed has nothing to compare
+  # against; it is reported there and would only produce a second, derived
+  # error here.
+  case " $FAILED " in *" $CHART "*) continue ;; esac
+  [ -s "$WORK/$CHART.json" ] || continue
+
+  echo "::group::ambient-mesh switch: $CHART"
+  ROW_FAIL=""
+  ON_JSON="$WORK/ambient-$CHART-on.json"
+  if ! RCJ_WITH_OVERLAY=1 render_chart_json "$CHART" "$ON_JSON" \
+      --set "$AMBIENT_VALUE=true"; then
+    sed 's/^/  /' "$WORK/xerr"
+    echo "  FAIL   $CHART does not render with $AMBIENT_VALUE=true"
+    echo "::error title=ambient-mesh $CHART::$CHART does not render with $AMBIENT_VALUE=true. Since Gitea #143 every chart in this repository offers that value, and networkPolicy carries additionalProperties: false in all of them - so an unknown key does not get ignored, it fails the render. Add the key to values.yaml AND to values.schema.json (schema and template belong together per chart), and render the ingress port 15008 plus, inside the source-restricted branch, the ipBlock 169.254.7.127/32. Copy template-chart/templates/xxx.networkpolicy.yaml."
+    ROW_FAIL=1
+  fi
+
+  # The policy's name is built from Chart.yaml's name, which is NOT always the
+  # directory name - template-chart is called "xxx", and the whole point of that
+  # chart is that it is a copy template. Reading the directory name here found
+  # exactly that on the first run, so the name comes from Chart.yaml.
+  CHART_NAME="$(sed -n 's/^name:[[:space:]]*//p' "$ROOT/$CHART/Chart.yaml" \
+    | head -1 | tr -d '[:space:]')"
+  if [ -z "$CHART_NAME" ]; then
+    echo "  FAIL   $CHART has no name in Chart.yaml"
+    echo "::error title=ambient-mesh $CHART::$CHART/Chart.yaml declares no name, so the NetworkPolicy's conventional name could not be derived."
+    ROW_FAIL=1
+  fi
+
+  if [ -z "$ROW_FAIL" ]; then
+    set +e
+    python3 - "$CHART" "$RELEASE" "$AMBIENT_VALUE" "$WORK/$CHART.json" "$ON_JSON" \
+        "$ROOT/$CHART" "$CHART_NAME" \
+        >"$WORK/ambient-$CHART.out" 2>"$WORK/ambient-$CHART.pyerr" <<'PY'
+import json, re, sys
+
+chart, release, gate, off_path, on_path, chart_dir, chart_name = sys.argv[1:8]
+
+PORT = 15008
+CIDR = "169.254.7.127/32"
+
+lines = []
+def ok(msg):   lines.append("OK|%s: %s" % (chart, msg))
+def skip(msg): lines.append("SKIP|%s: %s" % (chart, msg))
+def fail(msg): lines.append("FAIL|%s: %s" % (chart, msg))
+
+
+def docs_of(path):
+    return [d for d in (json.load(open(path)).get("docs") or []) if isinstance(d, dict)]
+
+
+def pod_templates(node):
+    out = []
+    if isinstance(node, dict):
+        for key, value in node.items():
+            if (key == "template" and isinstance(value, dict)
+                    and isinstance(value.get("spec"), dict)
+                    and "containers" in value["spec"]):
+                out.append(value)
+            out.extend(pod_templates(value))
+    elif isinstance(node, list):
+        for value in node:
+            out.extend(pod_templates(value))
+    return out
+
+
+def labels_of(template):
+    meta = template.get("metadata")
+    labels = meta.get("labels") if isinstance(meta, dict) else None
+    return labels if isinstance(labels, dict) else {}
+
+
+def matches(selector, labels):
+    for key, value in selector.items():
+        if key not in labels or str(labels[key]) != str(value):
+            return False
+    return True
+
+
+def container_ports(template):
+    out = []
+    spec = template.get("spec") or {}
+    for key in ("containers", "initContainers"):
+        for container in spec.get(key) or []:
+            if not isinstance(container, dict):
+                continue
+            for port in container.get("ports") or []:
+                if isinstance(port, dict) and port.get("containerPort") is not None:
+                    out.append((int(port["containerPort"]), port.get("name")))
+    return out
+
+
+def own_policies(path):
+    """This chart's OWN NetworkPolicies, by the <release>-<chart>- name
+    convention - so a bundled subchart's policy can neither satisfy nor break an
+    assertion the parent chart is responsible for."""
+    prefix = "%s-%s-" % (release, chart_name)
+    out = []
+    for doc in docs_of(path):
+        if doc.get("kind") != "NetworkPolicy":
+            continue
+        name = ((doc.get("metadata") or {}).get("name")) or ""
+        if name.startswith(prefix):
+            out.append((name, doc))
+    return out
+
+
+def ingress_rules(doc):
+    spec = doc.get("spec") if isinstance(doc.get("spec"), dict) else {}
+    return [r for r in (spec.get("ingress") or []) if isinstance(r, dict)]
+
+
+def rule_ports(rule):
+    """[(number_or_name, protocol)] for the rule's ports."""
+    out = []
+    for port in rule.get("ports") or []:
+        if not isinstance(port, dict):
+            continue
+        value = port.get("port")
+        proto = str(port.get("protocol") or "TCP")
+        if isinstance(value, int) or (isinstance(value, str) and value.isdigit()):
+            out.append((int(value), proto))
+        elif value is not None:
+            out.append((value, proto))
+    return out
+
+
+def cidrs_of(rule):
+    out = []
+    for peer in rule.get("from") or []:
+        if isinstance(peer, dict) and isinstance(peer.get("ipBlock"), dict):
+            out.append(str(peer["ipBlock"].get("cidr")))
+    return out
+
+
+# --- The value is declared, in BOTH places it has to be --------------------
+# The render with the switch on already proves the SCHEMA declares it - an
+# undeclared key cannot pass additionalProperties: false. What that render
+# cannot prove is that values.yaml ships the documented default, and a value
+# only reachable through --set is not a value an operator finds.
+values_yaml = ""
+try:
+    values_yaml = open("%s/values.yaml" % chart_dir, encoding="utf-8").read()
+except IOError:
+    pass
+if re.search(r"^\s*ambientMesh\s*:", values_yaml, re.M):
+    ok("values.yaml ships the ambientMesh key with its default, and the render "
+       "with %s=true succeeded - which is itself the proof that "
+       "values.schema.json declares it, since networkPolicy carries "
+       "additionalProperties: false" % gate)
+else:
+    fail("renders with %s=true - so the schema knows the key - but values.yaml "
+         "contains no ambientMesh key at all. A value that exists only in the "
+         "schema is reachable through --set and findable nowhere: the defaults "
+         "are what an operator reads. Add it with its comment, default false, "
+         "as in template-chart." % gate)
+
+off_policies = own_policies(off_path)
+on_policies = own_policies(on_path)
+
+if not off_policies or not on_policies:
+    fail("renders no NetworkPolicy named %s-%s-* (off: %d, on: %d), so the "
+         "ambient-mesh exception could not be held against this chart's own "
+         "policy. The convention is {{ .Release.Name }}-{{ .Chart.Name }}-<kind>; "
+         "if the name changed, this assertion has to follow - it is not skipped, "
+         "because an exception granted to an object this test cannot identify is "
+         "worth less than no exception at all."
+         % (release, chart_name, len(off_policies), len(on_policies)))
+else:
+    # --- The switch OFF half ------------------------------------------------
+    off_hits, off_cidr_hits = [], []
+    for name, doc in off_policies:
+        for i, rule in enumerate(ingress_rules(doc)):
+            for value, proto in rule_ports(rule):
+                if value == PORT:
+                    off_hits.append("%s ingress[%d] %s/%s" % (name, i, value, proto))
+            for cidr in cidrs_of(rule):
+                if cidr == CIDR:
+                    off_cidr_hits.append("%s ingress[%d]" % (name, i))
+    if off_hits:
+        fail("allows ingress port %d with %s at its default (false): %s. The "
+             "exception assertion 2 grants for %d is gated on that value, and "
+             "this is the half that keeps it from being a hole - a port rendered "
+             "unconditionally would otherwise be found present by this assertion "
+             "and never held against the pod spec by assertion 2. Gate the port "
+             "on the value." % (PORT, gate, ", ".join(off_hits), PORT))
+    elif off_cidr_hits:
+        fail("carries the ipBlock %s with %s at its default (false): %s. The "
+             "probe-source half of the switch is for the mesh only, where the "
+             "kubelet's probes are redirected; outside it the address matches "
+             "nothing and makes the rule read as if it did something."
+             % (CIDR, gate, ", ".join(off_cidr_hits)))
+    else:
+        ok("with %s at its default the policy allows neither port %d nor the "
+           "ipBlock %s - the negative half of assertion 2's named exception, "
+           "without which the exception would be a hole" % (gate, PORT, CIDR))
+
+    # --- The switch ON half -------------------------------------------------
+    templates = []
+    for doc in docs_of(on_path):
+        templates.extend(pod_templates(doc))
+
+    found_port, bad_ports, restricted_without_cidr, carriers = [], [], [], []
+    for name, doc in on_policies:
+        spec = doc.get("spec") if isinstance(doc.get("spec"), dict) else {}
+        selector = spec.get("podSelector")
+        match_labels = (selector or {}).get("matchLabels") or {}
+        selected = [t for t in templates if matches(match_labels, labels_of(t))]
+        declared = set()
+        for t in selected:
+            for number, pname in container_ports(t):
+                declared.add(number)
+                if pname:
+                    declared.add(pname)
+        for i, rule in enumerate(ingress_rules(doc)):
+            for value, proto in rule_ports(rule):
+                if value == PORT:
+                    found_port.append("%s ingress[%d] %d/%s" % (name, i, PORT, proto))
+                    carriers.append((name, i, rule))
+                    if proto != "TCP":
+                        bad_ports.append("%s ingress[%d] renders %d as %s, but "
+                                         "HBONE is TCP" % (name, i, PORT, proto))
+                elif value not in declared and selected:
+                    bad_ports.append(
+                        "%s ingress[%d] allows %r, which is no containerPort of "
+                        "the selected pod spec" % (name, i, value))
+
+    if not found_port:
+        fail("renders no ingress port %d with %s=true, so the value is "
+             "decorative: the chart accepts it, the schema declares it, and the "
+             "policy that comes out is the same one as before. A pod whose "
+             "namespace has joined the ambient mesh is then unreachable, because "
+             "ambient does not bypass NetworkPolicy enforcement." % (PORT, gate))
+    else:
+        ok("with %s=true the policy allows %s - ztunnel's HBONE tunnel, the one "
+           "port assertion 2 grants a named exception for"
+           % (gate, ", ".join(found_port)))
+
+    if bad_ports:
+        fail("with %s=true the policy allows ingress port(s) that are NOT "
+             "containerPorts of the selected pod spec and are NOT the one number "
+             "the exception covers (%d): %s. This is the line between a named "
+             "exception and the free port key that was rejected as design A of "
+             "#143 - with the switch on, every OTHER wrong port still fails, "
+             "including a NodePort number written where a pod port belongs."
+             % (gate, PORT, "; ".join(bad_ports)))
+    elif found_port:
+        ok("with %s=true, port %d is still the ONLY ingress port of this chart "
+           "that is not a containerPort of the selected pod spec - the exception "
+           "covers exactly one number and nothing rode in beside it"
+           % (gate, PORT))
+
+    # The inseparability of the two halves, where the render can reach it.
+    for name, i, rule in carriers:
+        if not rule.get("from"):
+            continue
+        if CIDR in cidrs_of(rule):
+            ok("%s ingress[%d] carries port %d AND the ipBlock %s in the same "
+               "\"from\" list - the two halves of the switch are inseparable, "
+               "which is the reason it is one value and not two"
+               % (name, i, PORT, CIDR))
+        else:
+            restricted_without_cidr.append(
+                "%s ingress[%d] (peers: %s)"
+                % (name, i, ", ".join(cidrs_of(rule)) or "no ipBlock at all"))
+    if restricted_without_cidr:
+        fail("with %s=true the rule carrying port %d restricts its sources but "
+             "does not include the ipBlock %s: %s. Inside the mesh the kubelet's "
+             "probes are redirected and arrive from that address instead of from "
+             "the worker node's, so a restricted rule without it drops every "
+             "probe - the pod never becomes Ready and the symptom reads as a "
+             "broken application rather than as a policy. That is assertion 4 of "
+             "this script, re-created one value further along, and it is the "
+             "whole reason the two halves are a single switch."
+             % (gate, PORT, CIDR, "; ".join(restricted_without_cidr)))
+    if not any(r.get("from") for _n, _i, r in carriers):
+        skip("the rule carrying port %d has no \"from\" list in this render, so "
+             "the probe-source half (%s) is not reachable here and was not "
+             "checked. That is the state of every chart's DEFAULTS - probeSources "
+             "is empty everywhere - and it is why ci/networkpolicy/<chart>.yaml "
+             "exists: template-chart and gotenberg render the restricted branch "
+             "and the half IS checked there. Reported, not counted as passing."
+             % (PORT, CIDR))
+
+for line in lines:
+    print(line)
+PY
+    RC=$?
+    set -e
+    if [ "$RC" -gt 1 ]; then
+      sed 's/^/  /' "$WORK/ambient-$CHART.pyerr"
+      echo "::error title=ambient-mesh $CHART::the ambient-mesh comparison itself failed for $CHART (python exit $RC). Treated as a failure, never as a pass."
+      ROW_FAIL=1
+    else
+      while IFS='|' read -r VERDICT DETAIL; do
+        case "$VERDICT" in
+          OK)
+            TOTAL_ASSERTIONS=$((TOTAL_ASSERTIONS + 1))
+            echo "  ok     $DETAIL"
+            ;;
+          SKIP) echo "  gap    $DETAIL" ;;
+          FAIL)
+            ROW_FAIL=1
+            echo "  FAIL   $DETAIL"
+            echo "::error title=ambient-mesh $CHART::$DETAIL"
+            ;;
+          *) [ -n "$VERDICT" ] && echo "  $VERDICT $DETAIL" ;;
+        esac
+      done <"$WORK/ambient-$CHART.out"
+    fi
+  fi
+
+  echo "::endgroup::"
+  if [ -n "$ROW_FAIL" ]; then
+    FAILED="$FAILED $CHART(ambient-mesh)"
+    echo "ambient-mesh $CHART: FAILED"
+  else
+    AMBIENT_OK="${AMBIENT_OK:-} $CHART"
+    echo "ambient-mesh $CHART: ok"
+  fi
+done
+
 # --- Report ------------------------------------------------------------------
 count() { set -- $1; echo "$#"; }
 listing() { set -- $1; [ "$#" -eq 0 ] && { echo "–"; return; }; printf '`%s`' "$1"; shift; for C in "$@"; do printf ', `%s`' "$C"; done; echo; }
@@ -1225,6 +1635,7 @@ SUMMARY="${GITHUB_STEP_SUMMARY:-/dev/null}"
   echo "| Result | Count | Charts |"
   echo "|---|---|---|"
   echo "| Policies checked | $(count "$CHECKED") | $(listing "$CHECKED") |"
+  echo "| Ambient-mesh switch held (Gitea #143) | $(count "${AMBIENT_OK:-}") | $(listing "${AMBIENT_OK:-}") |"
   echo "| Failed | $(count "$FAILED") | $(listing "$FAILED") |"
   echo
   echo "$TOTAL_ASSERTIONS assertion(s) held in total."