From badfaa32cb52dca199ee0bf48553a7e2f3f5a96d Mon Sep 17 00:00:00 2001 From: Justin Klein Date: Mon, 19 Oct 2026 13:01:03 +0000 Subject: [PATCH] feat(template-chart): networkPolicy.ingress.ambientMesh, plus its checked CI exception Gitea #143. A pod cannot be reached inside Istio's ambient mesh unless its NetworkPolicy lets ztunnel's HBONE tunnel in - ambient explicitly does not bypass policy enforcement - and 15008 is deliberately no containerPort of the application, so assertion 2 of ci/run-networkpolicy-test.sh rejected it. The switch is a named intention rather than a free port list, and it renders two things that cannot be had separately: 15008/TCP inbound, and - where probeSources is non-empty - 169.254.7.127/32 as an additional peer, because inside the mesh the kubelet's probes are redirected away from the worker node's address. Whoever opens 15008 by hand has not added the probe address, and the first narrowed probeSources then drops every probe: the pod never becomes Ready and the symptom reads as a broken application. That is assertion 4 of the same script, one value further along. Assertion 2 keeps its edge. It gets ONE named exception for ONE port number, gated on one value, and it is not granted in the main loop at all - which renders every chart with the value at its default, so 15008 stays an ordinary failure there. The free-port variant was design A of #143 and would have opened the assertion for all 30 charts. Assertion 7 holds the exception to five things on two renders per chart: the port and the ipBlock are absent while the switch is off, 15008 is present while it is on, 15008 is still the ONLY ingress port that is not a containerPort, and wherever the rule carrying it restricts sources it also carries the ipBlock. The last one is reachable because these renders apply ci/networkpolicy/.yaml, where probeSources is non-empty. Built into the existing script, as for #136: the exception weakens assertion 2 and belongs beside it - a separate script could go green while the exception it is paired with is wrong, with nothing connecting the two. The job already runs over every chart on every pull request. Found by running it: the policy's name comes from Chart.yaml's name, not from the directory - template-chart is called "xxx". Not proven, and said so in values.yaml, the template header and the script's header: that 15008 alone suffices comes from Istio's documentation, and the probe-source half reaches no release, because probeSources is empty in every chart here. No namespace of this cluster carries istio.io/dataplane-mode, so there is no path on which either could be measured. What is proven is the render and the assertions. --- ci/run-networkpolicy-test.sh | 411 ++++++++++++++++++ .../templates/xxx.networkpolicy.yaml | 70 +++ template-chart/values.schema.json | 11 + template-chart/values.yaml | 56 +++ 4 files changed, 548 insertions(+) diff --git a/ci/run-networkpolicy-test.sh b/ci/run-networkpolicy-test.sh index 4acfab7..19398fb 100755 --- a/ci/run-networkpolicy-test.sh +++ b/ci/run-networkpolicy-test.sh @@ -83,6 +83,15 @@ # because a green tick for a code path nobody entered is worse than an # honest gap (README, "Two rules for every check added here"). # - PORT RANGES (endPort). Same treatment: reported, never counted. +# - THAT 15008 IS ENOUGH TO REACH A POD IN THE AMBIENT MESH. Assertion 7 +# asserts the RENDER of networkPolicy.ingress.ambientMesh in both +# directions, and nothing beyond it. That the HBONE port alone suffices +# comes from Istio's documentation; measured on this cluster is only which +# listeners ztunnel opens, and no namespace carries istio.io/dataplane-mode +# yet, so there is no path on which it could be measured end to end. The +# probe-source half (169.254.7.127/32) is in the same position: its render is +# asserted, its effect is not, because probeSources is empty in every chart +# of this repository. # # AND THE ANTI-VACUUM GUARDS, because this test could be green over nothing: # @@ -492,6 +501,20 @@ for policy in policies: # port is a port on the DESTINATION and has nothing to do with this pod's # containerPorts, so holding egress against them would fail every chart that # reaches a database. + # + # THE ONE NAMED EXCEPTION, AND IT IS NOT GRANTED HERE (Gitea #143). Port + # 15008 - ztunnel's HBONE tunnel - is terminated by the mesh and not by the + # application, so it is legitimately no containerPort while + # networkPolicy.ingress.ambientMesh is true. This loop renders every chart + # with that value at its DEFAULT, which is false, so for this assertion + # 15008 is an ordinary number and stays an ordinary failure - that is the + # negative half of the exception and it costs nothing. The exception itself + # lives in assertion 7, on a SECOND render with the value on, and it permits + # exactly that one number. It is deliberately not implemented as "skip + # operator-supplied ports when a gate is set": that was design A in #143 and + # would have opened this assertion for all 30 charts, letting every mistyped + # ingress port through - including the NodePort mistake this assertion + # exists for. for i, rule in enumerate(spec.get("ingress") or []): if not isinstance(rule, dict): fail("%s ingress[%d] is not a mapping" % (tag, i)) @@ -870,6 +893,11 @@ EGRESS_PEER_ROWS="paperless-ngx|apache-tika,gotenberg|paperless.files.tikaGotenb # leaves the parsed JSON in $2. The render/split/parse steps are repeated here # instead of refactored out of the loop on purpose: the five assertions above # are load-bearing and were not worth disturbing for this. +# RCJ_WITH_OVERLAY=1 additionally applies ci/networkpolicy/.yaml, last, +# exactly as the main loop does. Off by default so the cross-chart rows below +# keep rendering the consumer the way its release does; assertion 7 switches it +# on, because the overlay is the ONLY way the source-restricted branch - and +# with it the probe-source half of the ambient switch - is reachable at all. render_chart_json() { RCJ_CHART="$1" RCJ_OUT="$2" @@ -883,6 +911,9 @@ render_chart_json() { RCJ_VALUES=(-f "$CI_DIR/trivy/values/$RCJ_CHART.yaml") RCJ_LABEL="ci/trivy/values/$RCJ_CHART.yaml" fi + if [ "${RCJ_WITH_OVERLAY:-}" = "1" ] && [ -f "$OVERLAY_DIR/$RCJ_CHART.yaml" ]; then + RCJ_VALUES+=(-f "$OVERLAY_DIR/$RCJ_CHART.yaml") + fi "$ROOT/ci/helm-dependencies.sh" "$ROOT/$RCJ_CHART" >/dev/null || return 1 RCJ_MANIFEST="$RCJ_OUT.manifest.yaml" helm template "$RELEASE" "$ROOT/$RCJ_CHART" \ @@ -1210,6 +1241,385 @@ PY fi done +# --- Assertion 7: the ambient-mesh switch (Gitea #143) ----------------------- +# networkPolicy.ingress.ambientMesh renders the two things a pod needs before it +# can be reached inside Istio's ambient mesh, and the whole design question of +# #143 was how to allow the first of them without blunting assertion 2: +# +# 15008/TCP inbound - ztunnel's HBONE tunnel. Terminated by the +# mesh, so it is NOT a containerPort of the +# application. Assertion 2 would reject it, and +# it is right to reject it by default. +# 169.254.7.127/32 as a peer - where probeSources is non-empty. Inside the +# mesh the kubelet's probes are redirected and +# no longer arrive from the worker node's +# address. +# +# WHAT THIS ASSERTION IS FOR. The exception assertion 2 grants is NAMED and +# single-valued: one port number, gated on one value. The danger of any +# exception is that it stops being checked and becomes the hole it was carved +# around, so the exception is held to five things at once, on TWO renders of +# every chart: +# +# switch OFF (the main loop's render, reused) +# - 15008 must be ABSENT. Without this half the exception is a hole: a +# chart could render the port unconditionally and nothing would notice, +# because assertion 7 would find it present and assertion 2 is where it +# would have failed. +# - 169.254.7.127/32 must be absent too, for the same reason. +# switch ON +# - 15008/TCP must be PRESENT, or the value is decorative. +# - 15008 must be the ONLY ingress port that is not a containerPort. This is +# the line between a named exception and design A of #143: every other +# mistyped or NodePort number still fails here, with the switch on. +# - wherever the rule carrying 15008 restricts sources, that rule's "from" +# must contain 169.254.7.127/32. This is the inseparability of the two +# halves, asserted rather than trusted - and it is REACHABLE, because +# these renders apply ci/networkpolicy/.yaml, which is where +# probeSources is non-empty (template-chart, gotenberg). +# +# And the policy is found by the NAME CONVENTION, --*, the same +# way assertion 6 finds it: an exception must not be granted to an object this +# test did not identify. A chart whose policy stops following the convention +# fails here instead of silently losing its coverage. +# +# BUILT INTO THIS SCRIPT rather than added as a new test with its own pipeline +# job - the same decision as assertion 6 for #136, and here it is stronger than +# convenience. The exception WEAKENS assertion 2, so it belongs next to it: a +# separate script can be green while the exception it is paired with is wrong, +# and nothing would connect the two. Everything else is already here as well - +# the job runs over every chart on every pull request and so cannot be skipped +# by a path filter, helm is already the YAML parser through ci/yaml2json, and +# the verdict, grouping and summary plumbing is shared. +# +# EVERY CHART MUST OFFER THE VALUE, and a chart that does not fails. Same +# mechanism as assertion 0: the mistake arrives by copy into a NEW chart whose +# own diff nobody reads for it, which is exactly how garage (#81) and keycloak +# (#83) arrived with no policy at all after #5 had been filed. +AMBIENT_VALUE="networkPolicy.ingress.ambientMesh" +for CHART in "${CHARTS[@]}"; do + # A chart whose main-loop render already failed has nothing to compare + # against; it is reported there and would only produce a second, derived + # error here. + case " $FAILED " in *" $CHART "*) continue ;; esac + [ -s "$WORK/$CHART.json" ] || continue + + echo "::group::ambient-mesh switch: $CHART" + ROW_FAIL="" + ON_JSON="$WORK/ambient-$CHART-on.json" + if ! RCJ_WITH_OVERLAY=1 render_chart_json "$CHART" "$ON_JSON" \ + --set "$AMBIENT_VALUE=true"; then + sed 's/^/ /' "$WORK/xerr" + echo " FAIL $CHART does not render with $AMBIENT_VALUE=true" + echo "::error title=ambient-mesh $CHART::$CHART does not render with $AMBIENT_VALUE=true. Since Gitea #143 every chart in this repository offers that value, and networkPolicy carries additionalProperties: false in all of them - so an unknown key does not get ignored, it fails the render. Add the key to values.yaml AND to values.schema.json (schema and template belong together per chart), and render the ingress port 15008 plus, inside the source-restricted branch, the ipBlock 169.254.7.127/32. Copy template-chart/templates/xxx.networkpolicy.yaml." + ROW_FAIL=1 + fi + + # The policy's name is built from Chart.yaml's name, which is NOT always the + # directory name - template-chart is called "xxx", and the whole point of that + # chart is that it is a copy template. Reading the directory name here found + # exactly that on the first run, so the name comes from Chart.yaml. + CHART_NAME="$(sed -n 's/^name:[[:space:]]*//p' "$ROOT/$CHART/Chart.yaml" \ + | head -1 | tr -d '[:space:]')" + if [ -z "$CHART_NAME" ]; then + echo " FAIL $CHART has no name in Chart.yaml" + echo "::error title=ambient-mesh $CHART::$CHART/Chart.yaml declares no name, so the NetworkPolicy's conventional name could not be derived." + ROW_FAIL=1 + fi + + if [ -z "$ROW_FAIL" ]; then + set +e + python3 - "$CHART" "$RELEASE" "$AMBIENT_VALUE" "$WORK/$CHART.json" "$ON_JSON" \ + "$ROOT/$CHART" "$CHART_NAME" \ + >"$WORK/ambient-$CHART.out" 2>"$WORK/ambient-$CHART.pyerr" <<'PY' +import json, re, sys + +chart, release, gate, off_path, on_path, chart_dir, chart_name = sys.argv[1:8] + +PORT = 15008 +CIDR = "169.254.7.127/32" + +lines = [] +def ok(msg): lines.append("OK|%s: %s" % (chart, msg)) +def skip(msg): lines.append("SKIP|%s: %s" % (chart, msg)) +def fail(msg): lines.append("FAIL|%s: %s" % (chart, msg)) + + +def docs_of(path): + return [d for d in (json.load(open(path)).get("docs") or []) if isinstance(d, dict)] + + +def pod_templates(node): + out = [] + if isinstance(node, dict): + for key, value in node.items(): + if (key == "template" and isinstance(value, dict) + and isinstance(value.get("spec"), dict) + and "containers" in value["spec"]): + out.append(value) + out.extend(pod_templates(value)) + elif isinstance(node, list): + for value in node: + out.extend(pod_templates(value)) + return out + + +def labels_of(template): + meta = template.get("metadata") + labels = meta.get("labels") if isinstance(meta, dict) else None + return labels if isinstance(labels, dict) else {} + + +def matches(selector, labels): + for key, value in selector.items(): + if key not in labels or str(labels[key]) != str(value): + return False + return True + + +def container_ports(template): + out = [] + spec = template.get("spec") or {} + for key in ("containers", "initContainers"): + for container in spec.get(key) or []: + if not isinstance(container, dict): + continue + for port in container.get("ports") or []: + if isinstance(port, dict) and port.get("containerPort") is not None: + out.append((int(port["containerPort"]), port.get("name"))) + return out + + +def own_policies(path): + """This chart's OWN NetworkPolicies, by the -- name + convention - so a bundled subchart's policy can neither satisfy nor break an + assertion the parent chart is responsible for.""" + prefix = "%s-%s-" % (release, chart_name) + out = [] + for doc in docs_of(path): + if doc.get("kind") != "NetworkPolicy": + continue + name = ((doc.get("metadata") or {}).get("name")) or "" + if name.startswith(prefix): + out.append((name, doc)) + return out + + +def ingress_rules(doc): + spec = doc.get("spec") if isinstance(doc.get("spec"), dict) else {} + return [r for r in (spec.get("ingress") or []) if isinstance(r, dict)] + + +def rule_ports(rule): + """[(number_or_name, protocol)] for the rule's ports.""" + out = [] + for port in rule.get("ports") or []: + if not isinstance(port, dict): + continue + value = port.get("port") + proto = str(port.get("protocol") or "TCP") + if isinstance(value, int) or (isinstance(value, str) and value.isdigit()): + out.append((int(value), proto)) + elif value is not None: + out.append((value, proto)) + return out + + +def cidrs_of(rule): + out = [] + for peer in rule.get("from") or []: + if isinstance(peer, dict) and isinstance(peer.get("ipBlock"), dict): + out.append(str(peer["ipBlock"].get("cidr"))) + return out + + +# --- The value is declared, in BOTH places it has to be -------------------- +# The render with the switch on already proves the SCHEMA declares it - an +# undeclared key cannot pass additionalProperties: false. What that render +# cannot prove is that values.yaml ships the documented default, and a value +# only reachable through --set is not a value an operator finds. +values_yaml = "" +try: + values_yaml = open("%s/values.yaml" % chart_dir, encoding="utf-8").read() +except IOError: + pass +if re.search(r"^\s*ambientMesh\s*:", values_yaml, re.M): + ok("values.yaml ships the ambientMesh key with its default, and the render " + "with %s=true succeeded - which is itself the proof that " + "values.schema.json declares it, since networkPolicy carries " + "additionalProperties: false" % gate) +else: + fail("renders with %s=true - so the schema knows the key - but values.yaml " + "contains no ambientMesh key at all. A value that exists only in the " + "schema is reachable through --set and findable nowhere: the defaults " + "are what an operator reads. Add it with its comment, default false, " + "as in template-chart." % gate) + +off_policies = own_policies(off_path) +on_policies = own_policies(on_path) + +if not off_policies or not on_policies: + fail("renders no NetworkPolicy named %s-%s-* (off: %d, on: %d), so the " + "ambient-mesh exception could not be held against this chart's own " + "policy. The convention is {{ .Release.Name }}-{{ .Chart.Name }}-; " + "if the name changed, this assertion has to follow - it is not skipped, " + "because an exception granted to an object this test cannot identify is " + "worth less than no exception at all." + % (release, chart_name, len(off_policies), len(on_policies))) +else: + # --- The switch OFF half ------------------------------------------------ + off_hits, off_cidr_hits = [], [] + for name, doc in off_policies: + for i, rule in enumerate(ingress_rules(doc)): + for value, proto in rule_ports(rule): + if value == PORT: + off_hits.append("%s ingress[%d] %s/%s" % (name, i, value, proto)) + for cidr in cidrs_of(rule): + if cidr == CIDR: + off_cidr_hits.append("%s ingress[%d]" % (name, i)) + if off_hits: + fail("allows ingress port %d with %s at its default (false): %s. The " + "exception assertion 2 grants for %d is gated on that value, and " + "this is the half that keeps it from being a hole - a port rendered " + "unconditionally would otherwise be found present by this assertion " + "and never held against the pod spec by assertion 2. Gate the port " + "on the value." % (PORT, gate, ", ".join(off_hits), PORT)) + elif off_cidr_hits: + fail("carries the ipBlock %s with %s at its default (false): %s. The " + "probe-source half of the switch is for the mesh only, where the " + "kubelet's probes are redirected; outside it the address matches " + "nothing and makes the rule read as if it did something." + % (CIDR, gate, ", ".join(off_cidr_hits))) + else: + ok("with %s at its default the policy allows neither port %d nor the " + "ipBlock %s - the negative half of assertion 2's named exception, " + "without which the exception would be a hole" % (gate, PORT, CIDR)) + + # --- The switch ON half ------------------------------------------------- + templates = [] + for doc in docs_of(on_path): + templates.extend(pod_templates(doc)) + + found_port, bad_ports, restricted_without_cidr, carriers = [], [], [], [] + for name, doc in on_policies: + spec = doc.get("spec") if isinstance(doc.get("spec"), dict) else {} + selector = spec.get("podSelector") + match_labels = (selector or {}).get("matchLabels") or {} + selected = [t for t in templates if matches(match_labels, labels_of(t))] + declared = set() + for t in selected: + for number, pname in container_ports(t): + declared.add(number) + if pname: + declared.add(pname) + for i, rule in enumerate(ingress_rules(doc)): + for value, proto in rule_ports(rule): + if value == PORT: + found_port.append("%s ingress[%d] %d/%s" % (name, i, PORT, proto)) + carriers.append((name, i, rule)) + if proto != "TCP": + bad_ports.append("%s ingress[%d] renders %d as %s, but " + "HBONE is TCP" % (name, i, PORT, proto)) + elif value not in declared and selected: + bad_ports.append( + "%s ingress[%d] allows %r, which is no containerPort of " + "the selected pod spec" % (name, i, value)) + + if not found_port: + fail("renders no ingress port %d with %s=true, so the value is " + "decorative: the chart accepts it, the schema declares it, and the " + "policy that comes out is the same one as before. A pod whose " + "namespace has joined the ambient mesh is then unreachable, because " + "ambient does not bypass NetworkPolicy enforcement." % (PORT, gate)) + else: + ok("with %s=true the policy allows %s - ztunnel's HBONE tunnel, the one " + "port assertion 2 grants a named exception for" + % (gate, ", ".join(found_port))) + + if bad_ports: + fail("with %s=true the policy allows ingress port(s) that are NOT " + "containerPorts of the selected pod spec and are NOT the one number " + "the exception covers (%d): %s. This is the line between a named " + "exception and the free port key that was rejected as design A of " + "#143 - with the switch on, every OTHER wrong port still fails, " + "including a NodePort number written where a pod port belongs." + % (gate, PORT, "; ".join(bad_ports))) + elif found_port: + ok("with %s=true, port %d is still the ONLY ingress port of this chart " + "that is not a containerPort of the selected pod spec - the exception " + "covers exactly one number and nothing rode in beside it" + % (gate, PORT)) + + # The inseparability of the two halves, where the render can reach it. + for name, i, rule in carriers: + if not rule.get("from"): + continue + if CIDR in cidrs_of(rule): + ok("%s ingress[%d] carries port %d AND the ipBlock %s in the same " + "\"from\" list - the two halves of the switch are inseparable, " + "which is the reason it is one value and not two" + % (name, i, PORT, CIDR)) + else: + restricted_without_cidr.append( + "%s ingress[%d] (peers: %s)" + % (name, i, ", ".join(cidrs_of(rule)) or "no ipBlock at all")) + if restricted_without_cidr: + fail("with %s=true the rule carrying port %d restricts its sources but " + "does not include the ipBlock %s: %s. Inside the mesh the kubelet's " + "probes are redirected and arrive from that address instead of from " + "the worker node's, so a restricted rule without it drops every " + "probe - the pod never becomes Ready and the symptom reads as a " + "broken application rather than as a policy. That is assertion 4 of " + "this script, re-created one value further along, and it is the " + "whole reason the two halves are a single switch." + % (gate, PORT, CIDR, "; ".join(restricted_without_cidr))) + if not any(r.get("from") for _n, _i, r in carriers): + skip("the rule carrying port %d has no \"from\" list in this render, so " + "the probe-source half (%s) is not reachable here and was not " + "checked. That is the state of every chart's DEFAULTS - probeSources " + "is empty everywhere - and it is why ci/networkpolicy/.yaml " + "exists: template-chart and gotenberg render the restricted branch " + "and the half IS checked there. Reported, not counted as passing." + % (PORT, CIDR)) + +for line in lines: + print(line) +PY + RC=$? + set -e + if [ "$RC" -gt 1 ]; then + sed 's/^/ /' "$WORK/ambient-$CHART.pyerr" + echo "::error title=ambient-mesh $CHART::the ambient-mesh comparison itself failed for $CHART (python exit $RC). Treated as a failure, never as a pass." + ROW_FAIL=1 + else + while IFS='|' read -r VERDICT DETAIL; do + case "$VERDICT" in + OK) + TOTAL_ASSERTIONS=$((TOTAL_ASSERTIONS + 1)) + echo " ok $DETAIL" + ;; + SKIP) echo " gap $DETAIL" ;; + FAIL) + ROW_FAIL=1 + echo " FAIL $DETAIL" + echo "::error title=ambient-mesh $CHART::$DETAIL" + ;; + *) [ -n "$VERDICT" ] && echo " $VERDICT $DETAIL" ;; + esac + done <"$WORK/ambient-$CHART.out" + fi + fi + + echo "::endgroup::" + if [ -n "$ROW_FAIL" ]; then + FAILED="$FAILED $CHART(ambient-mesh)" + echo "ambient-mesh $CHART: FAILED" + else + AMBIENT_OK="${AMBIENT_OK:-} $CHART" + echo "ambient-mesh $CHART: ok" + fi +done + # --- Report ------------------------------------------------------------------ count() { set -- $1; echo "$#"; } listing() { set -- $1; [ "$#" -eq 0 ] && { echo "–"; return; }; printf '`%s`' "$1"; shift; for C in "$@"; do printf ', `%s`' "$C"; done; echo; } @@ -1225,6 +1635,7 @@ SUMMARY="${GITHUB_STEP_SUMMARY:-/dev/null}" echo "| Result | Count | Charts |" echo "|---|---|---|" echo "| Policies checked | $(count "$CHECKED") | $(listing "$CHECKED") |" + echo "| Ambient-mesh switch held (Gitea #143) | $(count "${AMBIENT_OK:-}") | $(listing "${AMBIENT_OK:-}") |" echo "| Failed | $(count "$FAILED") | $(listing "$FAILED") |" echo echo "$TOTAL_ASSERTIONS assertion(s) held in total." diff --git a/template-chart/templates/xxx.networkpolicy.yaml b/template-chart/templates/xxx.networkpolicy.yaml index 023f1fd..90051ce 100644 --- a/template-chart/templates/xxx.networkpolicy.yaml +++ b/template-chart/templates/xxx.networkpolicy.yaml @@ -96,6 +96,24 @@ spec this policy selects. {{- $probeSources := $np.probeSources | default list }} {{- $allowedNs := $np.allowedNamespaces | default list }} {{- $ic := $np.ingressController | default dict }} +{{- /* networkPolicy.ingress - the policy's INGRESS DIRECTION, and not to be + confused with networkPolicy.ingressController one line up, nor with the + chart's top-level .Values.ingress. Three different things whose names + overlap as substrings: + + .Values.ingress the Ingress OBJECT this chart renders + .Values.networkPolicy.ingress switches on the policy's inbound side + .Values.networkPolicy.ingressController + the SOURCE SELECTOR for the ingress + controller's pods, a sibling of + networkPolicy.ingress and NO PART OF + IT + + Named here because a search for "ingress" matches all three, and one + round of this work already reported ingressController as an inbound + escape hatch on the strength of the substring alone. */}} +{{- $npIngress := $np.ingress | default dict }} +{{- $ambientMesh := $npIngress.ambientMesh }} {{- $routed := or .Values.ingress.enabled .Values.httpRoute.enabled }} apiVersion: networking.k8s.io/v1 kind: NetworkPolicy @@ -129,6 +147,29 @@ spec: - protocol: {{ .protocol | default "TCP" }} port: {{ .containerPort }} {{- end }} + {{- if $ambientMesh }} + {{- /* FIRST HALF OF networkPolicy.ingress.ambientMesh. 15008/TCP is + ztunnel's HBONE tunnel, the port every inbound mesh connection + arrives on once the namespace carries istio.io/dataplane-mode. + Istio's own documentation is explicit that ambient "does not and + will never bypass Kubernetes NetworkPolicy enforcement", so + without this port a pod in the mesh is simply unreachable. + + It is the one port in this rule that is NOT a containerPort of + the selected pod spec - ztunnel terminates it, not the + application - so ci/run-networkpolicy-test.sh holds a NAMED + exception for exactly this number, gated on exactly this value. + Anything else that is not a containerPort still fails, and so + does 15008 itself while the value is off. + + NOT MEASURED, and the chart should not pretend otherwise: that + 15008 ALONE suffices comes from Istio's documentation. What has + been measured on this cluster is only which listeners ztunnel + opens. Provable at the first namespace that joins the mesh, and + no namespace carries the label yet. */}} + - protocol: TCP + port: 15008 + {{- end }} {{- if $probeSources }} {{- /* RESTRICTED, because the operator supplied probeSources. The kubelet's probes come from the IP of the WORKER NODE running the @@ -168,6 +209,35 @@ spec: - ipBlock: cidr: {{ . | quote }} {{- end }} + {{- if $ambientMesh }} + {{- /* SECOND HALF OF networkPolicy.ingress.ambientMesh, and the reason + the switch is one value instead of two. Inside the mesh the + kubelet's probes are redirected and no longer arrive from the + worker node's address: they arrive from 169.254.7.127, which is + therefore the probeSources entry nobody would think to add. + + The two halves are deliberately inseparable. An operator who + only opens 15008 by hand has a policy that passes every review - + and the moment probeSources is non-empty, every probe is dropped, + the pod never becomes Ready, and the symptom reads as a broken + application rather than as a policy. That is assertion 4 of + ci/run-networkpolicy-test.sh, re-created one value further along. + One switch cannot be set halfway. + + Only rendered in this branch, because this is the only branch + with a "from" list at all: with probeSources empty the rule is + open and needs no peer. The IPv6 counterpart + fd16:9254:7127:1337:ffff:ffff:ffff:ffff is deliberately absent - + this cluster is SingleStack IPv4, and an ipBlock of the wrong + family is accepted by the API server and matches nothing. + + UNPROVEN IN OPERATION: probeSources is empty in every chart of + this repository (stalwart sets it to the empty list explicitly), + so no release reaches this branch today. What CI proves is its + render, through the overlay in ci/networkpolicy. */}} + - ipBlock: + cidr: 169.254.7.127/32 + {{- end }} {{- else }} {{- /* NO "from" SELECTOR, WHICH MEANS OPEN - deliberately, and only inside the cluster, because this port carries no NodePort. The diff --git a/template-chart/values.schema.json b/template-chart/values.schema.json index fcbcdef..a1fcc0b 100644 --- a/template-chart/values.schema.json +++ b/template-chart/values.schema.json @@ -45,6 +45,17 @@ } } }, + "ingress": { + "description": "The policy's INBOUND side. Three names here overlap as substrings and are not the same thing: the top-level ingress is the Ingress OBJECT this chart renders, networkPolicy.ingress is this block, and networkPolicy.ingressController is the SOURCE SELECTOR for the ingress controller's pods - a SIBLING of this block and no part of it.", + "type": ["object", "null"], + "additionalProperties": false, + "properties": { + "ambientMesh": { + "description": "Opens the pod for Istio's ambient mesh; off by default and additive, so the render is byte-identical while it is false. Renders 15008/TCP inbound - ztunnel's HBONE tunnel, which ambient explicitly does not let bypass NetworkPolicy enforcement - AND, where probeSources is non-empty, 169.254.7.127/32 as an additional probe source, because inside the mesh the kubelet's probes are redirected away from the worker node's address. The two halves are inseparable on purpose: whoever opens 15008 by hand has not added the probe address, and the first narrowed probeSources then drops every probe, so the pod never becomes Ready and the symptom reads as a broken application rather than as a policy. Unproven in operation: that 15008 alone suffices comes from Istio's documentation, and no namespace of this cluster carries istio.io/dataplane-mode yet.", + "type": ["boolean", "null"] + } + } + }, "egress": { "type": ["object", "null"], "additionalProperties": false, diff --git a/template-chart/values.yaml b/template-chart/values.yaml index 85ef0de..0812102 100644 --- a/template-chart/values.yaml +++ b/template-chart/values.yaml @@ -272,6 +272,62 @@ networkPolicy: podLabels: app.kubernetes.io/name: rke2-traefik + # The policy's INBOUND side. Mind the three names that overlap as substrings, + # because a search for "ingress" matches all of them and they are not the + # same thing: + # + # ingress (top level) the Ingress OBJECT this chart + # renders - host, TLS, className + # networkPolicy.ingress this block: what the policy lets IN + # networkPolicy.ingressController the SOURCE SELECTOR for the controller's + # pods. A SIBLING of this block and NO PART + # OF IT, despite the name. + ingress: + # Opens the pod for Istio's ambient mesh. OFF by default, and additive: + # with it off this chart renders byte for byte as it did before the value + # existed. + # + # WHAT IT RENDERS, and the two halves are the point: + # + # 1. 15008/TCP inbound - ztunnel's HBONE tunnel. Every inbound mesh + # connection arrives on it, and Istio's documentation is explicit that + # ambient "does not and will never bypass Kubernetes NetworkPolicy + # enforcement". Without this port a pod whose namespace has joined the + # mesh is unreachable. + # 2. 169.254.7.127/32 as an additional probe source - but ONLY where + # probeSources is non-empty, because only then does the rule have a + # "from" list at all. Inside the mesh the kubelet's probes are + # redirected and no longer come from the worker node's address. + # + # WHY ONE SWITCH AND NOT TWO. Point 2 is the one nobody thinks of. An + # operator who adds "15008" to a port list by hand writes a policy that + # passes review, and the first time probeSources is narrowed every probe is + # dropped: the pod never becomes Ready and the symptom reads as a broken + # application rather than as a policy. A switch whose halves can only be + # had together cannot be set halfway - which is why this is a named + # intention and not a free port list. + # + # TWO THINGS IT DOES NOT PROVE, and they belong next to the value rather + # than in a commit message: + # + # - That 15008 alone is enough. That comes from Istio's documentation. + # Measured on this cluster is only which listeners ztunnel opens + # ([::]:15008 inbound, plus its own admin and stats ports). Provable at + # the first namespace that joins the mesh; none carries + # istio.io/dataplane-mode yet. + # - That the probe-source half works. probeSources is empty in every + # chart here, so no release reaches that branch. CI proves its render, + # not its effect. + # + # If the policy restricts sources, note that 15008 is restricted with them: + # the port joins the rule that carries the probe port, so it is reachable + # from that rule's peers only. Whether ztunnel needs a peer of its own is + # open - it runs with its own pod IPs in istio-system (measured: no + # hostNetwork on the DaemonSet, Istio 1.30.6), so if one turns out to be + # needed it is expressible as a namespaceSelector plus podSelector and does + # not need an unknowable node-IP CIDR. + ambientMesh: false + egress: # Additional outbound TCP ports. DNS (UDP and TCP 53) is always allowed and # has no switch - a knob whose only other setting breaks every name lookup