diff --git a/ci/run-networkpolicy-test.sh b/ci/run-networkpolicy-test.sh index fd77156..f79c7a3 100755 --- a/ci/run-networkpolicy-test.sh +++ b/ci/run-networkpolicy-test.sh @@ -1261,17 +1261,26 @@ done # around, so the exception is held to five things at once, on TWO renders of # every chart: # +# the chart's DECLARED DEFAULTS (read back through ci/yaml2json) +# - enabled must default to false, and the port and the probe source must be +# exactly the numbers pinned below. A chart whose default drifted would +# otherwise ship a port no other chart ships while the exception kept being +# granted to the pinned one. # switch OFF (the main loop's render, reused) -# - 15008 must be ABSENT. Without this half the exception is a hole: a +# - the pinned port must be ABSENT. Without this half the exception is a hole: a # chart could render the port unconditionally and nothing would notice, # because assertion 7 would find it present and assertion 2 is where it # would have failed. # - 169.254.7.127/32 must be absent too, for the same reason. -# switch ON -# - 15008/TCP must be PRESENT, or the value is decorative. -# - 15008 must be the ONLY ingress port that is not a containerPort. This is -# the line between a named exception and design A of #143: every other -# mistyped or NodePort number still fails here, with the switch on. +# switch ON (only "enabled=true" is set, never a port override - the pin is a +# statement about the defaults WE ship) +# - the PINNED port must be PRESENT, or the value is decorative. +# - the PINNED port must be the ONLY ingress port that is not a +# containerPort. This is the line between a named exception and design A +# of #143: every other mistyped or NodePort number still fails here, with +# the switch on. Note "pinned": since #143 made the port a VALUE, checking +# the exception against the rendered value would permit whatever the value +# said, which is design A's hole under design C's name. # - wherever the rule carrying 15008 restricts sources, that rule's "from" # must contain 169.254.7.127/32. This is the inseparability of the two # halves, asserted rather than trusted - and it is REACHABLE, because @@ -1314,6 +1323,45 @@ done # mechanism as assertion 0: the mistake arrives by copy into a NEW chart whose # own diff nobody reads for it, which is exactly how garage (#81) and keycloak # (#83) arrived with no policy at all after #5 had been filed. +# THE PINNED EXPECTATION, and it is pinned HERE rather than read from the chart +# on purpose (Gitea #143). Both numbers are values with documented defaults, so +# that an operator can correct an Istio constant without waiting for a chart +# release - but an exception that permits "whatever the value says" is design A's +# hole wearing design C's name: a mistyped port would pass simply by sitting in +# the configurable field, and that is the single reason design A was rejected. +# +# So the test states what WE believe Istio's numbers to be, and asserts two +# things against that statement: the render at CHART DEFAULTS produces exactly +# this port and exactly this ipBlock, AND every chart's values.yaml DECLARES +# these same defaults. A chart that drifts is red. +# +# Which makes this block the one place in the repository that says what we hold +# Istio's constants to be - and therefore also the answer to "where does +# 169.254.7.127 come from": +# +# 15008 Istio's HBONE port. Not registered with IANA for this, +# prescribed by no protocol, unknown to Kubernetes. Istio +# picked it and Istio can change it. +# 169.254.7.127/32 the address Istio's data plane rewrites the kubelet's probe +# source onto. LINK-LOCAL (169.254.0.0/16, RFC 3927), a range +# reserved for node-local use and never routed. Istio's IPv6 +# counterpart fd16:9254:7127:1337:ffff:ffff:ffff:ffff carries +# the SAME DIGITS plus 1337, which is what vendor constants +# look like. Neither the NetworkPolicy API nor the kubelet has +# any notion of it. +# +# BOTH READ FOR ztunnel 1.30.6. If Istio changes either, the failure is silent - +# a pod unreachable in the mesh, or probes dropped where probeSources is +# non-empty, which is nowhere today. Changing them means 30 values files AND +# these two lines, and these two lines are what force someone to look. +# +# An operator who overrides the value in their own release is outside this test's +# reach. That is true of every value in this repository and is not a new gap - +# and it is why these renders set ONLY "enabled=true" and never a port override: +# the pin is a statement about the defaults WE ship, not about what a release may +# choose. +AMBIENT_PORT=15008 +AMBIENT_CIDR="169.254.7.127/32" AMBIENT_VALUE="networkPolicy.ingress.ambientMesh" for CHART in "${CHARTS[@]}"; do # A chart whose main-loop render already failed has nothing to compare @@ -1326,10 +1374,10 @@ for CHART in "${CHARTS[@]}"; do ROW_FAIL="" ON_JSON="$WORK/ambient-$CHART-on.json" if ! RCJ_WITH_OVERLAY=1 render_chart_json "$CHART" "$ON_JSON" \ - --set "$AMBIENT_VALUE=true"; then + --set "$AMBIENT_VALUE.enabled=true"; then sed 's/^/ /' "$WORK/xerr" - echo " FAIL $CHART does not render with $AMBIENT_VALUE=true" - echo "::error title=ambient-mesh $CHART::$CHART does not render with $AMBIENT_VALUE=true. Since Gitea #143 every chart in this repository offers that value, and networkPolicy carries additionalProperties: false in all of them - so an unknown key does not get ignored, it fails the render. Add the key to values.yaml AND to values.schema.json (schema and template belong together per chart), and render the ingress port 15008 plus, inside the source-restricted branch, the ipBlock 169.254.7.127/32. Copy template-chart/templates/xxx.networkpolicy.yaml." + echo " FAIL $CHART does not render with $AMBIENT_VALUE.enabled=true" + echo "::error title=ambient-mesh $CHART::$CHART does not render with $AMBIENT_VALUE.enabled=true. Since Gitea #143 every chart in this repository offers that value, and networkPolicy carries additionalProperties: false in all of them - so an unknown key does not get ignored, it fails the render. Add the object to values.yaml AND to values.schema.json (schema and template belong together per chart), with enabled: false, port: \$AMBIENT_PORT and redirectedProbeSources: [\$AMBIENT_CIDR], and render that port plus - inside the source-restricted branch - that ipBlock. Copy template-chart/templates/xxx.networkpolicy.yaml." ROW_FAIL=1 fi @@ -1345,17 +1393,36 @@ for CHART in "${CHARTS[@]}"; do ROW_FAIL=1 fi + # The chart's DECLARED defaults, read with the same parser as everything else + # (ci/yaml2json prints a values file back as JSON). Not a regex over + # values.yaml: the pinned expectation is about numbers and a list, and a + # substring test over a commented file is how the probeSources check in + # assertion 4 was once fooled by a deleted key whose comment survived. + VALUES_JSON="$WORK/$CHART.values.json" + if [ -z "$ROW_FAIL" ]; then + if ! helm template y2j "$YAML2JSON" -f "$ROOT/$CHART/values.yaml" 2>"$WORK/xerr" \ + | awk '/^\{/ { print; exit }' >"$VALUES_JSON" || [ ! -s "$VALUES_JSON" ]; then + sed 's/^/ /' "$WORK/xerr" + echo " FAIL $CHART/values.yaml could not be re-read as JSON" + echo "::error title=ambient-mesh $CHART::$CHART/values.yaml could not be re-read by ci/yaml2json, so the chart's DECLARED ambient-mesh defaults could not be compared with the numbers this test pins. That is a parse failure, not a policy finding." + ROW_FAIL=1 + fi + fi + if [ -z "$ROW_FAIL" ]; then set +e python3 - "$CHART" "$RELEASE" "$AMBIENT_VALUE" "$WORK/$CHART.json" "$ON_JSON" \ - "$ROOT/$CHART" "$CHART_NAME" \ + "$ROOT/$CHART" "$CHART_NAME" "$AMBIENT_PORT" "$AMBIENT_CIDR" "$VALUES_JSON" \ >"$WORK/ambient-$CHART.out" 2>"$WORK/ambient-$CHART.pyerr" <<'PY' import json, re, sys -chart, release, gate, off_path, on_path, chart_dir, chart_name = sys.argv[1:8] +(chart, release, gate, off_path, on_path, chart_dir, chart_name, + pinned_port, pinned_cidr, values_json) = sys.argv[1:11] -PORT = 15008 -CIDR = "169.254.7.127/32" +# Pinned by the caller, never read from the chart - see the long note above +# AMBIENT_PORT in this script. +PORT = int(pinned_port) +CIDR = pinned_cidr lines = [] def ok(msg): lines.append("OK|%s: %s" % (chart, msg)) @@ -1451,27 +1518,77 @@ def cidrs_of(rule): return out -# --- The value is declared, in BOTH places it has to be -------------------- -# The render with the switch on already proves the SCHEMA declares it - an +# --- The chart DECLARES the pinned defaults ------------------------------- +# The render with the switch on already proves the SCHEMA declares the key - an # undeclared key cannot pass additionalProperties: false. What that render -# cannot prove is that values.yaml ships the documented default, and a value -# only reachable through --set is not a value an operator finds. +# cannot prove is what the chart's own defaults SAY, and since #143 made both +# Istio constants configurable, that is the half the pinned exception rests on: +# the exception below is granted to the number THIS SCRIPT names, so a chart +# whose default has drifted away from it must be red here rather than quietly +# carrying a different port than every other chart. values_yaml = "" try: values_yaml = open("%s/values.yaml" % chart_dir, encoding="utf-8").read() except IOError: pass -if re.search(r"^\s*ambientMesh\s*:", values_yaml, re.M): - ok("values.yaml ships the ambientMesh key with its default, and the render " - "with %s=true succeeded - which is itself the proof that " - "values.schema.json declares it, since networkPolicy carries " - "additionalProperties: false" % gate) + +declared = json.load(open(values_json)) +node = (((declared.get("networkPolicy") or {}).get("ingress") or {}) + .get("ambientMesh")) +if not isinstance(node, dict): + fail("renders with %s.enabled=true - so the schema knows the key - but the " + "chart's own values.yaml declares no ambientMesh OBJECT (found %r). A " + "value that exists only in the schema is reachable through --set and " + "findable nowhere: the defaults are what an operator reads. Ship the " + "object with enabled: false, port: %d and redirectedProbeSources: " + "[%s], as in template-chart." % (gate, node, PORT, CIDR)) else: - fail("renders with %s=true - so the schema knows the key - but values.yaml " - "contains no ambientMesh key at all. A value that exists only in the " - "schema is reachable through --set and findable nowhere: the defaults " - "are what an operator reads. Add it with its comment, default false, " - "as in template-chart." % gate) + problems = [] + if node.get("enabled") not in (False, None): + problems.append("enabled is %r and must default to false - the switch is " + "additive and no release may get it by accident" + % node.get("enabled")) + if node.get("port") != PORT: + problems.append("port defaults to %r, but this test pins Istio's HBONE " + "port at %d" % (node.get("port"), PORT)) + # Conditional on the chart offering probeSources, exactly as the + # inseparability check below and assertion 4 are. A chart whose probe rule + # is never source-restricted renders no ipBlock, so this key would reach no + # rule - and a value that silently does nothing reads as a control, which is + # worse than one that is absent (README, "Values, and shipping only the live + # ones"). So for those charts the key must be ABSENT, and that is asserted + # rather than tolerated. + sources = node.get("redirectedProbeSources") + offers_ps = bool(re.search(r"^\s*probeSources\s*:", values_yaml, re.M)) + if offers_ps and sources != [CIDR]: + problems.append("redirectedProbeSources defaults to %r, but this test " + "pins Istio's redirected probe source at [%r]" + % (sources, CIDR)) + if not offers_ps and sources is not None: + problems.append("redirectedProbeSources is declared (%r) although this " + "chart offers no networkPolicy.probeSources, so its probe " + "rule is never source-restricted and no ipBlock is ever " + "rendered - the key would reach no rule and read as a " + "control that does nothing" % sources) + if problems: + fail("the chart's declared ambient-mesh defaults do not match the " + "numbers ci/run-networkpolicy-test.sh pins: %s. Both are ISTIO " + "CONSTANTS, and they are values precisely so an operator can " + "correct them without a chart release - but the exception assertion " + "2 grants the HBONE port is pinned to the number THIS TEST knows, " + "never to whatever the value says, because a value-driven exception " + "would let a mistyped port ride in through a configurable field. " + "That is the hole design A of #143 was rejected for. So: if Istio " + "changed the constant, change it in all 30 charts AND in the pinned " + "expectation here - the drift is meant to be loud." + % "; ".join(problems)) + else: + ok("declares the pinned ambient-mesh defaults verbatim - enabled false, " + "port %d, %s - so the named exception below is granted to a number " + "this chart really ships, and not to whatever a value happened to say" + % (PORT, ("redirectedProbeSources [%s]" % CIDR) if offers_ps else + "and no redirectedProbeSources, correctly, since this chart " + "renders no ipBlock at all")) off_policies = own_policies(off_path) on_policies = own_policies(on_path) @@ -1496,20 +1613,20 @@ else: if cidr == CIDR: off_cidr_hits.append("%s ingress[%d]" % (name, i)) if off_hits: - fail("allows ingress port %d with %s at its default (false): %s. The " + fail("allows ingress port %d with %s.enabled at its default (false): %s. The " "exception assertion 2 grants for %d is gated on that value, and " "this is the half that keeps it from being a hole - a port rendered " "unconditionally would otherwise be found present by this assertion " "and never held against the pod spec by assertion 2. Gate the port " "on the value." % (PORT, gate, ", ".join(off_hits), PORT)) elif off_cidr_hits: - fail("carries the ipBlock %s with %s at its default (false): %s. The " + fail("carries the ipBlock %s with %s.enabled at its default (false): %s. The " "probe-source half of the switch is for the mesh only, where the " "kubelet's probes are redirected; outside it the address matches " "nothing and makes the rule read as if it did something." % (CIDR, gate, ", ".join(off_cidr_hits))) else: - ok("with %s at its default the policy allows neither port %d nor the " + ok("with %s.enabled at its default the policy allows neither port %d nor the " "ipBlock %s - the negative half of assertion 2's named exception, " "without which the exception would be a hole" % (gate, PORT, CIDR)) @@ -1544,18 +1661,18 @@ else: "the selected pod spec" % (name, i, value)) if not found_port: - fail("renders no ingress port %d with %s=true, so the value is " + fail("renders no ingress port %d with %s.enabled=true, so the value is " "decorative: the chart accepts it, the schema declares it, and the " "policy that comes out is the same one as before. A pod whose " "namespace has joined the ambient mesh is then unreachable, because " "ambient does not bypass NetworkPolicy enforcement." % (PORT, gate)) else: - ok("with %s=true the policy allows %s - ztunnel's HBONE tunnel, the one " + ok("with %s.enabled=true the policy allows %s - ztunnel's HBONE tunnel, the one " "port assertion 2 grants a named exception for" % (gate, ", ".join(found_port))) if bad_ports: - fail("with %s=true the policy allows ingress port(s) that are NOT " + fail("with %s.enabled=true the policy allows ingress port(s) that are NOT " "containerPorts of the selected pod spec and are NOT the one number " "the exception covers (%d): %s. This is the line between a named " "exception and the free port key that was rejected as design A of " @@ -1563,7 +1680,7 @@ else: "including a NodePort number written where a pod port belongs." % (gate, PORT, "; ".join(bad_ports))) elif found_port: - ok("with %s=true, port %d is still the ONLY ingress port of this chart " + ok("with %s.enabled=true, port %d is still the ONLY ingress port of this chart " "that is not a containerPort of the selected pod spec - the exception " "covers exactly one number and nothing rode in beside it" % (gate, PORT)) @@ -1599,7 +1716,7 @@ else: "%s ingress[%d] (peers: %s)" % (name, i, ", ".join(cidrs_of(rule)) or "no ipBlock at all")) if restricted_without_cidr: - fail("with %s=true the rule carrying port %d restricts its sources but " + fail("with %s.enabled=true the rule carrying port %d restricts its sources but " "does not include the ipBlock %s: %s. Inside the mesh the kubelet's " "probes are redirected and arrive from that address instead of from " "the worker node's, so a restricted rule without it drops every " diff --git a/template-chart/templates/xxx.networkpolicy.yaml b/template-chart/templates/xxx.networkpolicy.yaml index 90051ce..80843d2 100644 --- a/template-chart/templates/xxx.networkpolicy.yaml +++ b/template-chart/templates/xxx.networkpolicy.yaml @@ -113,7 +113,23 @@ spec this policy selects. round of this work already reported ingressController as an inbound escape hatch on the strength of the substring alone. */}} {{- $npIngress := $np.ingress | default dict }} -{{- $ambientMesh := $npIngress.ambientMesh }} +{{- /* NULL-SAFE ON PURPOSE, and not as decoration (#99). ambientMesh is an + OBJECT, so every read below goes through a fallback that reproduces THIS + chart's declared defaults - because an operator who deletes the object, + or who writes only "enabled: true" and leaves the rest out, must land on + the defaults and not in a nil dereference. #136 shipped exactly that + defect one value away from here: the NOTES.txt branch dereferenced + .networkPolicy.egress directly, and a release that had dropped the key + failed to render. + + Note which fallback the probe-source list gets: "default" treats an EMPTY + list as unset, so clearing it restores the constant rather than removing + the ipBlock. That is deliberate - the two halves of this switch are not + separable, and the way to not open the port at all is enabled: false. */}} +{{- $ambient := $npIngress.ambientMesh | default dict }} +{{- $ambientEnabled := $ambient.enabled }} +{{- $ambientPort := $ambient.port | default 15008 }} +{{- $ambientProbeSources := $ambient.redirectedProbeSources | default (list "169.254.7.127/32") }} {{- $routed := or .Values.ingress.enabled .Values.httpRoute.enabled }} apiVersion: networking.k8s.io/v1 kind: NetworkPolicy @@ -147,20 +163,36 @@ spec: - protocol: {{ .protocol | default "TCP" }} port: {{ .containerPort }} {{- end }} - {{- if $ambientMesh }} - {{- /* FIRST HALF OF networkPolicy.ingress.ambientMesh. 15008/TCP is - ztunnel's HBONE tunnel, the port every inbound mesh connection - arrives on once the namespace carries istio.io/dataplane-mode. - Istio's own documentation is explicit that ambient "does not and - will never bypass Kubernetes NetworkPolicy enforcement", so - without this port a pod in the mesh is simply unreachable. + {{- if $ambientEnabled }} + {{- /* FIRST HALF OF networkPolicy.ingress.ambientMesh. The default is + 15008/TCP: ztunnel's HBONE tunnel, the port every inbound mesh + connection arrives on once the namespace carries + istio.io/dataplane-mode. Istio's own documentation is explicit + that ambient "does not and will never bypass Kubernetes + NetworkPolicy enforcement", so without this port a pod in the + mesh is simply unreachable. + + 15008 IS AN ISTIO CONSTANT, NOT A STANDARD. It is not registered + with IANA for this, Kubernetes knows nothing about it, and no + protocol prescribes it - Istio picked it for HBONE and Istio can + change it. That is precisely why it sits in a VALUE with a + documented default rather than as a literal here: an operator can + correct it without waiting for a chart release. It is the one port in this rule that is NOT a containerPort of the selected pod spec - ztunnel terminates it, not the application - so ci/run-networkpolicy-test.sh holds a NAMED - exception for exactly this number, gated on exactly this value. - Anything else that is not a containerPort still fails, and so - does 15008 itself while the value is off. + exception for it. Note what that exception is pinned to: the + number 15008 as written in THE TEST, plus an assertion that this + chart's values.yaml declares the same default. It is NOT "whatever + this value happens to say" - that would let a mistyped port ride + in through a configurable field, which is the hole design A of + #143 was rejected for. + + READ FOR ztunnel 1.30.6. If Istio changes the port, or the mesh is + replaced: the failure is a pod that cannot be reached inside the + mesh. Change the default in all 30 charts AND the pinned number in + the test - which is the thing that forces someone to look. NOT MEASURED, and the chart should not pretend otherwise: that 15008 ALONE suffices comes from Istio's documentation. What has @@ -168,7 +200,7 @@ spec: opens. Provable at the first namespace that joins the mesh, and no namespace carries the label yet. */}} - protocol: TCP - port: 15008 + port: {{ int $ambientPort }} {{- end }} {{- if $probeSources }} {{- /* RESTRICTED, because the operator supplied probeSources. The @@ -209,34 +241,64 @@ spec: - ipBlock: cidr: {{ . | quote }} {{- end }} - {{- if $ambientMesh }} + {{- if $ambientEnabled }} {{- /* SECOND HALF OF networkPolicy.ingress.ambientMesh, and the reason - the switch is one value instead of two. Inside the mesh the - kubelet's probes are redirected and no longer arrive from the - worker node's address: they arrive from 169.254.7.127, which is - therefore the probeSources entry nobody would think to add. + the switch is one value instead of two. Inside ISTIO's ambient + mesh the kubelet's probes are redirected BY ISTIO and no longer + arrive from the worker node's address: they arrive from + 169.254.7.127, which is therefore the probe source nobody would + think to add. - The two halves are deliberately inseparable. An operator who - only opens 15008 by hand has a policy that passes every review - - and the moment probeSources is non-empty, every probe is dropped, - the pod never becomes Ready, and the symptom reads as a broken + WHERE THAT ADDRESS COMES FROM, because it is worth being able to + recognise as vendor magic rather than as protocol: + + - It is LINK-LOCAL, 169.254.0.0/16 (RFC 3927) - a range + reserved for node-local use that is never routed. Exactly + the range one picks a vendor's magic address out of. + - Istio's IPv6 counterpart gives the game away: + fd16:9254:7127:1337:ffff:ffff:ffff:ffff carries the SAME + DIGITS - 9254:7127 is 169.254.7.127 - plus 1337. Numbers + built that way are vendor constants, not protocols. + - KUBERNETES DOES NOT KNOW THIS ADDRESS. Neither the + NetworkPolicy API nor the kubelet has any notion of it; only + Istio's own data plane rewrites the probe source onto it. + + READ FOR ztunnel 1.30.6. If Istio changes it, or the mesh is + replaced, THE FAILURE IS SILENT: probes would be dropped, and + only where probeSources is non-empty - which is nowhere today. + Nothing in this repository or in Kubernetes would report it. That + is why the address is a VALUE with a documented default, so it can + be corrected without a chart release, and why the test pins the + number it expects: changing it means touching 30 values files and + the test, and the test is what makes someone look. + + A LIST, not a string, and this is the one case where that does not + contradict "the value is a constant, not a configuration": on a + DUAL-STACK cluster BOTH entries are needed at the same time, not + one of two. This cluster is SingleStack IPv4, so the IPv6 + counterpart above is left out of the default - an ipBlock of the + wrong family is accepted by the API server and matches nothing. + Add it alongside, never instead, if the cluster gains IPv6. + + The two halves stay inseparable. An operator who only opens the + port by hand has a policy that passes every review - and the + moment probeSources is non-empty, every probe is dropped, the pod + never becomes Ready, and the symptom reads as a broken application rather than as a policy. That is assertion 4 of ci/run-networkpolicy-test.sh, re-created one value further along. - One switch cannot be set halfway. Only rendered in this branch, because this is the only branch with a "from" list at all: with probeSources empty the rule is - open and needs no peer. The IPv6 counterpart - fd16:9254:7127:1337:ffff:ffff:ffff:ffff is deliberately absent - - this cluster is SingleStack IPv4, and an ipBlock of the wrong - family is accepted by the API server and matches nothing. + open and needs no peer. UNPROVEN IN OPERATION: probeSources is empty in every chart of this repository (stalwart sets it to the empty list explicitly), so no release reaches this branch today. What CI proves is its render, through the overlay in ci/networkpolicy. */}} + {{- range $ambientProbeSources }} - ipBlock: - cidr: 169.254.7.127/32 + cidr: {{ . | quote }} + {{- end }} {{- end }} {{- else }} {{- /* NO "from" SELECTOR, WHICH MEANS OPEN - deliberately, and only diff --git a/template-chart/values.schema.json b/template-chart/values.schema.json index a1fcc0b..125e4f7 100644 --- a/template-chart/values.schema.json +++ b/template-chart/values.schema.json @@ -51,8 +51,28 @@ "additionalProperties": false, "properties": { "ambientMesh": { - "description": "Opens the pod for Istio's ambient mesh; off by default and additive, so the render is byte-identical while it is false. Renders 15008/TCP inbound - ztunnel's HBONE tunnel, which ambient explicitly does not let bypass NetworkPolicy enforcement - AND, where probeSources is non-empty, 169.254.7.127/32 as an additional probe source, because inside the mesh the kubelet's probes are redirected away from the worker node's address. The two halves are inseparable on purpose: whoever opens 15008 by hand has not added the probe address, and the first narrowed probeSources then drops every probe, so the pod never becomes Ready and the symptom reads as a broken application rather than as a policy. Unproven in operation: that 15008 alone suffices comes from Istio's documentation, and no namespace of this cluster carries istio.io/dataplane-mode yet.", - "type": ["boolean", "null"] + "description": "Opens the pod for Istio's ambient mesh; off by default and additive, so the render is byte-identical while enabled is false. Renders the HBONE port inbound AND, where networkPolicy.probeSources is non-empty, the redirected probe source in the same \"from\" list. The two halves live in one object under one switch on purpose: whoever opens the port by hand has not added the probe address, and the first narrowed probeSources then drops every probe, so the pod never becomes Ready and the symptom reads as a broken application rather than as a policy. Both numbers are ISTIO CONSTANTS rather than standards, which is why they are values with documented defaults and not template literals - a vendor constant can change, and an operator should be able to correct it without waiting for a chart release. Deleting the whole object is safe: every read falls back to the defaults the chart declares. Unproven in operation: that the HBONE port alone suffices comes from Istio's documentation, and no namespace of this cluster carries istio.io/dataplane-mode yet.", + "type": ["object", "null"], + "additionalProperties": false, + "properties": { + "enabled": { + "description": "The switch. Everything else in this object is inert while it is false.", + "type": ["boolean", "null"] + }, + "port": { + "description": "Istio's HBONE port, default 15008. A constant Istio chose, not a standard: not registered with IANA for this, prescribed by no protocol, and unknown to Kubernetes. Read for ztunnel 1.30.6. It is the one ingress port of these charts that is deliberately no containerPort - ztunnel terminates it, not the application - and the exception ci/run-networkpolicy-test.sh grants it is pinned to the number THAT TEST knows, plus an assertion that this chart declares the same default. Not to whatever this value says: a value-driven exception would let a mistyped port ride in through a configurable field.", + "type": ["integer", "null"], + "minimum": 1, + "maximum": 65535 + }, + "redirectedProbeSources": { + "description": "Sources the kubelet's probes arrive from once Istio has redirected them, default [169.254.7.127/32]. NOT the same value as networkPolicy.probeSources one level up: that one holds YOUR worker node CIDRs, is empty by default and means OPEN when empty, and the chart cannot know those addresses; this one holds ISTIO's rewrite address and ships filled, because unlike the node IPs it is a constant the chart can know. Recognisable as vendor magic rather than protocol: it is link-local (169.254.0.0/16, RFC 3927, never routed), Istio's IPv6 counterpart fd16:9254:7127:1337:ffff:ffff:ffff:ffff carries the same digits plus 1337, and Kubernetes itself knows the address in neither the NetworkPolicy API nor the kubelet. A list and not a string because a dual-stack cluster needs BOTH entries at once; this cluster is SingleStack IPv4, so the IPv6 counterpart is left out of the default - an ipBlock of the wrong family is accepted and matches nothing. If Istio changes the address the failure is SILENT: probes dropped, and only where probeSources is non-empty.", + "type": ["array", "null"], + "items": { + "type": "string" + } + } + } } } }, diff --git a/template-chart/values.yaml b/template-chart/values.yaml index 0812102..83ed712 100644 --- a/template-chart/values.yaml +++ b/template-chart/values.yaml @@ -283,50 +283,111 @@ networkPolicy: # pods. A SIBLING of this block and NO PART # OF IT, despite the name. ingress: - # Opens the pod for Istio's ambient mesh. OFF by default, and additive: - # with it off this chart renders byte for byte as it did before the value - # existed. + # Opens the pod for Istio's ambient mesh. OFF by default, and additive: with + # it off this chart renders byte for byte as it did before the value existed. # # WHAT IT RENDERS, and the two halves are the point: # - # 1. 15008/TCP inbound - ztunnel's HBONE tunnel. Every inbound mesh - # connection arrives on it, and Istio's documentation is explicit that - # ambient "does not and will never bypass Kubernetes NetworkPolicy - # enforcement". Without this port a pod whose namespace has joined the - # mesh is unreachable. - # 2. 169.254.7.127/32 as an additional probe source - but ONLY where - # probeSources is non-empty, because only then does the rule have a - # "from" list at all. Inside the mesh the kubelet's probes are - # redirected and no longer come from the worker node's address. + # 1. The HBONE port inbound (default 15008/TCP) - ztunnel's tunnel. Every + # inbound mesh connection arrives on it, and Istio's documentation is + # explicit that ambient "does not and will never bypass Kubernetes + # NetworkPolicy enforcement". Without this port a pod whose namespace + # has joined the mesh is unreachable. + # 2. The redirected probe source (default 169.254.7.127/32) - but ONLY + # where networkPolicy.probeSources is non-empty, because only then does + # the rule have a "from" list at all. Inside the mesh the kubelet's + # probes are redirected and no longer come from the worker node's + # address. # # WHY ONE SWITCH AND NOT TWO. Point 2 is the one nobody thinks of. An # operator who adds "15008" to a port list by hand writes a policy that # passes review, and the first time probeSources is narrowed every probe is # dropped: the pod never becomes Ready and the symptom reads as a broken - # application rather than as a policy. A switch whose halves can only be - # had together cannot be set halfway - which is why this is a named - # intention and not a free port list. + # application rather than as a policy. A switch whose halves can only be had + # together cannot be set halfway - which is why the two live in ONE object + # under one "enabled", and not as separate values that can drift apart. # - # TWO THINGS IT DOES NOT PROVE, and they belong next to the value rather - # than in a commit message: + # BOTH NUMBERS BELOW ARE ISTIO CONSTANTS, NOT STANDARDS, and they are values + # rather than template literals for exactly that reason: a vendor constant + # can change, and an operator should be able to correct it without waiting + # for a chart release. Their provenance is written on each one. # - # - That 15008 alone is enough. That comes from Istio's documentation. - # Measured on this cluster is only which listeners ztunnel opens - # ([::]:15008 inbound, plus its own admin and stats ports). Provable at - # the first namespace that joins the mesh; none carries - # istio.io/dataplane-mode yet. - # - That the probe-source half works. probeSources is empty in every - # chart here, so no release reaches that branch. CI proves its render, - # not its effect. - # - # If the policy restricts sources, note that 15008 is restricted with them: - # the port joins the rule that carries the probe port, so it is reachable - # from that rule's peers only. Whether ztunnel needs a peer of its own is - # open - it runs with its own pod IPs in istio-system (measured: no - # hostNetwork on the DaemonSet, Istio 1.30.6), so if one turns out to be - # needed it is expressible as a namespaceSelector plus podSelector and does - # not need an unknowable node-IP CIDR. - ambientMesh: false + # DELETING THIS OBJECT IS SAFE. Every read falls back to the defaults + # declared here (#99), so "ambientMesh: null", or an object carrying only + # "enabled: true", renders the documented defaults instead of failing. + ambientMesh: + # The switch. Everything else in this object is inert while it is false, + # and the chart renders exactly as it did before the object existed. + enabled: false + + # ISTIO'S HBONE PORT. A constant Istio chose, not a standard: it is not + # registered with IANA for this, no protocol prescribes it, and Kubernetes + # knows nothing about it. Read for ztunnel 1.30.6 (measured on this + # cluster: ztunnel opens [::]:15008 as its inbound listener). + # + # If Istio changes it, a pod in the mesh simply cannot be reached. Change + # it here in all 30 charts AND in the pinned expectation inside + # ci/run-networkpolicy-test.sh - the test is deliberately the thing that + # forces someone to look, because the exception it grants this port is + # pinned to the number IT knows and not to whatever this value says. A + # value-driven exception would let a mistyped port through, which is the + # hole design A of #143 was rejected for. + # + # NOT MEASURED: that this port ALONE suffices comes from Istio's + # documentation. Measured here is only which listeners ztunnel opens. + # Provable at the first namespace to join the mesh; none carries + # istio.io/dataplane-mode yet. + port: 15008 + + # SOURCES THE KUBELET'S PROBES ARRIVE FROM ONCE ISTIO HAS REDIRECTED THEM. + # + # NOT THE SAME VALUE AS networkPolicy.probeSources one level up, and the + # two are easy to confuse, so: + # + # networkPolicy.probeSources + # YOUR worker node CIDRs. Empty by default, and empty means the + # probe rule is OPEN. The chart cannot know these addresses. + # networkPolicy.ingress.ambientMesh.redirectedProbeSources + # ISTIO's rewrite address. Shipped filled, because unlike the node + # IPs it is a constant the chart CAN know. Only rendered where the + # rule above is restricted, i.e. where probeSources is non-empty. + # + # WHERE 169.254.7.127 COMES FROM, so it is recognisable as vendor magic + # rather than as protocol: + # + # - It is LINK-LOCAL, 169.254.0.0/16 (RFC 3927): a range reserved for + # node-local use that is never routed - exactly the range one picks a + # vendor's magic address out of. + # - Istio's IPv6 counterpart gives it away: + # fd16:9254:7127:1337:ffff:ffff:ffff:ffff carries the SAME DIGITS + # (9254:7127 is 169.254.7.127) plus 1337. Numbers built that way are + # vendor constants, not protocols. + # - KUBERNETES DOES NOT KNOW THIS ADDRESS. Neither the NetworkPolicy API + # nor the kubelet has any notion of it; only Istio's own data plane + # rewrites the probe source onto it. + # + # Read for ztunnel 1.30.6. IF ISTIO CHANGES IT, THE FAILURE IS SILENT: + # probes would be dropped, and only where probeSources is non-empty - which + # is nowhere in this repository today - so nothing here and nothing in + # Kubernetes would report it. What to re-check then: the address Istio's + # release notes name for probe redirection, and this list in all 30 charts + # plus the pinned expectation in ci/run-networkpolicy-test.sh. + # + # A LIST, not a string, and this is the one case where that does not + # contradict "a constant, not a configuration": on a DUAL-STACK cluster + # BOTH entries are needed at the same time, not one of two. This cluster is + # SingleStack IPv4, so the IPv6 counterpart is left out - an ipBlock of the + # wrong family is accepted by the API server and matches nothing. Add it + # ALONGSIDE, never instead, if the cluster gains IPv6. + # + # Clearing this list restores the default rather than removing the + # ipBlock, deliberately: the two halves of this switch are not separable. + # To not open the port at all, set enabled: false. + # + # UNPROVEN IN OPERATION: probeSources is empty in every chart here, so no + # release reaches this branch. CI proves the render, not the effect. + redirectedProbeSources: + - 169.254.7.127/32 egress: # Additional outbound TCP ports. DNS (UDP and TCP 53) is always allowed and