jlab-cluster/helm-charts
3
0 Fork 0
Code Issues 15 Pull Requests 2 Actions Packages Wiki Activity

feat(template-chart)!: ambientMesh becomes an object, and the CI exception gets a pinned expectation

Browse Source 
Gitea #143, maintainer's instruction. Both numbers this switch renders are
ISTIO CONSTANTS rather than standards, and a value with a documented default
is the better home for a vendor constant than a template literal: an operator
can correct it when Istio changes the number, without waiting for a chart
release.

    networkPolicy:
      ingress:
        ambientMesh:
          enabled: false
          port: 15008
          redirectedProbeSources:
            - 169.254.7.127/32

One object under one switch, so the coupling argument survives the change
instead of being undermined by it - two separate values could drift apart,
which is the thing this design exists to prevent. redirectedProbeSources is a
LIST because a dual-stack cluster needs BOTH of Istio's addresses at once, not
one of two; that is a nameable case rather than speculation, so it does not
contradict reason 4 of the issue. Istio's IPv6 counterpart is documented as the
dual-stack addition, with the reason it is absent today unchanged: SingleStack
IPv4, and an ipBlock of the wrong family is accepted and matches nothing.

Named redirectedProbeSources, not probeSources, because the confusion risk with
networkPolicy.probeSources one level up is larger than the ingress/
ingressController one: that value holds YOUR worker node CIDRs, is empty by
default and means OPEN when empty; this one holds ISTIO's rewrite address and
ships filled, because unlike the node IPs it is a constant the chart can know.
The distinction is written where both appear.

PROVENANCE, which the previous round explained the coupling of but never
attributed. 15008 is Istio's HBONE port - not IANA-registered for this, not
prescribed by any protocol, unknown to Kubernetes. 169.254.7.127 is link-local
(169.254.0.0/16, RFC 3927, never routed - the range one picks vendor magic out
of); Istio's IPv6 counterpart fd16:9254:7127:1337:ffff:ffff:ffff:ffff carries
the same digits plus 1337, which is what vendor constants look like; and
Kubernetes knows the address in neither the NetworkPolicy API nor the kubelet -
only Istio's data plane rewrites the probe source onto it. Both read for
ztunnel 1.30.6. If either changes, the failure is SILENT: a pod unreachable in
the mesh, or probes dropped where probeSources is non-empty, which is nowhere
today.

NULL-SAFE, per #99 and because #136 shipped this exact defect one value away
(its NOTES.txt branch dereferenced .networkPolicy.egress directly): every read
goes through a fallback reproducing the chart's own declared defaults, so a
deleted object or one carrying only "enabled: true" renders the defaults rather
than failing. Clearing the probe-source list restores the constant instead of
removing the ipBlock, deliberately - the halves are not separable, and
enabled: false is how you open nothing.

THE CONSEQUENCE, and it is the point of the CI half. With the port
configurable, the exception can no longer be checked against the rendered
value: "whatever the value says" is design A's hole wearing design C's name,
and a mistyped port would pass by sitting in the configurable field. So the
expectation is PINNED IN THE TEST - AMBIENT_PORT and AMBIENT_CIDR - and
assertion 7 now checks two things instead of one: that the render at chart
defaults produces exactly that port and that ipBlock, and that every chart's
values.yaml DECLARES the same defaults, read back through ci/yaml2json rather
than by regex. A chart that drifts is red. Changing an Istio constant means
touching 30 values files AND the pinned lines, and the pinned lines are what
force someone to look.

That block is therefore also the single place saying what we hold Istio's
numbers to be, so it carries the provenance too. The renders set only
enabled=true and never a port override, on purpose: the pin is a statement
about the defaults we ship, not about what a release may choose - an operator
overriding a value is outside this test's reach, as for every value here.

The declared-defaults check is conditional on the chart offering probeSources,
like the inseparability check and assertion 4: the five charts whose probe rule
is never source-restricted render no ipBlock, so for them the key must be
ABSENT - a value reaching no rule reads as a control that does nothing - and
that absence is asserted rather than tolerated. Their schemas drop it too.

No version change: nothing of this is published, the switch stays additive with
its default off, and the render at defaults is byte-identical - re-diffed per
chart against origin/main, not assumed.
This commit is contained in:
jklein committed 2026-10-19 17:56:58 +00:00
1 parent f2fda15e57
commit 22977c183f
4 files changed
+358 -98
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

ci/run-networkpolicy-test.sh
+152 -35
View File
@@ -1261,17 +1261,26 @@ done
# around, so the exception is held to five things at once, on TWO renders of # around, so the exception is held to five things at once, on TWO renders of
# every chart: # every chart:
# #
# the chart's DECLARED DEFAULTS (read back through ci/yaml2json)
# - enabled must default to false, and the port and the probe source must be
# exactly the numbers pinned below. A chart whose default drifted would
# otherwise ship a port no other chart ships while the exception kept being
# granted to the pinned one.
# switch OFF (the main loop's render, reused) # switch OFF (the main loop's render, reused)
# - 15008 must be ABSENT. Without this half the exception is a hole: a # - the pinned port must be ABSENT. Without this half the exception is a hole: a
# chart could render the port unconditionally and nothing would notice, # chart could render the port unconditionally and nothing would notice,
# because assertion 7 would find it present and assertion 2 is where it # because assertion 7 would find it present and assertion 2 is where it
# would have failed. # would have failed.
# - 169.254.7.127/32 must be absent too, for the same reason. # - 169.254.7.127/32 must be absent too, for the same reason.
# switch ON # switch ON (only "enabled=true" is set, never a port override - the pin is a
# - 15008/TCP must be PRESENT, or the value is decorative. # statement about the defaults WE ship)
# - 15008 must be the ONLY ingress port that is not a containerPort. This is # - the PINNED port must be PRESENT, or the value is decorative.
# the line between a named exception and design A of #143: every other # - the PINNED port must be the ONLY ingress port that is not a
# mistyped or NodePort number still fails here, with the switch on. # containerPort. This is the line between a named exception and design A
# of #143: every other mistyped or NodePort number still fails here, with
# the switch on. Note "pinned": since #143 made the port a VALUE, checking
# the exception against the rendered value would permit whatever the value
# said, which is design A's hole under design C's name.
# - wherever the rule carrying 15008 restricts sources, that rule's "from" # - wherever the rule carrying 15008 restricts sources, that rule's "from"
# must contain 169.254.7.127/32. This is the inseparability of the two # must contain 169.254.7.127/32. This is the inseparability of the two
# halves, asserted rather than trusted - and it is REACHABLE, because # halves, asserted rather than trusted - and it is REACHABLE, because
@@ -1314,6 +1323,45 @@ done
# mechanism as assertion 0: the mistake arrives by copy into a NEW chart whose # mechanism as assertion 0: the mistake arrives by copy into a NEW chart whose
# own diff nobody reads for it, which is exactly how garage (#81) and keycloak # own diff nobody reads for it, which is exactly how garage (#81) and keycloak
# (#83) arrived with no policy at all after #5 had been filed. # (#83) arrived with no policy at all after #5 had been filed.
# THE PINNED EXPECTATION, and it is pinned HERE rather than read from the chart
# on purpose (Gitea #143). Both numbers are values with documented defaults, so
# that an operator can correct an Istio constant without waiting for a chart
# release - but an exception that permits "whatever the value says" is design A's
# hole wearing design C's name: a mistyped port would pass simply by sitting in
# the configurable field, and that is the single reason design A was rejected.
#
# So the test states what WE believe Istio's numbers to be, and asserts two
# things against that statement: the render at CHART DEFAULTS produces exactly
# this port and exactly this ipBlock, AND every chart's values.yaml DECLARES
# these same defaults. A chart that drifts is red.
#
# Which makes this block the one place in the repository that says what we hold
# Istio's constants to be - and therefore also the answer to "where does
# 169.254.7.127 come from":
#
# 15008 Istio's HBONE port. Not registered with IANA for this,
# prescribed by no protocol, unknown to Kubernetes. Istio
# picked it and Istio can change it.
# 169.254.7.127/32 the address Istio's data plane rewrites the kubelet's probe
# source onto. LINK-LOCAL (169.254.0.0/16, RFC 3927), a range
# reserved for node-local use and never routed. Istio's IPv6
# counterpart fd16:9254:7127:1337:ffff:ffff:ffff:ffff carries
# the SAME DIGITS plus 1337, which is what vendor constants
# look like. Neither the NetworkPolicy API nor the kubelet has
# any notion of it.
#
# BOTH READ FOR ztunnel 1.30.6. If Istio changes either, the failure is silent -
# a pod unreachable in the mesh, or probes dropped where probeSources is
# non-empty, which is nowhere today. Changing them means 30 values files AND
# these two lines, and these two lines are what force someone to look.
#
# An operator who overrides the value in their own release is outside this test's
# reach. That is true of every value in this repository and is not a new gap -
# and it is why these renders set ONLY "enabled=true" and never a port override:
# the pin is a statement about the defaults WE ship, not about what a release may
# choose.
AMBIENT_PORT=15008
AMBIENT_CIDR="169.254.7.127/32"
AMBIENT_VALUE="networkPolicy.ingress.ambientMesh" AMBIENT_VALUE="networkPolicy.ingress.ambientMesh"
for CHART in "${CHARTS[@]}"; do for CHART in "${CHARTS[@]}"; do
# A chart whose main-loop render already failed has nothing to compare # A chart whose main-loop render already failed has nothing to compare
@@ -1326,10 +1374,10 @@ for CHART in "${CHARTS[@]}"; do
ROW_FAIL="" ROW_FAIL=""
ON_JSON="$WORK/ambient-$CHART-on.json" ON_JSON="$WORK/ambient-$CHART-on.json"
if ! RCJ_WITH_OVERLAY=1 render_chart_json "$CHART" "$ON_JSON" \ if ! RCJ_WITH_OVERLAY=1 render_chart_json "$CHART" "$ON_JSON" \
--set "$AMBIENT_VALUE=true"; then --set "$AMBIENT_VALUE.enabled=true"; then
sed 's/^/ /' "$WORK/xerr" sed 's/^/ /' "$WORK/xerr"
echo " FAIL $CHART does not render with $AMBIENT_VALUE=true" echo " FAIL $CHART does not render with $AMBIENT_VALUE.enabled=true"
echo "::error title=ambient-mesh $CHART::$CHART does not render with $AMBIENT_VALUE=true. Since Gitea #143 every chart in this repository offers that value, and networkPolicy carries additionalProperties: false in all of them - so an unknown key does not get ignored, it fails the render. Add the key to values.yaml AND to values.schema.json (schema and template belong together per chart), and render the ingress port 15008 plus, inside the source-restricted branch, the ipBlock 169.254.7.127/32. Copy template-chart/templates/xxx.networkpolicy.yaml." echo "::error title=ambient-mesh $CHART::$CHART does not render with $AMBIENT_VALUE.enabled=true. Since Gitea #143 every chart in this repository offers that value, and networkPolicy carries additionalProperties: false in all of them - so an unknown key does not get ignored, it fails the render. Add the object to values.yaml AND to values.schema.json (schema and template belong together per chart), with enabled: false, port: \$AMBIENT_PORT and redirectedProbeSources: [\$AMBIENT_CIDR], and render that port plus - inside the source-restricted branch - that ipBlock. Copy template-chart/templates/xxx.networkpolicy.yaml."
ROW_FAIL=1 ROW_FAIL=1
fi fi
@@ -1345,17 +1393,36 @@ for CHART in "${CHARTS[@]}"; do
ROW_FAIL=1 ROW_FAIL=1
fi fi
# The chart's DECLARED defaults, read with the same parser as everything else
# (ci/yaml2json prints a values file back as JSON). Not a regex over
# values.yaml: the pinned expectation is about numbers and a list, and a
# substring test over a commented file is how the probeSources check in
# assertion 4 was once fooled by a deleted key whose comment survived.
VALUES_JSON="$WORK/$CHART.values.json"
if [ -z "$ROW_FAIL" ]; then
if ! helm template y2j "$YAML2JSON" -f "$ROOT/$CHART/values.yaml" 2>"$WORK/xerr" \
| awk '/^\{/ { print; exit }' >"$VALUES_JSON" || [ ! -s "$VALUES_JSON" ]; then
sed 's/^/ /' "$WORK/xerr"
echo " FAIL $CHART/values.yaml could not be re-read as JSON"
echo "::error title=ambient-mesh $CHART::$CHART/values.yaml could not be re-read by ci/yaml2json, so the chart's DECLARED ambient-mesh defaults could not be compared with the numbers this test pins. That is a parse failure, not a policy finding."
ROW_FAIL=1
fi
fi
if [ -z "$ROW_FAIL" ]; then if [ -z "$ROW_FAIL" ]; then
set +e set +e
python3 - "$CHART" "$RELEASE" "$AMBIENT_VALUE" "$WORK/$CHART.json" "$ON_JSON" \ python3 - "$CHART" "$RELEASE" "$AMBIENT_VALUE" "$WORK/$CHART.json" "$ON_JSON" \
"$ROOT/$CHART" "$CHART_NAME" \ "$ROOT/$CHART" "$CHART_NAME" "$AMBIENT_PORT" "$AMBIENT_CIDR" "$VALUES_JSON" \
>"$WORK/ambient-$CHART.out" 2>"$WORK/ambient-$CHART.pyerr" <<'PY' >"$WORK/ambient-$CHART.out" 2>"$WORK/ambient-$CHART.pyerr" <<'PY'
import json, re, sys import json, re, sys
chart, release, gate, off_path, on_path, chart_dir, chart_name = sys.argv[1:8] (chart, release, gate, off_path, on_path, chart_dir, chart_name,
pinned_port, pinned_cidr, values_json) = sys.argv[1:11]
PORT = 15008 # Pinned by the caller, never read from the chart - see the long note above
CIDR = "169.254.7.127/32" # AMBIENT_PORT in this script.
PORT = int(pinned_port)
CIDR = pinned_cidr
lines = [] lines = []
def ok(msg): lines.append("OK|%s: %s" % (chart, msg)) def ok(msg): lines.append("OK|%s: %s" % (chart, msg))
@@ -1451,27 +1518,77 @@ def cidrs_of(rule):
return out return out
# --- The value is declared, in BOTH places it has to be -------------------- # --- The chart DECLARES the pinned defaults -------------------------------
# The render with the switch on already proves the SCHEMA declares it - an # The render with the switch on already proves the SCHEMA declares the key - an
# undeclared key cannot pass additionalProperties: false. What that render # undeclared key cannot pass additionalProperties: false. What that render
# cannot prove is that values.yaml ships the documented default, and a value # cannot prove is what the chart's own defaults SAY, and since #143 made both
# only reachable through --set is not a value an operator finds. # Istio constants configurable, that is the half the pinned exception rests on:
# the exception below is granted to the number THIS SCRIPT names, so a chart
# whose default has drifted away from it must be red here rather than quietly
# carrying a different port than every other chart.
values_yaml = "" values_yaml = ""
try: try:
values_yaml = open("%s/values.yaml" % chart_dir, encoding="utf-8").read() values_yaml = open("%s/values.yaml" % chart_dir, encoding="utf-8").read()
except IOError: except IOError:
pass pass
if re.search(r"^\s*ambientMesh\s*:", values_yaml, re.M):
ok("values.yaml ships the ambientMesh key with its default, and the render " declared = json.load(open(values_json))
"with %s=true succeeded - which is itself the proof that " node = (((declared.get("networkPolicy") or {}).get("ingress") or {})
"values.schema.json declares it, since networkPolicy carries " .get("ambientMesh"))
"additionalProperties: false" % gate) if not isinstance(node, dict):
fail("renders with %s.enabled=true - so the schema knows the key - but the "
"chart's own values.yaml declares no ambientMesh OBJECT (found %r). A "
"value that exists only in the schema is reachable through --set and "
"findable nowhere: the defaults are what an operator reads. Ship the "
"object with enabled: false, port: %d and redirectedProbeSources: "
"[%s], as in template-chart." % (gate, node, PORT, CIDR))
else: else:
fail("renders with %s=true - so the schema knows the key - but values.yaml " problems = []
"contains no ambientMesh key at all. A value that exists only in the " if node.get("enabled") not in (False, None):
"schema is reachable through --set and findable nowhere: the defaults " problems.append("enabled is %r and must default to false - the switch is "
"are what an operator reads. Add it with its comment, default false, " "additive and no release may get it by accident"
"as in template-chart." % gate) % node.get("enabled"))
if node.get("port") != PORT:
problems.append("port defaults to %r, but this test pins Istio's HBONE "
"port at %d" % (node.get("port"), PORT))
# Conditional on the chart offering probeSources, exactly as the
# inseparability check below and assertion 4 are. A chart whose probe rule
# is never source-restricted renders no ipBlock, so this key would reach no
# rule - and a value that silently does nothing reads as a control, which is
# worse than one that is absent (README, "Values, and shipping only the live
# ones"). So for those charts the key must be ABSENT, and that is asserted
# rather than tolerated.
sources = node.get("redirectedProbeSources")
offers_ps = bool(re.search(r"^\s*probeSources\s*:", values_yaml, re.M))
if offers_ps and sources != [CIDR]:
problems.append("redirectedProbeSources defaults to %r, but this test "
"pins Istio's redirected probe source at [%r]"
% (sources, CIDR))
if not offers_ps and sources is not None:
problems.append("redirectedProbeSources is declared (%r) although this "
"chart offers no networkPolicy.probeSources, so its probe "
"rule is never source-restricted and no ipBlock is ever "
"rendered - the key would reach no rule and read as a "
"control that does nothing" % sources)
if problems:
fail("the chart's declared ambient-mesh defaults do not match the "
"numbers ci/run-networkpolicy-test.sh pins: %s. Both are ISTIO "
"CONSTANTS, and they are values precisely so an operator can "
"correct them without a chart release - but the exception assertion "
"2 grants the HBONE port is pinned to the number THIS TEST knows, "
"never to whatever the value says, because a value-driven exception "
"would let a mistyped port ride in through a configurable field. "
"That is the hole design A of #143 was rejected for. So: if Istio "
"changed the constant, change it in all 30 charts AND in the pinned "
"expectation here - the drift is meant to be loud."
% "; ".join(problems))
else:
ok("declares the pinned ambient-mesh defaults verbatim - enabled false, "
"port %d, %s - so the named exception below is granted to a number "
"this chart really ships, and not to whatever a value happened to say"
% (PORT, ("redirectedProbeSources [%s]" % CIDR) if offers_ps else
"and no redirectedProbeSources, correctly, since this chart "
"renders no ipBlock at all"))
off_policies = own_policies(off_path) off_policies = own_policies(off_path)
on_policies = own_policies(on_path) on_policies = own_policies(on_path)
@@ -1496,20 +1613,20 @@ else:
if cidr == CIDR: if cidr == CIDR:
off_cidr_hits.append("%s ingress[%d]" % (name, i)) off_cidr_hits.append("%s ingress[%d]" % (name, i))
if off_hits: if off_hits:
fail("allows ingress port %d with %s at its default (false): %s. The " fail("allows ingress port %d with %s.enabled at its default (false): %s. The "
"exception assertion 2 grants for %d is gated on that value, and " "exception assertion 2 grants for %d is gated on that value, and "
"this is the half that keeps it from being a hole - a port rendered " "this is the half that keeps it from being a hole - a port rendered "
"unconditionally would otherwise be found present by this assertion " "unconditionally would otherwise be found present by this assertion "
"and never held against the pod spec by assertion 2. Gate the port " "and never held against the pod spec by assertion 2. Gate the port "
"on the value." % (PORT, gate, ", ".join(off_hits), PORT)) "on the value." % (PORT, gate, ", ".join(off_hits), PORT))
elif off_cidr_hits: elif off_cidr_hits:
fail("carries the ipBlock %s with %s at its default (false): %s. The " fail("carries the ipBlock %s with %s.enabled at its default (false): %s. The "
"probe-source half of the switch is for the mesh only, where the " "probe-source half of the switch is for the mesh only, where the "
"kubelet's probes are redirected; outside it the address matches " "kubelet's probes are redirected; outside it the address matches "
"nothing and makes the rule read as if it did something." "nothing and makes the rule read as if it did something."
% (CIDR, gate, ", ".join(off_cidr_hits))) % (CIDR, gate, ", ".join(off_cidr_hits)))
else: else:
ok("with %s at its default the policy allows neither port %d nor the " ok("with %s.enabled at its default the policy allows neither port %d nor the "
"ipBlock %s - the negative half of assertion 2's named exception, " "ipBlock %s - the negative half of assertion 2's named exception, "
"without which the exception would be a hole" % (gate, PORT, CIDR)) "without which the exception would be a hole" % (gate, PORT, CIDR))
@@ -1544,18 +1661,18 @@ else:
"the selected pod spec" % (name, i, value)) "the selected pod spec" % (name, i, value))
if not found_port: if not found_port:
fail("renders no ingress port %d with %s=true, so the value is " fail("renders no ingress port %d with %s.enabled=true, so the value is "
"decorative: the chart accepts it, the schema declares it, and the " "decorative: the chart accepts it, the schema declares it, and the "
"policy that comes out is the same one as before. A pod whose " "policy that comes out is the same one as before. A pod whose "
"namespace has joined the ambient mesh is then unreachable, because " "namespace has joined the ambient mesh is then unreachable, because "
"ambient does not bypass NetworkPolicy enforcement." % (PORT, gate)) "ambient does not bypass NetworkPolicy enforcement." % (PORT, gate))
else: else:
ok("with %s=true the policy allows %s - ztunnel's HBONE tunnel, the one " ok("with %s.enabled=true the policy allows %s - ztunnel's HBONE tunnel, the one "
"port assertion 2 grants a named exception for" "port assertion 2 grants a named exception for"
% (gate, ", ".join(found_port))) % (gate, ", ".join(found_port)))
if bad_ports: if bad_ports:
fail("with %s=true the policy allows ingress port(s) that are NOT " fail("with %s.enabled=true the policy allows ingress port(s) that are NOT "
"containerPorts of the selected pod spec and are NOT the one number " "containerPorts of the selected pod spec and are NOT the one number "
"the exception covers (%d): %s. This is the line between a named " "the exception covers (%d): %s. This is the line between a named "
"exception and the free port key that was rejected as design A of " "exception and the free port key that was rejected as design A of "
@@ -1563,7 +1680,7 @@ else:
"including a NodePort number written where a pod port belongs." "including a NodePort number written where a pod port belongs."
% (gate, PORT, "; ".join(bad_ports))) % (gate, PORT, "; ".join(bad_ports)))
elif found_port: elif found_port:
ok("with %s=true, port %d is still the ONLY ingress port of this chart " ok("with %s.enabled=true, port %d is still the ONLY ingress port of this chart "
"that is not a containerPort of the selected pod spec - the exception " "that is not a containerPort of the selected pod spec - the exception "
"covers exactly one number and nothing rode in beside it" "covers exactly one number and nothing rode in beside it"
% (gate, PORT)) % (gate, PORT))
@@ -1599,7 +1716,7 @@ else:
"%s ingress[%d] (peers: %s)" "%s ingress[%d] (peers: %s)"
% (name, i, ", ".join(cidrs_of(rule)) or "no ipBlock at all")) % (name, i, ", ".join(cidrs_of(rule)) or "no ipBlock at all"))
if restricted_without_cidr: if restricted_without_cidr:
fail("with %s=true the rule carrying port %d restricts its sources but " fail("with %s.enabled=true the rule carrying port %d restricts its sources but "
"does not include the ipBlock %s: %s. Inside the mesh the kubelet's " "does not include the ipBlock %s: %s. Inside the mesh the kubelet's "
"probes are redirected and arrive from that address instead of from " "probes are redirected and arrive from that address instead of from "
"the worker node's, so a restricted rule without it drops every " "the worker node's, so a restricted rule without it drops every "
template-chart/templates/xxx.networkpolicy.yaml
+89 -27
View File
@@ -113,7 +113,23 @@ spec this policy selects.
round of this work already reported ingressController as an inbound round of this work already reported ingressController as an inbound
escape hatch on the strength of the substring alone. */}} escape hatch on the strength of the substring alone. */}}
{{- $npIngress := $np.ingress | default dict }} {{- $npIngress := $np.ingress | default dict }}
{{- $ambientMesh := $npIngress.ambientMesh }} {{- /* NULL-SAFE ON PURPOSE, and not as decoration (#99). ambientMesh is an
OBJECT, so every read below goes through a fallback that reproduces THIS
chart's declared defaults - because an operator who deletes the object,
or who writes only "enabled: true" and leaves the rest out, must land on
the defaults and not in a nil dereference. #136 shipped exactly that
defect one value away from here: the NOTES.txt branch dereferenced
.networkPolicy.egress directly, and a release that had dropped the key
failed to render.
Note which fallback the probe-source list gets: "default" treats an EMPTY
list as unset, so clearing it restores the constant rather than removing
the ipBlock. That is deliberate - the two halves of this switch are not
separable, and the way to not open the port at all is enabled: false. */}}
{{- $ambient := $npIngress.ambientMesh | default dict }}
{{- $ambientEnabled := $ambient.enabled }}
{{- $ambientPort := $ambient.port | default 15008 }}
{{- $ambientProbeSources := $ambient.redirectedProbeSources | default (list "169.254.7.127/32") }}
{{- $routed := or .Values.ingress.enabled .Values.httpRoute.enabled }} {{- $routed := or .Values.ingress.enabled .Values.httpRoute.enabled }}
apiVersion: networking.k8s.io/v1 apiVersion: networking.k8s.io/v1
kind: NetworkPolicy kind: NetworkPolicy
@@ -147,20 +163,36 @@ spec:
- protocol: {{ .protocol | default "TCP" }} - protocol: {{ .protocol | default "TCP" }}
port: {{ .containerPort }} port: {{ .containerPort }}
{{- end }} {{- end }}
{{- if $ambientMesh }} {{- if $ambientEnabled }}
{{- /* FIRST HALF OF networkPolicy.ingress.ambientMesh. 15008/TCP is {{- /* FIRST HALF OF networkPolicy.ingress.ambientMesh. The default is
ztunnel's HBONE tunnel, the port every inbound mesh connection 15008/TCP: ztunnel's HBONE tunnel, the port every inbound mesh
arrives on once the namespace carries istio.io/dataplane-mode. connection arrives on once the namespace carries
Istio's own documentation is explicit that ambient "does not and istio.io/dataplane-mode. Istio's own documentation is explicit
will never bypass Kubernetes NetworkPolicy enforcement", so that ambient "does not and will never bypass Kubernetes
without this port a pod in the mesh is simply unreachable. NetworkPolicy enforcement", so without this port a pod in the
mesh is simply unreachable.
15008 IS AN ISTIO CONSTANT, NOT A STANDARD. It is not registered
with IANA for this, Kubernetes knows nothing about it, and no
protocol prescribes it - Istio picked it for HBONE and Istio can
change it. That is precisely why it sits in a VALUE with a
documented default rather than as a literal here: an operator can
correct it without waiting for a chart release.
It is the one port in this rule that is NOT a containerPort of It is the one port in this rule that is NOT a containerPort of
the selected pod spec - ztunnel terminates it, not the the selected pod spec - ztunnel terminates it, not the
application - so ci/run-networkpolicy-test.sh holds a NAMED application - so ci/run-networkpolicy-test.sh holds a NAMED
exception for exactly this number, gated on exactly this value. exception for it. Note what that exception is pinned to: the
Anything else that is not a containerPort still fails, and so number 15008 as written in THE TEST, plus an assertion that this
does 15008 itself while the value is off. chart's values.yaml declares the same default. It is NOT "whatever
this value happens to say" - that would let a mistyped port ride
in through a configurable field, which is the hole design A of
#143 was rejected for.
READ FOR ztunnel 1.30.6. If Istio changes the port, or the mesh is
replaced: the failure is a pod that cannot be reached inside the
mesh. Change the default in all 30 charts AND the pinned number in
the test - which is the thing that forces someone to look.
NOT MEASURED, and the chart should not pretend otherwise: that NOT MEASURED, and the chart should not pretend otherwise: that
15008 ALONE suffices comes from Istio's documentation. What has 15008 ALONE suffices comes from Istio's documentation. What has
@@ -168,7 +200,7 @@ spec:
opens. Provable at the first namespace that joins the mesh, and opens. Provable at the first namespace that joins the mesh, and
no namespace carries the label yet. */}} no namespace carries the label yet. */}}
- protocol: TCP - protocol: TCP
port: 15008 port: {{ int $ambientPort }}
{{- end }} {{- end }}
{{- if $probeSources }} {{- if $probeSources }}
{{- /* RESTRICTED, because the operator supplied probeSources. The {{- /* RESTRICTED, because the operator supplied probeSources. The
@@ -209,34 +241,64 @@ spec:
- ipBlock: - ipBlock:
cidr: {{ . | quote }} cidr: {{ . | quote }}
{{- end }} {{- end }}
{{- if $ambientMesh }} {{- if $ambientEnabled }}
{{- /* SECOND HALF OF networkPolicy.ingress.ambientMesh, and the reason {{- /* SECOND HALF OF networkPolicy.ingress.ambientMesh, and the reason
the switch is one value instead of two. Inside the mesh the the switch is one value instead of two. Inside ISTIO's ambient
kubelet's probes are redirected and no longer arrive from the mesh the kubelet's probes are redirected BY ISTIO and no longer
worker node's address: they arrive from 169.254.7.127, which is arrive from the worker node's address: they arrive from
therefore the probeSources entry nobody would think to add. 169.254.7.127, which is therefore the probe source nobody would
think to add.
The two halves are deliberately inseparable. An operator who WHERE THAT ADDRESS COMES FROM, because it is worth being able to
only opens 15008 by hand has a policy that passes every review - recognise as vendor magic rather than as protocol:
and the moment probeSources is non-empty, every probe is dropped,
the pod never becomes Ready, and the symptom reads as a broken - It is LINK-LOCAL, 169.254.0.0/16 (RFC 3927) - a range
reserved for node-local use that is never routed. Exactly
the range one picks a vendor's magic address out of.
- Istio's IPv6 counterpart gives the game away:
fd16:9254:7127:1337:ffff:ffff:ffff:ffff carries the SAME
DIGITS - 9254:7127 is 169.254.7.127 - plus 1337. Numbers
built that way are vendor constants, not protocols.
- KUBERNETES DOES NOT KNOW THIS ADDRESS. Neither the
NetworkPolicy API nor the kubelet has any notion of it; only
Istio's own data plane rewrites the probe source onto it.
READ FOR ztunnel 1.30.6. If Istio changes it, or the mesh is
replaced, THE FAILURE IS SILENT: probes would be dropped, and
only where probeSources is non-empty - which is nowhere today.
Nothing in this repository or in Kubernetes would report it. That
is why the address is a VALUE with a documented default, so it can
be corrected without a chart release, and why the test pins the
number it expects: changing it means touching 30 values files and
the test, and the test is what makes someone look.
A LIST, not a string, and this is the one case where that does not
contradict "the value is a constant, not a configuration": on a
DUAL-STACK cluster BOTH entries are needed at the same time, not
one of two. This cluster is SingleStack IPv4, so the IPv6
counterpart above is left out of the default - an ipBlock of the
wrong family is accepted by the API server and matches nothing.
Add it alongside, never instead, if the cluster gains IPv6.
The two halves stay inseparable. An operator who only opens the
port by hand has a policy that passes every review - and the
moment probeSources is non-empty, every probe is dropped, the pod
never becomes Ready, and the symptom reads as a broken
application rather than as a policy. That is assertion 4 of application rather than as a policy. That is assertion 4 of
ci/run-networkpolicy-test.sh, re-created one value further along. ci/run-networkpolicy-test.sh, re-created one value further along.
One switch cannot be set halfway.
Only rendered in this branch, because this is the only branch Only rendered in this branch, because this is the only branch
with a "from" list at all: with probeSources empty the rule is with a "from" list at all: with probeSources empty the rule is
open and needs no peer. The IPv6 counterpart open and needs no peer.
fd16:9254:7127:1337:ffff:ffff:ffff:ffff is deliberately absent -
this cluster is SingleStack IPv4, and an ipBlock of the wrong
family is accepted by the API server and matches nothing.
UNPROVEN IN OPERATION: probeSources is empty in every chart of UNPROVEN IN OPERATION: probeSources is empty in every chart of
this repository (stalwart sets it to the empty list explicitly), this repository (stalwart sets it to the empty list explicitly),
so no release reaches this branch today. What CI proves is its so no release reaches this branch today. What CI proves is its
render, through the overlay in ci/networkpolicy. */}} render, through the overlay in ci/networkpolicy. */}}
{{- range $ambientProbeSources }}
- ipBlock: - ipBlock:
cidr: 169.254.7.127/32 cidr: {{ . | quote }}
{{- end }}
{{- end }} {{- end }}
{{- else }} {{- else }}
{{- /* NO "from" SELECTOR, WHICH MEANS OPEN - deliberately, and only {{- /* NO "from" SELECTOR, WHICH MEANS OPEN - deliberately, and only
template-chart/values.schema.json
+22 -2
View File
@@ -51,8 +51,28 @@
"additionalProperties": false, "additionalProperties": false,
"properties": { "properties": {
"ambientMesh": { "ambientMesh": {
"description": "Opens the pod for Istio's ambient mesh; off by default and additive, so the render is byte-identical while it is false. Renders 15008/TCP inbound - ztunnel's HBONE tunnel, which ambient explicitly does not let bypass NetworkPolicy enforcement - AND, where probeSources is non-empty, 169.254.7.127/32 as an additional probe source, because inside the mesh the kubelet's probes are redirected away from the worker node's address. The two halves are inseparable on purpose: whoever opens 15008 by hand has not added the probe address, and the first narrowed probeSources then drops every probe, so the pod never becomes Ready and the symptom reads as a broken application rather than as a policy. Unproven in operation: that 15008 alone suffices comes from Istio's documentation, and no namespace of this cluster carries istio.io/dataplane-mode yet.", "description": "Opens the pod for Istio's ambient mesh; off by default and additive, so the render is byte-identical while enabled is false. Renders the HBONE port inbound AND, where networkPolicy.probeSources is non-empty, the redirected probe source in the same \"from\" list. The two halves live in one object under one switch on purpose: whoever opens the port by hand has not added the probe address, and the first narrowed probeSources then drops every probe, so the pod never becomes Ready and the symptom reads as a broken application rather than as a policy. Both numbers are ISTIO CONSTANTS rather than standards, which is why they are values with documented defaults and not template literals - a vendor constant can change, and an operator should be able to correct it without waiting for a chart release. Deleting the whole object is safe: every read falls back to the defaults the chart declares. Unproven in operation: that the HBONE port alone suffices comes from Istio's documentation, and no namespace of this cluster carries istio.io/dataplane-mode yet.",
"type": ["boolean", "null"] "type": ["object", "null"],
"additionalProperties": false,
"properties": {
"enabled": {
"description": "The switch. Everything else in this object is inert while it is false.",
"type": ["boolean", "null"]
},
"port": {
"description": "Istio's HBONE port, default 15008. A constant Istio chose, not a standard: not registered with IANA for this, prescribed by no protocol, and unknown to Kubernetes. Read for ztunnel 1.30.6. It is the one ingress port of these charts that is deliberately no containerPort - ztunnel terminates it, not the application - and the exception ci/run-networkpolicy-test.sh grants it is pinned to the number THAT TEST knows, plus an assertion that this chart declares the same default. Not to whatever this value says: a value-driven exception would let a mistyped port ride in through a configurable field.",
"type": ["integer", "null"],
"minimum": 1,
"maximum": 65535
},
"redirectedProbeSources": {
"description": "Sources the kubelet's probes arrive from once Istio has redirected them, default [169.254.7.127/32]. NOT the same value as networkPolicy.probeSources one level up: that one holds YOUR worker node CIDRs, is empty by default and means OPEN when empty, and the chart cannot know those addresses; this one holds ISTIO's rewrite address and ships filled, because unlike the node IPs it is a constant the chart can know. Recognisable as vendor magic rather than protocol: it is link-local (169.254.0.0/16, RFC 3927, never routed), Istio's IPv6 counterpart fd16:9254:7127:1337:ffff:ffff:ffff:ffff carries the same digits plus 1337, and Kubernetes itself knows the address in neither the NetworkPolicy API nor the kubelet. A list and not a string because a dual-stack cluster needs BOTH entries at once; this cluster is SingleStack IPv4, so the IPv6 counterpart is left out of the default - an ipBlock of the wrong family is accepted and matches nothing. If Istio changes the address the failure is SILENT: probes dropped, and only where probeSources is non-empty.",
"type": ["array", "null"],
"items": {
"type": "string"
}
}
}
} }
} }
}, },
template-chart/values.yaml
+95 -34
View File
@@ -283,50 +283,111 @@ networkPolicy:
# pods. A SIBLING of this block and NO PART # pods. A SIBLING of this block and NO PART
# OF IT, despite the name. # OF IT, despite the name.
ingress: ingress:
# Opens the pod for Istio's ambient mesh. OFF by default, and additive: # Opens the pod for Istio's ambient mesh. OFF by default, and additive: with
# with it off this chart renders byte for byte as it did before the value # it off this chart renders byte for byte as it did before the value existed.
# existed.
# #
# WHAT IT RENDERS, and the two halves are the point: # WHAT IT RENDERS, and the two halves are the point:
# #
# 1. 15008/TCP inbound - ztunnel's HBONE tunnel. Every inbound mesh # 1. The HBONE port inbound (default 15008/TCP) - ztunnel's tunnel. Every
# connection arrives on it, and Istio's documentation is explicit that # inbound mesh connection arrives on it, and Istio's documentation is
# ambient "does not and will never bypass Kubernetes NetworkPolicy # explicit that ambient "does not and will never bypass Kubernetes
# enforcement". Without this port a pod whose namespace has joined the # NetworkPolicy enforcement". Without this port a pod whose namespace
# mesh is unreachable. # has joined the mesh is unreachable.
# 2. 169.254.7.127/32 as an additional probe source - but ONLY where # 2. The redirected probe source (default 169.254.7.127/32) - but ONLY
# probeSources is non-empty, because only then does the rule have a # where networkPolicy.probeSources is non-empty, because only then does
# "from" list at all. Inside the mesh the kubelet's probes are # the rule have a "from" list at all. Inside the mesh the kubelet's
# redirected and no longer come from the worker node's address. # probes are redirected and no longer come from the worker node's
# address.
# #
# WHY ONE SWITCH AND NOT TWO. Point 2 is the one nobody thinks of. An # WHY ONE SWITCH AND NOT TWO. Point 2 is the one nobody thinks of. An
# operator who adds "15008" to a port list by hand writes a policy that # operator who adds "15008" to a port list by hand writes a policy that
# passes review, and the first time probeSources is narrowed every probe is # passes review, and the first time probeSources is narrowed every probe is
# dropped: the pod never becomes Ready and the symptom reads as a broken # dropped: the pod never becomes Ready and the symptom reads as a broken
# application rather than as a policy. A switch whose halves can only be # application rather than as a policy. A switch whose halves can only be had
# had together cannot be set halfway - which is why this is a named # together cannot be set halfway - which is why the two live in ONE object
# intention and not a free port list. # under one "enabled", and not as separate values that can drift apart.
# #
# TWO THINGS IT DOES NOT PROVE, and they belong next to the value rather # BOTH NUMBERS BELOW ARE ISTIO CONSTANTS, NOT STANDARDS, and they are values
# than in a commit message: # rather than template literals for exactly that reason: a vendor constant
# can change, and an operator should be able to correct it without waiting
# for a chart release. Their provenance is written on each one.
# #
# - That 15008 alone is enough. That comes from Istio's documentation. # DELETING THIS OBJECT IS SAFE. Every read falls back to the defaults
# Measured on this cluster is only which listeners ztunnel opens # declared here (#99), so "ambientMesh: null", or an object carrying only
# ([::]:15008 inbound, plus its own admin and stats ports). Provable at # "enabled: true", renders the documented defaults instead of failing.
# the first namespace that joins the mesh; none carries ambientMesh:
# istio.io/dataplane-mode yet. # The switch. Everything else in this object is inert while it is false,
# - That the probe-source half works. probeSources is empty in every # and the chart renders exactly as it did before the object existed.
# chart here, so no release reaches that branch. CI proves its render, enabled: false
# not its effect.
# # ISTIO'S HBONE PORT. A constant Istio chose, not a standard: it is not
# If the policy restricts sources, note that 15008 is restricted with them: # registered with IANA for this, no protocol prescribes it, and Kubernetes
# the port joins the rule that carries the probe port, so it is reachable # knows nothing about it. Read for ztunnel 1.30.6 (measured on this
# from that rule's peers only. Whether ztunnel needs a peer of its own is # cluster: ztunnel opens [::]:15008 as its inbound listener).
# open - it runs with its own pod IPs in istio-system (measured: no #
# hostNetwork on the DaemonSet, Istio 1.30.6), so if one turns out to be # If Istio changes it, a pod in the mesh simply cannot be reached. Change
# needed it is expressible as a namespaceSelector plus podSelector and does # it here in all 30 charts AND in the pinned expectation inside
# not need an unknowable node-IP CIDR. # ci/run-networkpolicy-test.sh - the test is deliberately the thing that
ambientMesh: false # forces someone to look, because the exception it grants this port is
# pinned to the number IT knows and not to whatever this value says. A
# value-driven exception would let a mistyped port through, which is the
# hole design A of #143 was rejected for.
#
# NOT MEASURED: that this port ALONE suffices comes from Istio's
# documentation. Measured here is only which listeners ztunnel opens.
# Provable at the first namespace to join the mesh; none carries
# istio.io/dataplane-mode yet.
port: 15008
# SOURCES THE KUBELET'S PROBES ARRIVE FROM ONCE ISTIO HAS REDIRECTED THEM.
#
# NOT THE SAME VALUE AS networkPolicy.probeSources one level up, and the
# two are easy to confuse, so:
#
# networkPolicy.probeSources
# YOUR worker node CIDRs. Empty by default, and empty means the
# probe rule is OPEN. The chart cannot know these addresses.
# networkPolicy.ingress.ambientMesh.redirectedProbeSources
# ISTIO's rewrite address. Shipped filled, because unlike the node
# IPs it is a constant the chart CAN know. Only rendered where the
# rule above is restricted, i.e. where probeSources is non-empty.
#
# WHERE 169.254.7.127 COMES FROM, so it is recognisable as vendor magic
# rather than as protocol:
#
# - It is LINK-LOCAL, 169.254.0.0/16 (RFC 3927): a range reserved for
# node-local use that is never routed - exactly the range one picks a
# vendor's magic address out of.
# - Istio's IPv6 counterpart gives it away:
# fd16:9254:7127:1337:ffff:ffff:ffff:ffff carries the SAME DIGITS
# (9254:7127 is 169.254.7.127) plus 1337. Numbers built that way are
# vendor constants, not protocols.
# - KUBERNETES DOES NOT KNOW THIS ADDRESS. Neither the NetworkPolicy API
# nor the kubelet has any notion of it; only Istio's own data plane
# rewrites the probe source onto it.
#
# Read for ztunnel 1.30.6. IF ISTIO CHANGES IT, THE FAILURE IS SILENT:
# probes would be dropped, and only where probeSources is non-empty - which
# is nowhere in this repository today - so nothing here and nothing in
# Kubernetes would report it. What to re-check then: the address Istio's
# release notes name for probe redirection, and this list in all 30 charts
# plus the pinned expectation in ci/run-networkpolicy-test.sh.
#
# A LIST, not a string, and this is the one case where that does not
# contradict "a constant, not a configuration": on a DUAL-STACK cluster
# BOTH entries are needed at the same time, not one of two. This cluster is
# SingleStack IPv4, so the IPv6 counterpart is left out - an ipBlock of the
# wrong family is accepted by the API server and matches nothing. Add it
# ALONGSIDE, never instead, if the cluster gains IPv6.
#
# Clearing this list restores the default rather than removing the
# ipBlock, deliberately: the two halves of this switch are not separable.
# To not open the port at all, set enabled: false.
#
# UNPROVEN IN OPERATION: probeSources is empty in every chart here, so no
# release reaches this branch. CI proves the render, not the effect.
redirectedProbeSources:
- 169.254.7.127/32
egress: egress:
# Additional outbound TCP ports. DNS (UDP and TCP 53) is always allowed and # Additional outbound TCP ports. DNS (UDP and TCP 53) is always allowed and