diff --git a/README.md b/README.md index f1a3b95..74bf255 100644 --- a/README.md +++ b/README.md @@ -231,7 +231,7 @@ Measured locally: **30 charts, 408 scalars verified, 0 failed**, no cluster. The | no ingress rule allows every source by accident | `from: []` allows everything, identically to having no `from` at all, while reading in a diff and in `kubectl` output as a restriction — the one mistake in this class that makes a policy *weaker* while looking *stricter* | nobody | | every port a **network** probe addresses is either unrestricted, or the chart offers `networkPolicy.probeSources` **and** says in `NOTES.txt` what a wrong CIDR costs | the kubelet probes from the IP of the worker node running the pod, which no chart can know. A restricted probe port without that address drops every probe: the pod never becomes `Ready`, the rollout times out, and the symptom reads as a broken application | the install-test, in production's CNI — **but not in CI**, see below | | **cross-chart egress**: where a chart dials another chart *of this repository*, its egress rule must contain the **pod** ports behind that chart's Services (#136) | an egress rule naming the **Service** port. kube-proxy does the ClusterIP's destination NAT *before* the policy is evaluated at the pod's interface, so the policy sees `podIP:targetPort` and the rule matches nothing wherever `port` and `targetPort` differ | nobody: the symptom is a **timeout**, not `connection refused`, and no error mentions a NetworkPolicy — `paperless-ngx` 14.0.0 shipped it and cost an outage on 2026-10-15 | -| **the ambient-mesh switch**, on two renders per chart: with `networkPolicy.ingress.ambientMesh` off, neither `15008` nor `169.254.7.127/32` appears; with it on, `15008` is present and is still the **only** ingress port that is no `containerPort`, and a source-restricted rule carrying it also carries the `ipBlock` (#143) | the named exception decaying into the hole it was carved around. Without the off-half a chart could render `15008` unconditionally and nothing would notice; without the only-port half the exception becomes the free port key that was rejected, and every mistyped ingress port passes with the switch on | nobody — and in production as a pod unreachable in the mesh, or one whose every probe is dropped the first time `probeSources` is narrowed | +| **the ambient-mesh switch** (#143), on two renders per chart plus the chart's declared defaults: those defaults equal the port and `ipBlock` the test **pins**; with the switch off neither appears; with it on the pinned port is present and is still the **only** ingress port that is no `containerPort`, and a source-restricted rule carrying it also carries the `ipBlock` | the named exception decaying into the hole it was carved around. Without the off-half a chart could render the port unconditionally and nothing would notice; without the only-port half the exception becomes the free port key that was rejected; and **without the pin** — now that the port is a value — the exception would permit whatever the value said, so a typo would ride in through a configurable field | nobody — and in production as a pod unreachable in the mesh, or one whose every probe is dropped the first time `probeSources` is narrowed | Egress ports are otherwise **deliberately not** held against the pod's `containerPort`s: an egress port is a port on the *destination*, and that comparison would fail every chart that reaches a database. @@ -248,7 +248,7 @@ Four things to know before touching it: Measured locally: **30 charts, 333 assertions held, 0 failed**, no cluster, ~25 seconds (the ambient-mesh assertion added a second render per chart, and roughly 10 seconds with it). All twelve fail paths were exercised rather than assumed — and one earned its keep immediately: the `probeSources` check was a substring test, so a chart whose key had been **deleted** while the long comment above it still discussed the value passed it. It now looks for the key. The cross-chart assertion added two of those 210, and its own six fail paths were executed the same way: a required port removed from the rule, a wrong number in its place, a peer chart renamed away, the policy's name no longer following the `--` convention, the peers rendering no Service at all (which would make the requirement empty, and is treated as a failure because an empty requirement reads exactly like a satisfied one), and the gated rule made unconditional — which is the failure path of the *gate-off* half, since with the gate off and the rule properly gated the assertion must stay silent. -**The ambient-mesh assertion (#143) was built the same way, and six of its fail paths were executed**: a second, different port added under the exception (rejected — the exception covers exactly one number); `15008` rendered ungated, caught by assertion 2 in the main loop, which renders the switch at its default; the `ipBlock` rendered ungated, which **only** this assertion can catch, because assertion 2 never looks at `from` peers; the `ipBlock` half deleted while `15008` stayed, i.e. the half-set switch the design exists to make impossible, reachable because `ci/networkpolicy/template-chart.yaml` fills `probeSources`; the policy's name taken off the `--` convention; and the value present in the schema but absent from `values.yaml`, which a `--set` render alone would not have noticed. The name check earned its keep on the first run rather than in a rehearsal: it read the **directory** name, and `template-chart`'s chart is called `xxx`. +**The ambient-mesh assertion (#143) was built the same way, and ten of its fail paths were executed**: a second, different port added under the exception (rejected — the exception covers exactly one number); `15008` rendered ungated, caught by assertion 2 in the main loop, which renders the switch at its default; the `ipBlock` rendered ungated, which **only** this assertion can catch, because assertion 2 never looks at `from` peers; the `ipBlock` half deleted while `15008` stayed, i.e. the half-set switch the design exists to make impossible, reachable because `ci/networkpolicy/template-chart.yaml` fills `probeSources`; the policy's name taken off the `--` convention; the value present in the schema but absent from `values.yaml`, which a `--set` render alone would not have noticed; a chart whose **declared default port deviates** (three independent checks fire at once: the declared-defaults comparison, the pinned port missing from the render, and the deviating port not being covered by the exception); a port-only chart **declaring `redirectedProbeSources`**, which reaches no rule there; a **deleted `ambientMesh` object** and one carrying only `enabled: true` (both must *render*, on the chart's own defaults — and the guards that make that work were each shown to be load-bearing by removing them); and a **release-level port override**, which must render and does, while the pinned exception demonstrably does **not** stretch to cover it. The name check earned its keep on the first run rather than in a rehearsal: it read the **directory** name, and `template-chart`'s chart is called `xxx`. **What this test does not and cannot cover, named so the gap is not mistaken for coverage:** that traffic still flows. No ingress controller runs in the install-test cluster, so the path from Traefik to a pod is proven nowhere in this pipeline — and, measured on the PR for #5, **k3s exempts the kubelet's probe traffic from policy**, so a blocked probe path does *not* turn the install-test red either (`cyberchef` became `Ready` in 69s with its probe port restricted to an address no probe can come from). What the install-test *does* prove is enforcement of everything the application itself needs: with `outline`'s DNS egress rule removed, its pod never became `Ready` and the job failed after 5m59s. So the net under this convention is real, but it is an **egress** net — which raises the stakes for assertion 2, since a wrong ingress port is caught mechanically here or not at all. @@ -696,9 +696,18 @@ Every port of a workload falls into one of four, and this is the part to get rig #### Values -`networkPolicy.enabled`, `.allowedNamespaces`, `.probeSources`, `.ingressController`, `.ingress.ambientMesh`, `.egress.extraPorts` — and a chart ships **only the ones that reach a rule**, with `.enabled` and `.ingress.ambientMesh` the two every chart has. +`networkPolicy.enabled`, `.allowedNamespaces`, `.probeSources`, `.ingressController`, `.ingress.ambientMesh.*`, `.egress.extraPorts` — and a chart ships **only the ones that reach a rule**, with `.enabled` and `.ingress.ambientMesh` the two every chart has. -**Three of those names overlap as substrings and are three different things**, which is worth stating because a search for `ingress` matches all of them: top-level `ingress` is the Ingress **object** a chart renders, `networkPolicy.ingress` is the policy's **inbound side**, and `networkPolicy.ingressController` is the **source selector** for the controller's pods — a *sibling* of `networkPolicy.ingress` and no part of it. The 21 charts carrying `ingressController` had **no** inbound escape hatch before #143, and one search run during that work reported them as having one on the strength of the substring alone. A chart may add a **named** egress switch where a port depends on something the values cannot see: `paperless-ngx`' `.egress.mailImport` opens IMAPS `993`, off by default, because its mail accounts are rows in its own database and no value says whether a release fetches mail. The bar for one of those: it names what it opens, and it is **not** a second switch for a port some *other* value already decides — that is why the Tika/Gotenberg rule stayed gated on `paperless.files.tikaGotenberg.enabled` and got no policy switch of its own. Two values that can drift apart re-arm the outage the rule was fixed for. A value that silently does nothing is worse than one that is absent, because it reads as a control: `samba` and `satisfactory` carry `enabled` alone (every inbound rule is open to any source), and `teamspeak-server` has `allowedNamespaces` but no `probeSources`. +**Three of those names overlap as substrings and are three different things**, which is worth stating because a search for `ingress` matches all of them: top-level `ingress` is the Ingress **object** a chart renders, `networkPolicy.ingress` is the policy's **inbound side**, and `networkPolicy.ingressController` is the **source selector** for the controller's pods — a *sibling* of `networkPolicy.ingress` and no part of it. The 21 charts carrying `ingressController` had **no** inbound escape hatch before #143, and one search run during that work reported them as having one on the strength of the substring alone. + +**And a fourth name, whose confusion risk is the larger one**, because these two are both lists of probe sources and sit two levels apart: + +| Name | Whose addresses | Default | Empty means | +| --- | --- | --- | --- | +| `networkPolicy.probeSources` | **yours** — the worker node CIDRs, which no chart can know | `[]` | the probe rule is **open** | +| `networkPolicy.ingress.ambientMesh.redirectedProbeSources` | **Istio's** — the address its data plane rewrites the probe source onto | `[169.254.7.127/32]`, i.e. shipped *filled*, because unlike the node IPs it is a constant the chart **can** know | n/a — clearing it restores the default, deliberately: the two halves of the switch are not separable | + +It is named `redirectedProbeSources` and not `probeSources` for exactly that reason: one is where probes come from normally, the other where they come from *after Istio redirects them*. A chart may add a **named** egress switch where a port depends on something the values cannot see: `paperless-ngx`' `.egress.mailImport` opens IMAPS `993`, off by default, because its mail accounts are rows in its own database and no value says whether a release fetches mail. The bar for one of those: it names what it opens, and it is **not** a second switch for a port some *other* value already decides — that is why the Tika/Gotenberg rule stayed gated on `paperless.files.tikaGotenberg.enabled` and got no policy switch of its own. Two values that can drift apart re-arm the outage the rule was fixed for. A value that silently does nothing is worse than one that is absent, because it reads as a control: `samba` and `satisfactory` carry `enabled` alone (every inbound rule is open to any source), and `teamspeak-server` has `allowedNamespaces` but no `probeSources`. **No chart ships `allowedNamespaces` pre-filled, ever.** A chart that named `mc-router`, or `paperless-jk`, or `stalwart` in its defaults would be baking in an assumption about *this* cluster's namespace layout — which is exactly the kind of claim a chart cannot keep true. So the default is the empty list and the chart is **restrictive**; filling it is the **deployment adaptation**, made where the namespace layout is actually visible. That is why a sweep like this produces a list of release-side one-liners: those are the expected procedure, not a defect list, and the breaking-change major on each chart is what announces them. @@ -708,7 +717,30 @@ Every port of a workload falls into one of four, and this is the part to get rig #### The ambient-mesh switch, and the one named exception in the render test (#143) -**`networkPolicy.ingress.ambientMesh`, in all 30 charts, default `false`.** Istio runs on this cluster in ambient mode, and Istio's own documentation is explicit that *"ambient does not and will never bypass Kubernetes `NetworkPolicy` enforcement"* — so until this value existed, **no chart here could put a pod into the mesh at all**. Additive, unchanged default, unchanged render: **minor** per chart, and the render was diffed against the previous release per chart rather than asserted. +**`networkPolicy.ingress.ambientMesh`, in all 30 charts, switched off by default.** Istio runs on this cluster in ambient mode, and Istio's own documentation is explicit that *"ambient does not and will never bypass Kubernetes `NetworkPolicy` enforcement"* — so until this value existed, **no chart here could put a pod into the mesh at all**. Additive, unchanged default, unchanged render: **minor** per chart, and the render was diffed against the previous release per chart rather than asserted. + +**The shape, and why both numbers are values rather than literals:** + +```yaml +networkPolicy: + ingress: + ambientMesh: + enabled: false + port: 15008 + redirectedProbeSources: + - 169.254.7.127/32 +``` + +**Both numbers are Istio *constants*, not standards** — and a value with a documented default is the better home for a vendor constant than a template literal, because an operator can correct it when Istio changes the number without waiting for a chart release. Their provenance, written on each one in every chart rather than assumed: + +- **`15008`** is Istio's HBONE port. Not registered with IANA for this, prescribed by no protocol, and **unknown to Kubernetes**. Istio picked it; Istio can change it. +- **`169.254.7.127`** is the address Istio's data plane rewrites the kubelet's probe source onto, and it is recognisable as vendor magic rather than protocol three ways: it is **link-local**, `169.254.0.0/16` (RFC 3927), a range reserved for node-local use and **never routed** — exactly the range one picks a magic address out of; Istio's IPv6 counterpart `fd16:9254:7127:1337:ffff:ffff:ffff:ffff` carries the **same digits** (`9254:7127` *is* `169.254.7.127`) plus `1337`, which is what vendor constants look like; and **Kubernetes knows the address in neither the NetworkPolicy API nor the kubelet** — only Istio's own data plane does. + +**Both read for ztunnel 1.30.6.** If Istio changes either, **the failure is silent**: a pod unreachable inside the mesh, or — for the address — probes dropped, and only where `probeSources` is non-empty, which is nowhere today. Nothing in this repository and nothing in Kubernetes would report it. What to re-check then: the port and address Istio's release notes name, this default in all 30 charts, **and** the pinned expectation in `ci/run-networkpolicy-test.sh`. + +`redirectedProbeSources` is a **list** and not a string, and this is the one case where that does not contradict "the value is a constant, not a configuration": a **dual-stack** cluster needs **both** of Istio's addresses at the same time, not one of two. That is a nameable case rather than speculation. This cluster is SingleStack IPv4, so the IPv6 counterpart is left out of the default — an `ipBlock` of the wrong family is accepted by the API server and matches nothing — and it is documented as the addition to make *alongside*, never *instead*. + +**Deleting the object is safe**, per the null-safety rule (#99): every read goes through a fallback reproducing the chart's own declared defaults, so `ambientMesh: null`, or an object carrying only `enabled: true`, renders the documented defaults. Both guards are load-bearing and were executed, not assumed — the direct chained read `$npIngress.ambientMesh.enabled` fails with `nil pointer evaluating interface {}.enabled` on a deleted object, which is the shape of the defect #136 shipped one value away; and without the port fallback a nilled `port` renders **`port: 0`**, an accepted and silently wrong policy. **It renders two things, and that they are one value is the whole design.** @@ -719,7 +751,14 @@ Point 2 is the one nobody thinks of, which is why the two halves cannot be had s **The interesting part is the CI, because the obvious implementation was the worst one.** Assertion 2 of `ci/run-networkpolicy-test.sh` holds every ingress port against the selected pod spec's `containerPort`s, and `15008` is deliberately none — ztunnel terminates it, not the application. A free `ingress.extraPorts` key would have had to exempt *operator-supplied ports* from that assertion, which opens it for **all 30 charts**: every mistyped ingress port, the NodePort mistake included, would then pass. So the assertion keeps its edge and gets **one named exception for one port number, gated on one value** — and it is not granted in the main loop at all, which renders every chart with the value at its default, so `15008` stays an ordinary failure there. -**An exception without its own check is the hole it was carved around**, so assertion 7 holds it to five things on **two renders of every chart**: with the switch off, neither `15008` nor the `ipBlock` is present; with it on, `15008` is present, `15008` is still the **only** ingress port that is not a `containerPort`, and wherever the rule carrying it restricts sources it carries the `ipBlock` too. The policy is found by the `--` name convention, because an exception must not be granted to an object the test could not identify. It lives in the existing script rather than a new one for the reason #136's cross-chart assertion does, only more so: **the exception weakens assertion 2, so it belongs beside it** — a separate script can be green while the exception it is paired with is wrong, and nothing would connect the two. +**And since the port is a value, the exception cannot be checked against the rendered number.** "Whatever the value says" is design A's hole wearing design C's name: a mistyped port would pass simply by sitting in the configurable field. So **the expectation is pinned in the test** — `AMBIENT_PORT` and `AMBIENT_CIDR` — and assertion 7 checks two things against that statement: + +1. the render **at chart defaults** produces exactly that port and exactly that `ipBlock`, and +2. **every chart's `values.yaml` declares the same defaults**, read back through `ci/yaml2json` rather than by regex — because a substring test over a commented file is how the `probeSources` check was once fooled by a deleted key whose comment survived. + +**A chart that drifts is red.** If Istio changes a constant, it takes 30 values files **and** the two pinned lines — and the pinned lines are what force someone to look. That block is therefore also the single place in the repository that states what we hold Istio's numbers to be, which makes it the answer to "where does `169.254.7.127` come from" as well; the provenance is written there. The renders set **only** `enabled=true` and never a port override, deliberately: the pin is a statement about the defaults *we ship*, not about what a release may choose. An operator overriding the value in their own release is outside this test's reach, as is true of every value here — and that is not a new gap. + +**An exception without its own check is the hole it was carved around**, so assertion 7 holds it to six things on **two renders of every chart**: the declared defaults match the pin; with the switch off, neither the pinned port nor the `ipBlock` is present; with it on, the pinned port is present, it is still the **only** ingress port that is not a `containerPort`, and wherever the rule carrying it restricts sources it carries the `ipBlock` too. The probe-source half is conditional on the chart offering `probeSources`: the five charts whose probe rule is never source-restricted render no `ipBlock`, so for them the key must be **absent** — a value reaching no rule reads as a control that does nothing — and that absence is asserted rather than tolerated. The policy is found by the `--` name convention, because an exception must not be granted to an object the test could not identify. It lives in the existing script rather than a new one for the reason #136's cross-chart assertion does, only more so: **the exception weakens assertion 2, so it belongs beside it** — a separate script can be green while the exception it is paired with is wrong, and nothing would connect the two. **Two things this does not prove, and they are not footnotes.** That `15008` *alone* suffices comes from Istio's documentation; what is measured on this cluster is only which listeners ztunnel opens. And the probe-source half reaches no release, because `probeSources` is empty in every chart (`stalwart` sets it to `[]` explicitly) — CI proves its render through the overlays in `ci/networkpolicy`, not its effect. Both become provable at the first namespace to join the mesh, and **none carries `istio.io/dataplane-mode`** (measured 2026-10-19, 0 of 97). Whether ztunnel needs a source selector of its own is open for the same reason — but it is *not* the unknowable case it looked like: ztunnel runs with its own pod IPs in `istio-system` (no `hostNetwork` on the DaemonSet, Istio 1.30.6), so a peer for it would be a `namespaceSelector` plus `podSelector`, not a node-IP `ipBlock`.