From 8233617f57b32f29f621cb72b7701a0f5201c030 Mon Sep 17 00:00:00 2001 From: Justin Klein Date: Wed, 14 Oct 2026 14:48:21 +0000 Subject: [PATCH] docs: the NetworkPolicy convention and its render test README gains two sections and CLAUDE.md the binding compact summary. "NetworkPolicies (#5)" under Chart style guidelines carries the part that is actually hard to get right: the FOUR PORT CLASSES, and that the class decides the rule. A port a network probe addresses stays open until probeSources is set, because the kubelet probes from the worker node's IP and no chart can know it. A NodePort-published port stays open because the source is already masqueraded. Everything else can be - and is - restricted by source, which is why the charts with a real inbound restriction are exactly the ones whose probes use a different port: keycloak's 8080, garage's s3 and web, minecraft-server's RCON, teamspeak-server's query interface. And a cluster bus between a release's own replicas gets a bare podSelector, with the reminder that policyTypes Egress means the outbound side needs its own rule or the cluster never forms while every node looks healthy. The exec-probe distinction is written down as load-bearing rather than pedantic, because it is the whole reason valkey and minecraft-server are the tightest policies here and every HTTP-probed web app is looser. What the policies do NOT do has its own list, including the two that will surprise someone: most dependency ports are the STANDARD ones rather than the configured ones, because connection details live in 1Password items the charts never parse - with keycloak, stalwart and minecraft-router named as the exceptions that derive theirs from values; and a subchart's pods are not covered, with ytdlp-web-player's oauth2-proxy as the sharp case and paperless-ngx' valkey as the opposite. The three Ingress-only charts are listed with their individual arguments rather than a blanket excuse, and gotenberg is contrasted against them: it keeps a DNS-only cap on purpose, because it drives a headless Chromium on a port the whole cluster can reach. "NetworkPolicy render test" under CI/CD documents the gate, and ends with what the pipeline cannot prove - including the measurement from this PR that k3s exempts kubelet probe traffic from policy, so the install-test's net is an EGRESS net. That is stated there rather than implied, because it is the reason assertion 2 matters: a wrong ingress port is caught mechanically or not at all. Part of #5 --- CLAUDE.md | 1 + README.md | 80 +++++++++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 81 insertions(+) diff --git a/CLAUDE.md b/CLAUDE.md index c7ce967..f820a7a 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -116,6 +116,7 @@ The binding style guide for all current and future charts lives in **README.md ( - Every container offers an additive `env` block rendered through `tpl` with `value`, `secretKeyRef`, and `configMapKeyRef` forms. - Hardening: `automountServiceAccountToken: false`, non-root, `readOnlyRootFilesystem: true`, caps dropped, `allowPrivilegeEscalation: false`; contexts values-configurable (`securityContext.pod` / `securityContext.container`); writable paths as emptyDirs; image-forced deviations MUST be documented with a comment in the chart's `values.yaml`. - Probes: liveness/readiness/startup on every workload (exception: paperless consume CronJob), values-configurable with calibrated defaults. +- **Every chart renders a `NetworkPolicy`, `networkPolicy.enabled` defaults to `true`** (#5); adding one to an existing chart is a **major**. Values: `enabled`, `allowedNamespaces`, `probeSources`, `ingressController`, `egress.extraPorts` — and only the ones that actually reach a rule, because a value that does nothing reads as a control. Each port falls in one of four classes and the class decides the rule: a port a **network** probe addresses stays open (`no from`) until `probeSources` is set, because the kubelet probes from the worker node's IP and no chart can know it; a NodePort-published port stays open (the source is already masqueraded); anything else is restricted to the namespace, `allowedNamespaces` and — for HTTP — `ingressController`; a replica-to-replica bus gets a bare `podSelector`. **`exec` probes are not network probes** (no packet, so the port can be restricted — `valkey`, `minecraft-server`). **Pod ports, never NodePorts.** Never `from: []`. Reasoning in Go template comments, never `#` (those ship to the cluster), and such a comment must not spell out its own closing delimiter. Egress is by **port, never destination** — Canal has no FQDN rule and Calico-specific objects are out of scope — so most dependency ports are the *standard* ones and a dependency elsewhere is blocked; `policyTypes` names only what there is something to restrict (`valkey`, `homeassistant`, `satisfactory` are Ingress-only, each argued in its own file). `ci/run-networkpolicy-test.sh` is the gate. Full rules: README, "NetworkPolicies". - Resources: requests as values defaults; limits auto-computed by the `_helpers.tpl` helper (`limitFactor` default 3; no CPU limits unless set explicitly). - Replicas per workload (explicit `0` allowed) plus a chart-global `scaleDown` flag that overrides them. - `icon:` mandatory in every `Chart.yaml`; `helm lint --strict` without findings; CI fixtures under `ci//` or a justified exclusion. diff --git a/README.md b/README.md index b1f1102..b83e5be 100644 --- a/README.md +++ b/README.md @@ -220,6 +220,32 @@ Three things to know before touching it: Measured locally: **30 charts, 408 scalars verified, 0 failed**, no cluster. The fail path was exercised rather than assumed, in all three of its shapes, against a throwaway copy of the chart as it stands on `main`: the unquoted schedule aborted the render with `yaml: line 6: did not find expected alphabetic or numeric character` and the test reported it; with only the schedule quoted, the star-valued `env` entry aborted the render on the StatefulSet instead; and with only the bool-looking value left — a render that **succeeds** — the type assertion produced a `FAIL` naming the variable, the container and the exact API-server message. **Seven charts still render their `env` block bare** (`matrix-synapse`, `minecraft-router`, `minecraft-server`, `rallly`, `satisfactory`, `teamspeak-server`, and `paperless-ngx` until #11); the test is green for them only because their fixtures set no hostile `env` value, and the overlay is what makes the difference between covered and merely green. +### NetworkPolicy render test (all charts, every PR) + +`ci/run-networkpolicy-test.sh [chart …]` (no argument = all charts) renders every chart with the same fixtures the install-test uses and holds **every `NetworkPolicy` it renders against the pod specs, probes and Services of the same render**. Introduced with Gitea #5, which put a policy into all 29 installable charts at once with `networkPolicy.enabled: true` as the default — 29 objects whose every failure mode is silent, because the API server accepts a policy that filters **nothing** exactly as readily as one that filters **everything**. + +| Assertion | What goes wrong without it | Who notices today | +|---|---|---| +| the `podSelector` matches a pod spec of the same render | the object sits in the namespace, is listed by `kubectl`, reads as a boundary and filters nothing at all — usually because a label was renamed in the workload template, a different file and a different diff | nobody, ever: nothing is denied, so there is no symptom to trace | +| every **ingress** port is a `containerPort` of the selected pods | a NodePort number where a pod port belongs. The policy is evaluated at the pod's interface, *after* kube-proxy translated the port, so `30025` matches nothing while looking strict — and being the only rule for that port, it blocks every inbound connection instead | nobody in CI (see below), and in production as a service that went mute | +| no ingress rule allows every source by accident | `from: []` allows everything, identically to having no `from` at all, while reading in a diff and in `kubectl` output as a restriction — the one mistake in this class that makes a policy *weaker* while looking *stricter* | nobody | +| every port a **network** probe addresses is either unrestricted, or the chart offers `networkPolicy.probeSources` **and** says in `NOTES.txt` what a wrong CIDR costs | the kubelet probes from the IP of the worker node running the pod, which no chart can know. A restricted probe port without that address drops every probe: the pod never becomes `Ready`, the rollout times out, and the symptom reads as a broken application | the install-test, in production's CNI — **but not in CI**, see below | + +Egress ports are **deliberately not** held against the pod's `containerPort`s: an egress port is a port on the *destination*, and that comparison would fail every chart that reaches a database. + +Two guards beyond the four, both earned rather than designed. **A chart that renders no policy at all fails** — that is how this issue decayed in the first place: `garage` (#81) and `keycloak` (#83) were both created *after* #5 was filed and both arrived without a policy, `keycloak` with its unauthenticated management surface on 9000. And **a `#` comment inside a rendered policy fails**, because a `#` comment is part of the *object*: it travels into the cluster, shows up in `helm get manifest` and `kubectl get -o yaml`, and lands in every release diff. Reasoning belongs in Go template comments, which render nothing. Both precedent charts failed that check when it was first run, which is what it is for. + +Four things to know before touching it: + +- **Assertion 3 is stricter than it first reads, deliberately.** A *missing* `from` is the documented open form and both precedent charts use it on purpose, so banning it outright was not an option. It is accepted only where the render itself justifies being open: every port of the rule is addressed by a network probe, or is published by a `NodePort` Service in the same render. Anything else is an allow-all nobody chose. +- **`exec` probes are not network probes**, and that distinction carries real weight rather than being a technicality. An `exec` probe runs inside the container through the kubelet's runtime and puts no packet on the network, so a chart whose probes are all `exec` can restrict its port by source with **no `probeSources` at all** — which is why `valkey` and `minecraft-server` have the tightest policies in the repository and every web app has a looser one. +- **Subchart documents are skipped by the comment assertion**, for a structural reason and not a lenient one: a dependency is fetched from the Helm registry rather than from the copy in this repo, so the policy `paperless-ngx` and `stalwart` bundle is whatever `valkey` version is *published*, and nothing in the parent chart can change its rendered comments. The subchart's own PR is where it is caught, because this test runs over `valkey` too. +- **Anti-vacuum guard**, as in the tests above: `ci/networkpolicy/.yaml` is a render-only overlay applied **last**, whose job is to render the **source-restricted probe rule** — a branch no chart's own defaults ever reach, and the one containing the expensive mistake (a `namespaceSelector` and a `podSelector` written as two list entries instead of one would be an OR, opening the port to every pod in `kube-system`). A chart that *has* an overlay and still renders no restricted **probe** rule fails. That counter is specifically the probe rule and not "any restricted rule", because `keycloak` restricts two rules in its own defaults and would have satisfied a looser guard while the branch went unrendered. + +Measured locally: **30 charts, 206 assertions held, 0 failed**, no cluster, a few seconds. All twelve fail paths were exercised rather than assumed — and one earned its keep immediately: the `probeSources` check was a substring test, so a chart whose key had been **deleted** while the long comment above it still discussed the value passed it. It now looks for the key. + +**What this test does not and cannot cover, named so the gap is not mistaken for coverage:** that traffic still flows. No ingress controller runs in the install-test cluster, so the path from Traefik to a pod is proven nowhere in this pipeline — and, measured on the PR for #5, **k3s exempts the kubelet's probe traffic from policy**, so a blocked probe path does *not* turn the install-test red either (`cyberchef` became `Ready` in 69s with its probe port restricted to an address no probe can come from). What the install-test *does* prove is enforcement of everything the application itself needs: with `outline`'s DNS egress rule removed, its pod never became `Ready` and the job failed after 5m59s. So the net under this convention is real, but it is an **egress** net — which raises the stakes for assertion 2, since a wrong ingress port is caught mechanically here or not at all. + ### Install-test (PR pipeline stage 2) For every chart a PR changes, `ci/run-install-test.sh ` runs against a throwaway k3d cluster: it applies the stub `OnePasswordItem` CRD plus the chart's fixtures, `helm install`s the chart with its CI values, and waits for all Deployments/StatefulSets to roll out (per-chart timeout). Since probes and security hardening are active on all workloads, "Ready" means the app actually answers. On failure the job dumps pod descriptions, events and container logs. @@ -638,3 +664,57 @@ One more trap, and it is the reason `stalwart` gates its subchart on `valkeySubc Two implementation notes that cost time otherwise: a values key containing a dash is unreachable as `.Values.oauth2-proxy` (that parses as a subtraction) and needs `index .Values "oauth2-proxy"`, while `condition:` in `Chart.yaml` splits on dots only and needs no escaping. And inside a subchart `.Chart.Name` is the **subchart's** name — a value templated into it (`config.configFile` in this case) must spell the parent chart out literally; only `.Release.Name` is shared. Charts that need no dependency still have none, and `charts/` stays empty for all of them. `template-chart` remains dependency-free: the scaffold should not carry a `dependencies:` block a new chart has to remember to delete. + +### 15. NetworkPolicies (#5) + +**Every chart renders a `NetworkPolicy`, and `networkPolicy.enabled` defaults to `true`.** Reference implementation: `template-chart/templates/xxx.networkpolicy.yaml`. The default is the decision the whole convention rests on — a policy that ships switched off is reached only when somebody remembers to switch it on, and then the issue is closed while nothing is filtered. Adding one to an existing chart is therefore a **major** bump. + +**What the object is: a ceiling on reachability, not an authentication layer.** Three limits are real and worth having, and for most charts here the first two are where all of the value sits: + +1. **Only the declared ports are reachable at all.** Whatever else the process binds — a debug listener, a management interface the image opens by default, a port that appears in the next app version — is blocked without anyone having to notice it exists. +2. **Egress is capped by port**, so a compromised pod cannot open arbitrary outbound connections. +3. **Ingress is capped by source for every port no network probe touches.** + +#### The four port classes, and why the class decides the rule + +Every port of a workload falls into one of four, and this is the part to get right before writing anything: + +| Class | Rule | Why | +|---|---|---| +| **probe path** — a port a *network* probe addresses | **no `from`** by default; restricted once `networkPolicy.probeSources` is set | the kubelet probes from the IP of the **worker node running the pod**, an address no chart can know. A guessed CIDR that misses it drops every probe: the pod never becomes `Ready`, the rollout times out, and the symptom reads as a broken application rather than as a policy | +| **public** — published by a `NodePort` Service | **no `from`** | the senders are on the internet and, after NodePort masquerading, the source address is a cluster node anyway. A source selector could only ever be a list of node IPs — a restriction in appearance and a maintenance trap in fact | +| **closed** — no probe, not published outside | own namespace + `allowedNamespaces` (+ the ingress controller where the chart routes HTTP there) | this is the only class that yields a *real* inbound restriction, and the charts that have one are the ones whose probes use a *different* port: `keycloak`'s 8080, `garage`'s s3 and web, `minecraft-server`'s RCON, `teamspeak-server`'s query interface | +| **self** — a cluster bus between a release's own replicas | `podSelector` only, no `namespaceSelector` | the narrowest rule the object can express: "these labels, in this namespace". `keycloak`'s jgroups, `garage`'s rpc. Note that `policyTypes: Egress` means the **outbound** side needs its own rule, or the bus is open one way and the cluster never forms while every node looks healthy | + +**`exec` probes are not in the probe class**, and the distinction is load-bearing rather than pedantic: an `exec` probe runs inside the container through the kubelet's runtime and puts no packet on the network, so there is no kubelet traffic to accommodate. That is the entire reason `valkey` and `minecraft-server` have the tightest policies in the repository — every port restricted by source, no `probeSources` value at all — and every HTTP-probed web app has a looser one. + +#### Values + +`networkPolicy.enabled`, `.allowedNamespaces`, `.probeSources`, `.ingressController`, `.egress.extraPorts` — and a chart ships **only the ones that reach a rule**. A value that silently does nothing is worse than one that is absent, because it reads as a control: `samba` and `satisfactory` carry `enabled` alone (every inbound rule is open to any source), and `teamspeak-server` has `allowedNamespaces` but no `probeSources`. + +`ingressController` is a separate value with two keys rather than a line in `allowedNamespaces` for one reason: **its two selectors render as one `from` entry, which is an AND** — those pods in that namespace. Written as two list entries they would be an OR, and putting `kube-system` into `allowedNamespaces` instead would open the port to every pod in the cluster's own namespace. Its default is **measured, not assumed**: `ingress-nginx` is gone from this cluster, the only `IngressClass` is `traefik`, and the controller is the DaemonSet `rke2-traefik` in `kube-system` — which also serves the Gateway API here, so one source covers the Ingress and the HTTPRoute path alike. `ingress.className` says nothing about it: a class names an implementation, not the pods the traffic comes from. + +#### What these policies do not do — and it belongs in every chart, not just here + +- **Egress is coarse: by port, never by destination.** A `NetworkPolicy` matches `ipBlock` CIDRs and pod/namespace selectors and **never hostnames**, so "only to the database host" is not expressible at all. This cluster's CNI is Canal (Flannel plus Calico for policy) and has no FQDN rule; Calico's `GlobalNetworkSet` could express one but would bind every chart to Calico — the same argument that kept implementation-specific objects out of the Gateway-API sweep (#114) — so it is **not used**. This is a limit of the object, and each chart names what it costs *that* chart. +- **Most dependency ports are the standard ones, not the configured ones.** Databases, caches and OIDC issuers arrive as connection URLs or as fields inside 1Password items the charts never parse, so a chart cannot know where its dependency listens and cannot validate a guess. A dependency somewhere else is **blocked**, and `networkPolicy.egress.extraPorts` is the escape hatch. The exceptions are the two charts that hold their ports as values and therefore derive them: `keycloak` (`keycloak.db.port`) and `stalwart` (`dataStore.postgres.port`, `externalServices.*.port`); `minecraft-router` derives its backend ports from its own routing table the same way. +- **The probed port stays open inside the cluster by default.** For a single-port web app that means the policy is a port and egress ceiling, not pod-level isolation — say so in the chart rather than letting the object imply otherwise. +- **A subchart's pods are not covered.** The policy selects this chart's workload by label; a subchart renders its own pods with its own labels. `ytdlp-web-player`'s oauth2-proxy is the sharpest case — with the proxy on, the pod that actually faces the Ingress is unrestricted in both directions, and an upstream chart is consumed as it comes (section 14). `paperless-ngx`' `valkey` is the opposite: it brings its own, tighter policy. Name which case a chart is in. +- **And a policy is only a boundary where the CNI enforces one.** Unenforced, the API server still accepts it and nothing is filtered, so everything reads as configured. Verify by connecting from an unrelated namespace, not by reading the value. + +#### Where the convention says "Ingress only" + +`policyTypes` names only what there is something to restrict. `valkey` established it for a workload that opens **no** outbound connection: an egress rule would constrain nothing while making the object look stricter than it is. Three charts now invoke the same rule in the other direction, for outbound traffic nobody can enumerate — and each argues it in its own file rather than inheriting a blanket excuse: + +- **`homeassistant`**: outbound traffic is the union of every integration an operator enables (mDNS and SSDP discovery, MQTT, a Zigbee or Z-Wave bridge on whatever port its firmware chose, ESPHome, printers, speakers, inverters, cloud APIs, NTP). A cap would protect little — the devices sit on the node's own LAN — and would break integrations one at a time, months apart, each looking like a flaky device. +- **`satisfactory`**: Steam's content delivery is not a pinnable port set, *and* this is the one chart with no install-test at all (`ci/excluded-charts.txt`), so an egress cap here would be the only one in the sweep with no empirical check behind it, on the chart whose correct port set is least knowable. +- **`gotenberg`** goes the other way and is worth contrasting: it *keeps* a DNS-only cap deliberately, because it drives a headless Chromium and 3000 is its probe port — so an egress rule for 80/443 would turn a service the whole cluster can POST to into an SSRF pivot. The price is that HTML conversion silently loses remote assets, which is written into the chart because nothing will alert on it. + +#### Mechanics + +- **Pod ports, never NodePorts.** A `NetworkPolicy` is evaluated at the pod's network interface, *after* kube-proxy has translated a NodePort to the `targetPort`. `30025` where the pod listens on `25` matches nothing: accepted, strict-looking, and blocking every inbound connection instead. The render test asserts this mechanically against the selected pod spec's `containerPort`s. +- **Protocol as well as number.** `teamspeak-server`'s voice port is UDP and `satisfactory`'s game port is both — a TCP-only rule on either is accepted, looks complete, and drops the traffic that actually arrives. +- **`kubernetes.io/metadata.name`** is set by Kubernetes itself on every namespace (≥ 1.21), so namespace selectors need no labelling work on namespaces the chart does not own — and get no typo protection either. +- **Never `from: []`.** It allows everything, identically to having no key, while reading as a restriction. Write no `from` where open is meant and say so in a template comment. +- **Reasoning goes in Go template comments, not `#` comments.** A `#` comment is part of the rendered object: it travels into the cluster, appears in `helm get manifest` and `kubectl get -o yaml`, and lands in every release diff. The render test fails on a rendered `#`. One second-order trap, found while writing the reference implementation: prose inside a template comment must not spell out the comment's own closing delimiter, or the parser ends the comment early and the rest is emitted as manifest content — the render then fails with `did not find expected node content`, pointing at the template and not at the sentence. +- **A conditional `containerPort` needs a conditional rule.** `minecraft-server`'s RCON port is rendered only when `server.rcon.enabled` is true; a rule without the same gate names a port the pod does not declare, and the render test rejects it.