Gitea #143. A pod cannot be reached inside Istio's ambient mesh unless its NetworkPolicy lets ztunnel's HBONE tunnel in - ambient explicitly does not bypass policy enforcement - and 15008 is deliberately no containerPort of the application, so assertion 2 of ci/run-networkpolicy-test.sh rejected it. The switch is a named intention rather than a free port list, and it renders two things that cannot be had separately: 15008/TCP inbound, and - where probeSources is non-empty - 169.254.7.127/32 as an additional peer, because inside the mesh the kubelet's probes are redirected away from the worker node's address. Whoever opens 15008 by hand has not added the probe address, and the first narrowed probeSources then drops every probe: the pod never becomes Ready and the symptom reads as a broken application. That is assertion 4 of the same script, one value further along. Assertion 2 keeps its edge. It gets ONE named exception for ONE port number, gated on one value, and it is not granted in the main loop at all - which renders every chart with the value at its default, so 15008 stays an ordinary failure there. The free-port variant was design A of #143 and would have opened the assertion for all 30 charts. Assertion 7 holds the exception to five things on two renders per chart: the port and the ipBlock are absent while the switch is off, 15008 is present while it is on, 15008 is still the ONLY ingress port that is not a containerPort, and wherever the rule carrying it restricts sources it also carries the ipBlock. The last one is reachable because these renders apply ci/networkpolicy/<chart>.yaml, where probeSources is non-empty. Built into the existing script, as for #136: the exception weakens assertion 2 and belongs beside it - a separate script could go green while the exception it is paired with is wrong, with nothing connecting the two. The job already runs over every chart on every pull request. Found by running it: the policy's name comes from Chart.yaml's name, not from the directory - template-chart is called "xxx". Not proven, and said so in values.yaml, the template header and the script's header: that 15008 alone suffices comes from Istio's documentation, and the probe-source half reaches no release, because probeSources is empty in every chart here. No namespace of this cluster carries istio.io/dataplane-mode, so there is no path on which either could be measured. What is proven is the render and the assertions.
271 lines
14 KiB
YAML
271 lines
14 KiB
YAML
{{- /*
|
|
NetworkPolicy — the reference implementation for Gitea #5.
|
|
|
|
Copy this file with the rest of template-chart and replace every xxx: the
|
|
values paths (.Values.networkPolicy.*), the helper prefix and the podSelector
|
|
label, exactly as in the other template files.
|
|
|
|
EVERY COMMENT IN THIS FILE IS A TEMPLATE COMMENT, and that is not a style
|
|
choice. A "#" comment is part of the RENDERED manifest: it travels into the
|
|
cluster, shows up in `helm get manifest` and `kubectl get -o yaml`, and bloats
|
|
every diff a reviewer reads. Reasoning belongs in a Go template comment, which
|
|
renders nothing. This caught several rounds of this rollout.
|
|
|
|
And a second-order trap found while writing this header: the prose inside a
|
|
template comment must not spell out the comment's own closing delimiter. The
|
|
parser ends the comment at the first occurrence, and the remaining prose is
|
|
then emitted as manifest content - the render fails with "did not find
|
|
expected node content", pointing at the template and not at the sentence.
|
|
Describe the delimiters instead of quoting them.
|
|
|
|
WHAT THIS OBJECT DOES, AND - MORE IMPORTANTLY - WHAT IT DOES NOT
|
|
|
|
It is a CEILING ON REACHABILITY, not an authentication layer. Three limits are
|
|
real and worth having:
|
|
|
|
1. ONLY THE DECLARED PORTS ARE REACHABLE AT ALL. Whatever else the process
|
|
binds - a debug listener, a management interface the image opens by
|
|
default, a port that appears in the next app version - is blocked without
|
|
anyone having to notice it exists. This is the largest part of what a
|
|
default-on policy buys in this repository.
|
|
2. EGRESS IS CAPPED BY PORT, so a compromised pod cannot open arbitrary
|
|
outbound connections.
|
|
3. INGRESS IS CAPPED BY SOURCE for every port no network probe touches.
|
|
|
|
And three things it does NOT do, each of which has to be written down per
|
|
chart rather than quietly assumed:
|
|
|
|
a. EGRESS IS COARSE. A NetworkPolicy matches CIDRs and selectors and NEVER
|
|
hostnames, so "only to the database host" cannot be expressed. This
|
|
cluster's CNI is Canal (Flannel plus Calico for policy), which has no
|
|
FQDN rule; Calico's GlobalNetworkSet could express one but would bind
|
|
every chart to Calico - the same argument that kept implementation-
|
|
specific objects out of the Gateway-API sweep (Gitea #114). So each
|
|
egress rule below permits its PORT to any destination. That is a limit of
|
|
the object, not a shortcut.
|
|
b. THE PROBED PORT IS OPEN INSIDE THE CLUSTER BY DEFAULT. See the long note
|
|
on networkPolicy.probeSources in values.yaml. Narrowing it needs the
|
|
worker node IPs, which this chart cannot know.
|
|
c. A POLICY IS ONLY A BOUNDARY WHERE THE CNI ENFORCES ONE. An unenforced
|
|
policy is accepted by the API server and filters nothing, so everything
|
|
looks configured while nothing is filtered. Measured for this cluster
|
|
(Canal, 6/6) and for the CI cluster (k3d, which brings k3s' own policy
|
|
controller) - verify it rather than reading it off a value.
|
|
|
|
ONE CLUSTER SETTING THIS WHOLE CONVENTION DEPENDS ON AND CANNOT CHECK:
|
|
TRAEFIK MUST NOT RUN WITH hostNetwork.
|
|
|
|
Measured on 2026-10-14: the rke2-traefik DaemonSet has no hostNetwork, and its
|
|
pods carry real pod IPs (10.42.3.120, 10.42.4.94, 10.42.0.187; the pod CIDR is
|
|
10.42.0.0/16, while the nodes sit on entirely different addresses). ONLY
|
|
BECAUSE OF THAT does the podSelector in networkPolicy.ingressController match
|
|
it.
|
|
|
|
Switch the ingress controller to host networking and its pods share the node's
|
|
network namespace. They are then not selectable as pods at all, the
|
|
ingressController entry matches nothing, and EVERY WEB CHART IN THIS REPOSITORY
|
|
GOES DARK AT ONCE in the restricted configuration - while every policy is
|
|
still accepted and still reads as correct. The symptom to recognise is exactly
|
|
that shape: all web traffic stops together and every pod stays Ready, because
|
|
the probe path does not run through the controller.
|
|
|
|
Nothing here can detect this and nothing here can fail usefully on it, which is
|
|
the same class of dependency as --enable-certificate-owner-ref in #126. It is
|
|
written here, on the thing that depends on it, so whoever changes the
|
|
controller's networking meets the consequence.
|
|
|
|
AND A POLICY DOES APPLY TO TRAFFIC THAT CAME THROUGH A SERVICE, which is why
|
|
the ingressController rule works in the first place. A Service is only
|
|
destination NAT, and the policy is evaluated at the pod's interface, so a
|
|
ClusterIP exempts nothing. What the Service changes is the SOURCE ADDRESS: pod
|
|
-> ClusterIP -> pod PRESERVES it, so a podSelector or a namespaceSelector can
|
|
still match the sender. Traffic arriving from outside through a NodePort with
|
|
the default externalTrafficPolicy: Cluster is MASQUERADED to a node, which is
|
|
why such rules cannot be restricted by source at all and are left open instead.
|
|
|
|
THE PORT NUMBERS HERE ARE POD PORTS, NEVER NODEPORTS. A NetworkPolicy is
|
|
evaluated at the POD's network interface, after kube-proxy has already
|
|
translated a NodePort to the targetPort. Writing 30080 where the pod listens
|
|
on 8080 matches nothing: the object is accepted, looks strict, and blocks
|
|
every inbound connection instead. ci/run-networkpolicy-test.sh asserts this
|
|
mechanically - every port named here must exist as a containerPort of the pod
|
|
spec this policy selects.
|
|
*/ -}}
|
|
{{- if .Values.networkPolicy.enabled }}
|
|
{{- $np := .Values.networkPolicy }}
|
|
{{- $probeSources := $np.probeSources | default list }}
|
|
{{- $allowedNs := $np.allowedNamespaces | default list }}
|
|
{{- $ic := $np.ingressController | default dict }}
|
|
{{- /* networkPolicy.ingress - the policy's INGRESS DIRECTION, and not to be
|
|
confused with networkPolicy.ingressController one line up, nor with the
|
|
chart's top-level .Values.ingress. Three different things whose names
|
|
overlap as substrings:
|
|
|
|
.Values.ingress the Ingress OBJECT this chart renders
|
|
.Values.networkPolicy.ingress switches on the policy's inbound side
|
|
.Values.networkPolicy.ingressController
|
|
the SOURCE SELECTOR for the ingress
|
|
controller's pods, a sibling of
|
|
networkPolicy.ingress and NO PART OF
|
|
IT
|
|
|
|
Named here because a search for "ingress" matches all three, and one
|
|
round of this work already reported ingressController as an inbound
|
|
escape hatch on the strength of the substring alone. */}}
|
|
{{- $npIngress := $np.ingress | default dict }}
|
|
{{- $ambientMesh := $npIngress.ambientMesh }}
|
|
{{- $routed := or .Values.ingress.enabled .Values.httpRoute.enabled }}
|
|
apiVersion: networking.k8s.io/v1
|
|
kind: NetworkPolicy
|
|
metadata:
|
|
name: {{ .Release.Name }}-{{ .Chart.Name }}-networkpolicy
|
|
labels:
|
|
app: {{ .Release.Name }}-{{ .Chart.Name }}-networkpolicy
|
|
spec:
|
|
{{- /* Selects the Deployment's pods. A chart that keeps the StatefulSet
|
|
instead has to move this selector along with the Service's, and
|
|
ci/run-networkpolicy-test.sh fails if it selects no pod spec of the
|
|
same render - which is the mistake that would otherwise leave a
|
|
NetworkPolicy in the namespace filtering nothing at all. */}}
|
|
podSelector:
|
|
matchLabels:
|
|
app: {{ .Release.Name }}-{{ .Chart.Name }}-deployment
|
|
policyTypes:
|
|
- Ingress
|
|
{{- /* Egress is listed because there IS something to restrict here: this
|
|
workload needs DNS and nothing else by default. A chart whose
|
|
workload opens no outbound connection of its own must NOT list
|
|
Egress - valkey is the example - because an egress rule that
|
|
constrains nothing makes the object look stricter than it is. */}}
|
|
- Egress
|
|
ingress:
|
|
{{- /* The http port. It is the port all three probes address, so it is
|
|
the probe path, and the shape below is the whole reason
|
|
networkPolicy.probeSources exists. */}}
|
|
- ports:
|
|
{{- range .Values.xxx.ports }}
|
|
- protocol: {{ .protocol | default "TCP" }}
|
|
port: {{ .containerPort }}
|
|
{{- end }}
|
|
{{- if $ambientMesh }}
|
|
{{- /* FIRST HALF OF networkPolicy.ingress.ambientMesh. 15008/TCP is
|
|
ztunnel's HBONE tunnel, the port every inbound mesh connection
|
|
arrives on once the namespace carries istio.io/dataplane-mode.
|
|
Istio's own documentation is explicit that ambient "does not and
|
|
will never bypass Kubernetes NetworkPolicy enforcement", so
|
|
without this port a pod in the mesh is simply unreachable.
|
|
|
|
It is the one port in this rule that is NOT a containerPort of
|
|
the selected pod spec - ztunnel terminates it, not the
|
|
application - so ci/run-networkpolicy-test.sh holds a NAMED
|
|
exception for exactly this number, gated on exactly this value.
|
|
Anything else that is not a containerPort still fails, and so
|
|
does 15008 itself while the value is off.
|
|
|
|
NOT MEASURED, and the chart should not pretend otherwise: that
|
|
15008 ALONE suffices comes from Istio's documentation. What has
|
|
been measured on this cluster is only which listeners ztunnel
|
|
opens. Provable at the first namespace that joins the mesh, and
|
|
no namespace carries the label yet. */}}
|
|
- protocol: TCP
|
|
port: 15008
|
|
{{- end }}
|
|
{{- if $probeSources }}
|
|
{{- /* RESTRICTED, because the operator supplied probeSources. The
|
|
kubelet's probes come from the IP of the WORKER NODE running the
|
|
pod, so that address has to be in the list or the pod never
|
|
becomes Ready - and a pod that is never Ready reads as a broken
|
|
application rather than as a policy. The ingress controller is
|
|
added here too, or the Ingress goes dark the moment this list is
|
|
filled. */}}
|
|
from:
|
|
- namespaceSelector:
|
|
matchLabels:
|
|
{{- /* Set by Kubernetes itself on every namespace (>= 1.21), so
|
|
this needs no labelling work on namespaces this chart does
|
|
not own. */}}
|
|
kubernetes.io/metadata.name: {{ .Release.Namespace }}
|
|
{{- if and $routed $ic.namespace }}
|
|
{{- /* ONE from-entry with TWO selectors, which is an AND: pods
|
|
carrying these labels IN that namespace. Written as two list
|
|
entries it would be an OR and would open the port to the whole
|
|
controller namespace - kube-system here, i.e. to everything in
|
|
the cluster's own namespace. */}}
|
|
- namespaceSelector:
|
|
matchLabels:
|
|
kubernetes.io/metadata.name: {{ $ic.namespace | quote }}
|
|
{{- with $ic.podLabels }}
|
|
podSelector:
|
|
matchLabels:
|
|
{{- toYaml . | nindent 14 }}
|
|
{{- end }}
|
|
{{- end }}
|
|
{{- range $allowedNs }}
|
|
- namespaceSelector:
|
|
matchLabels:
|
|
kubernetes.io/metadata.name: {{ . | quote }}
|
|
{{- end }}
|
|
{{- range $probeSources }}
|
|
- ipBlock:
|
|
cidr: {{ . | quote }}
|
|
{{- end }}
|
|
{{- if $ambientMesh }}
|
|
{{- /* SECOND HALF OF networkPolicy.ingress.ambientMesh, and the reason
|
|
the switch is one value instead of two. Inside the mesh the
|
|
kubelet's probes are redirected and no longer arrive from the
|
|
worker node's address: they arrive from 169.254.7.127, which is
|
|
therefore the probeSources entry nobody would think to add.
|
|
|
|
The two halves are deliberately inseparable. An operator who
|
|
only opens 15008 by hand has a policy that passes every review -
|
|
and the moment probeSources is non-empty, every probe is dropped,
|
|
the pod never becomes Ready, and the symptom reads as a broken
|
|
application rather than as a policy. That is assertion 4 of
|
|
ci/run-networkpolicy-test.sh, re-created one value further along.
|
|
One switch cannot be set halfway.
|
|
|
|
Only rendered in this branch, because this is the only branch
|
|
with a "from" list at all: with probeSources empty the rule is
|
|
open and needs no peer. The IPv6 counterpart
|
|
fd16:9254:7127:1337:ffff:ffff:ffff:ffff is deliberately absent -
|
|
this cluster is SingleStack IPv4, and an ipBlock of the wrong
|
|
family is accepted by the API server and matches nothing.
|
|
|
|
UNPROVEN IN OPERATION: probeSources is empty in every chart of
|
|
this repository (stalwart sets it to the empty list explicitly),
|
|
so no release reaches this branch today. What CI proves is its
|
|
render, through the overlay in ci/networkpolicy. */}}
|
|
- ipBlock:
|
|
cidr: 169.254.7.127/32
|
|
{{- end }}
|
|
{{- else }}
|
|
{{- /* NO "from" SELECTOR, WHICH MEANS OPEN - deliberately, and only
|
|
inside the cluster, because this port carries no NodePort. The
|
|
alternative was to guess the worker node CIDRs, and a wrong guess
|
|
blocks every kubelet probe: the pod stays un-Ready, the rollout
|
|
times out, and the cause looks like the application. Note what an
|
|
EMPTY "from: []" would do instead: it also allows everything, but
|
|
while looking like a restriction. Hence no key at all, and
|
|
ci/run-networkpolicy-test.sh rejects the empty form. */}}
|
|
{{- end }}
|
|
egress:
|
|
{{- /* DNS, unconditionally. There is no value to switch this off, and
|
|
that is deliberate: a knob whose only other setting breaks every
|
|
name lookup in the pod is a trap and not a feature. */}}
|
|
- ports:
|
|
- protocol: UDP
|
|
port: 53
|
|
- protocol: TCP
|
|
port: 53
|
|
{{- with $np.egress }}
|
|
{{- with .extraPorts }}
|
|
{{- /* Ports only - see note (a) in the header. Any destination. */}}
|
|
- ports:
|
|
{{- range . }}
|
|
- protocol: TCP
|
|
port: {{ int . }}
|
|
{{- end }}
|
|
{{- end }}
|
|
{{- end }}
|
|
{{- end }}
|