jklein
5527ebfc15
Gitea #143, maintainer's instruction. The HBONE port and the redirected probe source are ISTIO CONSTANTS, not standards, so they move from template literals into values with documented defaults - an operator can then correct them when Istio changes a number, without waiting for a chart release. One object under one "enabled", so the two halves still cannot be set separately; two values could drift apart, which is what this design exists to prevent. Every read is null-safe (#99), so a deleted object or one carrying only "enabled: true" falls back to the defaults declared here rather than dereferencing nil - the defect #136 shipped one value away. Provenance now written on each constant: 15008 is Istio's HBONE port, not IANA-registered for this and unknown to Kubernetes; 169.254.7.127 is link-local (RFC 3927, never routed), its IPv6 counterpart carries the same digits plus 1337, and neither the NetworkPolicy API nor the kubelet knows the address - only Istio's data plane rewrites the probe source onto it. Both read for ztunnel 1.30.6, and a change in either fails silently. The exception in ci/run-networkpolicy-test.sh stays pinned to the numbers that test names, and additionally asserts this chart declares the same defaults - never "whatever the value says", which would let a mistyped port in through a configurable field. No version change: nothing is published yet, the default is still off, and the render at defaults is byte-identical - re-diffed against origin/main.
362 lines
17 KiB
YAML
362 lines
17 KiB
YAML
{{- /*
|
|
Two NetworkPolicies, one per workload (Gitea #5).
|
|
|
|
THIS CHART RENDERS TWO POD TEMPLATES AND THEY NEED DIFFERENT RULES, which is
|
|
why this file is written out rather than generated from one shape like most of
|
|
the sweep. A single policy selecting both would have to be the UNION of what
|
|
each needs - Synapse's federation egress granted to the delegation server, and
|
|
the delegation server's ports granted to Synapse - and a policy that is the
|
|
union of two workloads' needs is strictly weaker than two policies, while
|
|
looking like one decision.
|
|
|
|
<release>-<chart>-networkpolicy the Synapse StatefulSet
|
|
<release>-<chart>-delegation-networkpolicy the delegation nginx Deployment
|
|
|
|
The split matters most on egress. Synapse federates: it opens connections to
|
|
other people's homeservers, to push gateways and to identity servers, so its
|
|
egress rule is necessarily wide. The delegation server serves two static JSON
|
|
documents under /.well-known/matrix and calls NOTHING - so its egress is DNS
|
|
and nothing else, which is as tight as this object gets. Merging them would
|
|
have handed Synapse's reach to an nginx whose entire job is to return a
|
|
constant.
|
|
|
|
BOTH WORKLOADS SIT BEHIND THE SAME INGRESS, on different paths (/ to Synapse,
|
|
/.well-known/matrix to the delegation server), so both carry the ingress
|
|
controller in their restricted branch - and both would go dark together if it
|
|
were wrong.
|
|
|
|
AND BOTH PROBE PORTS STAY OPEN BY DEFAULT, because all three probes of each
|
|
workload address its own HTTP port. The kubelet probes from the IP of the
|
|
worker node running the pod, which no chart can know; narrowing is
|
|
networkPolicy.probeSources, and it applies to both policies at once.
|
|
|
|
WHAT THESE POLICIES DO NOT DO:
|
|
- EGRESS IS BY PORT, NEVER BY DESTINATION. A NetworkPolicy matches CIDRs and
|
|
selectors and never hostnames, so "only the homeservers we federate with"
|
|
cannot be expressed - and with Canal (Flannel plus Calico) there is no FQDN
|
|
rule to reach for. Federation therefore means 443 and 8448 to any address,
|
|
which is the honest description: Synapse may contact any homeserver on the
|
|
internet, because that is what federation is.
|
|
- THE DATABASE PORT IS A GUESS. Synapse's Postgres connection lives inside
|
|
secrets.configSecretRef, a 1Password item this chart never parses, so 5432
|
|
is the standard port and not the configured one.
|
|
- A POLICY IS ONLY A BOUNDARY WHERE THE CNI ENFORCES ONE.
|
|
*/ -}}
|
|
{{- if .Values.networkPolicy.enabled }}
|
|
{{- $np := .Values.networkPolicy }}
|
|
{{- $ns := .Release.Namespace }}
|
|
{{- $probeSources := $np.probeSources | default list }}
|
|
{{- $allowedNs := $np.allowedNamespaces | default list }}
|
|
{{- $ic := $np.ingressController | default dict }}
|
|
{{- $routed := or .Values.ingress.enabled .Values.httpRoute.enabled }}
|
|
{{- /* networkPolicy.ingress - the policy's INBOUND side, and not the same thing
|
|
as networkPolicy.ingressController (the SOURCE SELECTOR for the ingress
|
|
controller's pods, a SIBLING of it and no part of it) nor as the
|
|
top-level .Values.ingress (the Ingress object). All three match a search
|
|
for "ingress"; one round of this work already mistook the second for an
|
|
inbound escape hatch on the strength of the substring. */}}
|
|
{{- $npIngress := $np.ingress | default dict }}
|
|
{{- /* NULL-SAFE ON PURPOSE (#99): ambientMesh is an OBJECT, so every read goes
|
|
through a fallback reproducing THIS chart's declared defaults. An
|
|
operator who deletes the object, or writes only "enabled: true", must
|
|
land on the defaults and not in a nil dereference - #136 shipped exactly
|
|
that defect one value away, where the NOTES.txt branch dereferenced
|
|
.networkPolicy.egress directly. Clearing the probe-source list restores
|
|
the constant rather than removing the ipBlock, deliberately: the two
|
|
halves are not separable, and enabled: false is how you open nothing. */}}
|
|
{{- $ambient := $npIngress.ambientMesh | default dict }}
|
|
{{- $ambientEnabled := $ambient.enabled }}
|
|
{{- $ambientPort := $ambient.port | default 15008 }}
|
|
{{- $ambientProbeSources := $ambient.redirectedProbeSources | default (list "169.254.7.127/32") }}
|
|
apiVersion: networking.k8s.io/v1
|
|
kind: NetworkPolicy
|
|
metadata:
|
|
name: {{ .Release.Name }}-{{ .Chart.Name }}-networkpolicy
|
|
labels:
|
|
app: {{ .Release.Name }}-{{ .Chart.Name }}-networkpolicy
|
|
spec:
|
|
{{- /*
|
|
Selects the Synapse StatefulSet's pods - NOT the delegation Deployment's,
|
|
which has its own policy below. ci/run-networkpolicy-test.sh reports which pod
|
|
templates each selector matches, so a label renamed in one of the two
|
|
workloads shows up as a finding rather than as a policy that filters nothing.
|
|
*/}}
|
|
podSelector:
|
|
matchLabels:
|
|
app: {{ .Release.Name }}-{{ .Chart.Name }}-sfs
|
|
policyTypes:
|
|
- Ingress
|
|
- Egress
|
|
ingress:
|
|
{{- /*
|
|
8008 - Synapse's client and federation API, the port the Service forwards 80
|
|
to, and the port all three probes address. The probe path.
|
|
*/}}
|
|
- ports:
|
|
- protocol: TCP
|
|
port: 8008
|
|
{{- if $ambientEnabled }}
|
|
{{- /* FIRST HALF OF networkPolicy.ingress.ambientMesh (Gitea #143). 15008/TCP
|
|
is ztunnel's HBONE tunnel, the port every inbound connection arrives on
|
|
once the namespace carries istio.io/dataplane-mode - ambient "does not
|
|
and will never bypass Kubernetes NetworkPolicy enforcement", so without
|
|
it a pod in the mesh is simply unreachable.
|
|
|
|
It is the one port here that is NOT a containerPort of the selected pod
|
|
spec: ztunnel terminates it, not the application. ci/run-networkpolicy-
|
|
test.sh therefore holds a NAMED exception for exactly this number, gated
|
|
on exactly this value - and assertion 7 of that script proves, on a
|
|
second render, that 15008 stays the ONLY such port and that it is absent
|
|
while the value is false.
|
|
|
|
AN ISTIO CONSTANT, NOT A STANDARD: 15008 is not registered with
|
|
IANA for this, no protocol prescribes it, and Kubernetes knows
|
|
nothing about it - Istio picked it for HBONE and Istio can change
|
|
it. Hence a VALUE with a documented default rather than a literal
|
|
here, so an operator can correct it without a chart release. Read
|
|
for ztunnel 1.30.6. The exception ci/run-networkpolicy-test.sh
|
|
grants is pinned to the number THAT TEST names, plus a check that
|
|
this chart declares the same default - never to whatever the value
|
|
says, because that would let a mistyped port ride in through a
|
|
configurable field (design A of #143).
|
|
|
|
NOT MEASURED: that 15008 alone suffices comes from Istio's
|
|
documentation. Measured on this cluster is only which listeners ztunnel
|
|
opens. Provable at the first namespace to join the mesh; none has. */}}
|
|
- protocol: TCP
|
|
port: {{ int $ambientPort }}
|
|
{{- end }}
|
|
{{- if $probeSources }}
|
|
{{- /*
|
|
RESTRICTED, because the operator supplied probeSources. The kubelet's
|
|
probes come from the IP of the worker node running the pod, so that
|
|
address has to be in the list or the pod never becomes Ready - and a pod
|
|
that is never Ready reads as a broken application rather than as a policy.
|
|
|
|
MIND THE FEDERATION SIDE OF THIS RULE. Inbound federation traffic arrives
|
|
through the Ingress, so the controller entry below is what keeps other
|
|
homeservers able to reach this one. Restricting this rule without it does
|
|
not merely hide a web UI: it stops federation inbound, which looks like
|
|
other servers being unreachable rather than like a local policy.
|
|
*/}}
|
|
from:
|
|
- namespaceSelector:
|
|
matchLabels:
|
|
{{- /*
|
|
Set by Kubernetes itself on every namespace (>= 1.21), so this
|
|
needs no labelling work on a namespace this chart does not own.
|
|
*/}}
|
|
kubernetes.io/metadata.name: {{ $ns }}
|
|
{{- if and $routed $ic.namespace }}
|
|
{{- /*
|
|
ONE from-entry with TWO selectors, which is an AND: pods carrying these
|
|
labels IN that namespace. Written as two list entries it would be an OR,
|
|
and would open the port to every pod in kube-system.
|
|
*/}}
|
|
- namespaceSelector:
|
|
matchLabels:
|
|
kubernetes.io/metadata.name: {{ $ic.namespace | quote }}
|
|
{{- with $ic.podLabels }}
|
|
podSelector:
|
|
matchLabels:
|
|
{{- toYaml . | nindent 14 }}
|
|
{{- end }}
|
|
{{- end }}
|
|
{{- range $allowedNs }}
|
|
- namespaceSelector:
|
|
matchLabels:
|
|
kubernetes.io/metadata.name: {{ . | quote }}
|
|
{{- end }}
|
|
{{- range $probeSources }}
|
|
- ipBlock:
|
|
cidr: {{ . | quote }}
|
|
{{- end }}
|
|
{{- if $ambientEnabled }}
|
|
{{- /* SECOND HALF OF networkPolicy.ingress.ambientMesh, and the reason it is
|
|
one value and not two. Inside the mesh the kubelet's probes are
|
|
redirected and no longer arrive from the worker node's address: they
|
|
arrive from 169.254.7.127.
|
|
|
|
An operator who only opens 15008 by hand writes a policy that passes
|
|
review - and the moment probeSources is non-empty every probe is
|
|
dropped, the pod never becomes Ready, and the symptom reads as a broken
|
|
application rather than as a policy. That is assertion 4 of
|
|
ci/run-networkpolicy-test.sh one value further along. A switch whose
|
|
halves can only be had together cannot be set halfway.
|
|
|
|
Only in this branch, because this is the only branch with a "from" list:
|
|
with no sources restricted the rule is open and needs no peer. The IPv6
|
|
counterpart is deliberately absent - this cluster is SingleStack IPv4,
|
|
and an ipBlock of the wrong family is accepted and matches nothing.
|
|
|
|
WHERE 169.254.7.127 COMES FROM - vendor magic, not protocol:
|
|
it is LINK-LOCAL (169.254.0.0/16, RFC 3927), a range reserved for
|
|
node-local use and never routed; Istio's IPv6 counterpart
|
|
fd16:9254:7127:1337:ffff:ffff:ffff:ffff carries the SAME DIGITS
|
|
plus 1337; and KUBERNETES DOES NOT KNOW THE ADDRESS - neither the
|
|
NetworkPolicy API nor the kubelet - only Istio's data plane
|
|
rewrites the probe source onto it. Read for ztunnel 1.30.6, and if
|
|
Istio changes it THE FAILURE IS SILENT: probes dropped, and only
|
|
where probeSources is non-empty, which is nowhere today.
|
|
|
|
A LIST, not a string, and the one case where that does not
|
|
contradict "a constant, not a configuration": a DUAL-STACK cluster
|
|
needs BOTH entries at once. This cluster is SingleStack IPv4, so
|
|
the IPv6 counterpart is left out - an ipBlock of the wrong family
|
|
is accepted and matches nothing. Add it ALONGSIDE, never instead.
|
|
|
|
UNPROVEN IN OPERATION: probeSources is empty in every chart here, so no
|
|
release reaches this branch. CI proves its render, through the overlays
|
|
in ci/networkpolicy. */}}
|
|
{{- range $ambientProbeSources }}
|
|
- ipBlock:
|
|
cidr: {{ . | quote }}
|
|
{{- end }}
|
|
{{- end }}
|
|
{{- else }}
|
|
{{- /*
|
|
NO "from" SELECTOR, WHICH MEANS OPEN - deliberately, and only inside the
|
|
cluster, because this port carries no NodePort. The alternative was to
|
|
guess the worker node CIDRs, and a wrong guess blocks every kubelet probe:
|
|
the pod stays un-Ready, the rollout times out, and the cause looks like
|
|
the application. An empty "from: []" would be worse than either - it
|
|
allows everything too, but while reading as a restriction.
|
|
*/}}
|
|
{{- end }}
|
|
egress:
|
|
{{- /*
|
|
DNS, unconditionally. There is no value to switch this off, and that is
|
|
deliberate: a knob whose only other setting breaks every name lookup in the
|
|
pod is a trap and not a feature. Synapse needs it more than most - every
|
|
federated server is resolved by name, including its SRV and .well-known
|
|
delegation lookups.
|
|
*/}}
|
|
- ports:
|
|
- protocol: UDP
|
|
port: 53
|
|
- protocol: TCP
|
|
port: 53
|
|
{{- /*
|
|
PostgreSQL on the STANDARD port. The connection lives inside
|
|
secrets.configSecretRef, a 1Password item this chart never parses, so this
|
|
is the conventional port and not the configured one - a Postgres elsewhere
|
|
is blocked and belongs in networkPolicy.egress.extraPorts.
|
|
*/}}
|
|
- ports:
|
|
- protocol: TCP
|
|
port: 5432
|
|
{{- /*
|
|
FEDERATION, and this is the rule that cannot be narrowed. 443 and 8448 to
|
|
ANY destination: 8448 is the Matrix federation port, 443 is where a
|
|
delegating homeserver puts its federation API instead, and both are also
|
|
what Synapse uses to fetch remote media, to reach a push gateway and to
|
|
reach an identity server.
|
|
|
|
"Any destination" is not laziness here, it is the specification: a
|
|
federating homeserver may contact any other homeserver on the internet, and
|
|
this object cannot match a hostname in any case. What the cap still buys is
|
|
real - Synapse cannot open a connection on any OTHER port, so a compromised
|
|
instance reaches no database, no SSH and no mail server elsewhere in the
|
|
cluster.
|
|
*/}}
|
|
- ports:
|
|
- protocol: TCP
|
|
port: 443
|
|
- protocol: TCP
|
|
port: 8448
|
|
{{- with $np.egress }}
|
|
{{- with .extraPorts }}
|
|
{{- /*
|
|
Ports only, to any destination - see the header. The operator's escape hatch
|
|
from the egress cap, and the only alternative to switching the whole policy
|
|
off.
|
|
*/}}
|
|
- ports:
|
|
{{- range . }}
|
|
- protocol: TCP
|
|
port: {{ int . }}
|
|
{{- end }}
|
|
{{- end }}
|
|
{{- end }}
|
|
---
|
|
apiVersion: networking.k8s.io/v1
|
|
kind: NetworkPolicy
|
|
metadata:
|
|
name: {{ .Release.Name }}-{{ .Chart.Name }}-delegation-networkpolicy
|
|
labels:
|
|
app: {{ .Release.Name }}-{{ .Chart.Name }}-delegation-networkpolicy
|
|
spec:
|
|
{{- /* Selects the delegation nginx Deployment's pods only. */}}
|
|
podSelector:
|
|
matchLabels:
|
|
app: {{ .Release.Name }}-{{ .Chart.Name }}-delegation-nginx-deployment
|
|
policyTypes:
|
|
- Ingress
|
|
- Egress
|
|
ingress:
|
|
{{- /*
|
|
8080 - the only port this pod declares, the port the delegation Service
|
|
forwards 80 to, and the port all three probes address. The probe path.
|
|
*/}}
|
|
- ports:
|
|
- protocol: TCP
|
|
port: 8080
|
|
{{- if $probeSources }}
|
|
{{- /*
|
|
RESTRICTED, because the operator supplied probeSources - the same value
|
|
governs both policies in this chart. The kubelet probes from the node's
|
|
own IP, so it has to be in the list or this pod never becomes Ready.
|
|
|
|
AND THE CONSUMERS OF THIS ONE ARE OTHER PEOPLE'S HOMESERVERS. The
|
|
delegation documents under /.well-known/matrix are fetched by every remote
|
|
server that wants to federate with this one, through the Ingress. Restrict
|
|
this rule without the controller entry and federation stops being
|
|
discoverable - a failure that shows up on OTHER servers, not here.
|
|
*/}}
|
|
from:
|
|
- namespaceSelector:
|
|
matchLabels:
|
|
kubernetes.io/metadata.name: {{ $ns }}
|
|
{{- if and $routed $ic.namespace }}
|
|
- namespaceSelector:
|
|
matchLabels:
|
|
kubernetes.io/metadata.name: {{ $ic.namespace | quote }}
|
|
{{- with $ic.podLabels }}
|
|
podSelector:
|
|
matchLabels:
|
|
{{- toYaml . | nindent 14 }}
|
|
{{- end }}
|
|
{{- end }}
|
|
{{- range $allowedNs }}
|
|
- namespaceSelector:
|
|
matchLabels:
|
|
kubernetes.io/metadata.name: {{ . | quote }}
|
|
{{- end }}
|
|
{{- range $probeSources }}
|
|
- ipBlock:
|
|
cidr: {{ . | quote }}
|
|
{{- end }}
|
|
{{- else }}
|
|
{{- /* Open inside the cluster, as above - the probe path. */}}
|
|
{{- end }}
|
|
egress:
|
|
{{- /*
|
|
DNS AND NOTHING ELSE, and this is the tightest egress rule in the chart on
|
|
purpose. This nginx serves two static JSON documents and calls nobody; it
|
|
does not touch the database, it does not federate, it has no upstream. Even
|
|
DNS is only here because the container resolves at start.
|
|
|
|
This is exactly what the two-policy split buys. Folded into Synapse's
|
|
policy, this pod would have been permitted 443, 8448 and 5432 - full
|
|
federation reach and the database - for a workload whose entire job is to
|
|
return a constant. networkPolicy.egress.extraPorts deliberately does NOT
|
|
apply here: an entry meant for Synapse's database has no business widening
|
|
this.
|
|
*/}}
|
|
- ports:
|
|
- protocol: UDP
|
|
port: 53
|
|
- protocol: TCP
|
|
port: 53
|
|
{{- end }}
|