jklein
b0572834e0
Gitea #143. Additive, default false, so the render is byte-identical to the previous release - proven by diffing both renders against origin/main rather than asserted. Switched on it adds 15008/TCP, ztunnel's HBONE tunnel, to the rule carrying the probe port, and - inside that rule's source-restricted branch - 169.254.7.127/32, the address the kubelet's probes arrive from once the pod is in the mesh. Ambient does not bypass NetworkPolicy enforcement, so without the port a pod whose namespace has joined the mesh is unreachable. The two halves are one value because the second is the one nobody thinks of: 15008 added by hand leaves a policy that passes review and drops every probe the first time probeSources is narrowed, and a pod that never becomes Ready reads as a broken application rather than as a policy. Minor, not major: additive value, unchanged default, unchanged render.
322 lines
14 KiB
YAML
322 lines
14 KiB
YAML
{{- /*
|
|
Two NetworkPolicies, one per workload (Gitea #5).
|
|
|
|
THIS CHART RENDERS TWO POD TEMPLATES AND THEY NEED DIFFERENT RULES, which is
|
|
why this file is written out rather than generated from one shape like most of
|
|
the sweep. A single policy selecting both would have to be the UNION of what
|
|
each needs - Synapse's federation egress granted to the delegation server, and
|
|
the delegation server's ports granted to Synapse - and a policy that is the
|
|
union of two workloads' needs is strictly weaker than two policies, while
|
|
looking like one decision.
|
|
|
|
<release>-<chart>-networkpolicy the Synapse StatefulSet
|
|
<release>-<chart>-delegation-networkpolicy the delegation nginx Deployment
|
|
|
|
The split matters most on egress. Synapse federates: it opens connections to
|
|
other people's homeservers, to push gateways and to identity servers, so its
|
|
egress rule is necessarily wide. The delegation server serves two static JSON
|
|
documents under /.well-known/matrix and calls NOTHING - so its egress is DNS
|
|
and nothing else, which is as tight as this object gets. Merging them would
|
|
have handed Synapse's reach to an nginx whose entire job is to return a
|
|
constant.
|
|
|
|
BOTH WORKLOADS SIT BEHIND THE SAME INGRESS, on different paths (/ to Synapse,
|
|
/.well-known/matrix to the delegation server), so both carry the ingress
|
|
controller in their restricted branch - and both would go dark together if it
|
|
were wrong.
|
|
|
|
AND BOTH PROBE PORTS STAY OPEN BY DEFAULT, because all three probes of each
|
|
workload address its own HTTP port. The kubelet probes from the IP of the
|
|
worker node running the pod, which no chart can know; narrowing is
|
|
networkPolicy.probeSources, and it applies to both policies at once.
|
|
|
|
WHAT THESE POLICIES DO NOT DO:
|
|
- EGRESS IS BY PORT, NEVER BY DESTINATION. A NetworkPolicy matches CIDRs and
|
|
selectors and never hostnames, so "only the homeservers we federate with"
|
|
cannot be expressed - and with Canal (Flannel plus Calico) there is no FQDN
|
|
rule to reach for. Federation therefore means 443 and 8448 to any address,
|
|
which is the honest description: Synapse may contact any homeserver on the
|
|
internet, because that is what federation is.
|
|
- THE DATABASE PORT IS A GUESS. Synapse's Postgres connection lives inside
|
|
secrets.configSecretRef, a 1Password item this chart never parses, so 5432
|
|
is the standard port and not the configured one.
|
|
- A POLICY IS ONLY A BOUNDARY WHERE THE CNI ENFORCES ONE.
|
|
*/ -}}
|
|
{{- if .Values.networkPolicy.enabled }}
|
|
{{- $np := .Values.networkPolicy }}
|
|
{{- $ns := .Release.Namespace }}
|
|
{{- $probeSources := $np.probeSources | default list }}
|
|
{{- $allowedNs := $np.allowedNamespaces | default list }}
|
|
{{- $ic := $np.ingressController | default dict }}
|
|
{{- $routed := or .Values.ingress.enabled .Values.httpRoute.enabled }}
|
|
{{- /* networkPolicy.ingress - the policy's INBOUND side, and not the same thing
|
|
as networkPolicy.ingressController (the SOURCE SELECTOR for the ingress
|
|
controller's pods, a SIBLING of it and no part of it) nor as the
|
|
top-level .Values.ingress (the Ingress object). All three match a search
|
|
for "ingress"; one round of this work already mistook the second for an
|
|
inbound escape hatch on the strength of the substring. */}}
|
|
{{- $npIngress := $np.ingress | default dict }}
|
|
{{- $ambientMesh := $npIngress.ambientMesh }}
|
|
apiVersion: networking.k8s.io/v1
|
|
kind: NetworkPolicy
|
|
metadata:
|
|
name: {{ .Release.Name }}-{{ .Chart.Name }}-networkpolicy
|
|
labels:
|
|
app: {{ .Release.Name }}-{{ .Chart.Name }}-networkpolicy
|
|
spec:
|
|
{{- /*
|
|
Selects the Synapse StatefulSet's pods - NOT the delegation Deployment's,
|
|
which has its own policy below. ci/run-networkpolicy-test.sh reports which pod
|
|
templates each selector matches, so a label renamed in one of the two
|
|
workloads shows up as a finding rather than as a policy that filters nothing.
|
|
*/}}
|
|
podSelector:
|
|
matchLabels:
|
|
app: {{ .Release.Name }}-{{ .Chart.Name }}-sfs
|
|
policyTypes:
|
|
- Ingress
|
|
- Egress
|
|
ingress:
|
|
{{- /*
|
|
8008 - Synapse's client and federation API, the port the Service forwards 80
|
|
to, and the port all three probes address. The probe path.
|
|
*/}}
|
|
- ports:
|
|
- protocol: TCP
|
|
port: 8008
|
|
{{- if $ambientMesh }}
|
|
{{- /* FIRST HALF OF networkPolicy.ingress.ambientMesh (Gitea #143). 15008/TCP
|
|
is ztunnel's HBONE tunnel, the port every inbound connection arrives on
|
|
once the namespace carries istio.io/dataplane-mode - ambient "does not
|
|
and will never bypass Kubernetes NetworkPolicy enforcement", so without
|
|
it a pod in the mesh is simply unreachable.
|
|
|
|
It is the one port here that is NOT a containerPort of the selected pod
|
|
spec: ztunnel terminates it, not the application. ci/run-networkpolicy-
|
|
test.sh therefore holds a NAMED exception for exactly this number, gated
|
|
on exactly this value - and assertion 7 of that script proves, on a
|
|
second render, that 15008 stays the ONLY such port and that it is absent
|
|
while the value is false.
|
|
|
|
NOT MEASURED: that 15008 alone suffices comes from Istio's
|
|
documentation. Measured on this cluster is only which listeners ztunnel
|
|
opens. Provable at the first namespace to join the mesh; none has. */}}
|
|
- protocol: TCP
|
|
port: 15008
|
|
{{- end }}
|
|
{{- if $probeSources }}
|
|
{{- /*
|
|
RESTRICTED, because the operator supplied probeSources. The kubelet's
|
|
probes come from the IP of the worker node running the pod, so that
|
|
address has to be in the list or the pod never becomes Ready - and a pod
|
|
that is never Ready reads as a broken application rather than as a policy.
|
|
|
|
MIND THE FEDERATION SIDE OF THIS RULE. Inbound federation traffic arrives
|
|
through the Ingress, so the controller entry below is what keeps other
|
|
homeservers able to reach this one. Restricting this rule without it does
|
|
not merely hide a web UI: it stops federation inbound, which looks like
|
|
other servers being unreachable rather than like a local policy.
|
|
*/}}
|
|
from:
|
|
- namespaceSelector:
|
|
matchLabels:
|
|
{{- /*
|
|
Set by Kubernetes itself on every namespace (>= 1.21), so this
|
|
needs no labelling work on a namespace this chart does not own.
|
|
*/}}
|
|
kubernetes.io/metadata.name: {{ $ns }}
|
|
{{- if and $routed $ic.namespace }}
|
|
{{- /*
|
|
ONE from-entry with TWO selectors, which is an AND: pods carrying these
|
|
labels IN that namespace. Written as two list entries it would be an OR,
|
|
and would open the port to every pod in kube-system.
|
|
*/}}
|
|
- namespaceSelector:
|
|
matchLabels:
|
|
kubernetes.io/metadata.name: {{ $ic.namespace | quote }}
|
|
{{- with $ic.podLabels }}
|
|
podSelector:
|
|
matchLabels:
|
|
{{- toYaml . | nindent 14 }}
|
|
{{- end }}
|
|
{{- end }}
|
|
{{- range $allowedNs }}
|
|
- namespaceSelector:
|
|
matchLabels:
|
|
kubernetes.io/metadata.name: {{ . | quote }}
|
|
{{- end }}
|
|
{{- range $probeSources }}
|
|
- ipBlock:
|
|
cidr: {{ . | quote }}
|
|
{{- end }}
|
|
{{- if $ambientMesh }}
|
|
{{- /* SECOND HALF OF networkPolicy.ingress.ambientMesh, and the reason it is
|
|
one value and not two. Inside the mesh the kubelet's probes are
|
|
redirected and no longer arrive from the worker node's address: they
|
|
arrive from 169.254.7.127.
|
|
|
|
An operator who only opens 15008 by hand writes a policy that passes
|
|
review - and the moment probeSources is non-empty every probe is
|
|
dropped, the pod never becomes Ready, and the symptom reads as a broken
|
|
application rather than as a policy. That is assertion 4 of
|
|
ci/run-networkpolicy-test.sh one value further along. A switch whose
|
|
halves can only be had together cannot be set halfway.
|
|
|
|
Only in this branch, because this is the only branch with a "from" list:
|
|
with no sources restricted the rule is open and needs no peer. The IPv6
|
|
counterpart is deliberately absent - this cluster is SingleStack IPv4,
|
|
and an ipBlock of the wrong family is accepted and matches nothing.
|
|
|
|
UNPROVEN IN OPERATION: probeSources is empty in every chart here, so no
|
|
release reaches this branch. CI proves its render, through the overlays
|
|
in ci/networkpolicy. */}}
|
|
- ipBlock:
|
|
cidr: 169.254.7.127/32
|
|
{{- end }}
|
|
{{- else }}
|
|
{{- /*
|
|
NO "from" SELECTOR, WHICH MEANS OPEN - deliberately, and only inside the
|
|
cluster, because this port carries no NodePort. The alternative was to
|
|
guess the worker node CIDRs, and a wrong guess blocks every kubelet probe:
|
|
the pod stays un-Ready, the rollout times out, and the cause looks like
|
|
the application. An empty "from: []" would be worse than either - it
|
|
allows everything too, but while reading as a restriction.
|
|
*/}}
|
|
{{- end }}
|
|
egress:
|
|
{{- /*
|
|
DNS, unconditionally. There is no value to switch this off, and that is
|
|
deliberate: a knob whose only other setting breaks every name lookup in the
|
|
pod is a trap and not a feature. Synapse needs it more than most - every
|
|
federated server is resolved by name, including its SRV and .well-known
|
|
delegation lookups.
|
|
*/}}
|
|
- ports:
|
|
- protocol: UDP
|
|
port: 53
|
|
- protocol: TCP
|
|
port: 53
|
|
{{- /*
|
|
PostgreSQL on the STANDARD port. The connection lives inside
|
|
secrets.configSecretRef, a 1Password item this chart never parses, so this
|
|
is the conventional port and not the configured one - a Postgres elsewhere
|
|
is blocked and belongs in networkPolicy.egress.extraPorts.
|
|
*/}}
|
|
- ports:
|
|
- protocol: TCP
|
|
port: 5432
|
|
{{- /*
|
|
FEDERATION, and this is the rule that cannot be narrowed. 443 and 8448 to
|
|
ANY destination: 8448 is the Matrix federation port, 443 is where a
|
|
delegating homeserver puts its federation API instead, and both are also
|
|
what Synapse uses to fetch remote media, to reach a push gateway and to
|
|
reach an identity server.
|
|
|
|
"Any destination" is not laziness here, it is the specification: a
|
|
federating homeserver may contact any other homeserver on the internet, and
|
|
this object cannot match a hostname in any case. What the cap still buys is
|
|
real - Synapse cannot open a connection on any OTHER port, so a compromised
|
|
instance reaches no database, no SSH and no mail server elsewhere in the
|
|
cluster.
|
|
*/}}
|
|
- ports:
|
|
- protocol: TCP
|
|
port: 443
|
|
- protocol: TCP
|
|
port: 8448
|
|
{{- with $np.egress }}
|
|
{{- with .extraPorts }}
|
|
{{- /*
|
|
Ports only, to any destination - see the header. The operator's escape hatch
|
|
from the egress cap, and the only alternative to switching the whole policy
|
|
off.
|
|
*/}}
|
|
- ports:
|
|
{{- range . }}
|
|
- protocol: TCP
|
|
port: {{ int . }}
|
|
{{- end }}
|
|
{{- end }}
|
|
{{- end }}
|
|
---
|
|
apiVersion: networking.k8s.io/v1
|
|
kind: NetworkPolicy
|
|
metadata:
|
|
name: {{ .Release.Name }}-{{ .Chart.Name }}-delegation-networkpolicy
|
|
labels:
|
|
app: {{ .Release.Name }}-{{ .Chart.Name }}-delegation-networkpolicy
|
|
spec:
|
|
{{- /* Selects the delegation nginx Deployment's pods only. */}}
|
|
podSelector:
|
|
matchLabels:
|
|
app: {{ .Release.Name }}-{{ .Chart.Name }}-delegation-nginx-deployment
|
|
policyTypes:
|
|
- Ingress
|
|
- Egress
|
|
ingress:
|
|
{{- /*
|
|
8080 - the only port this pod declares, the port the delegation Service
|
|
forwards 80 to, and the port all three probes address. The probe path.
|
|
*/}}
|
|
- ports:
|
|
- protocol: TCP
|
|
port: 8080
|
|
{{- if $probeSources }}
|
|
{{- /*
|
|
RESTRICTED, because the operator supplied probeSources - the same value
|
|
governs both policies in this chart. The kubelet probes from the node's
|
|
own IP, so it has to be in the list or this pod never becomes Ready.
|
|
|
|
AND THE CONSUMERS OF THIS ONE ARE OTHER PEOPLE'S HOMESERVERS. The
|
|
delegation documents under /.well-known/matrix are fetched by every remote
|
|
server that wants to federate with this one, through the Ingress. Restrict
|
|
this rule without the controller entry and federation stops being
|
|
discoverable - a failure that shows up on OTHER servers, not here.
|
|
*/}}
|
|
from:
|
|
- namespaceSelector:
|
|
matchLabels:
|
|
kubernetes.io/metadata.name: {{ $ns }}
|
|
{{- if and $routed $ic.namespace }}
|
|
- namespaceSelector:
|
|
matchLabels:
|
|
kubernetes.io/metadata.name: {{ $ic.namespace | quote }}
|
|
{{- with $ic.podLabels }}
|
|
podSelector:
|
|
matchLabels:
|
|
{{- toYaml . | nindent 14 }}
|
|
{{- end }}
|
|
{{- end }}
|
|
{{- range $allowedNs }}
|
|
- namespaceSelector:
|
|
matchLabels:
|
|
kubernetes.io/metadata.name: {{ . | quote }}
|
|
{{- end }}
|
|
{{- range $probeSources }}
|
|
- ipBlock:
|
|
cidr: {{ . | quote }}
|
|
{{- end }}
|
|
{{- else }}
|
|
{{- /* Open inside the cluster, as above - the probe path. */}}
|
|
{{- end }}
|
|
egress:
|
|
{{- /*
|
|
DNS AND NOTHING ELSE, and this is the tightest egress rule in the chart on
|
|
purpose. This nginx serves two static JSON documents and calls nobody; it
|
|
does not touch the database, it does not federate, it has no upstream. Even
|
|
DNS is only here because the container resolves at start.
|
|
|
|
This is exactly what the two-policy split buys. Folded into Synapse's
|
|
policy, this pod would have been permitted 443, 8448 and 5432 - full
|
|
federation reach and the database - for a workload whose entire job is to
|
|
return a constant. networkPolicy.egress.extraPorts deliberately does NOT
|
|
apply here: an entry meant for Synapse's database has no business widening
|
|
this.
|
|
*/}}
|
|
- ports:
|
|
- protocol: UDP
|
|
port: 53
|
|
- protocol: TCP
|
|
port: 53
|
|
{{- end }}
|