jklein
2d7c11d137
Gitea #143, maintainer's instruction. The HBONE port and the redirected probe source are ISTIO CONSTANTS, not standards, so they move from template literals into values with documented defaults - an operator can then correct them when Istio changes a number, without waiting for a chart release. One object under one "enabled", so the two halves still cannot be set separately; two values could drift apart, which is what this design exists to prevent. Every read is null-safe (#99), so a deleted object or one carrying only "enabled: true" falls back to the defaults declared here rather than dereferencing nil - the defect #136 shipped one value away. Provenance now written on each constant: 15008 is Istio's HBONE port, not IANA-registered for this and unknown to Kubernetes; 169.254.7.127 is link-local (RFC 3927, never routed), its IPv6 counterpart carries the same digits plus 1337, and neither the NetworkPolicy API nor the kubelet knows the address - only Istio's data plane rewrites the probe source onto it. Both read for ztunnel 1.30.6, and a change in either fails silently. The exception in ci/run-networkpolicy-test.sh stays pinned to the numbers that test names, and additionally asserts this chart declares the same defaults - never "whatever the value says", which would let a mistyped port in through a configurable field. No version change: nothing is published yet, the default is still off, and the render at defaults is byte-identical - re-diffed against origin/main.
374 lines
19 KiB
YAML
374 lines
19 KiB
YAML
{{- /*
|
|
NetworkPolicy for keycloak (Gitea #5).
|
|
|
|
THE RICHEST POLICY OF THIS ROLLOUT, AND THE ONE WITH THE LARGEST BLAST RADIUS.
|
|
Read both halves of that before changing anything here.
|
|
|
|
Keycloak is the only chart in the repository whose three ports fall into three
|
|
DIFFERENT classes, so it is the one place the convention is visible in full:
|
|
|
|
http (8080) NOT addressed by any probe, so it CAN be restricted by
|
|
source - and it is. Own namespace, allowedNamespaces, and
|
|
the ingress controller. This is the only real inbound
|
|
restriction in the sweep, and it is possible only because
|
|
the probes use a different port.
|
|
management (9000) the port all three probes address, so it stays open by
|
|
default. That is unfortunate here specifically: 9000
|
|
carries the health and metrics surface UNAUTHENTICATED,
|
|
and "no Ingress" was its only protection before this
|
|
object existed. The policy improves it from "anything that
|
|
can route to the pod" to "anything inside the cluster",
|
|
and networkPolicy.probeSources is what closes the rest -
|
|
at the price of needing the worker node IPs.
|
|
jgroups (7800) the cluster bus between this release's own replicas.
|
|
Restricted to the pods this policy itself selects, which
|
|
is the narrowest rule this object can express.
|
|
|
|
All three port numbers and the database port are DERIVED from the chart's own
|
|
values through keycloak.settings, not written down twice: an operator who moves
|
|
keycloak.ports.http moves the policy with it. The database port is
|
|
keycloak.db.port, so unlike most charts in this sweep the egress rule here is
|
|
the CONFIGURED port and not a guess.
|
|
|
|
THE BLAST RADIUS, stated plainly because it is the largest in this issue: this
|
|
release is the cluster's identity provider. If the ingress-controller source
|
|
below is wrong, every login everywhere breaks, and NOTHING IN CI CAN CATCH IT -
|
|
no ingress controller runs in the install-test, and the probes use 9000, so the
|
|
pod becomes Ready either way. The source was measured on the live cluster
|
|
(DaemonSet rke2-traefik in kube-system, which also serves the Gateway API
|
|
here), but measured labels are not proven traffic. After the first upgrade,
|
|
verify with an actual login rather than with kubectl.
|
|
|
|
AND THE SECOND TRAP, which is the same shape: a consumer that reaches Keycloak
|
|
on its IN-CLUSTER Service address rather than through its public URL is a pod
|
|
in another namespace, and the http rule blocks it. Those namespaces belong in
|
|
allowedNamespaces. Consumers that use the public issuer URL arrive through
|
|
Traefik and are already covered.
|
|
|
|
WHAT THIS OBJECT IS: a ceiling on reachability, not an authentication layer.
|
|
Only the ports declared below are reachable at all - whatever else the process
|
|
binds, now or after the next app update, is blocked without anyone having to
|
|
notice it exists.
|
|
|
|
WHAT IT IS NOT:
|
|
- EGRESS IS COARSE. A NetworkPolicy matches CIDRs and selectors and NEVER
|
|
hostnames, so "only to the database host" cannot be expressed at all. This
|
|
cluster's CNI is Canal (Flannel plus Calico for policy) and has no FQDN
|
|
rule; Calico's GlobalNetworkSet could express one but would bind this chart
|
|
to Calico, the same argument that kept implementation-specific objects out
|
|
of the Gateway-API sweep (#114). Each egress rule therefore permits its
|
|
PORT to any destination.
|
|
- NOT A BOUNDARY WHERE THE CNI DOES NOT ENFORCE ONE. An unenforced policy is
|
|
accepted by the API server and filters nothing, so everything reads as
|
|
configured while nothing is filtered.
|
|
- THE REASONING IS IN TEMPLATE COMMENTS, not in "#" comments: those are part
|
|
of the rendered object, travel into the cluster and appear in every release
|
|
diff. ci/run-networkpolicy-test.sh fails on a rendered "#".
|
|
|
|
THE PORT NUMBERS BELOW ARE POD PORTS, NEVER NODEPORTS. A NetworkPolicy is
|
|
evaluated at the pod's network interface, after kube-proxy has already
|
|
translated a NodePort to the targetPort, so a NodePort number here would match
|
|
nothing while looking strict - and, being the only rule for that port, would
|
|
block every inbound connection instead. ci/run-networkpolicy-test.sh asserts
|
|
every port here against the containerPorts of the pod spec this policy
|
|
selects.
|
|
*/ -}}
|
|
{{- if .Values.networkPolicy.enabled }}
|
|
{{- $settings := fromYaml (include "keycloak.settings" .) }}
|
|
{{- $np := .Values.networkPolicy }}
|
|
{{- $ns := .Release.Namespace }}
|
|
{{- $probeSources := $np.probeSources | default list }}
|
|
{{- $allowedNs := $np.allowedNamespaces | default list }}
|
|
{{- $ic := $np.ingressController | default dict }}
|
|
{{- $routed := or .Values.ingress.enabled .Values.httpRoute.enabled }}
|
|
{{- /* networkPolicy.ingress - the policy's INBOUND side, and not the same thing
|
|
as networkPolicy.ingressController (the SOURCE SELECTOR for the ingress
|
|
controller's pods, a SIBLING of it and no part of it) nor as the
|
|
top-level .Values.ingress (the Ingress object). All three match a search
|
|
for "ingress"; one round of this work already mistook the second for an
|
|
inbound escape hatch on the strength of the substring. */}}
|
|
{{- $npIngress := $np.ingress | default dict }}
|
|
{{- /* NULL-SAFE ON PURPOSE (#99): ambientMesh is an OBJECT, so every read goes
|
|
through a fallback reproducing THIS chart's declared defaults. An
|
|
operator who deletes the object, or writes only "enabled: true", must
|
|
land on the defaults and not in a nil dereference - #136 shipped exactly
|
|
that defect one value away, where the NOTES.txt branch dereferenced
|
|
.networkPolicy.egress directly. Clearing the probe-source list restores
|
|
the constant rather than removing the ipBlock, deliberately: the two
|
|
halves are not separable, and enabled: false is how you open nothing. */}}
|
|
{{- $ambient := $npIngress.ambientMesh | default dict }}
|
|
{{- $ambientEnabled := $ambient.enabled }}
|
|
{{- $ambientPort := $ambient.port | default 15008 }}
|
|
{{- $ambientProbeSources := $ambient.redirectedProbeSources | default (list "169.254.7.127/32") }}
|
|
apiVersion: networking.k8s.io/v1
|
|
kind: NetworkPolicy
|
|
metadata:
|
|
name: {{ .Release.Name }}-{{ .Chart.Name }}-networkpolicy
|
|
labels:
|
|
app: {{ .Release.Name }}-{{ .Chart.Name }}-networkpolicy
|
|
spec:
|
|
{{- /*
|
|
Selects the Deployment's pods, and the same label the PodDisruptionBudget and
|
|
the two Services use. ci/run-networkpolicy-test.sh fails if it matches no pod
|
|
spec of the same render.
|
|
*/}}
|
|
podSelector:
|
|
matchLabels:
|
|
app: {{ .Release.Name }}-{{ .Chart.Name }}-deployment
|
|
policyTypes:
|
|
- Ingress
|
|
{{- /*
|
|
Egress is listed because there is a great deal to restrict and almost all of it
|
|
is known: the database on its configured port, the cluster bus, and HTTPS. What
|
|
is NOT permitted and is the likely first surprise: SMTP. Keycloak sends mail
|
|
(verification, password reset) through a server configured per realm, which
|
|
this chart cannot see - so it needs networkPolicy.egress.extraPorts.
|
|
*/}}
|
|
- Egress
|
|
ingress:
|
|
{{- /*
|
|
THE HTTP PORT, AND THE ONE GENUINELY RESTRICTED INBOUND RULE IN THIS SWEEP.
|
|
|
|
It can be restricted because no probe addresses it - the probes use the
|
|
management port - so there is no kubelet traffic to accommodate and no need for
|
|
probeSources here. Allowed: this release's own namespace, whatever
|
|
allowedNamespaces adds, and the ingress controller.
|
|
|
|
Both selectors of the controller entry sit in ONE list item, which makes them
|
|
an AND: those pods in that namespace. Split into two items they would be an OR
|
|
and would open Keycloak's login endpoint to every pod in kube-system.
|
|
|
|
If this rule is wrong, every login in the cluster breaks and no part of CI will
|
|
say so. See the header.
|
|
*/}}
|
|
- ports:
|
|
- protocol: TCP
|
|
port: {{ $settings.ports.http | int64 }}
|
|
from:
|
|
- namespaceSelector:
|
|
matchLabels:
|
|
{{- /*
|
|
Set by Kubernetes itself on every namespace (>= 1.21), so this needs no
|
|
labelling work on a namespace this chart does not own.
|
|
*/}}
|
|
kubernetes.io/metadata.name: {{ $ns }}
|
|
{{- if and $routed $ic.namespace }}
|
|
{{- /*
|
|
ONE from-entry with TWO selectors, which is an AND: pods carrying these labels
|
|
IN that namespace. Written as two list entries it would be an OR, and would
|
|
open the port to every pod in kube-system.
|
|
*/}}
|
|
- namespaceSelector:
|
|
matchLabels:
|
|
kubernetes.io/metadata.name: {{ $ic.namespace | quote }}
|
|
{{- with $ic.podLabels }}
|
|
podSelector:
|
|
matchLabels:
|
|
{{- toYaml . | nindent 14 }}
|
|
{{- end }}
|
|
{{- end }}
|
|
{{- range $allowedNs }}
|
|
- namespaceSelector:
|
|
matchLabels:
|
|
kubernetes.io/metadata.name: {{ . | quote }}
|
|
{{- end }}
|
|
{{- /*
|
|
THE MANAGEMENT PORT - health, readiness and metrics, and all three probes
|
|
address it. So it is the probe path and stays open by default.
|
|
|
|
That is worth naming rather than glossing: this surface is UNAUTHENTICATED, and
|
|
before this object existed its only protection was that no Ingress pointed at
|
|
it. What this policy adds is a smaller ceiling, not a closed door - the port is
|
|
reachable from inside the cluster. Closing it means listing the worker node IPs
|
|
in networkPolicy.probeSources, with the usual cost of getting them wrong.
|
|
*/}}
|
|
- ports:
|
|
- protocol: TCP
|
|
port: {{ $settings.ports.management | int64 }}
|
|
{{- if $ambientEnabled }}
|
|
{{- /* FIRST HALF OF networkPolicy.ingress.ambientMesh (Gitea #143). 15008/TCP
|
|
is ztunnel's HBONE tunnel, the port every inbound connection arrives on
|
|
once the namespace carries istio.io/dataplane-mode - ambient "does not
|
|
and will never bypass Kubernetes NetworkPolicy enforcement", so without
|
|
it a pod in the mesh is simply unreachable.
|
|
|
|
It is the one port here that is NOT a containerPort of the selected pod
|
|
spec: ztunnel terminates it, not the application. ci/run-networkpolicy-
|
|
test.sh therefore holds a NAMED exception for exactly this number, gated
|
|
on exactly this value - and assertion 7 of that script proves, on a
|
|
second render, that 15008 stays the ONLY such port and that it is absent
|
|
while the value is false.
|
|
|
|
AN ISTIO CONSTANT, NOT A STANDARD: 15008 is not registered with
|
|
IANA for this, no protocol prescribes it, and Kubernetes knows
|
|
nothing about it - Istio picked it for HBONE and Istio can change
|
|
it. Hence a VALUE with a documented default rather than a literal
|
|
here, so an operator can correct it without a chart release. Read
|
|
for ztunnel 1.30.6. The exception ci/run-networkpolicy-test.sh
|
|
grants is pinned to the number THAT TEST names, plus a check that
|
|
this chart declares the same default - never to whatever the value
|
|
says, because that would let a mistyped port ride in through a
|
|
configurable field (design A of #143).
|
|
|
|
NOT MEASURED: that 15008 alone suffices comes from Istio's
|
|
documentation. Measured on this cluster is only which listeners ztunnel
|
|
opens. Provable at the first namespace to join the mesh; none has. */}}
|
|
- protocol: TCP
|
|
port: {{ int $ambientPort }}
|
|
{{- end }}
|
|
{{- if $probeSources }}
|
|
{{- /*
|
|
RESTRICTED, because the operator supplied probeSources. The kubelet's probes
|
|
come from the IP of the WORKER NODE running the pod, so that address has to be
|
|
in the list or the pod never becomes Ready - and a pod that is never Ready
|
|
reads as a broken application rather than as a policy.
|
|
|
|
No ingress-controller entry on this rule: nothing routes to the management port
|
|
from outside, and it should stay that way.
|
|
*/}}
|
|
from:
|
|
- namespaceSelector:
|
|
matchLabels:
|
|
{{- /*
|
|
Set by Kubernetes itself on every namespace (>= 1.21), so this needs no
|
|
labelling work on a namespace this chart does not own.
|
|
*/}}
|
|
kubernetes.io/metadata.name: {{ $ns }}
|
|
{{- range $allowedNs }}
|
|
- namespaceSelector:
|
|
matchLabels:
|
|
kubernetes.io/metadata.name: {{ . | quote }}
|
|
{{- end }}
|
|
{{- range $probeSources }}
|
|
- ipBlock:
|
|
cidr: {{ . | quote }}
|
|
{{- end }}
|
|
{{- if $ambientEnabled }}
|
|
{{- /* SECOND HALF OF networkPolicy.ingress.ambientMesh, and the reason it is
|
|
one value and not two. Inside the mesh the kubelet's probes are
|
|
redirected and no longer arrive from the worker node's address: they
|
|
arrive from 169.254.7.127.
|
|
|
|
An operator who only opens 15008 by hand writes a policy that passes
|
|
review - and the moment probeSources is non-empty every probe is
|
|
dropped, the pod never becomes Ready, and the symptom reads as a broken
|
|
application rather than as a policy. That is assertion 4 of
|
|
ci/run-networkpolicy-test.sh one value further along. A switch whose
|
|
halves can only be had together cannot be set halfway.
|
|
|
|
Only in this branch, because this is the only branch with a "from" list:
|
|
with no sources restricted the rule is open and needs no peer. The IPv6
|
|
counterpart is deliberately absent - this cluster is SingleStack IPv4,
|
|
and an ipBlock of the wrong family is accepted and matches nothing.
|
|
|
|
WHERE 169.254.7.127 COMES FROM - vendor magic, not protocol:
|
|
it is LINK-LOCAL (169.254.0.0/16, RFC 3927), a range reserved for
|
|
node-local use and never routed; Istio's IPv6 counterpart
|
|
fd16:9254:7127:1337:ffff:ffff:ffff:ffff carries the SAME DIGITS
|
|
plus 1337; and KUBERNETES DOES NOT KNOW THE ADDRESS - neither the
|
|
NetworkPolicy API nor the kubelet - only Istio's data plane
|
|
rewrites the probe source onto it. Read for ztunnel 1.30.6, and if
|
|
Istio changes it THE FAILURE IS SILENT: probes dropped, and only
|
|
where probeSources is non-empty, which is nowhere today.
|
|
|
|
A LIST, not a string, and the one case where that does not
|
|
contradict "a constant, not a configuration": a DUAL-STACK cluster
|
|
needs BOTH entries at once. This cluster is SingleStack IPv4, so
|
|
the IPv6 counterpart is left out - an ipBlock of the wrong family
|
|
is accepted and matches nothing. Add it ALONGSIDE, never instead.
|
|
|
|
UNPROVEN IN OPERATION: probeSources is empty in every chart here, so no
|
|
release reaches this branch. CI proves its render, through the overlays
|
|
in ci/networkpolicy. */}}
|
|
{{- range $ambientProbeSources }}
|
|
- ipBlock:
|
|
cidr: {{ . | quote }}
|
|
{{- end }}
|
|
{{- end }}
|
|
{{- else }}
|
|
{{- /*
|
|
NO "from" SELECTOR, WHICH MEANS OPEN - deliberately, and only inside the
|
|
cluster, because this port carries no NodePort. The alternative was to guess
|
|
the worker node CIDRs, and a wrong guess blocks every kubelet probe: the pod
|
|
stays un-Ready, the rollout times out, and the cause looks like the
|
|
application. An empty "from: []" would be worse than either - it allows
|
|
everything too, but while reading as a restriction. Narrow it with
|
|
networkPolicy.probeSources.
|
|
*/}}
|
|
{{- end }}
|
|
{{- /*
|
|
THE CLUSTER BUS between this release's own replicas, and the narrowest rule
|
|
this object can express: a podSelector with no namespaceSelector, which means
|
|
"pods with these labels IN THIS NAMESPACE" - i.e. exactly the pods this policy
|
|
already selects.
|
|
|
|
Infinispan discovery here is jdbc-ping (see values.yaml), so no multicast and
|
|
no extra port is involved; the peers find each other through the database and
|
|
then connect on this port. The matching egress rule below is what lets them
|
|
dial OUT - without it the cluster would form in one direction only.
|
|
*/}}
|
|
- ports:
|
|
- protocol: TCP
|
|
port: {{ $settings.ports.jgroups | int64 }}
|
|
from:
|
|
- podSelector:
|
|
matchLabels:
|
|
app: {{ .Release.Name }}-{{ .Chart.Name }}-deployment
|
|
egress:
|
|
{{- /*
|
|
DNS, unconditionally. There is no value to switch this off, and that is
|
|
deliberate: a knob whose only other setting breaks every name lookup in
|
|
the pod is a trap and not a feature.
|
|
*/}}
|
|
- ports:
|
|
- protocol: UDP
|
|
port: 53
|
|
- protocol: TCP
|
|
port: 53
|
|
{{- /*
|
|
The external PostgreSQL, on THE CONFIGURED PORT - keycloak.db.port, derived
|
|
through keycloak.settings. This chart is the exception in the sweep: its
|
|
database port is a value rather than a field inside an opaque secret, so this
|
|
rule follows the configuration instead of guessing at 5432.
|
|
|
|
It also carries Infinispan's jdbc-ping discovery, which is why a blocked
|
|
database here costs the cluster as well as the data.
|
|
*/}}
|
|
- ports:
|
|
- protocol: TCP
|
|
port: {{ $settings.db.port | int64 }}
|
|
{{- /*
|
|
Outbound cluster bus. Needed as well as the ingress rule above: policyTypes
|
|
lists Egress, so a replica dialling its peer is subject to this policy too, and
|
|
without this rule the bus would be permitted inbound and blocked outbound - a
|
|
cluster that never forms, with two nodes that each look healthy.
|
|
*/}}
|
|
- ports:
|
|
- protocol: TCP
|
|
port: {{ $settings.ports.jgroups | int64 }}
|
|
{{- /*
|
|
Outbound HTTPS. Keycloak needs it for identity brokering and for external
|
|
identity-provider metadata, both configured inside REALMS this chart never
|
|
sees - so the port has to be permitted here or a federated login fails with no
|
|
hint pointing at this object. Also covers an outbound webhook or a remote JWKS.
|
|
Ports only: these are hostnames.
|
|
*/}}
|
|
- ports:
|
|
- protocol: TCP
|
|
port: 443
|
|
{{- with $np.egress }}
|
|
{{- with .extraPorts }}
|
|
{{- /*
|
|
Ports only, to any destination - see the header. This is the operator's
|
|
escape hatch from the egress cap, and the only alternative to switching
|
|
the whole policy off.
|
|
*/}}
|
|
- ports:
|
|
{{- range . }}
|
|
- protocol: TCP
|
|
port: {{ int . }}
|
|
{{- end }}
|
|
{{- end }}
|
|
{{- end }}
|
|
{{- end }}
|