From 530227405c98530908e2c09bef6aab804ddd20fc Mon Sep 17 00:00:00 2001 From: Justin Klein Date: Thu, 15 Oct 2026 17:39:24 +0000 Subject: [PATCH] fix(paperless-ngx): egress names the Tika/Gotenberg POD ports, not the Service port The rule allowed outbound 80, the port the apache-tika and gotenberg Services listen on, and justified it in a comment: an egress rule names the port on the destination as the client dials it, and the client dials the Service. That is wrong. kube-proxy does a ClusterIP Service's destination NAT BEFORE the egress policy is evaluated at the pod's network interface, so the policy sees podIP:targetPort and never clusterIP:port - and those two charts forward 80 to 9998 and 3000. Deployed as 14.0.0+up3.3.1 it cut Paperless off from both services on 2026-10-15: mail import, EML, HTML and Office documents stopped, plain PDFs and images kept working, and the error was httpx.ConnectTimeout with no mention of a NetworkPolicy. The rule now names 9998, 3000 and 80. The 80 stays because it costs nothing and covers a foreign Tika that really does listen on 80. The refuted sentence is quoted rather than deleted: it reads perfectly, which is how it survived review, and the timeout-not-refused symptom is written down with it. No policy switch of its own: the rule stays gated on paperless.files.tikaGotenberg.enabled, because a second value that can drift from the feature re-arms this very outage. New networkPolicy.egress.mailImport opens IMAPS 993, off by default - paperless' mail accounts are rows in its database, so the chart cannot derive the rule. Plain IMAP (143) and POP3 stay in extraPorts, and 993 is noted as a real destination port, because a mail server outside the cluster has no DNAT in the path. Minor: one additive value, and a fix that widens a rule. Nothing breaks, no value becomes required, appVersion unchanged. Part of #136 --- paperless-ngx/Chart.yaml | 2 +- paperless-ngx/templates/NOTES.txt | 26 +++- .../paperless-server.networkpolicy.yaml | 115 ++++++++++++++---- paperless-ngx/values.schema.json | 6 +- paperless-ngx/values.yaml | 62 +++++++--- 5 files changed, 168 insertions(+), 43 deletions(-) diff --git a/paperless-ngx/Chart.yaml b/paperless-ngx/Chart.yaml index 561e022..2059ed2 100644 --- a/paperless-ngx/Chart.yaml +++ b/paperless-ngx/Chart.yaml @@ -5,7 +5,7 @@ icon: https://raw.githubusercontent.com/paperless-ngx/paperless-ngx/dev/src-ui/s type: application -version: 14.0.0+up3.3.1 +version: 14.1.0+up3.3.1 appVersion: 3.3.1 diff --git a/paperless-ngx/templates/NOTES.txt b/paperless-ngx/templates/NOTES.txt index 0c46e49..aaf7aad 100644 --- a/paperless-ngx/templates/NOTES.txt +++ b/paperless-ngx/templates/NOTES.txt @@ -536,14 +536,32 @@ BACKUP IS TWO THINGS AT THE SAME TIME NETWORKPOLICY IS ACTIVE — WHAT IT CAPS, AND WHAT IT DOES NOT {{ .Release.Name }}-{{ .Chart.Name }}-networkpolicy selects this workload's pods. Inbound: TCP 8000 only. Outbound: DNS, TCP 5432 (Postgres), 6379 - (Redis/Valkey), 443{{ if .Values.paperless.files.tikaGotenberg.enabled }}, 80 (the Tika and Gotenberg Services){{ end }}{{ with .Values.networkPolicy.egress }}{{ with .extraPorts }}, plus TCP + (Redis/Valkey), 443{{ if .Values.paperless.files.tikaGotenberg.enabled }}, 9998 and 3000 (the Tika and Gotenberg POD ports) plus 80{{ end }}{{ with .Values.networkPolicy.egress }}{{ if .mailImport }}, 993 (IMAPS, mail import){{ end }}{{ with .extraPorts }}, plus TCP {{ join ", " . }}{{ end }}{{ end }}. +{{- if .Values.paperless.files.tikaGotenberg.enabled }} + + THE TIKA/GOTENBERG RULE NAMES POD PORTS, AND THAT IS THE FIX FROM GITEA #136. + It named the Services' 80 alone until chart 14.1.0, which cut Paperless off + from both services: kube-proxy does a ClusterIP Service's destination NAT + BEFORE the egress policy is evaluated at the pod's interface, so the policy + sees podIP:targetPort. The symptom was a TIMEOUT and not "connection refused", + and no error mentioned a NetworkPolicy. If this release reaches a Tika or + Gotenberg that listens on some other port, that port goes into + networkPolicy.egress.extraPorts. +{{- end }} +{{- if (.Values.networkPolicy.egress).mailImport }} + + MAIL IMPORT IS PERMITTED on 993 (IMAPS), because + networkPolicy.egress.mailImport is on. Plain IMAP (143) and POP3 are NOT + covered — add them to networkPolicy.egress.extraPorts if an account uses them. +{{- else }} MAIL FETCHING IS NOT PERMITTED. Paperless' mail accounts are rows in its database, configured in the web UI, so this chart cannot see them. A release - that fetches mail needs IMAP (993, or 143) in - networkPolicy.egress.extraPorts — otherwise a mail rule silently stops - importing, with a healthy pod behind it. + that fetches mail needs networkPolicy.egress.mailImport: true for IMAPS (993), + or 143/POP3 in networkPolicy.egress.extraPorts — otherwise a mail rule + silently stops importing, with a healthy pod behind it. +{{- end }} AND TWO PODS OF THIS RELEASE ARE NOT COVERED BY THIS OBJECT: - the valkey subchart's pod, which carries its own labels. It needs no diff --git a/paperless-ngx/templates/paperless-server.networkpolicy.yaml b/paperless-ngx/templates/paperless-server.networkpolicy.yaml index c9a6619..da99126 100644 --- a/paperless-ngx/templates/paperless-server.networkpolicy.yaml +++ b/paperless-ngx/templates/paperless-server.networkpolicy.yaml @@ -27,11 +27,12 @@ theoretical: opens no connection that a policy would have anything to say about. If that ever changes, the CronJob needs labels before it can need a policy. -3. MAIL FETCHING IS NOT PERMITTED. Paperless' mail accounts live in its - DATABASE, configured through the web UI, so this chart cannot see them. A - release that fetches mail needs IMAP (993 or 143) in - networkPolicy.egress.extraPorts, and the symptom otherwise is a mail rule - that silently stops importing. +3. MAIL FETCHING IS NOT PERMITTED UNLESS THE OPERATOR SAYS SO. Paperless' mail + accounts live in its DATABASE, configured through the web UI, so this chart + cannot see them and cannot derive the rule. networkPolicy.egress.mailImport + opens IMAPS (993) and is off by default; plain IMAP (143) and POP3 belong in + networkPolicy.egress.extraPorts. Left off on a release that does fetch mail, + the symptom is a mail rule that silently stops importing. WHAT THIS OBJECT IS: a ceiling on reachability, not an authentication layer. Only the ports declared below are reachable at all - whatever else the process @@ -53,13 +54,22 @@ WHAT IT IS NOT: of the rendered object, travel into the cluster and appear in every release diff. ci/run-networkpolicy-test.sh fails on a rendered "#". -THE PORT NUMBERS BELOW ARE POD PORTS, NEVER NODEPORTS. A NetworkPolicy is -evaluated at the pod's network interface, after kube-proxy has already -translated a NodePort to the targetPort, so a NodePort number here would match -nothing while looking strict - and, being the only rule for that port, would -block every inbound connection instead. ci/run-networkpolicy-test.sh asserts -every port here against the containerPorts of the pod spec this policy -selects. +THE PORT NUMBERS BELOW ARE POD PORTS, NEVER NODEPORTS AND NEVER SERVICE PORTS. +A NetworkPolicy is evaluated at the pod's network interface, after kube-proxy +has already done its destination NAT - of a NodePort on the way in, and of a +ClusterIP on the way OUT. So a NodePort number in an ingress rule matches +nothing while looking strict (and, being the only rule for that port, blocks +every inbound connection instead), and a SERVICE port in an egress rule matches +nothing wherever the Service's port and its targetPort differ. The second half +is not hypothetical: it is Gitea #136, it cost an outage on 2026-10-15, and the +sentence that caused it is quoted as refuted at the Tika/Gotenberg rule below. + +ci/run-networkpolicy-test.sh asserts every INGRESS port here against the +containerPorts of the pod spec this policy selects. Egress cannot be checked +that way - an egress port belongs to the DESTINATION, which for Postgres, Redis +and the web is not even in the render. The one exception is Tika and Gotenberg, +which are two charts of THIS repository: the same test renders them and asserts +that the egress rule below contains their containerPorts. */ -}} {{- if .Values.networkPolicy.enabled }} {{- $np := .Values.networkPolicy }} @@ -88,9 +98,10 @@ spec: - Ingress {{- /* Egress is listed because there is a lot to restrict and most of it is known. - What is NOT permitted and is the plausible surprise: IMAP. Paperless' mail - accounts are rows in its database, configured in the web UI, so the chart - cannot see them - see networkPolicy.egress.extraPorts. + What is NOT permitted by default and is the plausible surprise: mail. Paperless' + mail accounts are rows in its database, configured in the web UI, so the chart + cannot see them - networkPolicy.egress.mailImport opens IMAPS 993, and anything + else (143, POP3) goes into networkPolicy.egress.extraPorts. */}} - Egress ingress: @@ -184,17 +195,51 @@ spec: port: 6379 {{- if .Values.paperless.files.tikaGotenberg.enabled }} {{- /* - TIKA AND GOTENBERG, on port 80 - which is what their Services listen on in this - repository, even though their containers use 9998 and 3000. That distinction is - the whole reason this rule says 80: an EGRESS rule names the port on the - DESTINATION as the client dials it, and the client dials the Service. + TIKA AND GOTENBERG, ON THEIR POD PORTS 9998 AND 3000 - and 80 as well, for a + third case named at the end. Gitea #136: this rule used to name 80 ALONE, and + the release that shipped it cut Paperless off from both services on + 2026-10-15. Mail import, EML, HTML and Office documents stopped; plain PDFs + and images kept working, which is why it did not look like a network fault. - Rendered only while paperless.files.tikaGotenberg.enabled is true, so a release - without them carries no rule that permits nothing. Endpoints on another port - - the containers' own 9998 and 3000, if addressed directly rather than through - their Services - need networkPolicy.egress.extraPorts. + THE SENTENCE THAT STOOD HERE IS WRONG. It is quoted rather than deleted, + because it reads perfectly and that is precisely how it survived review: + + "an EGRESS rule names the port on the DESTINATION as the client dials + it, and the client dials the Service." + + kube-proxy translates the destination of a ClusterIP Service (DNAT) BEFORE + the egress policy is evaluated at the pod's network interface. The policy + therefore sees podIP:targetPort and never clusterIP:port, so an egress rule + names the POD port. A Service port only ever matches by coincidence, where + port and targetPort are the same number - which is the case for 5432, 6379 + and 443 above and below, and is the reason nothing else here broke. + + THE SYMPTOM IS A TIMEOUT, NOT "connection refused", and that is the half to + remember: a missing service REFUSES, a dropped packet RUNS INTO THE TIMEOUT. + What this produced was httpx.ConnectTimeout out of paperless' parser, naming + the Tika URL and no NetworkPolicy at all - nothing in the error points here. + + WHY 80 STAYS: it costs nothing and covers an installation whose Tika really + does listen on 80 - a foreign one behind a proxy, where 80 IS the pod port. + + WHY THERE IS NO SWITCH OF ITS OWN for this rule, and that is deliberate: it + is gated on the FEATURE value, because a second value that can drift from the + feature re-arms exactly this outage. Whoever enables Tika and forgets the + policy switch lands back here, with the same message that does not point at + the policy. Rendered only while paperless.files.tikaGotenberg.enabled is + true, so a release without them still carries no rule that permits nothing. + + 9998 and 3000 are the containerPorts the apache-tika and gotenberg charts of + this repository declare, so they are fixed rather than guessed - and + ci/run-networkpolicy-test.sh renders both of those charts and asserts that + this rule contains them. That is the one assertion a single-chart render + cannot make, and it is why this mistake was not caught the first time. */}} - ports: + - protocol: TCP + port: 9998 + - protocol: TCP + port: 3000 - protocol: TCP port: 80 {{- end }} @@ -207,6 +252,30 @@ spec: - protocol: TCP port: 443 {{- with $np.egress }} + {{- if .mailImport }} + {{- /* + IMAPS for mail import, OFF BY DEFAULT - because this chart cannot know + whether a release fetches mail. Paperless' mail accounts are rows in its + DATABASE, entered in the web UI, and nothing in the values hints that they + exist. So this value is an operator's statement ("this release imports + mail") and not a derivation, which is also why it cannot be folded into + another switch. + + 993 ONLY. Plain IMAP (143) and POP3 (110, 995) are deliberately NOT included: + a value named mailImport that quietly opened four ports would be wider than + its name, and the plaintext variants are not what a new account uses. They go + into networkPolicy.egress.extraPorts. + + AND THIS PORT IS THE REAL DESTINATION PORT, unlike the Tika rule above. A + mail server outside the cluster is reached with no Service and therefore no + DNAT in the path, so 993 is both what paperless dials and what the policy + sees. What decides which number an egress rule needs is the DNAT, not the + direction and not the protocol - that is the whole lesson of #136. + */}} + - ports: + - protocol: TCP + port: 993 + {{- end }} {{- with .extraPorts }} {{- /* Ports only, to any destination - see the header. This is the operator's diff --git a/paperless-ngx/values.schema.json b/paperless-ngx/values.schema.json index efd1364..5ff881a 100644 --- a/paperless-ngx/values.schema.json +++ b/paperless-ngx/values.schema.json @@ -91,8 +91,12 @@ "type": ["object", "null"], "additionalProperties": false, "properties": { + "mailImport": { + "description": "Open outbound IMAPS (993) for paperless' mail import. Off by default, and a value rather than a derivation because paperless' mail accounts are rows in its DATABASE, entered in the web UI - nothing in these values says whether a release fetches mail. 993 only: plain IMAP (143) and POP3 (110, 995) are deliberately not covered and belong in extraPorts. Unlike the Tika/Gotenberg rule this is the real destination port, because a mail server outside the cluster is reached with no Service and therefore no destination NAT in the path (Gitea #136).", + "type": ["boolean", "null"] + }, "extraPorts": { - "description": "Additional outbound TCP ports, to ANY destination - ports and not destinations, because a NetworkPolicy cannot express a hostname. DNS is always allowed and has no switch. Without an entry here an outbound port the chart does not already derive is blocked, which is the cap's whole purpose and also its upgrade risk.", + "description": "Additional outbound TCP ports, to ANY destination - ports and not destinations, because a NetworkPolicy cannot express a hostname. DNS is always allowed and has no switch. Without an entry here an outbound port the chart does not already derive is blocked, which is the cap's whole purpose and also its upgrade risk. For a destination INSIDE the cluster these are POD ports: kube-proxy translates a ClusterIP Service's port away before the egress policy is evaluated at the pod's interface, so a Service port matches nothing wherever port and targetPort differ (Gitea #136).", "type": ["array", "null"], "items": { "type": "integer" diff --git a/paperless-ngx/values.yaml b/paperless-ngx/values.yaml index 165efca..ee4cea0 100644 --- a/paperless-ngx/values.yaml +++ b/paperless-ngx/values.yaml @@ -549,13 +549,28 @@ valkey: networkPolicy: # WHAT THIS POLICY ALLOWS: inbound 8000 only, and outbound DNS plus TCP 5432 # (Postgres), 6379 (Redis/Valkey), 443, and - only while - # paperless.files.tikaGotenberg.enabled is true - 80 for the Tika and - # Gotenberg Services. + # paperless.files.tikaGotenberg.enabled is true - 9998, 3000 and 80 for Tika + # and Gotenberg. # - # NOTE WHY THAT RULE SAYS 80 AND NOT 9998/3000: an egress rule names the port - # on the DESTINATION as the client dials it, and paperless dials those - # charts' SERVICES, which listen on 80 and forward to the containers' own - # ports. + # WHY THAT RULE NAMES 9998/3000 AND NOT JUST THE SERVICES' 80 (Gitea #136, an + # outage on 2026-10-15). The note that used to stand here said: + # + # "an egress rule names the port on the DESTINATION as the client dials it, + # and paperless dials those charts' SERVICES, which listen on 80 and + # forward to the containers' own ports." + # + # That is wrong, and it is kept as a quotation because it sounds right - which + # is why it was never checked. kube-proxy does the ClusterIP Service's + # destination NAT BEFORE the egress policy is evaluated at the pod's network + # interface, so the policy sees podIP:targetPort, never clusterIP:port. An + # egress rule therefore names the POD port: 9998 for apache-tika, 3000 for + # gotenberg, both read from those charts' containerPorts. The 80 stays because + # it costs nothing and covers a foreign Tika that really does listen on 80. + # + # AND THE FAILURE MODE IS A TIMEOUT, NOT "connection refused": a missing + # service refuses, a dropped packet runs into the timeout. 5432, 6379 and 443 + # were unaffected only because their Service port and pod port are the same + # number. # # THREE THINGS IT DOES NOT COVER: # - The VALKEY SUBCHART'S POD. It carries its own labels, so this policy @@ -566,10 +581,10 @@ networkPolicy: # only because the job moves files between mounted volumes and the SMB # share arrives as a CSI volume mounted by the NODE - the pod opens no # connection of its own. If that changes, the CronJob needs labels first. - # - MAIL FETCHING. Paperless' mail accounts are rows in its database, - # configured in the web UI, so this chart cannot see them. A release that - # fetches mail needs IMAP in egress.extraPorts, or a mail rule silently - # stops importing. + # - MAIL FETCHING, unless egress.mailImport is switched on. Paperless' mail + # accounts are rows in its database, configured in the web UI, so this + # chart cannot see them and cannot derive the rule. Left off on a release + # that does fetch mail, a mail rule silently stops importing. # # And the usual two: 5432 and 6379 are the STANDARD ports rather than the # configured ones, and a policy is only a boundary where the CNI enforces @@ -622,6 +637,21 @@ networkPolicy: podLabels: app.kubernetes.io/name: rke2-traefik egress: + # Outbound IMAPS (993) for paperless' mail import. OFF, and it has to be a + # value rather than something the chart derives: the mail accounts are ROWS + # IN THE DATABASE, entered in the web UI, so nothing in these values says + # whether a release fetches mail. Switching this on is the operator saying + # that it does. + # + # 993 ONLY, deliberately. Plain IMAP (143) and POP3 (110, 995) are not + # covered - a value named mailImport that opened four ports would be wider + # than its name - and belong in extraPorts below. + # + # NOTE THE DIFFERENCE FROM THE TIKA RULE: 993 is the real destination port, + # because a mail server outside the cluster is reached with no Service and + # so no destination NAT in the path. What decides whether an egress rule + # needs the Service port or the pod port is that NAT, and nothing else. + mailImport: false # Additional outbound TCP ports. DNS (UDP and TCP 53) is always allowed and # has no switch - a knob whose only other setting breaks every name lookup # in the pod is a trap, not a feature. @@ -632,8 +662,12 @@ networkPolicy: # permits its port to ANY destination. What the cap still buys is real: a # compromised pod cannot open arbitrary outbound connections. # - # IMAP (993, or 143 for the plaintext case) is the entry this chart needs - # most, for mail accounts configured in the web UI. Also a Postgres or - # Redis on a non-standard port, or a Tika/Gotenberg addressed on its - # container port (9998 / 3000) instead of through its Service. + # PLAIN IMAP (143) and POP3 (110, 995) go here - IMAPS 993 has its own + # switch above. Also a Postgres or Redis on a non-standard port, and a Tika + # or Gotenberg that listens on neither its chart's container port nor 80. + # + # These are POD ports for anything inside the cluster, for the reason given + # at the top of this block: a ClusterIP Service's port is translated away + # before the policy sees the packet. For a destination outside the cluster + # there is no translation and the port is simply the one being dialled. extraPorts: []