fix(paperless-ngx): the egress rule names the Tika/Gotenberg pod ports, and CI now asserts it across charts #138

Merged

jklein merged 3 commits from fix/paperless-egress-pod-ports into main 2026-10-15 22:06:05 +02:00

Conversation 2 Commits 3 Files Changed 8

+580 -49

jlab-bot commented 2026-10-15 19:47:36 +02:00

Member

Copy link

Copy source

paperless-ngx' egress rule for Tika and Gotenberg named 80, the port those two charts' Services listen on, and justified it in a template comment. The justification was wrong, and the release that shipped it cut Paperless off from both services on 2026-10-15 — mail import, EML, HTML and Office documents stopped, plain PDFs and images kept working, and the error was httpx.ConnectTimeout with no mention of a NetworkPolicy.

kube-proxy does a ClusterIP Service's destination NAT before the egress policy is evaluated at the pod's network interface. The policy sees podIP:targetPort, never clusterIP:port — so an egress rule names the pod port, and the two charts forward 80 to 9998 and 3000.

Fixes #136

What changed

1. The rule names 9998, 3000 and 80. The two pod ports are read from the charts that declare them — apache-tika/templates/apache-tika.deployment.yaml has containerPort: 9998, gotenberg/templates/gotenberg.deployment.yaml has containerPort: 3000, and both Services are port: 80 -> targetPort: <named>. The 80 stays: it costs nothing and covers a foreign Tika that really does listen on 80, where 80 is the pod port.

2. No policy switch of its own. The rule stays gated on paperless.files.tikaGotenberg.enabled, and that is in the comment with its reason: a second value that can drift from the feature re-arms exactly this outage — whoever enables Tika and forgets the policy switch lands back here, with the same message that does not point at the policy.

3. The comment is replaced, and the old sentence is quoted as refuted rather than deleted:

"an EGRESS rule names the port on the DESTINATION as the client dials it, and the client dials the Service."

It reads perfectly, which is how it survived review — deleting it loses that. Written down with it: the mechanism, and that the symptom is a timeout, not connection refused (a missing service refuses, a dropped packet runs out). 5432, 6379 and 443 were unaffected only because their Service port and pod port are the same number.

4. New networkPolicy.egress.mailImport, opens 993 (IMAPS), off by default, carried through values.schema.json and NOTES.txt. It has to be a value rather than a derivation: paperless' mail accounts are rows in its database, entered in the web UI, so nothing in the values says whether a release fetches mail. Plain IMAP (143) and POP3 are deliberately not covered and belong in extraPorts — a value named mailImport that opened four ports would be wider than its name. And 993 is noted as the real destination port, because a mail server outside the cluster is reached with no Service and so no DNAT in the path: what decides which number a rule needs is the NAT, not the direction.

5. A cross-chart CI assertion in ci/run-networkpolicy-test.sh (details below).

6. Version 14.1.0+up3.3.1 — minor: one additive value, and a fix that widens a rule. Nothing breaks, no value becomes required, appVersion unchanged.

The assertion, and why it is the fifth row rather than a sixth test

The four assertions from #5 hold ports against the pod spec the policy selects — right for ingress, meaningless for egress, where the destination is in no render this repository has. Within one chart this mistake is not catchable.

Across two it is, and paperless-ngx → apache-tika, gotenberg is the only place in the repository where a chart dials another chart's Service. Verified rather than assumed: four policy templates name port: 80 (gotenberg, paperless-ngx, rss-bridge, ytdlp-web-player) and the other three aim at the internet, where there is no DNAT and 80 is the real port. They are untouched.

So each row of an explicit EGRESS_PEER_ROWS table is checked on two renders of the consumer:

• gate on → the egress ports must contain every pod port behind the peers' Services, resolved Service port -> targetPort -> containerPort, which is exactly the translation the policy is evaluated after. The evidence line prints the chain, so a reader sees the DNAT and not two numbers:

  ok  with paperless.files.tikaGotenberg.enabled=true, netpol-paperless-ngx-networkpolicy
      permits every pod port behind the peer charts' Services:
      gotenberg netpol-gotenberg-service: port 80 -> targetPort 'gotenberg-port' -> pod port 3000;
      apache-tika netpol-apache-tika-service: port 80 -> targetPort 'tika-port' -> pod port 9998
  

• gate off → the requirement must not apply, and those ports must be gone, so a gated rule cannot quietly become one that permits what the release cannot use.

The table is explicit rather than discovered because no render can know which Service another chart dials — the address is a value or a 1Password field. A row naming a chart that no longer exists fails instead of being skipped.

Built into the existing script, not added as a new one with its own job: that job already runs over every chart on every PR (no path filter can skip it), Helm is already the YAML parser through ci/yaml2json, and the verdict/grouping/summary plumbing is shared. A second script would duplicate all of it to add one comparison. python3 standard library only — the runners have no yq, no pyyaml, no Docker.

Fail paths executed, not assumed

Path	Result
9998 removed from the rule	fails, naming the port and the DNAT chain, exit 1
9999 in place of 9998	fails the same way, exit 1
peer chart apache-tika renamed away	fails (peer chart apache-tika does not exist), not skipped
the policy's name no longer <release>-<chart>-<kind>	fails: the rule this assertion is about was not found
peers render no Service at all	fails — an empty requirement reads exactly like a satisfied one
the rule made unconditional	gate-on half green, gate-off half fails — the fail path of the gate
the rule gated as shipped, gate off	assertion stays silent, as required

Measured: 30 charts, 210 assertions held, 0 failed, ~15 s, no cluster (208 before this PR, so the new assertion is two of them). README's stated figure was 206 and is corrected. The four other render tests and run-bump-level-test.sh are green, helm lint --strict is clean, and the schema bites on both a wrong type and a case typo (mailimport).

What is not proven here, so the gap is not mistaken for coverage

The install-test does not exercise the corrected path. ci/paperless-ngx/test-values.yaml never sets paperless.files.tikaGotenberg.enabled and the chart default is false, so the install-test installs paperless without Tika and Gotenberg — the rule is not even rendered there. What is proven is the render (the diff below) and the new assertion. Enforcement of the corrected path was proven in the cluster instead, on cluster-config PR #551, which is also what unblocked production.

Out of scope, deliberately

• The three other charts naming port: 80 — their destination is the internet, they are correct, untouched.
• jk-cluster-config: not touched, no PR there. The running release is unblockable with networkPolicy.egress.extraPorts: [9998, 3000], which is the maintainer's call.

Render diff

   egress:
     - ports: [53/UDP, 53/TCP]
     - ports: [5432]
     - ports: [6379]
-    - ports: [80]
+    - ports: [9998, 3000, 80]
     - ports: [443]
+    - ports: [993]          # only with networkPolicy.egress.mailImport: true

With the gate off, the egress set is unchanged at 53, 5432, 6379, 443.

Docs

README.md §15 (the new assertion, the egress/DNAT mechanics, the bar for a named egress switch) and the job comment in validate-charts.yml. The wiki's paperless-ngx page, CI-CD and Lessons-Learned are updated in the same round.

`paperless-ngx`' egress rule for Tika and Gotenberg named **80**, the port those two charts' *Services* listen on, and justified it in a template comment. The justification was wrong, and the release that shipped it cut Paperless off from both services on 2026-10-15 — mail import, EML, HTML and Office documents stopped, plain PDFs and images kept working, and the error was `httpx.ConnectTimeout` with no mention of a NetworkPolicy. `kube-proxy` does a ClusterIP Service's destination NAT **before** the egress policy is evaluated at the pod's network interface. The policy sees `podIP:targetPort`, never `clusterIP:port` — so an egress rule names the **pod** port, and the two charts forward `80` to `9998` and `3000`. Fixes #136 ## What changed **1. The rule names `9998`, `3000` and `80`.** The two pod ports are read from the charts that declare them — `apache-tika/templates/apache-tika.deployment.yaml` has `containerPort: 9998`, `gotenberg/templates/gotenberg.deployment.yaml` has `containerPort: 3000`, and both Services are `port: 80 -> targetPort: <named>`. The `80` stays: it costs nothing and covers a foreign Tika that really does listen on 80, where 80 *is* the pod port. **2. No policy switch of its own.** The rule stays gated on `paperless.files.tikaGotenberg.enabled`, and that is in the comment with its reason: a second value that can drift from the feature re-arms exactly this outage — whoever enables Tika and forgets the policy switch lands back here, with the same message that does not point at the policy. **3. The comment is replaced, and the old sentence is quoted as refuted rather than deleted:** > "an EGRESS rule names the port on the DESTINATION as the client dials it, and the client dials the Service." It reads perfectly, which is how it survived review — deleting it loses that. Written down with it: the mechanism, and that **the symptom is a timeout, not `connection refused`** (a missing service refuses, a dropped packet runs out). `5432`, `6379` and `443` were unaffected only because their Service port and pod port are the same number. **4. New `networkPolicy.egress.mailImport`**, opens **993** (IMAPS), **off** by default, carried through `values.schema.json` and `NOTES.txt`. It has to be a value rather than a derivation: paperless' mail accounts are rows in its *database*, entered in the web UI, so nothing in the values says whether a release fetches mail. Plain IMAP (143) and POP3 are deliberately **not** covered and belong in `extraPorts` — a value named `mailImport` that opened four ports would be wider than its name. And `993` is noted as the *real* destination port, because a mail server outside the cluster is reached with no Service and so no DNAT in the path: what decides which number a rule needs is the NAT, not the direction. **5. A cross-chart CI assertion** in `ci/run-networkpolicy-test.sh` (details below). **6. Version `14.1.0+up3.3.1`** — minor: one additive value, and a fix that *widens* a rule. Nothing breaks, no value becomes required, `appVersion` unchanged. ## The assertion, and why it is the fifth row rather than a sixth test The four assertions from #5 hold ports against the pod spec the policy **selects** — right for ingress, meaningless for egress, where the destination is in no render this repository has. Within one chart this mistake is **not** catchable. Across two it is, and `paperless-ngx` → `apache-tika`, `gotenberg` is the **only** place in the repository where a chart dials another chart's Service. Verified rather than assumed: four policy templates name `port: 80` (`gotenberg`, `paperless-ngx`, `rss-bridge`, `ytdlp-web-player`) and the other three aim at the **internet**, where there is no DNAT and 80 is the real port. They are untouched. So each row of an explicit `EGRESS_PEER_ROWS` table is checked on **two** renders of the consumer: - gate **on** → the egress ports must *contain* every pod port behind the peers' Services, resolved `Service port -> targetPort -> containerPort`, which is exactly the translation the policy is evaluated after. The evidence line prints the chain, so a reader sees the DNAT and not two numbers: ``` ok with paperless.files.tikaGotenberg.enabled=true, netpol-paperless-ngx-networkpolicy permits every pod port behind the peer charts' Services: gotenberg netpol-gotenberg-service: port 80 -> targetPort 'gotenberg-port' -> pod port 3000; apache-tika netpol-apache-tika-service: port 80 -> targetPort 'tika-port' -> pod port 9998 ``` - gate **off** → the requirement must **not** apply, *and* those ports must be gone, so a gated rule cannot quietly become one that permits what the release cannot use. The table is explicit rather than discovered because no render can know which Service another chart dials — the address is a value or a 1Password field. A row naming a chart that no longer exists **fails** instead of being skipped. **Built into the existing script, not added as a new one with its own job:** that job already runs over every chart on every PR (no path filter can skip it), Helm is already the YAML parser through `ci/yaml2json`, and the verdict/grouping/summary plumbing is shared. A second script would duplicate all of it to add one comparison. `python3` standard library only — the runners have no `yq`, no `pyyaml`, no Docker. ### Fail paths executed, not assumed | Path | Result | |---|---| | `9998` removed from the rule | fails, naming the port and the DNAT chain, exit 1 | | `9999` in place of `9998` | fails the same way, exit 1 | | peer chart `apache-tika` renamed away | fails (`peer chart apache-tika does not exist`), not skipped | | the policy's name no longer `<release>-<chart>-<kind>` | fails: the rule this assertion is about was not found | | peers render no Service at all | fails — an empty requirement reads exactly like a satisfied one | | the rule made **unconditional** | gate-on half green, **gate-off half fails** — the fail path of the gate | | the rule gated as shipped, gate off | assertion stays silent, as required | **Measured: 30 charts, 210 assertions held, 0 failed, ~15 s, no cluster** (208 before this PR, so the new assertion is two of them). README's stated figure was `206` and is corrected. The four other render tests and `run-bump-level-test.sh` are green, `helm lint --strict` is clean, and the schema bites on both a wrong type and a case typo (`mailimport`). ## What is *not* proven here, so the gap is not mistaken for coverage **The install-test does not exercise the corrected path.** `ci/paperless-ngx/test-values.yaml` never sets `paperless.files.tikaGotenberg.enabled` and the chart default is `false`, so the install-test installs paperless **without** Tika and Gotenberg — the rule is not even rendered there. What is proven is the *render* (the diff below) and the new assertion. Enforcement of the corrected path was proven in the cluster instead, on `cluster-config` PR #551, which is also what unblocked production. ## Out of scope, deliberately - The three other charts naming `port: 80` — their destination is the internet, they are correct, untouched. - `jk-cluster-config`: not touched, no PR there. The running release is unblockable with `networkPolicy.egress.extraPorts: [9998, 3000]`, which is the maintainer's call. ## Render diff ```diff egress: - ports: [53/UDP, 53/TCP] - ports: [5432] - ports: [6379] - - ports: [80] + - ports: [9998, 3000, 80] - ports: [443] + - ports: [993] # only with networkPolicy.egress.mailImport: true ``` With the gate off, the egress set is unchanged at `53, 5432, 6379, 443`. ## Docs `README.md` §15 (the new assertion, the egress/DNAT mechanics, the bar for a named egress switch) and the job comment in `validate-charts.yml`. The wiki's `paperless-ngx` page, `CI-CD` and `Lessons-Learned` are updated in the same round.

jlab-bot added 2 commits 2026-10-15 19:47:37 +02:00

fix(paperless-ngx): egress names the Tika/Gotenberg POD ports, not the Service port ... 530227405c

The rule allowed outbound 80, the port the apache-tika and gotenberg Services
listen on, and justified it in a comment: an egress rule names the port on the
destination as the client dials it, and the client dials the Service. That is
wrong. kube-proxy does a ClusterIP Service's destination NAT BEFORE the egress
policy is evaluated at the pod's network interface, so the policy sees
podIP:targetPort and never clusterIP:port - and those two charts forward 80 to
9998 and 3000. Deployed as 14.0.0+up3.3.1 it cut Paperless off from both
services on 2026-10-15: mail import, EML, HTML and Office documents stopped,
plain PDFs and images kept working, and the error was httpx.ConnectTimeout with
no mention of a NetworkPolicy.

The rule now names 9998, 3000 and 80. The 80 stays because it costs nothing and
covers a foreign Tika that really does listen on 80. The refuted sentence is
quoted rather than deleted: it reads perfectly, which is how it survived review,
and the timeout-not-refused symptom is written down with it.

No policy switch of its own: the rule stays gated on
paperless.files.tikaGotenberg.enabled, because a second value that can drift
from the feature re-arms this very outage.

New networkPolicy.egress.mailImport opens IMAPS 993, off by default - paperless'
mail accounts are rows in its database, so the chart cannot derive the rule.
Plain IMAP (143) and POP3 stay in extraPorts, and 993 is noted as a real
destination port, because a mail server outside the cluster has no DNAT in the
path.

Minor: one additive value, and a fix that widens a rule. Nothing breaks, no
value becomes required, appVersion unchanged.

Part of #136

test(ci): assert a chart's egress ports against the POD ports of the chart it dials ... 1894e7d465

The four assertions from Gitea #5 hold ports against the pod spec the policy
SELECTS. That is right for ingress and meaningless for egress: an egress port
belongs to the destination, and for a database, a cache or the web that
destination is in no render this repository has. So the mistake behind #136 -
an egress rule naming a ClusterIP Service's port, which kube-proxy has already
translated away by the time the policy is evaluated at the pod's interface - was
not catchable within one chart.

It is catchable across two, and paperless-ngx -> apache-tika, gotenberg is the
only place in the repository where a chart dials another chart's Service (the
three other policies naming port 80 aim at the internet, where there is no DNAT
and 80 is the real port). The pairs are an explicit table rather than a
discovery, because no render can know which Service another chart dials - the
address is a value or a 1Password field - and a row naming a chart that no
longer exists fails instead of being skipped.

Each row is checked on two renders of the consumer: with the feature gate ON the
egress ports must CONTAIN every pod port behind the peers' Services, resolved
Service port -> targetPort -> containerPort, which is exactly the translation the
policy is evaluated after; with the gate OFF the requirement must not apply and
those ports must be absent, so a gated rule cannot quietly become one that
permits what the release cannot use. The evidence line prints the chain
("port 80 -> targetPort 'tika-port' -> pod port 9998"), so a reader sees the
DNAT rather than two numbers.

Built into run-networkpolicy-test.sh rather than added as a new script with its
own job: that job already runs over every chart on every PR, Helm is already the
YAML parser through ci/yaml2json, and the verdict and summary plumbing is
shared. python3 standard library only, no yq, no pyyaml, no Docker.

Six fail paths executed, not assumed: a required port removed, a wrong number in
its place, the peer chart renamed away, the policy name no longer following the
<release>-<chart>-<kind> convention, the peers rendering no Service at all (an
empty requirement reads like a satisfied one, so it fails), and the rule made
unconditional, which is the fail path of the gate-off half.

Measured: 30 charts, 210 assertions held (208 before), 0 failed, ~15s, no
cluster. README's stated figure was 206 and is corrected here.

Part of #136

jlab-bot commented 2026-10-15 19:53:30 +02:00

Author

Member

Copy link

Copy source

CI green, and the new assertion ran there rather than only locally. Validate Helm Charts / NetworkPolicy, run 994 job 4101, 12 s:

::group::cross-chart egress: paperless-ngx -> apache-tika gotenberg
  ok  paperless-ngx: with paperless.files.tikaGotenberg.enabled=true,
      netpol-paperless-ngx-networkpolicy permits every pod port behind the peer charts Services:
      gotenberg netpol-gotenberg-service: port 80 -> targetPort "gotenberg-port" -> pod port 3000;
      apache-tika netpol-apache-tika-service: port 80 -> targetPort "tika-port" -> pod port 9998
  ok  paperless-ngx: with paperless.files.tikaGotenberg.enabled=false the rule is absent
      (egress: 53, 443, 5432, 6379), so the containment requirement correctly does not apply and
      no release without the feature carries a rule that permits nothing.
::endgroup::
cross-chart egress paperless-ngx: ok
failed:     0
assertions held: 210

All 11 checks succeeded: NetworkPolicy, Install-test paperless-ngx (3m2s), Build paperless-ngx, Bump level, Trivy config, Selector match, HTTPRoute match, Resource quantities, YAML scalars, Detect changed charts.

The one thing that green install-test does NOT prove, restated because it is easy to read the other way: ci/paperless-ngx/test-values.yaml never sets paperless.files.tikaGotenberg.enabled and the chart default is false, so the install-test installs paperless without Tika and Gotenberg and the corrected rule is not even rendered in that cluster. Proven here: the render, and the cross-chart assertion above. Enforcement of the corrected path was proven in the cluster on cluster-config PR #551.

Wiki updated in the same round (commit 42e5f41): the paperless-ngx page (the refuted sentence quoted, the pod-port rule, mailImport, and the Tika gate being off in CI), CI-CD (the fifth assertion row, the gate-on/gate-off pair, 210 assertions), Chart-Style-Guide section 15 (pod ports in egress too, and the bar for a named egress switch) and Lessons-Learned (the lesson, plus how the wrong sentence passed every check while the correct mechanism already stood in the style guide, two days older).

CI green, and the new assertion ran there rather than only locally. `Validate Helm Charts / NetworkPolicy`, run 994 job 4101, 12 s: ``` ::group::cross-chart egress: paperless-ngx -> apache-tika gotenberg ok paperless-ngx: with paperless.files.tikaGotenberg.enabled=true, netpol-paperless-ngx-networkpolicy permits every pod port behind the peer charts Services: gotenberg netpol-gotenberg-service: port 80 -> targetPort "gotenberg-port" -> pod port 3000; apache-tika netpol-apache-tika-service: port 80 -> targetPort "tika-port" -> pod port 9998 ok paperless-ngx: with paperless.files.tikaGotenberg.enabled=false the rule is absent (egress: 53, 443, 5432, 6379), so the containment requirement correctly does not apply and no release without the feature carries a rule that permits nothing. ::endgroup:: cross-chart egress paperless-ngx: ok failed: 0 assertions held: 210 ``` All 11 checks succeeded: `NetworkPolicy`, `Install-test paperless-ngx` (3m2s), `Build paperless-ngx`, `Bump level`, `Trivy config`, `Selector match`, `HTTPRoute match`, `Resource quantities`, `YAML scalars`, `Detect changed charts`. **The one thing that green install-test does NOT prove**, restated because it is easy to read the other way: `ci/paperless-ngx/test-values.yaml` never sets `paperless.files.tikaGotenberg.enabled` and the chart default is `false`, so the install-test installs paperless **without** Tika and Gotenberg and the corrected rule is not even rendered in that cluster. Proven here: the render, and the cross-chart assertion above. Enforcement of the corrected path was proven in the cluster on cluster-config PR #551. Wiki updated in the same round (commit `42e5f41`): the `paperless-ngx` page (the refuted sentence quoted, the pod-port rule, `mailImport`, and the Tika gate being off in CI), `CI-CD` (the fifth assertion row, the gate-on/gate-off pair, 210 assertions), `Chart-Style-Guide` section 15 (pod ports in egress too, and the bar for a named egress switch) and `Lessons-Learned` (the lesson, plus how the wrong sentence passed every check while the correct mechanism already stood in the style guide, two days older).

jklein added 1 commit 2026-10-15 19:54:16 +02:00

docs(paperless-ngx): the policy header said TWO and listed three ... 95f3e8c1a1

Pre-existing, and corrected here rather than left because the third item is the
mail rule this PR rewrites. values.yaml already said THREE.

Part of #136

jklein approved these changes 2026-10-15 22:06:01 +02:00

jklein merged commit 95f3e8c1a1 into main 2026-10-15 22:06:05 +02:00

jklein deleted branch fix/paperless-egress-pod-ports 2026-10-15 22:06:06 +02:00

Sign in to join this conversation.

Reviewers

No Reviewers

jklein

Labels

Clear labels

inhouse-chart

Inhouse-Chart aus der JLab-Git-Paketregistry

renovate

Von Renovate angelegt

No labels

Milestone

No items

No Milestone

Assignees

Clear assignees

jklein (Justin Klein) jlab-bot (jlab-git-bot) jlab-renovate-bot (jlab-renovate-bot)

No Assignees

2 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: jlab-cluster/helm-charts#138