From 1e684ca6ce5d39d5443d8020a12cd2322dd9cf50 Mon Sep 17 00:00:00 2001 From: Justin Klein Date: Mon, 12 Oct 2026 22:02:42 +0000 Subject: [PATCH] freesailarr, Renovate, Lessons learned: the grouped stack PR and the security default a patch bump switched off The chart's trustedNetworks value for prowlarr, radarr and sonarr, with the consequence written as it is: the default is empty, so the chart does not restore the forwarded-header trust that prowlarr 2.6.3 and radarr 6.4.3 removed, and sonarr has been without it since before 4.0.19. Renovate: the first grouped chart, the two rules it takes because matchUpdateTypes and separateMinorPatch are rejected in one rule, and the measured branch split without the second rule. Lessons learned: a plain app-version bump switched off a security default, and that is the one update class the wiki obligation exempts. --- Lessons-Learned.md | 29 +++++++++++++++++++++++++++++ Renovate.md | 45 +++++++++++++++++++++++++++++++++++++++++++-- freesailarr.md | 30 ++++++++++++++++++++++++++++++ 3 files changed, 102 insertions(+), 2 deletions(-) diff --git a/Lessons-Learned.md b/Lessons-Learned.md index 78e3d82..181ed10 100644 --- a/Lessons-Learned.md +++ b/Lessons-Learned.md @@ -608,6 +608,7 @@ The investigation produced a **correct search** and a **wrong conclusion**, and - **A values flag can be correct and still be the wrong question.** When one value answers two questions that happen to coincide, the day they stop coinciding is a day nothing fails: the flag still renders, the pod still starts, and the second question is now answered wrong. Adding a second mode to anything is the moment to ask which of its readers were reading it for the *other* reason. - **A search over the chart answers what the chart does; what the running system does is a second measurement.** The chart is one input among several — the cluster repository's values, the controller's flags, an admission webhook, a previous operator's hand edit. Where a finding about a chart is meant to carry a claim about *behaviour*, read the live object or the repository that supplies its values; a correct search over the wrong scope produces a confident wrong answer. - **An operator can close a chart's gap invisibly, and the chart still looks correct.** Where the closure lives in another repository, the chart is incomplete, the system works, and the change that removes the closure contains no trace of it — which is why "this has always worked" is evidence about a deployment and not about a chart. +- **The size of a version number is a claim about an API, not about a default.** Upstream's semver answers "will your integration still compile", and a flipped security default fits inside a patch without breaking that promise. So the update class a pipeline computes decides who looks at the change — and the class with no documentation duty is the one where a changed default arrives unread. - **A "nothing else reads this" note is a measurement, not a property.** It is true on the day it is written, it is what later decisions rest on, and the change that falsifies it is the change obliged to rewrite it — in every copy, the schema description included. ## Four of six charts leaned on an annotation the chart never set @@ -642,3 +643,31 @@ The chart still renders neither, and the three reasons are the transferable part **A guard can matter *more* in the new mode, and that is a reason to look at what it protects.** ytdlp-web-player was the last open chart on the `ingress.enabled`-derived screen, and the find was not an environment variable but a **render guard**: the one refusing to publish the application and its bundled oauth2-proxy on the same host. In Ingress mode ignoring that guard fails loudly — ingress-nginx's admission webhook rejects the second Ingress and the Fleet bundle lands in `ErrApplied`. In Gateway mode **the webhook never fires**: it compares Ingress objects with Ingress objects, and an `HTTPRoute` is a different *kind*. Two valid objects then serve one hostname over two data paths that are not equivalent — the proxy's Ingress through authentication, the chart's route **straight to the application** — and which one a client reaches depends on what DNS resolves to. So a guard whose Ingress-mode job was to pre-empt a noisy webhook becomes, in Gateway mode, the only thing between a values file and an **authentication bypass that comes and goes with a DNS record**. When screening for a derived value, ask what it *protects*, not only what it computes. **And a shape read off an issue's summary table is a hypothesis.** Three of #114's four descriptions of the group-3 shapes were wrong — bulwark has one Ingress rather than two and has no `rewrite-target` at all; matrix-synapse renders one object with two host rules rather than a loop over apps; stalwart's routing object is not per node group. Two of the three had already been copied into `README.md` and into the header comment of `ci/run-httproute-match-test.sh`, where they read as settled fact and would have been copied again. The bulwark one has a mechanism worth naming: the table was built by grepping for the string `rewrite-target`, and every occurrence of that string in bulwark is an explanation of **why the annotation does not help it**. A grep counts refutations as occurrences. + +## The security default that a patch bump switched off + +**Where:** [freesailarr](freesailarr), in the round that collapsed eight simultaneous Renovate PRs into one. `prowlarr` 2.5.2 → 2.6.5 and `radarr` 6.3.0 → 6.4.4 — and `sonarr`, where it had already happened and nobody had noticed. + +**What it is:** the \*arr applications used to hard-code the private networks whose `X-Forwarded-*` headers they trust. Read at the tags this chart pins rather than from the release notes alone — `src/NzbDrone.Host/Startup.cs` at prowlarr `v2.5.2.5491`: + +```csharp +options.KnownNetworks.Add(new IPNetwork(IPAddress.Parse("10.0.0.0"), 8)); +options.KnownNetworks.Add(new IPNetwork(IPAddress.Parse("172.16.0.0"), 12)); +options.KnownNetworks.Add(new IPNetwork(IPAddress.Parse("192.168.0.0"), 16)); +options.KnownNetworks.Add(new IPNetwork(IPAddress.Parse("fc00::"), 7)); +options.KnownNetworks.Add(new IPNetwork(IPAddress.Parse("fe80::"), 10)); +``` + +prowlarr 2.6.3 and radarr 6.4.3 removed all five and replaced them with a `TrustedNetworks` setting whose own default is `string.Empty`; `ForwardedHeadersConfigurator.Configure` returns before registering a single `KnownNetwork` while it is empty. Upstream's reason is sound — trusting `X-Forwarded-For` from any private network is a spoofing vector — and the consequence behind an in-cluster ingress controller is still a behaviour change: the client address the app logs and displays is the controller's pod IP, the scheme reads as `http` behind TLS termination, and under "Disabled for Local Addresses" a proxied request stops counting as local and is asked to authenticate. + +**The part that is a lesson rather than an upstream note: for `sonarr` this had already shipped, inside a patch.** 4.0.19 — the version the chart was already running — has no hard-coded networks and the same empty-default `TrustedNetworks`. It arrived as what the convention calls a *plain app-version bump*: the class `ci/renovate-bump-chart.sh` turns into a chart **patch** automatically, and the one class the wiki obligation explicitly exempts. So nothing documented it and nothing reviewed it. The two remaining apps were caught only because their jumps crossed a **minor** boundary, which the convention does treat as possibly behaviour-changing — the safety net that worked was the version number's *size*, which is a statement about upstream's API compatibility and not about its defaults. + +**Three transferable pieces:** + +1. **A one-line tag diff is not a small change; it is an unread one.** What carried this was a diff shaped `- tag: 4.0.19` / `+ tag: 4.0.20` — nothing in it can show a default being inverted. Where a bump is classified automatically, the classification is also what decides whether anybody looks. +2. **Read the source at both tags when a default is in question.** The release note says *"New: Add Trusted Networks setting"*, which reads as purely additive; the removal of the five networks is visible only in the diff of `Startup.cs`. A new setting and a changed default are the same commit here, and only one of them is in the headline. +3. **A guessed tag's 404 looks exactly like "not affected".** The first check of sonarr used an invented tag, `v4.0.20.2912`: the file 404'd and the grep came back empty — the identical result to a version that never had the code. The real tags are `v4.0.20.3014` and `v4.0.19.3011`, and against those the answer inverted. Where absence is the finding, confirm that the thing you are reading exists. + +**What the chart does about it, and what it deliberately does not.** A `.trustedNetworks` value was added for all three apps — sonarr included, because repairing two and knowingly leaving the third is the same silent pattern — rendered as `__SERVER__TRUSTEDNETWORKS` and **only when non-empty**, since an empty variable is indistinguishable from an unset one to the app but looks handled to a reader of the manifest. Its default is empty, so **the chart does not restore the old behaviour**: it follows upstream's intent and makes the state visible, with a `NOTES.txt` warning that fires in exactly the combination that costs something — the app published through an Ingress *or* an HTTPRoute, and the value empty. `AllowedHosts`, added by the same upstream commits, is deliberately not exposed: its default is `string.Empty` too and the app then accepts any host, so there is nothing to repair. If anyone ever sets it, the trap to know is that the app appends only `localhost`, `127.0.0.1`, `[::1]` and its own hostname, so neither the pod IP nor the Service DNS name is covered — and the kubelet probes the pod *by IP*, which an incomplete list answers with HTTP 400. + +**Left open on purpose:** whether the convention that exempts a plain app-version bump from the wiki obligation should change. It is the mechanism that let this through, and it is a decision of its own rather than an appendix to a version PR. diff --git a/Renovate.md b/Renovate.md index c440da0..0bd7a67 100644 --- a/Renovate.md +++ b/Renovate.md @@ -16,7 +16,7 @@ The `jlab-renovate` GitHub App that used to do this is switched off along with t |---|---| | Chart app versions (`appVersion:` in `Chart.yaml`) | one `customManagers` entry per chart, hard-coding the image repository | | Sidecar and init images in `values.yaml` (nginx, guacd, busybox, alpine) | built-in `helm-values` manager | -| **All** images of `freesailarr`, its main one included | built-in `helm-values` manager — the chart pins every tag in `values.yaml` and has no app-version block. Exception: `mediathekarr` is pinned by **digest** and has its own `customManagers` block | +| **All** images of `freesailarr`, its main one included | built-in `helm-values` manager — the chart pins every tag in `values.yaml` and has no app-version block. Exception: `mediathekarr` is pinned by **digest** and has its own `customManagers` block. All ten arrive **in one grouped PR**, see [The one grouped chart](#the-one-grouped-chart) | | The oauth2-proxy subchart of `ytdlp-web-player` | built-in `helmv3` manager — repository URL and version sit together in `Chart.yaml`, so no custom manager is needed | | GitHub Actions | built-in `github-actions` manager | | Tool versions pinned as workflow env vars (`TRIVY_VERSION`, `KUBECTL_VERSION`) | a `# renovate: datasource=… depName=…` comment above the line, through a `customManagers` entry for `.github/workflows/*.yml` | @@ -70,6 +70,45 @@ So **the procedure is documented instead of prevented**, which puts an obligatio The chart's `NOTES.txt` carries the same sequence, so it reaches whoever runs the release and not only whoever reads this page. One thing is deliberately left open there and here: upstream's 26.6 release notes call patch-level rolling updates supported and default-on, while the 26.8.0 `update-compatibility` page says a rolling update is possible "when the Keycloak version is the same" and uses a **patch** pair as its own incompatible example. Until that is resolved, run the check for patch updates too. +## The one grouped chart + +[freesailarr](freesailarr) is the only chart whose updates are **grouped**: all ten of its images arrive as one PR per cycle, `groupName: "freesailarr stack"`. Every other chart gets one PR per image, which for a one-image chart is already the same thing — so this is a rule about a ten-image chart, not a change of policy. It is also the only `groupName` in `renovate.json5`, and it reverses what the comment above the app-version managers says about the `helm-values` manager proposing "one update per image". That sentence described what Renovate does; it was never an argument for what this chart wants. + +**Measured on 2026-10-12, which is why the rule exists.** Eight of the ten images had an open Renovate PR at the same moment (#102 bazarr, #103 qbittorrent, #104 sonarr, #105 flaresolverr, #106 recyclarr, #107 prowlarr, #108 radarr, #111 seerr). Each edits `freesailarr/values.yaml` **and**, through the [version-bump hook](#the-version-bump-hook), `freesailarr/Chart.yaml` — so the API reported `mergeable: false` for all eight: they rewrite the same `version:` line. One can merge; the other seven then need a rebase each, and every rebase restarts the most expensive install-test in this repository (a real Wireguard tunnel, ten containers in one pod). + +A second symptom came with it, and it is the sharper one: all eight carried `version: 3.1.1+up1.0.0 → 3.1.2+up1.0.0` while `main` already stood at `3.3.0+up1.0.0`. The hook derives everything from the **base commit**, and those branches were cut before the Gateway-API rounds landed — so merging one unrebased would have moved the chart version *backwards* by two minors and republished it under a version the registry already had. A rebase repairs that, which is exactly the problem: **nothing fails if the rebase does not happen first**, so the repair depends on the bot reaching the branch before a human reaches the merge button. The mutual blocking is the durable argument; the backwards version depends on that race. + +Grouped, there is one branch, one `version:` computed once from one base, one publish, one install-test run. + +**Four decisions, each deliberate:** + +- **Matched by file name, not by a list of image names.** What creates the collision is "touches `freesailarr/Chart.yaml`", and that is true of every dependency Renovate finds in that `values.yaml`, whatever its manager or datasource. A `matchPackageNames` list would be a second place to keep the chart's image set current, and its failure mode is silence — an image added later never joins the group, with no error and no warning. A valid rule is not a matching rule. +- **`mediathekarr` is in the group**, although it comes from its own `customManagers` block rather than from `helm-values`. Its digest PR edits the same two files and blocks the others identically, so keeping it outside would preserve a smaller copy of the problem. Its own block is untouched: a group decides which *branch* an update lands on, not how it is found. `digest` is in `matchUpdateTypes` precisely for it. +- **`gluetun` and the `busybox` init image are in too**, against the instinct to hold them apart because gluetun is the tunnel every other container depends on and busybox is "only" an init container. Neither property has anything to do with the blocking, and a grouped PR hides nothing — the review still sees every changed line, just not eight times over. +- **Majors stay out**, via `matchUpdateTypes: ["patch", "minor", "digest"]`. [Versioning](Versioning#app-major-jumps-are-taken-one-line-at-a-time) requires an app major to be taken stepwise in its own PR and `separateMultipleMajor` produces that sequence; a group throwing a major in with seven patches would undo both, and a major is the update where a review most needs one image at a time. + +**It takes two `packageRules`, and that is not cosmetic.** Renovate refuses the two settings in one rule: + +``` +packageRules[n]: packageRules cannot combine both matchUpdateTypes and separateMinorPatch +``` + +a config-validation **error**, not a warning — found by running the dry-run rather than trusting the draft. It is consistent: `separateMinorPatch` is what *decides* the update-type split, so one rule may not both perform the split and select on its outcome. So one rule groups the non-major update types, and a second sets `separateMinorPatch: false` for the same file. + +Without that second rule the group still arrives as **two** PRs. Verified (`--platform=local --dry-run=full`, real registries, against a checkout holding those eight updates): + +| | branch | contents | +|---|---|---| +| group rule alone | `renovate/freesailarr-stack` | prowlarr 2.6.5, radarr 6.4.4, seerr v3.5.0 (minor) | +| | `renovate/patch-freesailarr-stack` | bazarr, qbittorrent, sonarr, flaresolverr, recyclarr (patch) | +| both rules | `renovate/freesailarr-stack` | all eight | + +Note the shape: `separateMinorPatch` **prefixes** the branch name rather than suffixing it, which makes the second branch easy to miss when scanning a list. The global `separateMinorPatch: true` keeps its reason for every other chart — "one red install-test must not hold up unrelated updates" — and that reason simply does not hold inside a single chart, where the updates share a pod, a chart version and a publish. + +The `mediathekarr` membership was checked the way [its own pin](#the-one-digest-pin) demands, because with a current digest a working rule and a broken one produce the same empty result: with the real digest the run yields no mediathekarr update at all and proves nothing, while with the digest set to `sha256:0000…` the digest update appears **on the group branch** beside the other eight instead of on `renovate/docker.io-pcjones-mediathekarr-beta`. + +**What none of this shows.** Renovate runs as the shared instance outside this repository, so the available evidence is a config that validates and a lookup that groups as intended locally — not the behaviour of the next real run, and not the instance's Renovate version. + ## The one digest pin `freesailarr`'s `mediathekarr` image is the only one pinned by digest, because its tags carry no version at all: `latest` is a 2024 build and the upstream README points at the moving `beta`. The built-in `helm-values` manager reads `repository`/`tag` and skips it (`skipReason: invalid-value`), so the two managers cannot produce competing PRs. @@ -94,6 +133,8 @@ On the Gitea instance that entry **exists and was verified rather than hoped for A consequence of `freesailarr` pinning every tag in `values.yaml`: the script sees an unchanged `appVersion` and bumps the chart **patch**. That holds for all ten images alike, the digest-pinned one included, and it is the intended outcome — `appVersion` there is the chart's own marker rather than a tracked upstream version. +**An open point this leaves, recorded rather than solved.** That automatic patch is also a *classification*, and the convention exempts a plain app-version bump from the wiki obligation — so an image update that quietly changes a **default** is the one case nothing reviews and nothing documents. It has already happened: sonarr's 4.0.19 bump carried upstream's removal of the hard-coded trusted proxy networks and reached `main` unremarked — see [The security default that a patch bump switched off](Lessons-Learned#the-security-default-that-a-patch-bump-switched-off). Whether the convention should change is a decision of its own and is deliberately not proposed here. + ## Constrained dependencies **nginx** (the Synapse delegation sidecar) is limited to **even minors** via `allowedVersions`. nginx numbers its release branches by minor — even is stable, odd is mainline — and the chart belongs on stable; patch updates inside the stable branch still flow normally. The constraint tolerates a variant suffix (`1.30.4-alpine`) on purpose: docker versioning only offers candidates carrying the current tag's suffix, so a suffix-blind constraint would silently freeze nginx the day the sidecar moves to a slimmer base image — with no error and no warning. @@ -106,7 +147,7 @@ The config is **aligned with the one in `cluster-config`**, because the same `jl - **Automerge is global** — `automerge: true`, `automergeType: "pr"`, `platformAutomerge: true`, for every update type, **majors included**. What keeps that from being unattended merging is the branch protection, not the absence of automerge: `main` requires a **review** and requires **`validate-charts`** to pass. `platformAutomerge` arms the PR on the forge, and Gitea merges it once *both* are satisfied — an armed PR nobody reviews just sits there. This is exactly the condition the older `platformAutomerge: false` comment named as the point at which `true` becomes the better value; it arrived, so the value changed. `cluster-config` records the same reasoning, having retired its dashboard-approval step for it. - **`minimumReleaseAge: "3 days"`** for `patch`, `minor` and `digest` updates — Renovate does not open the PR before then. A merge here is never just a merge: a chart publish follows it straight away, so a release that is yanked hours after it appeared must not reach the registry. `cluster-config` spends the same three days on the dependencies where a merge is expensive (authentik, rancher); here that is all of them. **Majors carry no age gate** — they are held at the review, not by a delay. -- **Majors** arrive one line at a time (`separateMultipleMajor`): a v1 → v3 jump produces a PR for the latest v2 first, then one for v3 — see [Versioning](Versioning#app-major-jumps-are-taken-one-line-at-a-time). `separateMinorPatch: true` splits minor and patch into separate PRs as well, so one red install-test does not hold up unrelated updates. +- **Majors** arrive one line at a time (`separateMultipleMajor`): a v1 → v3 jump produces a PR for the latest v2 first, then one for v3 — see [Versioning](Versioning#app-major-jumps-are-taken-one-line-at-a-time). `separateMinorPatch: true` splits minor and patch into separate PRs as well, so one red install-test does not hold up unrelated updates. **It has exactly one exception**, `freesailarr`, where it is overridden to `false` — see [The one grouped chart](#the-one-grouped-chart). - **No PR caps** (`prConcurrentLimit: 100`, `prHourlyLimit: 0`). The cost is known and accepted rather than discovered: the first run after a quiet week can open many PRs at once, and every chart PR starts a k3d install-test, which on a Gitea runner **queues inside one job** instead of fanning out (see [CI/CD](CI-CD)). The alternative was updates trickling in four per hour, and nothing merges without a review anyway. - **`config:best-practices`** is in `extends`, as in `cluster-config`. Two of its members matter here: `helpers:pinGitHubActionDigests` moves the workflow files from `actions/checkout@v7` to SHA pins, and `docker:pinDigests` is switched off again for `custom.regex` and `helm-values`. An `appVersion:` is a **tag** by construction — the templates render `:{{ .Chart.AppVersion }}` and the chart version's `+up` suffix is built from the same string — so a digest has nowhere to go. Sidecar digests in `values.yaml` could be pinned; that is its own decision, since each digest change would be a chart patch plus a publish. - **Labels:** every Renovate PR carries `renovate`; app-version PRs additionally `app-version`, subchart PRs `subchart`. diff --git a/freesailarr.md b/freesailarr.md index 75f06f1..273a844 100644 --- a/freesailarr.md +++ b/freesailarr.md @@ -171,6 +171,7 @@ Nine app blocks (`qbittorrent`, `prowlarr`, `flaresolverr`, `radarr`, `recyclarr | `.ingress.annotations` | object | unset | **Replaces** the global `ingress.annotations` for this app (it is an override, not a merge); the cert-manager annotation is always added. | | `.storage.size` | string | `1Gi` | Size of this app's config `volumeClaimTemplate` (RWO). Not null-safe: an erased size fails the render. | | `.env` | list | `[]` | Extra environment variables **appended** to the ones the chart sets. Rendered through `tpl`; `value`, `secretKeyRef` and `configMapKeyRef` forms. **Additive, not overriding** — see [Environment variables](Chart-Style-Guide#10-environment-variables). | +| `.trustedNetworks` | string | `""` | **Only for `prowlarr`, `radarr` and `sonarr`.** Comma-separated CIDRs whose `X-Forwarded-*` headers the app honours, rendered as `__SERVER__TRUSTEDNETWORKS` and **only when non-empty**. Empty — the default — means **no proxy is trusted**; see [Forwarded headers are not trusted by default](#forwarded-headers-are-not-trusted-by-default). | | `.livenessProbe` | object | per app, see table | Merged onto this container's own defaults; `null` restores them rather than dropping the probe. | | `.readinessProbe` | object | per app | | | `.startupProbe` | object | per app | Gates liveness and readiness. | @@ -320,6 +321,35 @@ That short-circuit alone settles it. What an unanchored regex location would the **What CI can and cannot prove about this.** `ci/run-httproute-match-test.sh` asserts, per `HTTPRoute` object, that a rule stripping a prefix with `ReplacePrefixMatch` is accompanied by a rule matching that path `Exact`ly with a `RequestRedirect` filter — see [CI/CD](CI-CD). That proves the rendered **object** carries the redirect. It does **not** prove a 301 is ever served: no Gateway controller runs in CI, no route reaches `Accepted: True`, and nothing there evaluates matching precedence. +### Forwarded headers are not trusted by default + +`prowlarr`, `radarr` and `sonarr` trust **no** proxy out of the box. Their `trustedNetworks` value is empty by default, and while it is empty the app registers not a single trusted network — upstream's `ForwardedHeadersConfigurator.Configure` returns early. + +**The chart does not restore the behaviour these apps used to have.** Up to prowlarr 2.5.2 and radarr 6.3.0 they hard-coded `10.0.0.0/8`, `172.16.0.0/12`, `192.168.0.0/16`, `fc00::/7` and `fe80::/10` as trusted proxy networks; prowlarr 2.6.3 and radarr 6.4.3 removed all five in favour of a setting that defaults to empty. This chart follows that default deliberately, because trusting `X-Forwarded-For` from any private network is a spoofing vector. `sonarr` has been in this state since before 4.0.19 — it arrived unnoticed in a plain app-version bump, which is [its own lesson](Lessons-Learned#the-security-default-that-a-patch-bump-switched-off). + +What it costs while the value is empty **and** the app is published through an Ingress or an HTTPRoute: + +- the client address the app logs and shows is your ingress controller's pod IP, not the real client; +- the app sees the scheme as `http` even where TLS is terminated in front of it; +- with authentication set to *Disabled for Local Addresses*, a proxied request no longer counts as local and is asked to authenticate. + +`NOTES.txt` prints a warning in exactly that combination — published and empty — and stays quiet otherwise, so it is not a banner to learn to ignore. + +To honour the headers, set the network your ingress controller's pods sit in. **The chart cannot know it**: it is a property of the cluster, not of the release. + +```yaml +prowlarr: + trustedNetworks: "10.42.0.0/16" # e.g. the k3s pod CIDR +radarr: + trustedNetworks: "10.42.0.0/16" +sonarr: + trustedNetworks: "10.42.0.0/16" +``` + +Narrowing it to the controller's own pod IP is stricter and valid; a pod IP is not stable across reschedules, so a CIDR is the practical choice. Leaving it empty is a legitimate decision — it is only no longer a silent one. + +**`AllowedHosts` is deliberately not a value here.** Upstream added it in the same releases, its default is empty as well, and while it is empty the app accepts requests for any host (it logs *"Allowed Hosts is not configured, accepting requests for any host"*), so there is nothing to repair. One trap if you ever set it through `.env` anyway: the app appends only `localhost`, `127.0.0.1`, `[::1]` and its own hostname to the list, so neither the pod IP nor the in-cluster Service DNS name is covered — and the kubelet addresses the probes to the **pod IP**, which an incomplete list answers with HTTP 400. Readiness then fails on an app that is working. + ### Every Service points at the same pod IP One ClusterIP Service per enabled app that listens on a port (recyclarr has none). Because all containers share gluetun's namespace, the Services differ only in their port: they give stable per-app DNS names, **not** network separation. mediathekarr's Service carries both its ports; only the primary (5007) is ever addressed by a routing object.