freesailarr, Lessons learned: the release runs in the jellyfin namespace - correcting a limit I invented

A correction to my own measurement round, and it runs in the direction of
understating the evidence rather than overstating it.

The freesailarr page claimed, as the second of three limits, that there is
no freesailarr namespace in jk-cluster and no pods of it visible, and
concluded that where the backend runs is not determinable. The first
clause is literally true; the conclusion does not follow.

THE RELEASE RUNS IN NAMESPACE jellyfin. Verified: pod
freesailarr-freesailarr-sfs-0, 10/10 Running, 0 restarts, on jk-worker-2;
the StatefulSet's Helm annotations name release freesailarr in namespace
jellyfin; all ten containers readable (gluetun, qbittorrent, prowlarr,
flaresolverr, radarr, recyclarr, sonarr, seerr, bazarr, mediathekarr).

How the error was produced: by searching over namespace NAMES - for
freesailarr, and for anything containing "sail" or "arr". None matched,
because the release lives beside its neighbouring service rather than in a
namespace named after its chart. An empty result that read like a fact.

What this strengthens, and it is the point of the correction: the
freesailarr.published helper is now proven AT THE SOURCE rather than
deduced from a redirect through a proxy. RADARR__SERVER__URLBASE=/radarr
is set in the running container (with PROWLARR__ and SONARR__ likewise)
while kubectl get ingress -n jellyfin returns NO resources at all - not
this chart's object and not jellyfin's own. That combination is exactly
what the helper exists for: read from ingress.enabled alone the variable
would be empty in Gateway mode, and the UI would be broken behind a Ready
pod.

The two genuine limits stay, and are now scoped: the implementation cannot
be named (httproutes and gatewayclasses are RBAC-forbidden to the
read-only identity), and every HTTP measurement passed through the front
proxy at proxy.jk-dev.de - which applies to the HTTP rows and not to the
rows read from the API server.

The lesson is recorded with it: a release does not have to live in a
namespace named after its chart, and a query over namespace names does not
answer "is this running". Same genus as the #118 finding one level over - a
search that answers a different question than the one asked - and the
mirror image of the rewrite-target grep that counted bulwark's refutations
as occurrences. Treat an empty result as a question about the query before
accepting it as a finding.

No chart change and no version step: this is a wiki-only correction.

Lessons-Learned.md

@@ -685,13 +685,18 @@ On 2026-10-13 `freesailarr.jk-dev.de` was switched to Gateway mode, and the clai
 
 ### The lesson is not "it works" — it is where the measurement stopped
 
-Three limits, and each one is the interesting part:
+Two limits are real, and a third one that was first written here turned out to be a mistake of my own — which is the more instructive half.
 
 1. **The implementation cannot be named.** `httproutes` and `gatewayclasses` are RBAC-forbidden to the read-only identity. That the 301 comes from the chart's rule rather than from the operator's nginx snippet is **inferred** — from the snippet's carrier being gone (`ingress.annotations` are dropped in this mode) and from the response signature changing with it (before: nginx, `text/html`, 162 bytes, HSTS). A good inference is still an inference, and it was written down as one.
-2. **There is no `freesailarr` namespace in the cluster at all**, and no pods of it visible, yet the host answers. So "the host is served without an Ingress" is solid; "a particular `HTTPRoute` of this chart serves it" is not readable from here.
-3. **Every HTTP measurement passed through a front proxy.** The hosts resolve to a single address that is not the ingress/gateway load balancer. So the response signature is evidence about what the proxy returned, which is one hop removed from what the Gateway did.
+2. **Every HTTP measurement passed through a front proxy.** The hosts resolve to a single address that is not the ingress/gateway load balancer. So the response signature is evidence about what the proxy returned, one hop removed from what the Gateway did. This limit applies to the HTTP measurements and **not** to anything read from the API server.
 
-**The generalisable point:** a measurement that reaches the system is worth far more than a render — and it still has a boundary, which is usually a layer of infrastructure nobody mentioned. The honest form is to state what was observed, what was inferred from it, and which layer the observation actually came from. "It routes in production" would have been an overstatement of a true result; "the host is served, without any Ingress object, with the base paths and the redirect intact, measured through the front proxy, implementation unnameable" is the same news without the debt.
+**The third "limit" was wrong, and it was wrong in the direction of understating the evidence.** It read: there is no `freesailarr` namespace in the cluster and no pods of it visible, so where the backend runs is not determinable. The first clause is literally true and the conclusion does not follow. **The release runs in namespace `jellyfin`** — `freesailarr-freesailarr-sfs-0`, `10/10 Running`, 0 restarts, Helm annotations naming release `freesailarr` in namespace `jellyfin`. All ten containers are readable, and so is the evidence that mattered most: `RADARR__SERVER__URLBASE=/radarr` in the running pod, in a namespace with **no Ingress object at all**.
+
+How the error was produced: by searching over namespace **names** — for `freesailarr`, and for anything containing "sail" or "arr". None matched, because the release lives beside its neighbouring service rather than in a namespace named after its chart.
+
+**So: a release does not have to live in a namespace named after its chart, and a query over namespace names does not answer "is this running".** `kubectl get pods -A | grep <release>`, or the Helm release annotations, answers it. This is the same genus as the #118 finding one level over: a search that answers a *different question* than the one asked, and returns an empty result that reads like a fact. The `rewrite-target` grep that counted `bulwark`'s *refutations* as occurrences is the same mistake in the other direction — a non-empty result that reads like a fact.
+
+**The generalisable point, now with a sharper edge:** a measurement that reaches the system is worth far more than a render, and it still has a boundary — but check whether the boundary is the infrastructure's or your own query's. Two of the three limits here were the system's; the third was mine, and it was the one that made the evidence look weaker than it was. The honest form is to state what was observed, what was inferred, and which layer the observation came from — and to treat an empty result as a question about the query before accepting it as a finding.
 
 ### And the query-string answer is a data point, not a promise
 

freesailarr.md

@@ -286,25 +286,31 @@ Every PR of the Gateway-API sweep (Gitea #114) says in as many words that "it ro
 
 | | |
 |---|---|
-| `freesailarr` Ingress objects | **none**, in any namespace. Cross-checked: the same identity lists 10 Ingress objects across six other namespaces, so this is a real absence and not a visibility gap |
+| Where the release runs | **namespace `jellyfin`** — `freesailarr-freesailarr-sfs-0`, `10/10 Running`, 0 restarts, on `jk-worker-2`. The StatefulSet's Helm annotations name release `freesailarr` in namespace `jellyfin`. **The namespace is named after the neighbouring service, not after the chart** — see the note below |
+| `freesailarr` Ingress objects | **none**, and the measurement that matters is **in `jellyfin` itself**: `kubectl get ingress -n jellyfin` returns *no resources at all* — not this chart's, and not jellyfin's own. Cross-checked cluster-wide: the same identity lists 10 Ingress objects across six other namespaces, so this is a real absence and not a visibility gap |
+| `RADARR__SERVER__URLBASE` in the running pod | `/radarr` — and `PROWLARR__SERVER__URLBASE=/prowlarr`, `SONARR__SERVER__URLBASE=/sonarr`. Read off the pod spec, not inferred |
 | `GET /radarr` | `302`, `server: Kestrel`, `location: https://freesailarr.jk-dev.de/radarr/login?returnUrl=%2Fradarr` |
 | `GET /qbittorrent` | `301`, `location: …/qbittorrent/`, 17-byte body, no `content-type`, no HSTS, no `server` |
 | `GET /qbittorrent?foo=bar` | `301`, `location: …/qbittorrent/?foo=bar` — the query string survives |
 
 **What each one settles.**
 
-The `/radarr` answer is radarr itself, and `returnUrl=%2Fradarr` proves it knows its own base path. So the `freesailarr.published` helper is doing its job **in production**: without it `RADARR__SERVER__URLBASE` would be empty in Gateway mode — `ingress.enabled` is `false` there by construction — and the UI would be broken behind a `Ready` pod, assets and API calls linked at the host root. That is exactly the defect the helper was written for, and it was measured rather than asserted.
+**`freesailarr.published` is proven directly, from the cluster rather than from a redirect.** `RADARR__SERVER__URLBASE=/radarr` is set in the running container while **no Ingress object exists in that namespace** — which is precisely the combination the helper exists for. Read from `ingress.enabled` alone that variable would be **empty** in Gateway mode, because that value is `false` there by construction, and the UI would be broken behind a `Ready` pod with assets and API calls linked at the host root. The pod spec is the strongest form this evidence can take: it is the chart's output, read where it landed, with nothing in between.
+
+The `/radarr` answer is the application agreeing: `returnUrl=%2Fradarr` is radarr itself reporting the base path it was given.
 
 The two qBittorrent answers show the chart's own redirect rule working, and the query string surviving it. The specification still does not promise the latter — see [The trailing slash on qBittorrent's prefix](#the-trailing-slash-on-qbittorrents-prefix).
 
 **A consequence for the operator:** the nginx snippet in `jk-cluster-config` is now **without effect and can be removed**. Removing it earlier would have been wrong — in Ingress mode it was the only thing performing the redirect. It lost its effect not through a change in the chart but because the mode drops `ingress.annotations`.
 
-**The limits of these measurements, which are real.** None of this names the implementation, and two independent reasons prevent it:
+**Look for the release, not for a namespace named after it.** This release lives in `jellyfin`, next to the service that namespace *is* named for. A search over namespace **names** — for `freesailarr`, or for anything containing "sail" or "arr" — finds nothing and invites the conclusion that the workload is not in this cluster at all. It is, it is healthy, and all ten of its containers are readable. The question "is this running" is not answered by a query over names; it is answered by looking for the release, e.g. `kubectl get pods -A | grep freesailarr` or the Helm release annotations. Recorded in [Lessons learned](Lessons-Learned) because the same mistake is available on every chart here.
 
-- `httproutes` and `gatewayclasses` are **RBAC-forbidden** to the read-only identity (`system:serviceaccount:coder-workspaces:claude-readonly`), so the route and its controller cannot be read at all. That the 301 comes from the chart's rule is inferred from the carrier's disappearance plus the changed response signature, not read off an object.
-- There is **no `freesailarr` namespace in `jk-cluster`** and no pods of it visible, while the host answers — and both `freesailarr.jk-dev.de` and the other hosts of this cluster resolve to a single front proxy (`proxy.jk-dev.de`). So every HTTP measurement above passed through that proxy before reaching anything, and where the backend runs is not determinable from here.
+**The limits of these measurements, and they are narrower than the ones first written here.** Two things remain genuinely out of reach:
 
-What is therefore established: the host is served, it is served **without any Ingress object**, the sub-path apps know their base paths, and the bare-prefix redirect happens with the query intact. What is **not** established: which Gateway implementation does it, or that any particular `HTTPRoute` of this chart is the object doing it. The first proof that this repository's Gateway mode routes at all is still a proof about the *form*, obtained one layer further out than one would like.
+- **The implementation cannot be named.** `httproutes` and `gatewayclasses` are **RBAC-forbidden** to the read-only identity (`system:serviceaccount:coder-workspaces:claude-readonly`), so neither the route nor its controller can be read. That the 301 comes from the chart's rule is inferred from the carrier's disappearance plus the changed response signature, not read off an object.
+- **Every HTTP measurement passed through a front proxy.** `freesailarr.jk-dev.de` — like the other hosts of this cluster — resolves to `152.53.207.153`, `proxy.jk-dev.de`, which is not the ingress/gateway load balancer. So the response signatures above describe what that proxy returned, one hop out from whatever the Gateway did. This limit applies to the four HTTP rows and **not** to the three cluster rows, which were read from the API server directly.
+
+What is therefore established: the release runs in `jellyfin`, healthy; **no Ingress object exists in that namespace**; the sub-path applications have been given their base paths, read from the pod spec; the host is served over HTTPS; and the bare-prefix redirect happens with the query string intact. What is **not** established: which Gateway implementation serves it, or that a particular `HTTPRoute` of this chart is the object doing so. The first proof that this repository's Gateway mode routes at all is a proof about the *form* — but the chart's own half of it, the published-helper output, is now read at the source rather than deduced.
 
 ### Two routing shapes, and they mix