feat: Gateway-API as an alternative to the Ingress in the six special shapes (group 3) #120

Merged

jklein merged 7 commits from feat/gateway-api-group-3 into main 2026-10-13 10:59:10 +02:00

Conversation 1 Commits 7 Files Changed 42

+2945 -137

jlab-bot commented 2026-10-12 23:48:49 +02:00

Member

Copy link

Copy source

Group 3 of the #114 sweep: the six special shapes — keycloak, bulwark, matrix-synapse, stalwart, homeassistant, ytdlp-web-player. This finishes the sweep except for rss-bridge, which is frozen (#51), so #114 stays open after this merges.

Each chart gets the form the pilot fixed and groups 1 and 2 repeated: httpRoute off by default, a template rendering only HTTPRoute objects with parentRefs, render aborts on both modes at once (naming both value paths) and on an empty parentRefs, empty hostnames meaning [ingress.domain], a schema entry closed against unknown keys with an open parentRef definition, and a NOTES.txt that names the active mode and what it costs.

Minor for all six, additive, defaults unchanged — helm template with each chart's CI values is byte-identical to origin/main for all six. One judgement call is flagged below.

---

The central task: four of six charts leaned on an annotation the chart never sets

#118 taught that a chart can read as correct while the running system fills a gap of the chart with an operator-set annotation — and Gateway mode drops ingress.annotations, so it drops the compensation with it. At #118 that was found after shipping. Here it was measured beforehand, with read-only kubectl against jk-cluster, and it was not singular:

Chart	Measured on the live Ingress, 2026-10-12 — chart sets none of it
keycloak	proxy-buffer-size: 16k, proxy-buffers-number: 4 — on both chart-owned Ingress objects
matrix-synapse	proxy-body-size: 50m, plus nginx.org/client-max-body-size: 50m, plus a configuration-snippet of three more_set_headers
stalwart	enable-cors: true, cors-allow-origin: https://webmail.bekemeyer-klein.de, cors-allow-methods: GET, POST, OPTIONS, cors-allow-headers: Authorization, Content-Type
bulwark	proxy-body-size: 25m

Confirmed: a git grep over those six chart directories for proxy-buffer, app-root, more_set_headers, enable-cors, cors-allow, proxy-body-size, configuration-snippet hits documentation only (bulwark/values.yaml:733 recommends 25m, ytdlp-web-player/values.yaml:316 recommends 16k). No template sets any of them.

Per decision 1 of #114 no chart renders an implementation-specific policy object, so these are named, not rebuilt — in the Gateway branch of each NOTES.txt, in that chart's own terms.

Where I could not reproduce the briefing, and this is the valuable half

homeassistant and ytdlp-web-player: "nothing found" is right, but not for the reason it reads like.

• homeassistant is not deployed in jk-cluster at all — 23 Ingress objects cluster-wide, none this chart's, no workload of it in any namespace. So it is not that the chart's host carries no annotation; there is no host.
• ytdlp-web-player is deployed (namespace jlab-player, both Deployments Ready, jlab-player-oauth2-proxy and jlab-player-ytdlp-web-player-service) and has no Ingress whatsoever — neither the chart's nor the subchart's. What publishes it could not be determined: this workspace's identity is forbidden httproutes and secrets, so neither a hand-written HTTPRoute nor the release's Helm values are readable. Recorded as an open question in values.yaml and in the overlay rather than guessed at. Worth resolving before anybody plans a mode switch for that host.

keycloak's four realm Ingresses do not share the chart's hostname. Confirmed: four keycloak-realm-*-ingress objects with app-root, from release jk-cluster-config-fleet-cd-keycloak-realm-ingress, i.e. another repository; the chart's template renders exactly two. But they share the backend Service, not the hostname — they publish bk.id.bekemeyer-klein.de, id.mail.bekemeyer-klein.de, muth-event.id.bekemeyer-klein.de and id.klein.pub, four further names, two of them subdomains of the chart's public host. A wildcard listener for *.id.bekemeyer-klein.de would cover them. Also: three of the four carry the proxy-buffer pair, jk-family does not. Noted in the wiki as a migration hint. jk-cluster-config is not touched.

One extra live annotation the briefing did not name: matrix-synapse also carries nginx.org/client-max-body-size: 50m, the NGINX-Inc-flavoured twin of proxy-body-size — a hedge across both nginx ingress controllers (jellyfin and paperless-ngx carry the same pair). It disappears in Gateway mode for the same reason.

And one extra sibling-repo object: stalwart/mta-sts-ingress, from release jk-cluster-config-fleet-cd-stalwart-mta-sts, publishing mta-sts.bekemeyer-klein.de/.well-known/mta-sts.txt against this release's Service. Same situation as the keycloak realm objects, same treatment: not rendered here, not touched.

---

Three of #114's shape descriptions did not survive contact with the charts

This is the finding I would most want reviewed, because two of the three had already been copied into README.md and into the header comment of ci/run-httproute-match-test.sh, where they read as settled fact.

#114 said	Measured
keycloak and bulwark have two Ingresses each	keycloak does. bulwark has one.
bulwark is the second rewrite-target chart	It has none and cannot have one. NEXT_PUBLIC_BASE_PATH is compiled into its bundle, so it cannot be served under a sub-path and its route is one rule at / with no URLRewrite. Every occurrence of the string rewrite-target in that chart — values.yaml, the schema, the Ingress template — is an explanation of why it does not help. The grep that built the table counted refutations as occurrences.
matrix-synapse renders its Ingresses in a loop over several apps	That is freesailarr. It renders one Ingress with two host rules (homeserver at /, delegation nginx at /.well-known/matrix).
stalwart renders per node group	Its Services and StatefulSets do. The routing object points at exactly one group (ingress.nodeGroup) and must — the management surface is not safe to spread over a group that may not run the management tasks.

Consequences for the #119 assurance: bulwark therefore does not meet the prefix/redirect pairing — with no URLRewrite on a prefix other than /, its precondition is never met, and no redirect rule was added. freesailarr remains the only chart it applies to. The stale forward-looking claim is corrected in ci/run-httproute-match-test.sh, in README.md and in the CI-CD wiki page.

---

Per chart

keycloak — two routes, and a pre-existing defect fixed

Two HTTPRoute objects mirroring the two Ingresses. httpRoute.admin.parentRefs falls back to httpRoute.parentRefs exactly as ingress.admin.className falls back to ingress.className: "which data path" is a property of the Gateway in this API, not of the route, so an internal Gateway there is what an internal ingress class is here. Without the fallback the admin route could only ever hang off the same Gateway and the useful half of the second hostname would be gone in this mode.

The admin route stays gated on ingress.admin.enabled, because that value is not an Ingress switch — it also sets KC_HOSTNAME_ADMIN on the Deployment. Switching modes must not require turning it off.

Fixed along the way, and this is the one judgement call in the PR: the admin Ingress did not hang off ingress.enabled. Measured on origin/main:

$ helm template kc keycloak -f ci/keycloak/test-values.yaml --set ingress.enabled=false | grep -A3 'kind: Ingress' | grep name:
  name: kc-keycloak-admin-ingress

So ingress.enabled: false did not mean "no Ingress", and ingress.clusterIssuer stayed required for an object nobody asked for. It also made Gateway mode impossible: the admin Ingress would have gone on claiming the admin host next to the admin HTTPRoute — one host, two data paths, exactly what the mutual exclusion exists to prevent. Now gated on both. The default render is unchanged (ingress.enabled defaults to true); what changed is the combination ingress.enabled: false + ingress.admin.enabled: true, which could not be coherent. I kept it Minor on that basis — say so if you would rather have it Major or split out.

1.1.0+up26.8.0 → 1.2.0+up26.8.0. Loss named in its own terms: Keycloak's large headers (JWTs, SAML assertions, long Set-Cookie chains on the login flow) make a too-small proxy buffer answer 502 at the proxy while Keycloak logs success — the symptom points at the application, the cause is at the proxy.

bulwark — one route, one rule, at the root

1.0.0+up1.12.0 → 1.1.0+up1.12.0. See the table above for what this chart is not. bulwark.publicUrl now follows the active mode instead of always printing ingress.domain: the two differ exactly when httpRoute.hostnames is set, which is how a migration runs — printing the old host there would hand the operator an OIDC redirect URI for the hostname being migrated away from, a failure that surfaces as a rejected login at the identity provider and says nothing about this chart.

Corrected claim (the garage/calibre-web-automated pattern): the Ingress template, the helper and the schema all said nothing outside the Ingress template functionally reads ingress.domain. The route reads it.

matrix-synapse — two routes, and one place the Gateway API fits better

7.1.1+upv1.159.0 → 7.2.0+upv1.159.0. Two objects, because spec.hostnames applies to the whole object and these two hostnames want different paths and different backends. httpRoute.delegation.hostnames deliberately does not fall back to httpRoute.hostnames: the server name is what remote homeservers resolve, and the migration order — homeserver host first, server name last — depends on telling them apart.

Better fit, not a translation: the server name is normally an apex domain already serving a website, and ingress-nginx refuses a second Ingress on one hostname — which is why the delegation rule has to sit inside this chart's single Ingress object. HTTPRoutes on one listener and hostname are merged by the implementation, longest match winning, so that route can sit beside whatever owns the apex.

And the snippet is not load-bearing — this took checking rather than assuming. It looks exactly like what Synapse's x_forwarded depends on (client IP for rate limiting, federation). It is not: more_set_headers is the headers-more module's output-header directive — verbatim, "Replaces (if any) or adds (if not any) the specified output headers" — while the request-side directive is more_set_input_headers ("it operates on input headers (or request headers)"). Those three headers go on the response to the client; Synapse never sees them, and what it reads with x_forwarded: true is the request's X-Forwarded-For, which ingress-nginx supplies itself. ResponseHeaderModifier could not rebuild it anyway (no nginx variables) and should not: NOTES.txt says in as many words not to re-create it and not to read its absence as the cause of a rate-limiting or federation problem.

So I could not reproduce the briefing's claim that more than cosmetics hangs on it. This is a reading of the directive's documented semantics, stated as such — nothing A/B-tested a running Synapse. The real loss is proxy-body-size: 50m against a max_upload_size the chart cannot see (the config comes from 1Password).

stalwart — one route, the mail certificate stays, and the CORS answer is not "no counterpart"

1.1.1+upv0.16.25 → 1.2.0+upv0.16.25.

The only chart in the sweep that still renders a Certificate in Gateway mode. The mail certificate covers STARTTLS on 25 and implicit TLS on 465/993, which pass through neither an ingress controller nor a Gateway. So ingress.domain stays required, ingress.clusterIssuer is not freed, and the object must not be removed during a migration — deleting it drops SMTP and IMAP to a self-signed certificate while every Kubernetes object stays green. Verified in the render: Gateway mode emits Certificate + HTTPRoute and no Ingress.

CORS, and "no counterpart" would have been wrong. The allow-origin names bulwark, the webmail chart in this same group — so this is the one loss that breaks a second chart, client-side only (browser console error, nothing in the server's log). Measured by grepping the CRDs this repo pins (ci/gateway-api-crds.sh, gateway-api v1.6.3 standard channel) rather than inferred:

• HTTPRoute's filters carry a full cors field (GEP-1767) with allowOrigins/allowMethods/allowHeaders/allowCredentials/maxAge. Verbatim from that CRD: Support: Extended.
• responseHeaderModifier is in the same CRD and is also Extended, not Core.

There is a counterpart, and the chart still renders neither. Three reasons, in order:

1. No CORS values to render from. The origin and the method list are the operator's; the chart sets no CORS annotation in Ingress mode either, so a filter would exist in Gateway mode only. That asymmetry is its own defect and its own piece of work — flagged as a candidate for a separate issue rather than silently skipped; your call.
2. Extended support is not support. A rejected route is rejected whole (the reasoning #119 reached for statusCode), which here takes WebUI and JMAP down together. And the observed failure is quieter: Cilium accepts filters.type: CORS and forwards the preflight upstream anyway (cilium/cilium#45272) — Accepted: True, behaviour absent.
3. Half a CORS is worse than none, and the translatable half is the useless one here. A responseHeaderModifier can add the Access-Control-Allow-* headers; the preflight must be answered by the gateway, which the CORS filter's own contract requires and a header filter cannot do. And Authorization in the allowed headers makes every webmail request preflighted.

NOTES.txt therefore says: configure CORS at the Gateway with that implementation's own support before moving the host, and test from the webmail in a browser — curl sends no preflight, so it passes while the webmail fails.

homeassistant — the documented exception, kept

6.1.1+up2026.8.3 → 6.2.0+up2026.8.3. The exception stays; nothing unifies it. No httpRoute.hostnames key (the freesailarr shape, and the schema rejects it so the absence is enforced): hostnames and paths come from ingress.hosts, read in both modes, and the chart renders one HTTPRoute per entry with one rule per path. One object with several hostnames would be wrong, not merely different — rules apply to every hostname of the object, and per-host paths are this chart's whole point.

pathType: ImplementationSpecific has no counterpart and is this chart's default. Exact→Exact, Prefix→PathPrefix; ImplementationSpecific is rendered as PathPrefix on / only (every controller agrees there, so the default works out of the box) and aborts the render on any other path, because in ingress-nginx an ImplementationSpecific sub-path is a regex and PathPrefix is not — mapping it silently would change matching while the render stayed green, the #118 class exactly. RegularExpression deliberately not offered (Implementation-specific support level).

Route names carry the entry's index, not its hostname; reordering ingress.hosts renames routes and therefore recreates rather than edits them — harmless (a route holds no state or certificate), and written down so it is not read as a bug.

ytdlp-web-player — the last open ingress.enabled reader, and the guard mattered more

1.0.1+upv1.2.0 → 1.1.0+upv1.2.0. The briefing's prediction held: ytdlp-web-player.validate read ingress.enabled outside the Ingress template, so in Gateway mode — where ingress.enabled is false by construction — the guard went silent. Fixed with ytdlp-web-player.published = ingress.enabled OR httpRoute.enabled.

And the guard is needed more in Gateway mode, not less. It refuses to publish the app and the bundled oauth2-proxy on the same host:

• Ingress mode: ignoring it fails loudly — ingress-nginx's admission webhook rejects the second Ingress, Fleet bundle to ErrApplied.
• Gateway mode: the webhook never fires. It compares Ingress objects with Ingress objects, and an HTTPRoute is a different kind. Nothing rejects anything, both objects stay valid, and one hostname is served over two data paths that are not equivalent — the proxy's Ingress through authentication, the chart's route straight to the application. Which one a client reaches depends on what DNS resolves to. An authentication bypass that comes and goes with a DNS record, reported by nothing.

The Gateway-mode abort has its own message saying exactly that. Also named where somebody will look for it: the oauth2-proxy subchart has no Gateway support and this chart cannot give it one (upstream consumed as it comes), so in the recommended configuration the publishing object is the subchart's Ingress in both modes and setting httpRoute.enabled buys nothing. The 16k proxy-buffer-size this chart recommends is not lost by the switch — it belongs on the proxy's Ingress, which the subchart keeps rendering.

Corrected claim: gating the Ingress off used to free both domain and clusterIssuer. clusterIssuer still; domain only with both modes off.

I screened all six charts for this pattern, not only this one. The other five read ingress.enabled nowhere outside their Ingress templates and NOTES.txt.

---

Verification

• ci/helm-dependencies.sh first for stalwart (valkey) and ytdlp-web-player (oauth2-proxy), or every helm command exits 1.
• helm lint --strict on all six: 1 chart(s) linted, 0 chart(s) failed each. helm package for all six.
• Empty helm template diff against origin/main with each chart's CI values, six times — plus a seventh for homeassistant with ingress.enabled=true, since its default is false and the plain fixture renders no routing object at all.
• The trap groups 1 and 2 both hit was avoided: every explanatory comment in the new templates is inside {{/* */}} or {{- /* */}}, never #. That is what the empty diffs above confirm.
• ci/run-httproute-match-test.sh over all charts: 56 assertions held over 21 charts, 0 failed (37 over 15 before). Group 3's 19: keycloak 4, matrix-synapse 4, homeassistant 5, bulwark 2, stalwart 2, ytdlp-web-player 2. ci/run-quantity-render-test.sh (28 exercised, 0 failed) and ci/run-selector-match-test.sh (5 selectors, 0 failed) both clean.
• Schema in both directions: every overlay accepted; httpRoute.bogus rejected; httpRoute.delegation.bogus rejected; a parentRef without name rejected (missing property 'name'); and httpRoute.hostnames rejected on homeassistant, which deliberately has no such key.
• Fail paths exercised, not assumed, on all six: both modes at once (all naming the true default where they have one), empty parentRefs, Gateway mode with no hostname. Plus per chart: keycloak with ingress.admin.domain unset, and ingress.admin.enabled=false rendering one route instead of two; matrix-synapse with matrix.delegation.serverUrl unset, and delegation off rendering one route; stalwart with a nodeGroup that is not a key of nodes:; homeassistant with ImplementationSpecific on /sub (aborts), with ingress.hosts: [], with an entry lacking host, with an entry lacking paths, and with its untouched default rendering a valid route; ytdlp-web-player with the same-host collision in both modes, reached through httpRoute.hostnames as well as through the ingress.domain fallback, and with only the Deployment rendered.
• NOTES.txt reviewed by reading, in every branch of all six — Ingress, Gateway, neither, plus the proxy branch on ytdlp-web-player and the admin/delegation variants. Rendered through a scratch copy of each chart, since helm template does not emit notes. Two layout defects found and fixed that way (keycloak's Gateway block split the ENDPOINTS section; its neither-mode branch advertised an admin URL nothing published).
• stalwart with a group lacking http: the route's own guard is unreachable, because stalwart.sfs.yaml aborts first. Reported rather than claimed as exercised — the Ingress template's identical guard is equally unreachable on main.

What stays unproven, by construction

• That traffic flows. No Gateway controller runs in CI and none will, so no HTTPRoute ever reaches Accepted: True and "it routes" is not demonstrable here. What is: a schema-valid object against the real Gateway-API CRDs (install-test, paid for by cyberchef) and the render assertions above.
• NOTES.txt is read by no pipeline stage. helm template does not emit it and no script under ci/ reads it, so its text is asserted by nothing — review-only. A rendering error in it does fail helm template, which is all CI can see. Noted in a header comment in each of the six.
• Nothing was run in a cluster for this PR: no k3d, no Docker, no Trivy, no kubectl apply. The kubectl use was read-only (get ingress, get svc, get ns) as system:serviceaccount:coder-workspaces:claude-readonly.
• The CORS and body-size losses are documented, and documentation is not something a test can assert.
• How ytdlp-web-player is currently published is unknown — see above; httproutes and secrets are RBAC-forbidden here.

Also in this PR

README.md: the corrected shape table, the new assertion figures, the live-object screen, the more_set_headers finding, the CORS-filter measurement, the pathType rule, the closed ingress.enabled screen, the subchart note, and the homeassistant documented-exception paragraph extended to its Gateway mode.

Wiki, 10 pages (52588f0 → 6421d29): the six chart pages, Chart-Style-Guide (three new §4 rules plus the corrected shape table), CI-CD (56/21 and three additions to what an overlay must do), Lessons-Learned (the generalisation of #118 and its three qualifications), and Charts (one note, because the "Ingress" column now understates 21 charts).

Not touched: jk-cluster-config, rss-bridge, #12.

Commits are signed. Opened under the bot account via tea.

Part of #114

Group 3 of the #114 sweep: the six special shapes — `keycloak`, `bulwark`, `matrix-synapse`, `stalwart`, `homeassistant`, `ytdlp-web-player`. **This finishes the sweep except for `rss-bridge`**, which is frozen (#51), so **#114 stays open** after this merges. Each chart gets the form the pilot fixed and groups 1 and 2 repeated: `httpRoute` off by default, a template rendering **only** `HTTPRoute` objects with `parentRefs`, render aborts on both modes at once (naming both value paths) and on an empty `parentRefs`, empty `hostnames` meaning `[ingress.domain]`, a schema entry closed against unknown keys with an open `parentRef` definition, and a `NOTES.txt` that names the active mode and what it costs. **Minor for all six, additive, defaults unchanged** — `helm template` with each chart's CI values is byte-identical to `origin/main` for all six. One judgement call is flagged below. --- ## The central task: four of six charts leaned on an annotation the chart never sets #118 taught that a chart can read as correct while the running system fills a gap of the chart with an **operator-set** annotation — and Gateway mode drops `ingress.annotations`, so it drops the compensation with it. At #118 that was found *after* shipping. Here it was measured **beforehand**, with read-only `kubectl` against `jk-cluster`, and it was not singular: | Chart | Measured on the live Ingress, 2026-10-12 — chart sets none of it | |---|---| | `keycloak` | `proxy-buffer-size: 16k`, `proxy-buffers-number: 4` — on **both** chart-owned Ingress objects | | `matrix-synapse` | `proxy-body-size: 50m`, **plus `nginx.org/client-max-body-size: 50m`**, plus a `configuration-snippet` of three `more_set_headers` | | `stalwart` | `enable-cors: true`, `cors-allow-origin: https://webmail.bekemeyer-klein.de`, `cors-allow-methods: GET, POST, OPTIONS`, `cors-allow-headers: Authorization, Content-Type` | | `bulwark` | `proxy-body-size: 25m` | Confirmed: a `git grep` over those six chart directories for `proxy-buffer`, `app-root`, `more_set_headers`, `enable-cors`, `cors-allow`, `proxy-body-size`, `configuration-snippet` hits **documentation only** (`bulwark/values.yaml:733` recommends `25m`, `ytdlp-web-player/values.yaml:316` recommends `16k`). No template sets any of them. Per decision 1 of #114 no chart renders an implementation-specific policy object, so these are **named, not rebuilt** — in the Gateway branch of each `NOTES.txt`, in that chart's own terms. ### Where I could not reproduce the briefing, and this is the valuable half **`homeassistant` and `ytdlp-web-player`: "nothing found" is right, but not for the reason it reads like.** - `homeassistant` is **not deployed in `jk-cluster` at all** — 23 Ingress objects cluster-wide, none this chart's, no workload of it in any namespace. So it is not that the chart's host carries no annotation; there is no host. - `ytdlp-web-player` **is** deployed (namespace `jlab-player`, both Deployments Ready, `jlab-player-oauth2-proxy` and `jlab-player-ytdlp-web-player-service`) and has **no Ingress whatsoever** — neither the chart's nor the subchart's. **What publishes it could not be determined**: this workspace's identity is forbidden `httproutes` and `secrets`, so neither a hand-written `HTTPRoute` nor the release's Helm values are readable. Recorded as an open question in `values.yaml` and in the overlay rather than guessed at. Worth resolving before anybody plans a mode switch for that host. **`keycloak`'s four realm Ingresses do not share the chart's hostname.** Confirmed: four `keycloak-realm-*-ingress` objects with `app-root`, from release `jk-cluster-config-fleet-cd-keycloak-realm-ingress`, i.e. another repository; the chart's template renders exactly two. **But they share the backend *Service*, not the hostname** — they publish `bk.id.bekemeyer-klein.de`, `id.mail.bekemeyer-klein.de`, `muth-event.id.bekemeyer-klein.de` and `id.klein.pub`, four further names, two of them subdomains of the chart's public host. A wildcard listener for `*.id.bekemeyer-klein.de` would cover them. Also: three of the four carry the proxy-buffer pair, `jk-family` does not. Noted in the wiki as a migration hint. **`jk-cluster-config` is not touched.** **One extra live annotation the briefing did not name:** `matrix-synapse` also carries `nginx.org/client-max-body-size: 50m`, the NGINX-Inc-flavoured twin of `proxy-body-size` — a hedge across both nginx ingress controllers (`jellyfin` and `paperless-ngx` carry the same pair). It disappears in Gateway mode for the same reason. **And one extra sibling-repo object:** `stalwart/mta-sts-ingress`, from release `jk-cluster-config-fleet-cd-stalwart-mta-sts`, publishing `mta-sts.bekemeyer-klein.de/.well-known/mta-sts.txt` against this release's Service. Same situation as the keycloak realm objects, same treatment: not rendered here, not touched. --- ## Three of #114's shape descriptions did not survive contact with the charts This is the finding I would most want reviewed, because two of the three had already been copied into `README.md` **and** into the header comment of `ci/run-httproute-match-test.sh`, where they read as settled fact. | #114 said | Measured | |---|---| | `keycloak` and `bulwark` have **two** Ingresses each | `keycloak` does. **`bulwark` has one.** | | `bulwark` is the second `rewrite-target` chart | **It has none and cannot have one.** `NEXT_PUBLIC_BASE_PATH` is compiled into its bundle, so it cannot be served under a sub-path and its route is one rule at `/` with no `URLRewrite`. Every occurrence of the string `rewrite-target` in that chart — `values.yaml`, the schema, the Ingress template — is an explanation of *why it does not help*. The grep that built the table counted refutations as occurrences. | | `matrix-synapse` renders its Ingresses **in a loop over several apps** | That is `freesailarr`. It renders **one** Ingress with **two host rules** (homeserver at `/`, delegation nginx at `/.well-known/matrix`). | | `stalwart` renders **per node group** | Its Services and StatefulSets do. The routing object points at exactly one group (`ingress.nodeGroup`) and must — the management surface is not safe to spread over a group that may not run the management tasks. | **Consequences for the #119 assurance:** `bulwark` therefore does **not** meet the prefix/redirect pairing — with no `URLRewrite` on a prefix other than `/`, its precondition is never met, and **no redirect rule was added**. `freesailarr` remains the only chart it applies to. The stale forward-looking claim is corrected in `ci/run-httproute-match-test.sh`, in `README.md` and in the `CI-CD` wiki page. --- ## Per chart ### `keycloak` — two routes, and a pre-existing defect fixed Two `HTTPRoute` objects mirroring the two Ingresses. `httpRoute.admin.parentRefs` falls back to `httpRoute.parentRefs` exactly as `ingress.admin.className` falls back to `ingress.className`: **"which data path" is a property of the Gateway in this API, not of the route**, so an internal Gateway there is what an internal ingress class is here. Without the fallback the admin route could only ever hang off the same Gateway and the useful half of the second hostname would be gone in this mode. The admin route stays gated on `ingress.admin.enabled`, because that value is **not an Ingress switch** — it also sets `KC_HOSTNAME_ADMIN` on the Deployment. Switching modes must not require turning it off. **Fixed along the way, and this is the one judgement call in the PR: the admin Ingress did not hang off `ingress.enabled`.** Measured on `origin/main`: ``` $ helm template kc keycloak -f ci/keycloak/test-values.yaml --set ingress.enabled=false | grep -A3 'kind: Ingress' | grep name: name: kc-keycloak-admin-ingress ``` So `ingress.enabled: false` did not mean "no Ingress", and `ingress.clusterIssuer` stayed required for an object nobody asked for. It also made Gateway mode impossible: the admin Ingress would have gone on claiming the admin host next to the admin `HTTPRoute` — one host, two data paths, exactly what the mutual exclusion exists to prevent. Now gated on both. **The default render is unchanged** (`ingress.enabled` defaults to `true`); what changed is the combination `ingress.enabled: false` + `ingress.admin.enabled: true`, which could not be coherent. I kept it **Minor** on that basis — say so if you would rather have it Major or split out. `1.1.0+up26.8.0` → **`1.2.0+up26.8.0`**. Loss named in its own terms: Keycloak's large headers (JWTs, SAML assertions, long `Set-Cookie` chains on the login flow) make a too-small proxy buffer answer **502 at the proxy while Keycloak logs success** — the symptom points at the application, the cause is at the proxy. ### `bulwark` — one route, one rule, at the root `1.0.0+up1.12.0` → **`1.1.0+up1.12.0`**. See the table above for what this chart is not. `bulwark.publicUrl` now follows the **active** mode instead of always printing `ingress.domain`: the two differ exactly when `httpRoute.hostnames` is set, which is how a migration runs — printing the old host there would hand the operator an OIDC redirect URI for the hostname being migrated *away from*, a failure that surfaces as a rejected login at the identity provider and says nothing about this chart. **Corrected claim** (the `garage`/`calibre-web-automated` pattern): the Ingress template, the helper and the schema all said nothing outside the Ingress template functionally reads `ingress.domain`. The route reads it. ### `matrix-synapse` — two routes, and one place the Gateway API fits better `7.1.1+upv1.159.0` → **`7.2.0+upv1.159.0`**. Two objects, because `spec.hostnames` applies to the whole object and these two hostnames want different paths *and* different backends. `httpRoute.delegation.hostnames` deliberately does **not** fall back to `httpRoute.hostnames`: the server name is what *remote* homeservers resolve, and the migration order — homeserver host first, server name last — depends on telling them apart. **Better fit, not a translation:** the server name is normally an apex domain already serving a website, and ingress-nginx refuses a second Ingress on one hostname — which is why the delegation rule has to sit *inside* this chart's single Ingress object. HTTPRoutes on one listener and hostname are **merged** by the implementation, longest match winning, so that route can sit beside whatever owns the apex. **And the snippet is not load-bearing — this took checking rather than assuming.** It looks exactly like what Synapse's `x_forwarded` depends on (client IP for rate limiting, federation). It is not: `more_set_headers` is the headers-more module's **output**-header directive — verbatim, *"Replaces (if any) or adds (if not any) the specified output headers"* — while the request-side directive is `more_set_input_headers` (*"it operates on input headers (or request headers)"*). Those three headers go on the **response to the client**; Synapse never sees them, and what it reads with `x_forwarded: true` is the *request's* `X-Forwarded-For`, which ingress-nginx supplies itself. `ResponseHeaderModifier` could not rebuild it anyway (no nginx variables) and **should not**: `NOTES.txt` says in as many words not to re-create it and not to read its absence as the cause of a rate-limiting or federation problem. So **I could not reproduce the briefing's claim that more than cosmetics hangs on it.** This is a reading of the directive's documented semantics, stated as such — nothing A/B-tested a running Synapse. The real loss is `proxy-body-size: 50m` against a `max_upload_size` the chart cannot see (the config comes from 1Password). ### `stalwart` — one route, the mail certificate stays, and the CORS answer is not "no counterpart" `1.1.1+upv0.16.25` → **`1.2.0+upv0.16.25`**. **The only chart in the sweep that still renders a `Certificate` in Gateway mode.** The mail certificate covers STARTTLS on 25 and implicit TLS on 465/993, which pass through neither an ingress controller nor a Gateway. So `ingress.domain` stays required, `ingress.clusterIssuer` is not freed, and the object **must not be removed** during a migration — deleting it drops SMTP and IMAP to a self-signed certificate while every Kubernetes object stays green. Verified in the render: Gateway mode emits `Certificate` + `HTTPRoute` and no `Ingress`. **CORS, and "no counterpart" would have been wrong.** The allow-origin names **`bulwark`, the webmail chart in this same group** — so this is the one loss that breaks a second chart, client-side only (browser console error, nothing in the server's log). Measured by grepping the CRDs this repo pins (`ci/gateway-api-crds.sh`, gateway-api **v1.6.3** standard channel) rather than inferred: - HTTPRoute's filters carry a full **`cors`** field (GEP-1767) with `allowOrigins`/`allowMethods`/`allowHeaders`/`allowCredentials`/`maxAge`. Verbatim from that CRD: **`Support: Extended`**. - `responseHeaderModifier` is in the same CRD and is **also Extended, not Core**. There *is* a counterpart, and the chart still renders neither. Three reasons, in order: 1. **No CORS values to render from.** The origin and the method list are the operator's; the chart sets no CORS annotation in Ingress mode either, so a filter would exist in Gateway mode only. That asymmetry is its own defect and its own piece of work — **flagged as a candidate for a separate issue rather than silently skipped**; your call. 2. **Extended support is not support.** A rejected route is rejected **whole** (the reasoning #119 reached for `statusCode`), which here takes WebUI and JMAP down together. And the observed failure is quieter: Cilium accepts `filters.type: CORS` and forwards the preflight upstream anyway (`cilium/cilium#45272`) — `Accepted: True`, behaviour absent. 3. **Half a CORS is worse than none, and the translatable half is the useless one here.** A `responseHeaderModifier` can add the `Access-Control-Allow-*` headers; the **preflight** must be answered by the *gateway*, which the CORS filter's own contract requires and a header filter cannot do. And `Authorization` in the allowed headers makes **every** webmail request preflighted. `NOTES.txt` therefore says: configure CORS at the Gateway with that implementation's own support **before** moving the host, and **test from the webmail in a browser — `curl` sends no preflight**, so it passes while the webmail fails. ### `homeassistant` — the documented exception, kept `6.1.1+up2026.8.3` → **`6.2.0+up2026.8.3`**. The exception stays; nothing unifies it. **No `httpRoute.hostnames` key** (the `freesailarr` shape, and the schema rejects it so the absence is enforced): hostnames and paths come from `ingress.hosts`, read in **both** modes, and the chart renders **one `HTTPRoute` per entry** with one rule per path. One object with several hostnames would be *wrong*, not merely different — rules apply to every hostname of the object, and per-host paths are this chart's whole point. **`pathType: ImplementationSpecific` has no counterpart and is this chart's default.** `Exact`→`Exact`, `Prefix`→`PathPrefix`; `ImplementationSpecific` is rendered as `PathPrefix` **on `/` only** (every controller agrees there, so the default works out of the box) and **aborts the render on any other path**, because in ingress-nginx an `ImplementationSpecific` sub-path is a **regex** and `PathPrefix` is not — mapping it silently would change matching while the render stayed green, the #118 class exactly. `RegularExpression` deliberately not offered (Implementation-specific support level). Route names carry the entry's **index**, not its hostname; **reordering `ingress.hosts` renames routes** and therefore recreates rather than edits them — harmless (a route holds no state or certificate), and written down so it is not read as a bug. ### `ytdlp-web-player` — the last open `ingress.enabled` reader, and the guard mattered *more* `1.0.1+upv1.2.0` → **`1.1.0+upv1.2.0`**. The briefing's prediction held: `ytdlp-web-player.validate` read `ingress.enabled` outside the Ingress template, so in Gateway mode — where `ingress.enabled` is `false` by construction — the guard went silent. Fixed with `ytdlp-web-player.published` = `ingress.enabled OR httpRoute.enabled`. **And the guard is needed more in Gateway mode, not less.** It refuses to publish the app and the bundled oauth2-proxy on the same host: - **Ingress mode:** ignoring it fails loudly — ingress-nginx's admission webhook rejects the second Ingress, Fleet bundle to `ErrApplied`. - **Gateway mode: the webhook never fires.** It compares Ingress objects with Ingress objects, and an `HTTPRoute` is a different **kind**. Nothing rejects anything, both objects stay valid, and one hostname is served over two data paths that are **not equivalent** — the proxy's Ingress through **authentication**, the chart's route **straight to the application**. Which one a client reaches depends on what DNS resolves to. **An authentication bypass that comes and goes with a DNS record, reported by nothing.** The Gateway-mode abort has its own message saying exactly that. Also named where somebody will look for it: **the oauth2-proxy subchart has no Gateway support** and this chart cannot give it one (upstream consumed as it comes), so in the *recommended* configuration the publishing object is the **subchart's Ingress in both modes** and setting `httpRoute.enabled` buys nothing. The 16k `proxy-buffer-size` this chart recommends is **not** lost by the switch — it belongs on the proxy's Ingress, which the subchart keeps rendering. **Corrected claim:** gating the Ingress off used to free *both* `domain` and `clusterIssuer`. `clusterIssuer` still; `domain` only with both modes off. **I screened all six charts for this pattern, not only this one.** The other five read `ingress.enabled` nowhere outside their Ingress templates and `NOTES.txt`. --- ## Verification - `ci/helm-dependencies.sh` first for `stalwart` (valkey) and `ytdlp-web-player` (oauth2-proxy), or every helm command exits 1. - `helm lint --strict` on all six: `1 chart(s) linted, 0 chart(s) failed` each. `helm package` for all six. - **Empty `helm template` diff against `origin/main` with each chart's CI values, six times** — plus a seventh for `homeassistant` with `ingress.enabled=true`, since its default is `false` and the plain fixture renders no routing object at all. - **The trap groups 1 and 2 both hit was avoided:** every explanatory comment in the new templates is inside `{{/* */}}` or `{{- /* */}}`, never `#`. That is what the empty diffs above confirm. - `ci/run-httproute-match-test.sh` over all charts: **56 assertions held over 21 charts, 0 failed** (37 over 15 before). Group 3's 19: `keycloak` 4, `matrix-synapse` 4, `homeassistant` 5, `bulwark` 2, `stalwart` 2, `ytdlp-web-player` 2. `ci/run-quantity-render-test.sh` (28 exercised, 0 failed) and `ci/run-selector-match-test.sh` (5 selectors, 0 failed) both clean. - Schema in both directions: every overlay accepted; `httpRoute.bogus` rejected; `httpRoute.delegation.bogus` rejected; a `parentRef` without `name` rejected (`missing property 'name'`); and `httpRoute.hostnames` rejected on `homeassistant`, which deliberately has no such key. - **Fail paths exercised, not assumed**, on all six: both modes at once (all naming the `true` default where they have one), empty `parentRefs`, Gateway mode with no hostname. Plus per chart: `keycloak` with `ingress.admin.domain` unset, and `ingress.admin.enabled=false` rendering one route instead of two; `matrix-synapse` with `matrix.delegation.serverUrl` unset, and delegation off rendering one route; `stalwart` with a `nodeGroup` that is not a key of `nodes:`; `homeassistant` with `ImplementationSpecific` on `/sub` (aborts), with `ingress.hosts: []`, with an entry lacking `host`, with an entry lacking `paths`, and with its untouched default rendering a valid route; `ytdlp-web-player` with the same-host collision in **both** modes, reached through `httpRoute.hostnames` as well as through the `ingress.domain` fallback, and with only the Deployment rendered. - **`NOTES.txt` reviewed by reading, in every branch of all six** — Ingress, Gateway, neither, plus the proxy branch on `ytdlp-web-player` and the admin/delegation variants. Rendered through a scratch copy of each chart, since `helm template` does not emit notes. Two layout defects found and fixed that way (`keycloak`'s Gateway block split the ENDPOINTS section; its neither-mode branch advertised an admin URL nothing published). - `stalwart` with a group lacking `http`: **the route's own guard is unreachable**, because `stalwart.sfs.yaml` aborts first. Reported rather than claimed as exercised — the Ingress template's identical guard is equally unreachable on `main`. ## What stays unproven, by construction - **That traffic flows.** No Gateway controller runs in CI and none will, so **no `HTTPRoute` ever reaches `Accepted: True`** and "it routes" is not demonstrable here. What is: a schema-valid object against the real Gateway-API CRDs (install-test, paid for by `cyberchef`) and the render assertions above. - **`NOTES.txt` is read by no pipeline stage.** `helm template` does not emit it and no script under `ci/` reads it, so its text is asserted by nothing — review-only. A rendering *error* in it does fail `helm template`, which is all CI can see. Noted in a header comment in each of the six. - **Nothing was run in a cluster for this PR**: no k3d, no Docker, no Trivy, no `kubectl apply`. The `kubectl` use was read-only (`get ingress`, `get svc`, `get ns`) as `system:serviceaccount:coder-workspaces:claude-readonly`. - **The CORS and body-size losses are documented, and documentation is not something a test can assert.** - **How `ytdlp-web-player` is currently published is unknown** — see above; `httproutes` and `secrets` are RBAC-forbidden here. ## Also in this PR `README.md`: the corrected shape table, the new assertion figures, the live-object screen, the `more_set_headers` finding, the CORS-filter measurement, the `pathType` rule, the closed `ingress.enabled` screen, the subchart note, and the `homeassistant` documented-exception paragraph extended to its Gateway mode. **Wiki, 10 pages** (`52588f0` → `6421d29`): the six chart pages, `Chart-Style-Guide` (three new §4 rules plus the corrected shape table), `CI-CD` (56/21 and three additions to what an overlay must do), `Lessons-Learned` (the generalisation of #118 and its three qualifications), and `Charts` (one note, because the "Ingress" column now understates 21 charts). **Not touched:** `jk-cluster-config`, `rss-bridge`, `#12`. Commits are signed. Opened under the bot account via `tea`. Part of #114

jlab-bot added 7 commits 2026-10-12 23:48:50 +02:00

feat(keycloak): Gateway-API as an alternative to the two Ingresses ... f97e1e6d52

Group 3 of the #114 sweep, first chart. keycloak has TWO Ingress objects -
the public hostname and the admin hostname - so it gets TWO HTTPRoutes, with
httpRoute.admin.parentRefs falling back to httpRoute.parentRefs exactly as
ingress.admin.className falls back to ingress.className: "which data path" is
a property of the Gateway in this API, not of the route, so an internal
Gateway here is what an internal ingress class is in the other mode.

The admin route stays gated on ingress.admin.enabled, because that value is
not an Ingress switch - it also decides whether KC_HOSTNAME_ADMIN is set on
the Deployment, i.e. whether the admin hostname exists at all. Switching to
Gateway mode therefore must not require turning it off.

FIXED ALONG THE WAY: the admin Ingress did not hang off ingress.enabled.
Measured on origin/main, `helm template -f ci/keycloak/test-values.yaml --set
ingress.enabled=false` emitted kc-keycloak-admin-ingress - so
"ingress.enabled: false" did not mean "no Ingress", and ingress.clusterIssuer
stayed required for an object nobody asked for. It also made Gateway mode
impossible: the admin Ingress would have gone on claiming the admin host next
to the admin HTTPRoute, one host over two data paths, which is the exact
conflict the mutual exclusion exists to prevent. The default render is
unchanged; what changed is a combination that could not be coherent.

WHAT THIS MODE DROPS, MEASURED ON THE LIVE CLUSTER rather than inferred. On
2026-10-12 BOTH of this chart's Ingress objects in jk-cluster carried

    nginx.ingress.kubernetes.io/proxy-buffer-size:    16k
    nginx.ingress.kubernetes.io/proxy-buffers-number: 4

set by the OPERATOR through ingress.annotations and ingress.admin.annotations
- the chart sets neither, which is why reading the chart alone does not reveal
them. The Gateway API has no annotation channel and no counterpart for proxy
buffer sizing; it is implementation policy, not routing. Per decision 1 of
#114 the chart renders no policy object, so this is NAMED and not rebuilt, in
values.yaml, in the schema and in the Gateway branch of NOTES.txt, with the
consequence spelled out in Keycloak's own terms: large headers (JWTs, SAML
assertions, long Set-Cookie chains on the login flow) make a too-small buffer
answer 502 at the PROXY while Keycloak logs the request as successful.

Named and deliberately not touched: four further Ingress objects in the same
namespace (keycloak-realm-*-ingress, with app-root) belong to the Helm release
jk-cluster-config-fleet-cd-keycloak-realm-ingress, i.e. to another repository.
They point at this release's Service under four further hostnames. This chart
renders exactly the two routes its own two Ingresses cover.

Minor: additive, defaults unchanged. Empty helm template diff against
origin/main with ci/keycloak/test-values.yaml.

Part of #114

feat(bulwark): Gateway-API as an alternative to the Ingress ... f6eeff87c2

Group 3 of the #114 sweep. One route, one rule, at the root - and THREE
EXPECTATIONS ABOUT THIS CHART DID NOT SURVIVE CONTACT WITH IT:

  - #114 says bulwark has TWO Ingress objects. It has ONE. keycloak has two.
  - #114's measured table lists bulwark under rewrite-target. It has no
    rewrite-target annotation and CANNOT have one: NEXT_PUBLIC_BASE_PATH is
    compiled into the bundle, so the application cannot be served under a
    sub-path, and stripping a prefix at the proxy would yield a page that
    loads and then fetches everything from the wrong place. Every occurrence
    of the string "rewrite-target" in this chart - values.yaml, the schema,
    the Ingress template - is an explanation of exactly that. A grep for the
    string finds the refutations and counts them as occurrences.
  - ci/run-httproute-match-test.sh said in its header that bulwark was the
    second rewrite-target chart and would meet the #118 prefix/redirect
    assurance when group 3 landed. Group 3 landed and it does not: with no
    URLRewrite on a prefix other than "/", the assurance's precondition is
    never met here. The header is corrected rather than left standing, and it
    now says freesailarr is the only chart the pairing applies to.

So no redirect rule is added, and none is needed.

WHAT THIS MODE DROPS, MEASURED ON THE LIVE OBJECT rather than inferred. On
2026-10-12 the live Ingress on webmail.bekemeyer-klein.de carried

    nginx.ingress.kubernetes.io/proxy-body-size: 25m

supplied by the OPERATOR through ingress.annotations - values.yaml recommends
exactly that value and explains why the chart cannot pick it, so the chart
sets nothing and a reader of this repository alone would not find it. It is
the single annotation this installation depends on, and the Gateway API has no
body-size counterpart: implementation policy, not routing. Per decisions 1 and
2 of #114 it is NAMED, not rebuilt - in values.yaml, in the schema and in the
Gateway branch of NOTES.txt, in this application's own terms: an attachment
above the Gateway's default is rejected before Bulwark sees a byte, the 413
comes from the proxy, and it arrives after the user has written the mail and
attached the file.

CORRECTED CLAIM, the same one garage and calibre-web-automated needed: the
Ingress template, the bulwark.publicUrl helper and the schema all stated that
nothing outside the Ingress template FUNCTIONALLY reads ingress.domain. The
HTTPRoute reads it, so the claim held only while both modes were off. All
three now say so, and the route carries a required() of its own.

bulwark.publicUrl follows the ACTIVE mode instead of always printing
ingress.domain. The two differ exactly when httpRoute.hostnames is set, which
is how a migration off the Ingress runs - and printing the old host there
would hand the operator an OIDC redirect URI for the hostname being migrated
away from, a failure that surfaces as a rejected login at the identity
provider and says nothing about this chart.

Minor: additive, defaults unchanged. Empty helm template diff against
origin/main with ci/bulwark/test-values.yaml.

Part of #114

feat(matrix-synapse): Gateway-API as an alternative to the Ingress ... 470b90a393

Group 3 of the #114 sweep. TWO HTTPRoutes - and the shape is not the one the
issue described.

#114 says this chart renders its Ingresses in a LOOP OVER SEVERAL APPS. It
does not; that is freesailarr. It renders ONE Ingress object with TWO HOST
rules: the homeserver on ingress.domain at "/", and the delegation nginx on
matrix.delegation.serverUrl at "/.well-known/matrix". An HTTPRoute's
spec.hostnames applies to the WHOLE object, so two hostnames that want
different paths and different backends have to be two objects. That is the
only faithful translation, not a stylistic choice.

httpRoute.delegation.hostnames deliberately does NOT fall back to
httpRoute.hostnames. The server name is what REMOTE homeservers resolve and is
a different thing from the homeserver's own hostname; the migration order -
homeserver host first, server name last - depends on telling them apart, and a
chart-global hostname list covering both would erase the distinction.
httpRoute.delegation.parentRefs falls back to httpRoute.parentRefs, which
matters because the server name is usually an apex domain behind a different
Gateway.

ONE THING THIS MODE IMPROVES, and it is the awkward part of Matrix delegation.
Under ingress-nginx two Ingress objects cannot both claim one hostname - the
admission webhook rejects the second - which is why the delegation rule has to
live inside this chart's single Ingress object even though the apex domain
usually belongs to a website. HTTPRoutes attached to the same listener for the
same hostname are merged by the implementation, longest match winning, so the
/.well-known/matrix route can sit beside whatever owns that apex.

WHAT THIS MODE DROPS, MEASURED ON THE LIVE OBJECT ON 2026-10-12 - and ONE OF
THE TWO IS NOT WHAT IT LOOKS LIKE. The live Ingress carried, all of it
operator-supplied through ingress.annotations (this chart sets none of it):

    nginx.ingress.kubernetes.io/proxy-body-size:    50m
    nginx.org/client-max-body-size:                 50m
    nginx.ingress.kubernetes.io/configuration-snippet: |
      more_set_headers "X-Forwarded-For $remote_addr";
      more_set_headers "X-Forwarded-Proto $scheme";
      more_set_headers "Host $host:$server_port";

1. THE BODY SIZE IS THE REAL LOSS. Synapse's media repository takes uploads;
   its own ceiling is max_upload_size in homeserver.yaml, which this chart
   cannot see because the whole config arrives from 1Password. The effective
   limit is the smaller of the two, and in this mode the proxy half becomes
   the GATEWAY'S default - neither number anyone can point at is necessarily
   the one that decides. No counterpart in the Gateway API, so per decision 2
   of #114 it is documented, not rebuilt. nginx.org/client-max-body-size is
   the same limit spelled for NGINX Inc's controller, a hedge across both
   implementations, and it goes for the same reason. (#114's own measured
   table did not list matrix-synapse under proxy-body-size at all; the live
   object has it.)

2. THE SNIPPET IS NOT LOAD-BEARING, which took checking rather than assuming.
   It looks like what Synapse's x_forwarded setting depends on - client IP for
   rate limiting, and federation. It is not: more_set_headers is the
   headers-more module's OUTPUT-header directive ("Replaces (if any) or adds
   (if not any) the specified output headers"), the request-side directive
   being more_set_input_headers. So those three headers are added to the
   RESPONSE TO THE CLIENT; Synapse never sees them, and what it reads with
   x_forwarded: true is the REQUEST's X-Forwarded-For, which ingress-nginx
   supplies itself. The Gateway API's ResponseHeaderModifier could not rebuild
   it anyway (it knows no nginx variables), and should not: NOTES.txt says in
   as many words not to re-create it and not to read its absence as the cause
   of a rate-limiting or federation problem.

   Stated as what it is - a reading of the directive's documented semantics.
   Nothing here observed a Synapse instance with and without the snippet.

NOTES.txt now asks $published (ingress.enabled OR httpRoute.enabled) wherever
it used to ask ingress.enabled - the server-name/serverUrl mismatch warning,
the "clients work fine" aside and the scaled-to-zero paragraph were all really
asking whether anything publishes the release.

Minor: additive, defaults unchanged. Empty helm template diff against
origin/main with ci/matrix-synapse/test-values.yaml.

Part of #114

feat(stalwart): Gateway-API as an alternative to the Ingress ... cffc50e4ea

Group 3 of the #114 sweep. ONE route - not one per node group. #114 describes
this chart as rendering "per node group"; its SERVICES and StatefulSets are,
the routing object is not and must not be. The route targets exactly
ingress.nodeGroup's own http Service and keeps both of the Ingress's guards,
because the WebUI's setup and management screens are not safe to spread over a
group that may not run the management tasks. ingress.nodeGroup is therefore
shared by both modes: it names where the management surface is published from,
which is not a property of the Ingress.

THE MAIL CERTIFICATE STAYS, and this is the only chart in the sweep where
anything of the TLS path does. stalwart.certificate.yaml covers STARTTLS on 25
and implicit TLS on 465/993; those connections pass through neither an ingress
controller nor a Gateway. So in Gateway mode ingress.domain is still required
(the certificate's dnsNames read it), ingress.clusterIssuer is still required
unless certificate.clusterIssuer is set, and the Certificate must NOT be
removed as part of a migration - deleting it drops SMTP and IMAP back to
Stalwart's self-signed certificate, with clients seeing a name mismatch while
every Kubernetes object stays green. Verified in the render: Gateway mode emits
Certificate + HTTPRoute and no Ingress.

WHAT THIS MODE DROPS - AND HERE IT BREAKS A SECOND CHART OF THE SAME GROUP.
Measured on 2026-10-12, the live Ingress on mail.bekemeyer-klein.de carried,
all of it operator-supplied through ingress.annotations (this chart sets none
of it):

    enable-cors: true
    cors-allow-origin:  https://webmail.bekemeyer-klein.de
    cors-allow-methods: GET, POST, OPTIONS
    cors-allow-headers: Authorization, Content-Type

That origin is BULWARK, the webmail chart in this same group. Without the CORS
configuration the browser-side webmail cannot call this server's JMAP endpoint
at all, and the failure is entirely client-side: a console error, nothing in
the server's logs.

AND "no counterpart" WOULD HAVE BEEN WRONG. Measured by grepping the CRDs this
repo pins (gateway-api v1.6.3 standard channel, ci/gateway-api-crds.sh) rather
than inferred: HTTPRoute's filters carry a full 'cors' field (GEP-1767) with
allowOrigins/allowMethods/allowHeaders/allowCredentials/maxAge, and its
documented level - verbatim from that CRD - is "Support: Extended".
responseHeaderModifier is in the same CRD and is also Extended, not Core.

The chart still does not render it, for three reasons in order:

  1. The chart has NO CORS values. The live configuration is the operator's,
     and the chart sets no CORS annotations in Ingress mode either - so
     rendering a filter means inventing a values surface that would exist in
     Gateway mode only. That asymmetry is its own defect and its own work.
  2. Extended support is not support. A rejected route is rejected WHOLE,
     which would take the WebUI and JMAP down together (the reasoning #119
     reached for statusCode); and the observed failure is quieter - Cilium
     accepts filters.type: CORS and forwards the preflight upstream anyway
     (cilium/cilium#45272), i.e. Accepted: True with the behaviour absent.
  3. Half a CORS is worse than none, and the translatable half is the useless
     one HERE. A responseHeaderModifier can add the Access-Control-Allow-*
     headers; the PREFLIGHT has to be answered BY THE GATEWAY, which the CORS
     filter's own contract states and a bare header filter cannot do. Because
     the allowed headers include Authorization, EVERY webmail request is
     preflighted - so the expressible half is exactly the half that does not
     help.

Named, not rebuilt (decision 2 of #114), in values.yaml, in the schema and in
the Gateway branch of NOTES.txt, with the operational instruction: configure
CORS at the Gateway with that implementation's own support BEFORE moving this
host, and test it from the webmail in a browser - curl sends no preflight, so
it passes while the webmail fails.

NOTES.txt now asks $published (ingress.enabled OR httpRoute.enabled) where it
asked ingress.enabled, and two paragraphs that named the ingress controller
specifically now name the Gateway in that mode: the auto-ban step (switching
API does not avoid or reset it - one proxy address instead of another) and the
NetworkPolicy paragraph, where the namespace to allow is the Gateway's data
plane rather than an ingress controller's.

Minor: additive, defaults unchanged. Empty helm template diff against
origin/main with ci/stalwart/test-values.yaml.

Part of #114

feat(homeassistant): Gateway-API as an alternative to the Ingress ... d860704172

Group 3 of the #114 sweep, and the one chart where the convention had to bend
rather than the chart. homeassistant is the DOCUMENTED INGRESS EXCEPTION
(README): upstream's flexible values, a list of hosts each with its own list
of paths, instead of the repo's required domain/clusterIssuer pair. The
exception stays - nothing here unifies it.

NO httpRoute.hostnames KEY, which is the freesailarr shape from group 2 and
for the same reason: there is no ingress.domain to fall back to, and a
chart-global hostname list could not say which host entry - and therefore
which set of paths - it belonged to. So httpRoute carries only enabled and
parentRefs, the hostnames come from ingress.hosts, and the chart renders ONE
HTTPRoute PER ENTRY with one rule per path of that entry. One object with
several hostnames would be wrong, not merely different: an HTTPRoute's rules
apply to every hostname of the object, and per-host paths are this chart's
whole point. The schema rejects httpRoute.hostnames outright, so the absence
is enforced rather than merely intended.

ingress.hosts is therefore read in BOTH modes and is not an Ingress-only
value. ingress.className, ingress.tls and ingress.annotations are not read in
Gateway mode: class and certificate belong to the Gateway's listener, and
there is no annotation channel.

pathType IS THE SHARP EDGE, AND THE CHART'S DEFAULT IS THE AWKWARD VALUE.
Ingress has Exact/Prefix/ImplementationSpecific; the Gateway API has
Exact/PathPrefix/RegularExpression. Exact and Prefix map cleanly.
ImplementationSpecific - the default in values.yaml - means "ask the
controller" and has NO counterpart, so:

  - on "/" it renders as PathPrefix. Every controller treats a root
    ImplementationSpecific path as a prefix match, so the chart's own default
    works in this mode unchanged (verified: the untouched default renders a
    valid route).
  - on ANY OTHER path the render ABORTS, naming the value and asking for
    Prefix or Exact. In ingress-nginx - this chart's default class - an
    ImplementationSpecific sub-path is a REGEX, and PathPrefix is not one.
    Mapping it silently would change how the path matches while the render
    stayed green, which is exactly the class of defect #118 was.

RegularExpression is deliberately not offered as an escape hatch: its support
level is Implementation-specific, which decision 1 of #114 forbids.

NOTHING WAS MEASURED ON THE LIVE CLUSTER FOR THIS CHART, because there is
nothing to measure: homeassistant is NOT DEPLOYED in jk-cluster (checked
2026-10-12 - 23 Ingress objects cluster-wide, none this chart's, and no
workload of it in any namespace). So it is the one chart of the six with no
operator-supplied annotation to lose. values.yaml and NOTES.txt say that as
an absence of evidence about this installation rather than as a property of
the chart.

Route NAMES carry the entry's INDEX, not its hostname - a hostname is not a
dependable object-name fragment. Consequence named in values.yaml and in the
template: reordering ingress.hosts renames routes and therefore recreates
rather than edits them. Harmless, since a route holds no state and no
certificate.

ci/httproute/homeassistant.yaml has to SUPPLY HOSTS, which no other overlay in
the sweep does, and it adds coverage the install-test never had: this chart's
ingress.enabled defaults to FALSE and the fixture does not switch it on, so
no routing object was exercised at all before. Two entries on purpose - a
plain one, and one with two paths including an Exact - because that second
entry is what proves the chart renders one object per host rather than one
object with several hostnames. 5 assertions over 2 routes and 3 rules.

NOTES.txt asks $published where it asked ingress.enabled, and the
trusted_proxies section now warns that moving from an Ingress to a Gateway
CHANGES the proxy's pod network: a list carried over unchanged turns every
request into a 400 if the CIDR differs, or silently keeps attributing every
client to one address if it does not.

Minor: additive, defaults unchanged. Empty helm template diff against
origin/main, both with the fixture as it stands and with ingress.enabled=true.

Part of #114

feat(ytdlp-web-player): Gateway-API as an alternative to the Ingress ... f647597a69

Group 3 of the #114 sweep, last chart - and the one that carried the group-2
pattern this sweep was told to look for. It reads ingress.enabled OUTSIDE its
Ingress template, in ytdlp-web-player.validate, and in Gateway mode
ingress.enabled is false by construction, so that guard would have gone
silent. Fixed with ytdlp-web-player.published = ingress.enabled OR
httpRoute.enabled - the freesailarr.published / sure.publishedOverTls shape,
identical to ingress.enabled while httpRoute.enabled is false, which is why it
changes no existing render.

AND THE GUARD IS NEEDED MORE IN GATEWAY MODE, NOT LESS. It refuses to publish
the application and the oauth2-proxy on the same host:

  - In INGRESS mode the consequence of ignoring it is loud: ingress-nginx's
    admission webhook rejects the second Ingress and the Fleet bundle goes to
    ErrApplied.
  - In GATEWAY mode THE WEBHOOK NEVER FIRES. It compares Ingress objects with
    Ingress objects, and an HTTPRoute is a different kind - so nothing rejects
    anything, both objects stay valid, and one hostname is served over two
    data paths that are NOT equivalent: the proxy's Ingress goes through
    AUTHENTICATION and this chart's HTTPRoute goes straight to the
    application. Which one a client reaches depends on what DNS resolves to.
    That is an authentication bypass that comes and goes with a DNS record and
    that no Kubernetes object reports.

So the Gateway-mode abort has its own message saying exactly that. Both
messages exercised, plus the collision reached through httpRoute.hostnames
rather than through the ingress.domain fallback, and the guard still firing
when only the Deployment is rendered.

THE SUBCHART HAS NO GATEWAY SUPPORT, and this chart cannot give it one -
upstream's chart is consumed as it comes (README, "Chart dependencies"). It
renders an Ingress or nothing. So in the RECOMMENDED configuration, where the
proxy sits in the data path and owns the host, the publishing object is the
SUBCHART'S INGRESS in both of this chart's modes, and turning httpRoute on
buys nothing. The route is for the proxy-disabled case, or for the application
on a host of its own - which is the documented unauthenticated bypass. Said in
the template header, in values.yaml, in the schema and in the proxy branch of
NOTES.txt, because someone migrating hosts will otherwise set the value and
wonder why nothing moved.

The 16k proxy-buffer-size this chart recommends is NOT lost by this switch,
and that needed stating rather than assuming: it belongs on the PROXY's
Ingress (oauth2-proxy.ingress.annotations) as the second half of the 4kB
session-cookie trap, and the subchart keeps rendering an Ingress. This chart's
own ingress.annotations default is {} and it sets nothing but the cert-manager
annotation, so there is no chart-side annotation loss to name.

CORRECTED CLAIM: values.yaml and the schema both said that everything the
Ingress needs is read only inside its template, so switching it off freed BOTH
ingress.domain and ingress.clusterIssuer. clusterIssuer is still freed; domain
is now read by the HTTPRoute, so it is freed only with both modes off. The
reason the claim was made still holds and is kept - the application has no
base-URL setting and builds no absolute links.

NOTHING WAS MEASURED ON A LIVE INGRESS OF THIS CHART, because there is none.
Measured 2026-10-12: the release runs in namespace jlab-player with both
Deployments Ready, and NOT ONE of the cluster's 23 Ingress objects is in that
namespace - neither this chart's nor the subchart's. What publishes it instead
could not be determined: this workspace's identity may not list httproutes or
secrets, so neither a hand-written HTTPRoute nor the release's Helm values are
readable. Recorded as the open question it is, in values.yaml and in the
overlay, rather than guessed at.

Minor: additive, defaults unchanged. Empty helm template diff against
origin/main with ci/ytdlp-web-player/test-values.yaml.

Part of #114

docs(README): group 3 of the Gateway-API sweep, and four corrections it forced ... 23ff5d5b08

The figures: the HTTPRoute render test goes from 37 assertions over 15 charts
to **56 over 21**, with the per-chart breakdown, and the sweep is finished
except for the frozen rss-bridge - so #114 still cannot be closed while #51 is
open.

FOUR CORRECTIONS, because three of #114's descriptions of the group-3 shapes
and one of this file's own forward-looking claims did not survive contact with
the charts. The corrected list is a table in section 4, since that is what
somebody will copy from:

  - bulwark has ONE Ingress, not two. keycloak has two.
  - bulwark has NO rewrite-target and cannot have one: NEXT_PUBLIC_BASE_PATH is
    compiled into its bundle. #114's table reached "bulwark, freesailarr" by
    grepping for the string, and every occurrence in that chart is an
    explanation of why the annotation does not help. freesailarr is therefore
    still the only chart the #118 prefix/redirect pairing applies to - the
    HTTPRoute-test section said bulwark would meet it "when group 3 lands",
    and now says why it does not.
  - matrix-synapse does NOT render in a loop over several apps (that is
    freesailarr): one Ingress object with two HOST rules, which has to become
    two HTTPRoute objects because spec.hostnames applies to the whole object.
  - stalwart does not render a routing object per node group - its Services and
    StatefulSets do; one route targets ingress.nodeGroup's own http Service.

THREE NEW CONVENTIONS, written where the next sweep will read them:

  - SCREEN THE LIVE OBJECT, not only the chart. Four of group 3's six charts
    carried an annotation the chart never sets and the Gateway mode drops -
    keycloak's proxy buffers on BOTH its Ingresses, matrix-synapse's body size
    and snippet, stalwart's whole CORS block, bulwark's body size. None of them
    appears in a git grep of this repo. At #118 this was found after shipping;
    here it was measured beforehand with read-only kubectl, and that
    measurement is now part of the procedure.
  - READ THE DIRECTIVE before calling an annotation load-bearing.
    matrix-synapse's configuration-snippet uses more_set_headers, which is the
    headers-more module's OUTPUT-header directive - so those headers go on the
    response and Synapse never sees them. The scariest-looking loss in the
    group is not a loss, and its NOTES.txt says not to re-create it.
  - "NO COUNTERPART" IS A CLAIM TO CHECK AGAINST THE PINNED CRDs, and for CORS
    it is false. Grepped out of gateway-api v1.6.3 standard channel: HTTPRoute
    filters carry a full cors field (GEP-1767), Support: Extended, and
    responseHeaderModifier is Extended too rather than Core. stalwart still
    renders neither, and the three reasons are recorded - no CORS values to
    render from, Extended support is not support (Cilium accepts the filter and
    forwards the preflight anyway), and the preflight is the half a bare header
    filter cannot do while Authorization in the allowed headers makes every
    request preflighted.

Also: pathType ImplementationSpecific has no counterpart and is a chart DEFAULT
in homeassistant - PathPrefix on "/", render abort on any other path, and
RegularExpression deliberately not offered. The ingress.enabled-derived-value
screen is closed: ytdlp-web-player was the third and last find, and its guard
is the one that is needed MORE in Gateway mode, because the ingress-nginx
admission webhook never sees an HTTPRoute and the silent outcome is an
authentication bypass that depends on DNS. A subchart does not come along:
oauth2-proxy has no Gateway support, so in the recommended configuration the
subchart's Ingress publishes the release in both of the chart's modes. And
stalwart is the only chart in the sweep still rendering a Certificate in
Gateway mode, for its mail listeners.

The homeassistant documented-exception bullet now covers its Gateway mode: no
httpRoute.hostnames key (the second chart without it), one route per
ingress.hosts entry, index-based route names and what reordering the list
costs.

Part of #114

jklein approved these changes 2026-10-13 10:59:02 +02:00

jklein merged commit 6d79d8f270 into main 2026-10-13 10:59:10 +02:00

jklein deleted branch feat/gateway-api-group-3 2026-10-13 10:59:10 +02:00

jlab-bot referenced this pull request 2026-10-13 11:00:26 +02:00

keycloak: die vier Realm-Hostnamen als Chart-Feature statt in cluster-config #122

jlab-bot referenced this pull request 2026-10-13 11:01:28 +02:00

Gateway-API als Alternative zum Ingress in allen Inhouse-Charts #114

jlab-bot referenced this pull request 2026-10-13 11:54:08 +02:00

stalwart: MTA-STS-Routen als pflegbare Liste im Chart, Subdomaenen eingeschlossen #123

jlab-bot referenced this pull request 2026-10-13 13:25:49 +02:00

stalwart/bulwark: CORS als Chart-Werte - die Gateway-API hat einen CORS-Filter, aber nur Extended #129

Sign in to join this conversation.

Reviewers

No Reviewers

jklein

Labels

Clear labels

inhouse-chart

Inhouse-Chart aus der JLab-Git-Paketregistry

renovate

Von Renovate angelegt

No labels

Milestone

No items

No Milestone

Assignees

Clear assignees

jklein (Justin Klein) jlab-bot (jlab-git-bot) jlab-renovate-bot (jlab-renovate-bot)

No Assignees

2 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: jlab-cluster/helm-charts#120