feat: Gateway-API as an alternative to the Ingress in the six special shapes (group 3) #120

Merged

jklein merged 7 commits from feat/gateway-api-group-3 into main 2026-10-13 10:59:10 +02:00

Conversation 1 Commits 7 Files Changed 42

+2945 -137

7 Commits

Author	SHA1	Message	Date
jklein	23ff5d5b08	docs(README): group 3 of the Gateway-API sweep, and four corrections it forced ... The figures: the HTTPRoute render test goes from 37 assertions over 15 charts to **56 over 21**, with the per-chart breakdown, and the sweep is finished except for the frozen rss-bridge - so #114 still cannot be closed while #51 is open. FOUR CORRECTIONS, because three of #114's descriptions of the group-3 shapes and one of this file's own forward-looking claims did not survive contact with the charts. The corrected list is a table in section 4, since that is what somebody will copy from: - bulwark has ONE Ingress, not two. keycloak has two. - bulwark has NO rewrite-target and cannot have one: NEXT_PUBLIC_BASE_PATH is compiled into its bundle. #114's table reached "bulwark, freesailarr" by grepping for the string, and every occurrence in that chart is an explanation of why the annotation does not help. freesailarr is therefore still the only chart the #118 prefix/redirect pairing applies to - the HTTPRoute-test section said bulwark would meet it "when group 3 lands", and now says why it does not. - matrix-synapse does NOT render in a loop over several apps (that is freesailarr): one Ingress object with two HOST rules, which has to become two HTTPRoute objects because spec.hostnames applies to the whole object. - stalwart does not render a routing object per node group - its Services and StatefulSets do; one route targets ingress.nodeGroup's own http Service. THREE NEW CONVENTIONS, written where the next sweep will read them: - SCREEN THE LIVE OBJECT, not only the chart. Four of group 3's six charts carried an annotation the chart never sets and the Gateway mode drops - keycloak's proxy buffers on BOTH its Ingresses, matrix-synapse's body size and snippet, stalwart's whole CORS block, bulwark's body size. None of them appears in a git grep of this repo. At #118 this was found after shipping; here it was measured beforehand with read-only kubectl, and that measurement is now part of the procedure. - READ THE DIRECTIVE before calling an annotation load-bearing. matrix-synapse's configuration-snippet uses more_set_headers, which is the headers-more module's OUTPUT-header directive - so those headers go on the response and Synapse never sees them. The scariest-looking loss in the group is not a loss, and its NOTES.txt says not to re-create it. - "NO COUNTERPART" IS A CLAIM TO CHECK AGAINST THE PINNED CRDs, and for CORS it is false. Grepped out of gateway-api v1.6.3 standard channel: HTTPRoute filters carry a full cors field (GEP-1767), Support: Extended, and responseHeaderModifier is Extended too rather than Core. stalwart still renders neither, and the three reasons are recorded - no CORS values to render from, Extended support is not support (Cilium accepts the filter and forwards the preflight anyway), and the preflight is the half a bare header filter cannot do while Authorization in the allowed headers makes every request preflighted. Also: pathType ImplementationSpecific has no counterpart and is a chart DEFAULT in homeassistant - PathPrefix on "/", render abort on any other path, and RegularExpression deliberately not offered. The ingress.enabled-derived-value screen is closed: ytdlp-web-player was the third and last find, and its guard is the one that is needed MORE in Gateway mode, because the ingress-nginx admission webhook never sees an HTTPRoute and the silent outcome is an authentication bypass that depends on DNS. A subchart does not come along: oauth2-proxy has no Gateway support, so in the recommended configuration the subchart's Ingress publishes the release in both of the chart's modes. And stalwart is the only chart in the sweep still rendering a Certificate in Gateway mode, for its mail listeners. The homeassistant documented-exception bullet now covers its Gateway mode: no httpRoute.hostnames key (the second chart without it), one route per ingress.hosts entry, index-based route names and what reordering the list costs. Part of #114	2026-10-12 21:26:38 +00:00
jklein	f647597a69	feat(ytdlp-web-player): Gateway-API as an alternative to the Ingress ... Group 3 of the #114 sweep, last chart - and the one that carried the group-2 pattern this sweep was told to look for. It reads ingress.enabled OUTSIDE its Ingress template, in ytdlp-web-player.validate, and in Gateway mode ingress.enabled is false by construction, so that guard would have gone silent. Fixed with ytdlp-web-player.published = ingress.enabled OR httpRoute.enabled - the freesailarr.published / sure.publishedOverTls shape, identical to ingress.enabled while httpRoute.enabled is false, which is why it changes no existing render. AND THE GUARD IS NEEDED MORE IN GATEWAY MODE, NOT LESS. It refuses to publish the application and the oauth2-proxy on the same host: - In INGRESS mode the consequence of ignoring it is loud: ingress-nginx's admission webhook rejects the second Ingress and the Fleet bundle goes to ErrApplied. - In GATEWAY mode THE WEBHOOK NEVER FIRES. It compares Ingress objects with Ingress objects, and an HTTPRoute is a different kind - so nothing rejects anything, both objects stay valid, and one hostname is served over two data paths that are NOT equivalent: the proxy's Ingress goes through AUTHENTICATION and this chart's HTTPRoute goes straight to the application. Which one a client reaches depends on what DNS resolves to. That is an authentication bypass that comes and goes with a DNS record and that no Kubernetes object reports. So the Gateway-mode abort has its own message saying exactly that. Both messages exercised, plus the collision reached through httpRoute.hostnames rather than through the ingress.domain fallback, and the guard still firing when only the Deployment is rendered. THE SUBCHART HAS NO GATEWAY SUPPORT, and this chart cannot give it one - upstream's chart is consumed as it comes (README, "Chart dependencies"). It renders an Ingress or nothing. So in the RECOMMENDED configuration, where the proxy sits in the data path and owns the host, the publishing object is the SUBCHART'S INGRESS in both of this chart's modes, and turning httpRoute on buys nothing. The route is for the proxy-disabled case, or for the application on a host of its own - which is the documented unauthenticated bypass. Said in the template header, in values.yaml, in the schema and in the proxy branch of NOTES.txt, because someone migrating hosts will otherwise set the value and wonder why nothing moved. The 16k proxy-buffer-size this chart recommends is NOT lost by this switch, and that needed stating rather than assuming: it belongs on the PROXY's Ingress (oauth2-proxy.ingress.annotations) as the second half of the 4kB session-cookie trap, and the subchart keeps rendering an Ingress. This chart's own ingress.annotations default is {} and it sets nothing but the cert-manager annotation, so there is no chart-side annotation loss to name. CORRECTED CLAIM: values.yaml and the schema both said that everything the Ingress needs is read only inside its template, so switching it off freed BOTH ingress.domain and ingress.clusterIssuer. clusterIssuer is still freed; domain is now read by the HTTPRoute, so it is freed only with both modes off. The reason the claim was made still holds and is kept - the application has no base-URL setting and builds no absolute links. NOTHING WAS MEASURED ON A LIVE INGRESS OF THIS CHART, because there is none. Measured 2026-10-12: the release runs in namespace jlab-player with both Deployments Ready, and NOT ONE of the cluster's 23 Ingress objects is in that namespace - neither this chart's nor the subchart's. What publishes it instead could not be determined: this workspace's identity may not list httproutes or secrets, so neither a hand-written HTTPRoute nor the release's Helm values are readable. Recorded as the open question it is, in values.yaml and in the overlay, rather than guessed at. Minor: additive, defaults unchanged. Empty helm template diff against origin/main with ci/ytdlp-web-player/test-values.yaml. Part of #114	2026-10-12 21:22:49 +00:00
jklein	d860704172	feat(homeassistant): Gateway-API as an alternative to the Ingress ... Group 3 of the #114 sweep, and the one chart where the convention had to bend rather than the chart. homeassistant is the DOCUMENTED INGRESS EXCEPTION (README): upstream's flexible values, a list of hosts each with its own list of paths, instead of the repo's required domain/clusterIssuer pair. The exception stays - nothing here unifies it. NO httpRoute.hostnames KEY, which is the freesailarr shape from group 2 and for the same reason: there is no ingress.domain to fall back to, and a chart-global hostname list could not say which host entry - and therefore which set of paths - it belonged to. So httpRoute carries only enabled and parentRefs, the hostnames come from ingress.hosts, and the chart renders ONE HTTPRoute PER ENTRY with one rule per path of that entry. One object with several hostnames would be wrong, not merely different: an HTTPRoute's rules apply to every hostname of the object, and per-host paths are this chart's whole point. The schema rejects httpRoute.hostnames outright, so the absence is enforced rather than merely intended. ingress.hosts is therefore read in BOTH modes and is not an Ingress-only value. ingress.className, ingress.tls and ingress.annotations are not read in Gateway mode: class and certificate belong to the Gateway's listener, and there is no annotation channel. pathType IS THE SHARP EDGE, AND THE CHART'S DEFAULT IS THE AWKWARD VALUE. Ingress has Exact/Prefix/ImplementationSpecific; the Gateway API has Exact/PathPrefix/RegularExpression. Exact and Prefix map cleanly. ImplementationSpecific - the default in values.yaml - means "ask the controller" and has NO counterpart, so: - on "/" it renders as PathPrefix. Every controller treats a root ImplementationSpecific path as a prefix match, so the chart's own default works in this mode unchanged (verified: the untouched default renders a valid route). - on ANY OTHER path the render ABORTS, naming the value and asking for Prefix or Exact. In ingress-nginx - this chart's default class - an ImplementationSpecific sub-path is a REGEX, and PathPrefix is not one. Mapping it silently would change how the path matches while the render stayed green, which is exactly the class of defect #118 was. RegularExpression is deliberately not offered as an escape hatch: its support level is Implementation-specific, which decision 1 of #114 forbids. NOTHING WAS MEASURED ON THE LIVE CLUSTER FOR THIS CHART, because there is nothing to measure: homeassistant is NOT DEPLOYED in jk-cluster (checked 2026-10-12 - 23 Ingress objects cluster-wide, none this chart's, and no workload of it in any namespace). So it is the one chart of the six with no operator-supplied annotation to lose. values.yaml and NOTES.txt say that as an absence of evidence about this installation rather than as a property of the chart. Route NAMES carry the entry's INDEX, not its hostname - a hostname is not a dependable object-name fragment. Consequence named in values.yaml and in the template: reordering ingress.hosts renames routes and therefore recreates rather than edits them. Harmless, since a route holds no state and no certificate. ci/httproute/homeassistant.yaml has to SUPPLY HOSTS, which no other overlay in the sweep does, and it adds coverage the install-test never had: this chart's ingress.enabled defaults to FALSE and the fixture does not switch it on, so no routing object was exercised at all before. Two entries on purpose - a plain one, and one with two paths including an Exact - because that second entry is what proves the chart renders one object per host rather than one object with several hostnames. 5 assertions over 2 routes and 3 rules. NOTES.txt asks $published where it asked ingress.enabled, and the trusted_proxies section now warns that moving from an Ingress to a Gateway CHANGES the proxy's pod network: a list carried over unchanged turns every request into a 400 if the CIDR differs, or silently keeps attributing every client to one address if it does not. Minor: additive, defaults unchanged. Empty helm template diff against origin/main, both with the fixture as it stands and with ingress.enabled=true. Part of #114	2026-10-12 21:16:03 +00:00
jklein	cffc50e4ea	feat(stalwart): Gateway-API as an alternative to the Ingress ... Group 3 of the #114 sweep. ONE route - not one per node group. #114 describes this chart as rendering "per node group"; its SERVICES and StatefulSets are, the routing object is not and must not be. The route targets exactly ingress.nodeGroup's own http Service and keeps both of the Ingress's guards, because the WebUI's setup and management screens are not safe to spread over a group that may not run the management tasks. ingress.nodeGroup is therefore shared by both modes: it names where the management surface is published from, which is not a property of the Ingress. THE MAIL CERTIFICATE STAYS, and this is the only chart in the sweep where anything of the TLS path does. stalwart.certificate.yaml covers STARTTLS on 25 and implicit TLS on 465/993; those connections pass through neither an ingress controller nor a Gateway. So in Gateway mode ingress.domain is still required (the certificate's dnsNames read it), ingress.clusterIssuer is still required unless certificate.clusterIssuer is set, and the Certificate must NOT be removed as part of a migration - deleting it drops SMTP and IMAP back to Stalwart's self-signed certificate, with clients seeing a name mismatch while every Kubernetes object stays green. Verified in the render: Gateway mode emits Certificate + HTTPRoute and no Ingress. WHAT THIS MODE DROPS - AND HERE IT BREAKS A SECOND CHART OF THE SAME GROUP. Measured on 2026-10-12, the live Ingress on mail.bekemeyer-klein.de carried, all of it operator-supplied through ingress.annotations (this chart sets none of it): enable-cors: true cors-allow-origin: https://webmail.bekemeyer-klein.de cors-allow-methods: GET, POST, OPTIONS cors-allow-headers: Authorization, Content-Type That origin is BULWARK, the webmail chart in this same group. Without the CORS configuration the browser-side webmail cannot call this server's JMAP endpoint at all, and the failure is entirely client-side: a console error, nothing in the server's logs. AND "no counterpart" WOULD HAVE BEEN WRONG. Measured by grepping the CRDs this repo pins (gateway-api v1.6.3 standard channel, ci/gateway-api-crds.sh) rather than inferred: HTTPRoute's filters carry a full 'cors' field (GEP-1767) with allowOrigins/allowMethods/allowHeaders/allowCredentials/maxAge, and its documented level - verbatim from that CRD - is "Support: Extended". responseHeaderModifier is in the same CRD and is also Extended, not Core. The chart still does not render it, for three reasons in order: 1. The chart has NO CORS values. The live configuration is the operator's, and the chart sets no CORS annotations in Ingress mode either - so rendering a filter means inventing a values surface that would exist in Gateway mode only. That asymmetry is its own defect and its own work. 2. Extended support is not support. A rejected route is rejected WHOLE, which would take the WebUI and JMAP down together (the reasoning #119 reached for statusCode); and the observed failure is quieter - Cilium accepts filters.type: CORS and forwards the preflight upstream anyway (cilium/cilium#45272), i.e. Accepted: True with the behaviour absent. 3. Half a CORS is worse than none, and the translatable half is the useless one HERE. A responseHeaderModifier can add the Access-Control-Allow-* headers; the PREFLIGHT has to be answered BY THE GATEWAY, which the CORS filter's own contract states and a bare header filter cannot do. Because the allowed headers include Authorization, EVERY webmail request is preflighted - so the expressible half is exactly the half that does not help. Named, not rebuilt (decision 2 of #114), in values.yaml, in the schema and in the Gateway branch of NOTES.txt, with the operational instruction: configure CORS at the Gateway with that implementation's own support BEFORE moving this host, and test it from the webmail in a browser - curl sends no preflight, so it passes while the webmail fails. NOTES.txt now asks $published (ingress.enabled OR httpRoute.enabled) where it asked ingress.enabled, and two paragraphs that named the ingress controller specifically now name the Gateway in that mode: the auto-ban step (switching API does not avoid or reset it - one proxy address instead of another) and the NetworkPolicy paragraph, where the namespace to allow is the Gateway's data plane rather than an ingress controller's. Minor: additive, defaults unchanged. Empty helm template diff against origin/main with ci/stalwart/test-values.yaml. Part of #114	2026-10-12 21:10:00 +00:00
jklein	470b90a393	feat(matrix-synapse): Gateway-API as an alternative to the Ingress ... Group 3 of the #114 sweep. TWO HTTPRoutes - and the shape is not the one the issue described. #114 says this chart renders its Ingresses in a LOOP OVER SEVERAL APPS. It does not; that is freesailarr. It renders ONE Ingress object with TWO HOST rules: the homeserver on ingress.domain at "/", and the delegation nginx on matrix.delegation.serverUrl at "/.well-known/matrix". An HTTPRoute's spec.hostnames applies to the WHOLE object, so two hostnames that want different paths and different backends have to be two objects. That is the only faithful translation, not a stylistic choice. httpRoute.delegation.hostnames deliberately does NOT fall back to httpRoute.hostnames. The server name is what REMOTE homeservers resolve and is a different thing from the homeserver's own hostname; the migration order - homeserver host first, server name last - depends on telling them apart, and a chart-global hostname list covering both would erase the distinction. httpRoute.delegation.parentRefs falls back to httpRoute.parentRefs, which matters because the server name is usually an apex domain behind a different Gateway. ONE THING THIS MODE IMPROVES, and it is the awkward part of Matrix delegation. Under ingress-nginx two Ingress objects cannot both claim one hostname - the admission webhook rejects the second - which is why the delegation rule has to live inside this chart's single Ingress object even though the apex domain usually belongs to a website. HTTPRoutes attached to the same listener for the same hostname are merged by the implementation, longest match winning, so the /.well-known/matrix route can sit beside whatever owns that apex. WHAT THIS MODE DROPS, MEASURED ON THE LIVE OBJECT ON 2026-10-12 - and ONE OF THE TWO IS NOT WHAT IT LOOKS LIKE. The live Ingress carried, all of it operator-supplied through ingress.annotations (this chart sets none of it): nginx.ingress.kubernetes.io/proxy-body-size: 50m nginx.org/client-max-body-size: 50m nginx.ingress.kubernetes.io/configuration-snippet: | more_set_headers "X-Forwarded-For $remote_addr"; more_set_headers "X-Forwarded-Proto $scheme"; more_set_headers "Host $host:$server_port"; 1. THE BODY SIZE IS THE REAL LOSS. Synapse's media repository takes uploads; its own ceiling is max_upload_size in homeserver.yaml, which this chart cannot see because the whole config arrives from 1Password. The effective limit is the smaller of the two, and in this mode the proxy half becomes the GATEWAY'S default - neither number anyone can point at is necessarily the one that decides. No counterpart in the Gateway API, so per decision 2 of #114 it is documented, not rebuilt. nginx.org/client-max-body-size is the same limit spelled for NGINX Inc's controller, a hedge across both implementations, and it goes for the same reason. (#114's own measured table did not list matrix-synapse under proxy-body-size at all; the live object has it.) 2. THE SNIPPET IS NOT LOAD-BEARING, which took checking rather than assuming. It looks like what Synapse's x_forwarded setting depends on - client IP for rate limiting, and federation. It is not: more_set_headers is the headers-more module's OUTPUT-header directive ("Replaces (if any) or adds (if not any) the specified output headers"), the request-side directive being more_set_input_headers. So those three headers are added to the RESPONSE TO THE CLIENT; Synapse never sees them, and what it reads with x_forwarded: true is the REQUEST's X-Forwarded-For, which ingress-nginx supplies itself. The Gateway API's ResponseHeaderModifier could not rebuild it anyway (it knows no nginx variables), and should not: NOTES.txt says in as many words not to re-create it and not to read its absence as the cause of a rate-limiting or federation problem. Stated as what it is - a reading of the directive's documented semantics. Nothing here observed a Synapse instance with and without the snippet. NOTES.txt now asks $published (ingress.enabled OR httpRoute.enabled) wherever it used to ask ingress.enabled - the server-name/serverUrl mismatch warning, the "clients work fine" aside and the scaled-to-zero paragraph were all really asking whether anything publishes the release. Minor: additive, defaults unchanged. Empty helm template diff against origin/main with ci/matrix-synapse/test-values.yaml. Part of #114	2026-10-12 20:55:55 +00:00
jklein	f6eeff87c2	feat(bulwark): Gateway-API as an alternative to the Ingress ... Group 3 of the #114 sweep. One route, one rule, at the root - and THREE EXPECTATIONS ABOUT THIS CHART DID NOT SURVIVE CONTACT WITH IT: - #114 says bulwark has TWO Ingress objects. It has ONE. keycloak has two. - #114's measured table lists bulwark under rewrite-target. It has no rewrite-target annotation and CANNOT have one: NEXT_PUBLIC_BASE_PATH is compiled into the bundle, so the application cannot be served under a sub-path, and stripping a prefix at the proxy would yield a page that loads and then fetches everything from the wrong place. Every occurrence of the string "rewrite-target" in this chart - values.yaml, the schema, the Ingress template - is an explanation of exactly that. A grep for the string finds the refutations and counts them as occurrences. - ci/run-httproute-match-test.sh said in its header that bulwark was the second rewrite-target chart and would meet the #118 prefix/redirect assurance when group 3 landed. Group 3 landed and it does not: with no URLRewrite on a prefix other than "/", the assurance's precondition is never met here. The header is corrected rather than left standing, and it now says freesailarr is the only chart the pairing applies to. So no redirect rule is added, and none is needed. WHAT THIS MODE DROPS, MEASURED ON THE LIVE OBJECT rather than inferred. On 2026-10-12 the live Ingress on webmail.bekemeyer-klein.de carried nginx.ingress.kubernetes.io/proxy-body-size: 25m supplied by the OPERATOR through ingress.annotations - values.yaml recommends exactly that value and explains why the chart cannot pick it, so the chart sets nothing and a reader of this repository alone would not find it. It is the single annotation this installation depends on, and the Gateway API has no body-size counterpart: implementation policy, not routing. Per decisions 1 and 2 of #114 it is NAMED, not rebuilt - in values.yaml, in the schema and in the Gateway branch of NOTES.txt, in this application's own terms: an attachment above the Gateway's default is rejected before Bulwark sees a byte, the 413 comes from the proxy, and it arrives after the user has written the mail and attached the file. CORRECTED CLAIM, the same one garage and calibre-web-automated needed: the Ingress template, the bulwark.publicUrl helper and the schema all stated that nothing outside the Ingress template FUNCTIONALLY reads ingress.domain. The HTTPRoute reads it, so the claim held only while both modes were off. All three now say so, and the route carries a required() of its own. bulwark.publicUrl follows the ACTIVE mode instead of always printing ingress.domain. The two differ exactly when httpRoute.hostnames is set, which is how a migration off the Ingress runs - and printing the old host there would hand the operator an OIDC redirect URI for the hostname being migrated away from, a failure that surfaces as a rejected login at the identity provider and says nothing about this chart. Minor: additive, defaults unchanged. Empty helm template diff against origin/main with ci/bulwark/test-values.yaml. Part of #114	2026-10-12 20:49:56 +00:00
jklein	f97e1e6d52	feat(keycloak): Gateway-API as an alternative to the two Ingresses ... Group 3 of the #114 sweep, first chart. keycloak has TWO Ingress objects - the public hostname and the admin hostname - so it gets TWO HTTPRoutes, with httpRoute.admin.parentRefs falling back to httpRoute.parentRefs exactly as ingress.admin.className falls back to ingress.className: "which data path" is a property of the Gateway in this API, not of the route, so an internal Gateway here is what an internal ingress class is in the other mode. The admin route stays gated on ingress.admin.enabled, because that value is not an Ingress switch - it also decides whether KC_HOSTNAME_ADMIN is set on the Deployment, i.e. whether the admin hostname exists at all. Switching to Gateway mode therefore must not require turning it off. FIXED ALONG THE WAY: the admin Ingress did not hang off ingress.enabled. Measured on origin/main, `helm template -f ci/keycloak/test-values.yaml --set ingress.enabled=false` emitted kc-keycloak-admin-ingress - so "ingress.enabled: false" did not mean "no Ingress", and ingress.clusterIssuer stayed required for an object nobody asked for. It also made Gateway mode impossible: the admin Ingress would have gone on claiming the admin host next to the admin HTTPRoute, one host over two data paths, which is the exact conflict the mutual exclusion exists to prevent. The default render is unchanged; what changed is a combination that could not be coherent. WHAT THIS MODE DROPS, MEASURED ON THE LIVE CLUSTER rather than inferred. On 2026-10-12 BOTH of this chart's Ingress objects in jk-cluster carried nginx.ingress.kubernetes.io/proxy-buffer-size: 16k nginx.ingress.kubernetes.io/proxy-buffers-number: 4 set by the OPERATOR through ingress.annotations and ingress.admin.annotations - the chart sets neither, which is why reading the chart alone does not reveal them. The Gateway API has no annotation channel and no counterpart for proxy buffer sizing; it is implementation policy, not routing. Per decision 1 of #114 the chart renders no policy object, so this is NAMED and not rebuilt, in values.yaml, in the schema and in the Gateway branch of NOTES.txt, with the consequence spelled out in Keycloak's own terms: large headers (JWTs, SAML assertions, long Set-Cookie chains on the login flow) make a too-small buffer answer 502 at the PROXY while Keycloak logs the request as successful. Named and deliberately not touched: four further Ingress objects in the same namespace (keycloak-realm-*-ingress, with app-root) belong to the Helm release jk-cluster-config-fleet-cd-keycloak-realm-ingress, i.e. to another repository. They point at this release's Service under four further hostnames. This chart renders exactly the two routes its own two Ingresses cover. Minor: additive, defaults unchanged. Empty helm template diff against origin/main with ci/keycloak/test-values.yaml. Part of #114	2026-10-12 20:44:49 +00:00