Support for Gateway API

[pl4nty]

Has there been any discussions on implementing support for the k8s beta Gateway API? The API should be flexible enough to implement most Tunnel capabilities, and uses built-in resources so new users can easily onboard/offboard (eg ingress2gateway).

Not sure whether this project would be the best fit, or a separate project from scratch. As I imagine this design as a gateway controller implementation, rather than an operator.

[adyanth]

I haven't explored the Gateway API too much myself, but the reason for converting the operator from targeting the ingress to the service was that you can do anything you could by pointing to the ingress and more with a service. Same thing goes for the Gateway API, at the end, all of them end in a service, which is where you can point the tunnel to directly, since the cloudflared is already a reverse proxy.

Unless you are suggesting to make an ingress/gateway controller out of cloudflared, for which, there are custom inputs that it needs, like credentials and which tunnel for each ingress that does not have place in the GatewayAPI CRD. We would need to fall back to annotations (or another CRD linked via the GatewayClass), which is what we moved away to custom CRDs. And instead of making the CRD+Ingress/HTTPRoute combo, we are just making it CRD+Service combo, reducing a layer of complexity, since there is no point of an Ingress if you anyway need to specify details in the CRD. You could still have Ingress/HTTPRoute, they will still work locally, but cloudflared will not target it. Same argument about pointing to the ingress for getting certain features like authentication, you can still point to the ingress/gateway controller's service to get all those features.

[pl4nty]

Yeah, I'm suggesting a gateway controller to allow configuring cloudflared with Gateway API resources. Definitely agree about avoiding annotations and additional CRDs where possible. GatewayClass spec.ParametersReference might work for some parameters, but Policy Attachment seems designed for this usecase, and adds some reusability that isn't available with the current CRDs.

Needing additional CRDs for routing should be avoidable. Here's a quick mapping that might be possible:

Current API	Gateway API
Install from yaml	Add GatewayClass to yaml
Tunnel/ClusterTunnel	Gateway, with namespace filtering via spec.listeners[].allowedRoutes
Tunnel/ClusterTunnel spec	New TunnelPolicy CRD
TunnelBinding HTTP(S)/TCP/UDP	HTTPRoute/TLSRoute/TCPRoute/UDPRoute
TunnelBinding SSH/RDP	TCPRoute spec.parentRefs.sectionName?

A new CRD (eg TunnelPolicy) would be required to handle credentials etc, but policy attachment might simplify this overall. A TunnelPolicy could be applied to multiple gateways, so config would be reusable between tunnels (I have a lot of duplicated config with the current ClusterTunnels/Tunnels).

Ongoing use of a gateway would only need the various routes, and those come with extra functionality. Eg HTTPRoutes could allow easier implementation of cloudflared's simple path-based routing.

TCP/UDP routes are still experimental though, and the API as a whole only reached beta recently. So maybe not something for the short term. I've just had great experiences with the API on other platforms, and it's worked really well for other flexible routing platforms.

[adyanth]

Ahh, so instead of a TunnelBinding, use one of the Gateway API's CRDs. That is a good way if everything that Cloudflare supports can be mapped neatly without abusing the Gateway API's meaning of the fields. I would have to look into this a bit more. Thanks for the idea! I will keep this as an enhancement for now.

[dudo]

Gateway classes support a lot more than what cloudflared needs to be concerned with. GAPI just creates a service of type LoadBalancer, which you can point the daemon/operator to.

This feature feels like it's asking too much of cloudflared.

[pl4nty]

@dudo GAPI doesn't require a LoadBalancer, and its core functionality seems like a good match for cloudflare-operator at first glance. particularly since cloudflare-operator originally used ingress annotations, and seemingly moved to CRDs for extensibility (#51) which is a GAPI design goal.

With Cloudflare adding more tunnel features (eg path-based routing), the existing CRDs might be difficult to maintain, vs GAPI where these concepts are standardised

[pl4nty]

I've been hitting too many reconciliation bugs with this operator recently, so I'm going to try implementing GAPI separately with remote tunnel config. aiming for tunnel/hostname/service/path reconciliation first, since originRequest/warp-routing and cloudflared args/env may need an experimental GAPI policy CRD. full conformance may not be possible anyway due to missing Cloudflare features (eg error handling with specific HTTP codes)

[pl4nty]

My Gateway controller has been stable for around 6 months. I haven't implemented policy attachment or non-HTTP protocols yet though