ReferenceGrant %s not owned by this object

[divanikus]

I'm trying to run knative with cilium as a gateway. My scheme involves per namespace certificates generated by certmanager DNS01 challenge (wildcard TLS). I can start one ksvc with my configuration. But starting yet another ksvc in the same namespace just fails with ReferenceGrant %s not owned by this object. Seems like autoTLS with wildcard certs is just not supported by the current plugin. I see no options to overcome it.

My knative instance is deployed with operator:

apiVersion: operator.knative.dev/v1beta1
kind: KnativeServing
metadata:
  name: knative-serving
  namespace: knative-serving
spec:
  ingress:
    istio:
      enabled: false
  config:
    network:
      ingress-class: "gateway-api.ingress.networking.knative.dev"
      domain-template: "{{.Name}}.{{.Namespace}}.{{.Domain}}"
      auto-tls: "Enabled"
      http-protocol: "Redirected"
      default-external-scheme: "https"
      namespace-wildcard-cert-selector: "{\"matchExpressions\": [{\"key\":\"networking.knative.dev/enableWildcardCert\", \"operator\": \"In\", \"values\":[\"true\"]}]}"

Gateways:

---
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: knative-ingress-gateway
  namespace: kube-system
spec:
  gatewayClassName: cilium
  listeners:
  - protocol: HTTP
    name: http
    port: 80
    allowedRoutes:
      namespaces:
        from: All
---
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: knative-local-gateway
  namespace: kube-system
spec:
  gatewayClassName: cilium
  listeners:
  - protocol: HTTP
    name: http
    port: 8081
    allowedRoutes:
      namespaces:
        from: All

Gateway config:

apiVersion: v1
kind: ConfigMap
metadata:
  name: config-gateway
  namespace: knative-serving
data:
  external-gateways: |
    - class: cilium
      gateway: kube-system/knative-ingress-gateway
      service: kube-system/cilium-gateway-knative-ingress-gateway
      supported-features:
      - Gateway
      - GatewayPort8080
      - GatewayStaticAddresses
      - HTTPRoute
      - HTTPRouteDestinationPortMatching
      - HTTPRouteHostRewrite
      - HTTPRouteMethodMatching
      - HTTPRoutePathRedirect
      - HTTPRoutePathRewrite
      - HTTPRoutePortRedirect
      - HTTPRouteQueryParamMatching
      - HTTPRouteRequestMirror
      - HTTPRouteRequestMultipleMirrors
      - HTTPRouteResponseHeaderModification
      - HTTPRouteSchemeRedirect
      - Mesh
      - ReferenceGrant
      - TLSRoute
  local-gateways: |
    - class: cilium
      gateway: kube-system/knative-local-gateway
      service: kube-system/cilium-gateway-knative-local-gateway
      supported-features:
      - Gateway
      - GatewayPort8080
      - GatewayStaticAddresses
      - HTTPRoute
      - HTTPRouteDestinationPortMatching
      - HTTPRouteHostRewrite
      - HTTPRouteMethodMatching
      - HTTPRoutePathRedirect
      - HTTPRoutePathRewrite
      - HTTPRoutePortRedirect
      - HTTPRouteQueryParamMatching
      - HTTPRouteRequestMirror
      - HTTPRouteRequestMultipleMirrors
      - HTTPRouteResponseHeaderModification
      - HTTPRouteSchemeRedirect
      - Mesh
      - ReferenceGrant
      - TLSRoute

The plugin generates a ReferenceGrant:

apiVersion: gateway.networking.k8s.io/v1beta1
kind: ReferenceGrant
metadata:
  creationTimestamp: "2024-10-17T13:44:17Z"
  generation: 1
  name: default.example.io-kube-system
  namespace: default
  ownerReferences:
  - apiVersion: networking.internal.knative.dev/v1alpha1
    blockOwnerDeletion: true
    controller: true
    kind: Ingress
    name: hello
    uid: 3149c0ac-f6a8-47c3-8b5a-78955ada0a7b
  resourceVersion: "1113699"
  uid: 53784654-8e98-45ba-a0e7-93c50882b05a
spec:
  from:
  - group: gateway.networking.k8s.io
    kind: Gateway
    namespace: kube-system
  to:
  - group: ""
    kind: Secret
    name: default.example.io

As you might see, it has ownerReference of the first ksvc i have deployed. Deploying anything else fails.

Is there a way to overcome this? I've tried to create a ReferenceGrant manually, but ksvc deployment still fails with the same error, because it's not owned by it.

[divanikus]

The problem is that MakeReferenceGrant always creates it with the same name in my case: https://github.com/knative-extensions/net-gateway-api/blob/main/pkg/reconciler/ingress/resources/reference_grant.go#L29

You should probably add the ingress name there or check if we already have this grant for other services, idk.