KNative Serving- Activator throws me "error reverse proxying request;" When Sending many requests(80) at a time.

[msgurikar]

Hi,
I am using KNative serving 1.11.1 with Istio as Ingress gateway to do auto scaling of my app.
We begin sending requests many requests(80) at once, when there is only 1 pod running, requests hit Activator pod, during autoscaling process, activate throws throttle error and client recieves OS error. Here is the logs of Activator:

severity":"ERROR","timestamp":"2023-09-28T14:42:35.638529969Z","logger":"activator","caller":"network/error_handler.go:33","message":"error reverse proxying request; sockstat: sockets: used 48\nTCP: inuse 44 orphan 5 tw 16 alloc 252 mem 54\nUDP: inuse 0 mem 1\nUDPLITE: inuse 0\nRAW: inuse 0\nFRAG: inuse 0 memory 0\n","commit":"f60eb32","knative.dev/controller":"activator","knative.dev/pod":"activator-54f7b7d8dd-clz2d","knative.dev/key":"testsvc/test-service-latest","error":"client disconnected","stacktrace":"knative.dev/pkg/network.ErrorHandler.func1\n\tknative.dev/[email protected]/network/error_handler.go:33\nknative.dev/serving/pkg/activator/handler.(*activationHandler).proxyRequest.func1\n\tknative.dev/serving/pkg/activator/handler/handler.go:140\nnet/http/httputil.(*ReverseProxy).ServeHTTP\n\tnet/http/httputil/reverseproxy.go:475\nknative.dev/serving/pkg/activator/handler.(*activationHandler).proxyRequest\n\tknative.dev/serving/pkg/activator/handler/handler.go:143\nknative.dev/serving/pkg/activator/handler.(*activationHandler).ServeHTTP.func1\n\tknative.dev/serving/pkg/activator/handler/handler.go:96\nknative.dev/serving/pkg/activator/net.(*revisionThrottler).try.func1\n\tknative.dev/serving/pkg/activator/net/throttler.go:236\nknative.dev/serving/pkg/queue.(*Breaker).Maybe\n\tknative.dev/serving/pkg/queue/breaker.go:155\nknative.dev/serving/pkg/activator/net.(*revisionThrottler).try\n\tknative.dev/serving/pkg/activator/net/throttler.go:226\nknative.dev/serving/pkg/activator/net.(*Throttler).Try\n\tknative.dev/serving/pkg/activator/net/throttler.go:521\nknative.dev/serving/pkg/activator/handler.(*activationHandler).ServeHTTP\n\tknative.dev/serving/pkg/activator/handler/handler.go:89\nknative.dev/serving/pkg/http/handler.(*timeoutHandler).ServeHTTP.func4\n\tknative.dev/serving/pkg/http/handler/timeout.go:118"}
{"severity":"ERROR","timestamp":"2023-09-28T14:42:35.638501341Z","logger":"activator","caller":"network/error_handler.go:33","message":"error reverse proxying request; sockstat: sockets: used 48\nTCP: inuse 44 orphan 5 tw 16 alloc 252 mem 54\nUDP: inuse 0 mem 1\nUDPLITE: inuse 0\nRAW: inuse 0\nFRAG: inuse 0 memory 0\n","commit":"f60eb32","knative.dev/controller":"activator","knative.dev/pod":"activator-54f7b7d8dd-clz2d","knative.dev/key":"testsvc/test-service-latest","error":"client

Here is my KNative service yaml

apiVersion: serving.knative.dev/v1
kind: Service
metadata:
name: test-service
namespace: testsvc
spec:
template:
metadata:
name: test-service-latest
annotations:

    queue.sidecar.serving.knative.dev/memory-resource-request: "300Mi"
    queue.sidecar.serving.knative.dev/memory-resource-limit: "400Mi"        
    autoscaling.knative.dev/class: kpa.autoscaling.knative.dev     
    autoscaling.knative.dev/minScale: "1"
    autoscaling.knative.dev/maxScale: "80"        
    autoscaling.knative.dev/scaleDownDelay: "30m"
    
spec:
  containerConcurrency: 1
  timeoutSeconds: 300
  tolerations:
    - key: "sku"
      operator: "Equal"
      value: "qi"
      effect: "NoSchedule"
  nodeSelector:
    kubernetes.io/os: linux
    algo : qiefp-linux-F16s_v2      
  containers:        
  - image: xxxacr.azurecr.io/bis/testservice_res:latest
    imagePullPolicy: Always
    resources:
      requests:
        cpu: 950m
        memory: 950Mi
      limits:
        cpu: 1000m
        memory: 1000Mi
   
    ports:
      - name: h2c
        containerPort: 40083

Any suggestions to fix this problem will help. Thank you.