DAMS fcrepo unstability

[qjhart]

The fcrep instance of the dams is suspected to be the primary source for the slowdowns on sandbox. We need to understand what the underlying cause of this is.

Typical log explorer entry isn't super helpful (to me :), though maybe it's a function of the ro fcrepo not being able to log something?

{
  "textPayload": "02-Aug-2024 16:26:40.983 SEVERE [http-nio-8080-exec-107] org.glassfish.jersey.server.ServerRuntime$Responder.writeResponse An I/O error has occurred while writing a response message entity to the container output stream.",
  "insertId": "0lpyiv2dpvmqsohe",
  "resource": {
    "type": "k8s_container",
    "labels": {
      "pod_name": "fcrepo-ro-1",
      "cluster_name": "dams",
      "container_name": "service",
      "namespace_name": "default",
      "project_id": "ucdlib-dams",
      "location": "us-central1-a"
    }
  },
  "timestamp": "2024-08-02T16:26:40.986362192Z",
  "severity": "ERROR",
  "labels": {
    "k8s-pod/statefulset_kubernetes_io/pod-name": "fcrepo-ro-1",
    "k8s-pod/apps_kubernetes_io/pod-index": "1",
    "k8s-pod/controller-revision-hash": "fcrepo-ro-8489d69744",
    "k8s-pod/app": "fcrepo-ro",
    "compute.googleapis.com/resource_name": "gke-dams-scalable-pool-b1cc4b2e-pz9f"
  },
  "logName": "projects/ucdlib-dams/logs/stderr",
  "receiveTimestamp": "2024-08-02T16:26:44.092998606Z"
}