One of the grafana-agent unit stuck in executing status

[jeffreychang911]

Bug Description

In SQA Openstack deployment, we found one of the grafana-agent unit stuck in executing status for 4 hours until test timeout.
https://solutions.qa.canonical.com/testruns/bc1d3e50-fb3e-431d-87a4-c16283b7c092 is the test run.
https://oil-jenkins.canonical.com/artifacts/bc1d3e50-fb3e-431d-87a4-c16283b7c092/generated/generated/openstack/juju-crashdump-openstack-2025-01-22-21.13.14.tar.gz - contains debug logs

To Reproduce

This happens in SQA environment, doesn't have a simple reproduce step.

Environment

This is a Openstack PCB+ deployment with grafana-agent

Relevant log output

From debug-log, the unit doesn't have any more status change after 18:10, and stuck in "executing" for 4 hrs until timeout.
On other grafana-agent, I can see them switch to idle.

2025-01-22 18:10:00 DEBUG juju.worker.uniter agent.go:22 [AGENT-STATUS] executing: running peers-relation-joined hook for grafana-agent-container/32
2025-01-22 18:10:07 DEBUG juju.worker.uniter agent.go:22 [AGENT-STATUS] executing: running peers-relation-changed hook for grafana-agent-container/32
2025-01-22 18:10:14 DEBUG juju.worker.uniter agent.go:22 [AGENT-STATUS] executing: running peers-relation-joined hook for grafana-agent-container/34
2025-01-22 18:10:21 DEBUG juju.worker.uniter agent.go:22 [AGENT-STATUS] executing: running peers-relation-changed hook for grafana-agent-container/34
2025-01-22 18:10:28 DEBUG juju.worker.uniter agent.go:22 [AGENT-STATUS] executing: running peers-relation-joined hook for grafana-agent-container/4

Additional context

No response