one of grafana-agent unit has hook failed: "cos-agent-relation-joined"

[jeffreychang911]

Bug Description

SQA test run - https://solutions.qa.canonical.com/testruns/f3d91bff-6b0b-41df-a9b6-b8e423e9b69c
Crashdump - https://oil-jenkins.canonical.com/artifacts/f3d91bff-6b0b-41df-a9b6-b8e423e9b69c/generated/kafka/juju-crashdump-kafka-2025-01-27-14.09.29.tar.gz

SQA try to deploy grafana-agent with kafka, and one of the agent get error below.
grafana-agent-host/0* error idle 10.241.32.129 hook failed: "cos-agent-relation-joined" for grafana-agent-host:cos-agent

To Reproduce

SQA deploy 3 kafka units and has grafana-agent with them all and ntp.

Environment

grafana-agent 332, with juju 3/stable.

Relevant log output

from juju debug-log, we can see 
2025-01-27 14:10:07 ERROR unit.grafana-agent-host/0.juju-log server.go:326 cos-agent:16: Uncaught exception while in charm code:
Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-grafana-agent-host-0/charm/./src/charm.py", line 701, in <module>
    main(GrafanaAgentMachineCharm)
  File "/var/lib/juju/agents/unit-grafana-agent-host-0/charm/venv/ops/__init__.py", line 343, in __call__
    return _main.main(charm_class=charm_class, use_juju_for_storage=use_juju_for_storage)
  File "/var/lib/juju/agents/unit-grafana-agent-host-0/charm/venv/ops/_main.py", line 543, in main
    manager.run()
  File "/var/lib/juju/agents/unit-grafana-agent-host-0/charm/venv/ops/_main.py", line 529, in run
    self._emit()
  File "/var/lib/juju/agents/unit-grafana-agent-host-0/charm/venv/ops/_main.py", line 518, in _emit
    _emit_charm_event(self.charm, self.dispatcher.event_name, self._juju_context)
  File "/var/lib/juju/agents/unit-grafana-agent-host-0/charm/venv/ops/_main.py", line 134, in _emit_charm_event
    event_to_emit.emit(*args, **kwargs)
  File "/var/lib/juju/agents/unit-grafana-agent-host-0/charm/venv/ops/framework.py", line 347, in emit
    framework._emit(event)
  File "/var/lib/juju/agents/unit-grafana-agent-host-0/charm/venv/ops/framework.py", line 857, in _emit
    self._reemit(event_path)
  File "/var/lib/juju/agents/unit-grafana-agent-host-0/charm/venv/ops/framework.py", line 947, in _reemit
    custom_handler(event)
  File "/var/lib/juju/agents/unit-grafana-agent-host-0/charm/lib/charms/tempo_coordinator_k8s/v0/charm_tracing.py", line 1064, in wrapped_function
    return callable(*args, **kwargs)  # type: ignore
  File "/var/lib/juju/agents/unit-grafana-agent-host-0/charm/lib/charms/grafana_agent/v0/cos_agent.py", line 982, in _on_relation_data_changed
    self.on.data_changed.emit()  # pyright: ignore
  File "/var/lib/juju/agents/unit-grafana-agent-host-0/charm/venv/ops/framework.py", line 347, in emit
    framework._emit(event)
  File "/var/lib/juju/agents/unit-grafana-agent-host-0/charm/venv/ops/framework.py", line 857, in _emit
    self._reemit(event_path)
  File "/var/lib/juju/agents/unit-grafana-agent-host-0/charm/venv/ops/framework.py", line 947, in _reemit
    custom_handler(event)
  File "/var/lib/juju/agents/unit-grafana-agent-host-0/charm/lib/charms/tempo_coordinator_k8s/v0/charm_tracing.py", line 1064, in wrapped_function
    return callable(*args, **kwargs)  # type: ignore
  File "/var/lib/juju/agents/unit-grafana-agent-host-0/charm/./src/charm.py", line 214, in _on_cos_data_changed
    self._update_config()
  File "/var/lib/juju/agents/unit-grafana-agent-host-0/charm/lib/charms/tempo_coordinator_k8s/v0/charm_tracing.py", line 1064, in wrapped_function
    return callable(*args, **kwargs)  # type: ignore
  File "/var/lib/juju/agents/unit-grafana-agent-host-0/charm/src/grafana_agent.py", line 585, in _update_config
    config = self._generate_config()
  File "/var/lib/juju/agents/unit-grafana-agent-host-0/charm/lib/charms/tempo_coordinator_k8s/v0/charm_tracing.py", line 1064, in wrapped_function
    return callable(*args, **kwargs)  # type: ignore
  File "/var/lib/juju/agents/unit-grafana-agent-host-0/charm/src/grafana_agent.py", line 741, in _generate_config
    "logs": self._loki_config,
  File "/var/lib/juju/agents/unit-grafana-agent-host-0/charm/src/grafana_agent.py", line 1032, in _loki_config
    configs.extend(self._additional_log_configs)  # type: ignore
  File "/var/lib/juju/agents/unit-grafana-agent-host-0/charm/./src/charm.py", line 495, in _additional_log_configs
    + self._snap_plugs_logging_configs,
  File "/var/lib/juju/agents/unit-grafana-agent-host-0/charm/./src/charm.py", line 637, in _snap_plugs_logging_configs
    log_dirs = snap_yaml["slots"][endpoint.name]["source"]["read"]
KeyError: 'logs'
2025-01-27 14:10:07 ERROR juju.worker.uniter.operation runhook.go:180 hook "cos-agent-relation-joined" (via hook dispatching script: dispatch) failed: exit status 1

Additional context

No response