Opni Ingest Pipeline
The Opni ingest pipeline normalizes any log messages coming to the central Opni cluster to a standard format. The normalized logs can then be utilized by other Opni components including Opensearch and AIOps services. It is deployed as an OpenSearch ingest pipeline and implemented as a plugin of OpenSearch.
- Java
- the logging processor
- Normalize fields that contain log messages to a fixed field
log. - Normalize time or timestamp fields to a fixed field
time. - Assign random ID to each document.
- Normalize fields that contain log messages to a fixed field
- the AIOps processor
- Normalize a few key fields utilized by AIOps services.
- log_type: workload/controlplane/rancher/longhorn
- kubernetes_component
- pod_name
- namespace_name
- Push document to Nats.
- Normalize a few key fields utilized by AIOps services.
| Component | Type | Description |
|---|---|---|
| Opni agent | Opni agent | The data received by the Opensearch ingest pipeline comes from clusters with Opni agent installed and logging capability enabled. |
| Component | Type | Description |
|---|---|---|
| Opensearach | Database | |
| Nats | Nats subject | push logs to the Nats subject raw_logs . |
- Doesn't support batch process, only process 1 document at a time in each thread.
- (Known issue) Performance issue -- processing speed too slow -- caused by the random ID assignment.
- Unit Test
- Covered in the codebase, but more unit-test cases needed. Unit-test is triggered automatically every time the plugin binary is built.
- Manual Local Test
- Dev and test locally following the docs here: https://github.com/tybalex/opni-ingest-plugin/tree/main/AIOps-OpniPreProcessor#usage
- Integration Test
- Follow the docs to manually build the project and install in testing environment.
- E2e Test
- Covered by the GitHub workflow: https://github.com/tybalex/opni-ingest-plugin/tree/main/.github/workflows