20240301 panther

CHANGELOG.md

@@ -62,6 +62,8 @@ New Zentral Audit events to track configuration changes.
 
 New `zentral.core.stores.backends.snowflake` store backend for [Snowflake](https://www.snowflake.com/).
 
+New `zentral.core.stores.backends.panther` store backend for [Panther](https://panther.com/)
+
 #### One more thing…
 
 🚧 Alpha release of the new UI.

docs/architecture/index.md

@@ -100,7 +100,7 @@ A brief description of the Architecture decisions, Frameworks and tools used in
 
 **Asynchronous Processing**: Zentral uses asynchronous processing techniques, such as Celery, RabbitMQ and Google PubSub to handle incoming data and alerts. This approach ensures that the platform can handle large volumes of data and respond quickly to security events.
 
-**Storage Flexibility**: the platform supports a range of different technologies to store events, including Elasticsearch, Kinesis, Splunk, Humio, OpenSearch and Azure. This flexibility enables organizations to choose the database technology that best fits their needs and requirements.
+**Storage Flexibility**: the platform supports a range of different technologies to store events, including Elasticsearch, Kinesis, Panther, Splunk, Humio, OpenSearch and Azure. This flexibility enables organizations to choose the database technology that best fits their needs and requirements.
 
 **RESTful API**: Zentral provides a RESTful API that enables developers to build custom integrations and extensions, as well as to automate tasks and workflows within the platform (e.g. reporting).
 

docs/configuration/stores.md

@@ -31,6 +31,7 @@ The python module implementing the store, as a string. Currently available:
 * `zentral.core.stores.backends.humio`
 * `zentral.core.stores.backends.kinesis`
 * `zentral.core.stores.backends.opensearch`
+* `zentral.core.stores.backends.panther`
 * `zentral.core.stores.backends.snowflake`
 * `zentral.core.stores.backends.splunk`
 * `zentral.core.stores.backends.sumo_logic`
@@ -102,55 +103,78 @@ Use `included_event_filters` instead.
 
 A list of group names. Empty by default (i.e. all users will get the links). Can be used to display the links to the events in the store to only a subset of Zentral users, if not all users have direct access to the store.
 
-## OpenSearch backend options
+## HTTP backend options
 
-Use this store if you have a managed AWS OpenSearch domain. API calls to store and retrieve the events can be authenticated using the standard AWS signatures. It is recommended to use a role attached to the EC2 instance or to the container, but an IAM key and a secret can be provided.
+### `endpoint_url`
 
-### `aws_auth`
+**MANDATORY**
+
+The URL where the Zentral events will be POSTed.
+
+For example: `https://acme.service-now.com/api/now/import/zentral_events`.
+
+### `username`
 
 **OPTIONAL**
 
-A dictionary to configure the AWS authentication. If omitted, the API calls will not be authenticated. It must contain the AWS region in the `region` key. `access_key_id` and `secret_access_key` can also be used if the default AWS authentication via instance or container profile is not set.
+Username used for Basic Authentication. If used, `password` **MUST** be set too.
 
-### Simple example
+### `password`
 
-```json
-{
-  "backend": "zentral.core.stores.backends.opensearch",
-  "frontend": true,
-  "index": "zentral-events",
-  "hosts": ["https://example-00000000000000000000000000.us-east-1.es.amazonaws.com"],
-  "kibana_discover_url": "https://example-00000000000000000000000000.us-east-1.es.amazonaws.com/_dashboards",
-  "kibana_index_pattern_uuid": "00000000-0000-0000-0000-000000000000",
-  "aws_auth": {"region": "us-east-1"}
-}
-```
+**OPTIONAL**
 
-### Full example
+Password used for Basic Authentication. If used, `username` **MUST** be set too.
 
-In this example, a separate index is setup to receive the Osquery events. You could configure it with a different [OpenSearch policy](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/ism.html) to change the retention time or the storage type. Use the index name as the key, add `included_event_filters` and `excluded_event_filters`. Set a priority to make sure that only one index will be chosen by Zentral. Finally do not forget to add a default unfiltered index with the lowest priority. A `read_index` is also required in this kind of setup. It should point to an alias that is covering all the events you want to be able to retrieve in the Zentral GUI.
+### `headers`
+
+**OPTIONAL**
+
+A string / string dictionary of extra headers to be set for the HTTP requests. The `Content-Type` header is set to `application/json` by default.
+
+**WARNING** Basic Authentication via `username` and `password` conflicts with the configuration of the `Authorization` header.
+
+### `concurrency`
+
+**OPTIONAL**
+
+**WARNING** only works if the AWS SNS/SQS queues backend is used.
+
+An integer between 1 and 20, 1 by default. The number of threads to use when posting the events. This can increase the throughput of the store worker.
+
+### Full example
 
 ```json
 {
-  "backend": "zentral.core.stores.backends.opensearch",
-  "frontend": true,
-  "batch_size": 100,
-  "indices": {
-    "zentral-osquery": {
-      "priority": 10,
-      "included_event_filters": {
-        "tags": ["osquery"]
-      }
-    },
-    "zentral-other": {
-      "priority": 1
-    }
-  },
-  "read_index": "zentral-all",
-  "hosts": ["https://example-00000000000000000000000000.us-east-1.es.amazonaws.com"],
-  "kibana_discover_url": "https://example-00000000000000000000000000.us-east-1.es.amazonaws.com/_dashboards",
-  "kibana_index_pattern_uuid": "00000000-0000-0000-0000-000000000000",
-  "aws_auth": {"region": "us-east-1"}
+    "backend": "zentral.core.stores.backends.http",
+    "endpoint_url": "https://acme.service-now.com/api/now/import/zentral_events",
+    "username": "Zentral",
+    "password": "{{ env:SERVICE_NOW_API_PASSWORD }}",
+    "verify_tls": true,
+    "included_event_filters": [{
+      "event_type": [
+        "add_machine",
+        "add_machine_os_version",
+        "remove_machine_os_version",
+        "add_machine_system_info",
+        "remove_machine_system_info",
+        "add_machine_business_unit",
+        "remove_machine_business_unit",
+        "add_machine_group",
+        "remove_machine_group",
+        "add_machine_disk",
+        "remove_machine_disk",
+        "add_machine_network_interface",
+        "remove_machine_network_interface",
+        "add_machine_osx_app_instance",
+        "remove_machine_osx_app_instance",
+        "add_machine_deb_package",
+        "remove_machine_deb_package",
+        "add_machine_program_instance",
+        "remove_machine_program_instance",
+        "add_machine_principal_user",
+        "remove_machine_principal_user"
+      ]
+    }]
 }
 ```
 
@@ -219,78 +243,83 @@ A serialization format optimized for the use with Kinesis Firehose is also avail
 }
 ```
 
-## HTTP backend options
+## OpenSearch backend options
 
-### `endpoint_url`
+Use this store if you have a managed AWS OpenSearch domain. API calls to store and retrieve the events can be authenticated using the standard AWS signatures. It is recommended to use a role attached to the EC2 instance or to the container, but an IAM key and a secret can be provided.
 
-**MANDATORY**
+### `aws_auth`
 
-The URL where the Zentral events will be POSTed.
+**OPTIONAL**
 
-For example: `https://acme.service-now.com/api/now/import/zentral_events`.
+A dictionary to configure the AWS authentication. If omitted, the API calls will not be authenticated. It must contain the AWS region in the `region` key. `access_key_id` and `secret_access_key` can also be used if the default AWS authentication via instance or container profile is not set.
 
-### `username`
+### Simple example
 
-**OPTIONAL**
+```json
+{
+  "backend": "zentral.core.stores.backends.opensearch",
+  "frontend": true,
+  "index": "zentral-events",
+  "hosts": ["https://example-00000000000000000000000000.us-east-1.es.amazonaws.com"],
+  "kibana_discover_url": "https://example-00000000000000000000000000.us-east-1.es.amazonaws.com/_dashboards",
+  "kibana_index_pattern_uuid": "00000000-0000-0000-0000-000000000000",
+  "aws_auth": {"region": "us-east-1"}
+}
+```
 
-Username used for Basic Authentication. If used, `password` **MUST** be set too.
+### Full example
 
-### `password`
+In this example, a separate index is setup to receive the Osquery events. You could configure it with a different [OpenSearch policy](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/ism.html) to change the retention time or the storage type. Use the index name as the key, add `included_event_filters` and `excluded_event_filters`. Set a priority to make sure that only one index will be chosen by Zentral. Finally do not forget to add a default unfiltered index with the lowest priority. A `read_index` is also required in this kind of setup. It should point to an alias that is covering all the events you want to be able to retrieve in the Zentral GUI.
 
-**OPTIONAL**
+```json
+{
+  "backend": "zentral.core.stores.backends.opensearch",
+  "frontend": true,
+  "batch_size": 100,
+  "indices": {
+    "zentral-osquery": {
+      "priority": 10,
+      "included_event_filters": {
+        "tags": ["osquery"]
+      }
+    },
+    "zentral-other": {
+      "priority": 1
+    }
+  },
+  "read_index": "zentral-all",
+  "hosts": ["https://example-00000000000000000000000000.us-east-1.es.amazonaws.com"],
+  "kibana_discover_url": "https://example-00000000000000000000000000.us-east-1.es.amazonaws.com/_dashboards",
+  "kibana_index_pattern_uuid": "00000000-0000-0000-0000-000000000000",
+  "aws_auth": {"region": "us-east-1"}
+}
+```
 
-Password used for Basic Authentication. If used, `username` **MUST** be set too.
+## Panther backend options
 
-### `headers`
+Zentral can send events to a Panther HTTP log source, with Bearer authentication. A custom schema must be configured in Panther – use [`schema.yaml`](https://github.com/zentralopensource/zentral/tree/main/ee/zentral/core/stores/backends/panther/schema.yaml) from the Zentral repository.
 
-**OPTIONAL**
+### `endpoint_url`
 
-A string / string dictionary of extra headers to be set for the HTTP requests. The `Content-Type` header is set to `application/json` by default.
+**MANDATORY**
 
-**WARNING** Basic Authentication via `username` and `password` conflicts with the configuration of the `Authorization` header.
+The Panther [HTTP Log Source](https://docs.panther.com/data-onboarding/data-transports/http) URL.
 
-### `concurrency`
+For example: `https://logs.example.runpanther.net/http/00000000-0000-0000-0000-000000000000`.
 
-**OPTIONAL**
+### `bearer_token`
 
-**WARNING** only works if the AWS SNS/SQS queues backend is used.
+**MANDATORY**
 
-An integer between 1 and 20, 1 by default. The number of threads to use when posting the events. This can increase the throughput of the store worker.
+The token used for Bearer Authentication.
 
-### Full example
+### Example
 
 ```json
 {
-    "backend": "zentral.core.stores.backends.http",
-    "endpoint_url": "https://acme.service-now.com/api/now/import/zentral_events",
-    "username": "Zentral",
-    "password": "{{ env:SERVICE_NOW_API_PASSWORD }}",
-    "verify_tls": true,
-    "included_event_filters": [{
-      "event_type": [
-        "add_machine",
-        "add_machine_os_version",
-        "remove_machine_os_version",
-        "add_machine_system_info",
-        "remove_machine_system_info",
-        "add_machine_business_unit",
-        "remove_machine_business_unit",
-        "add_machine_group",
-        "remove_machine_group",
-        "add_machine_disk",
-        "remove_machine_disk",
-        "add_machine_network_interface",
-        "remove_machine_network_interface",
-        "add_machine_osx_app_instance",
-        "remove_machine_osx_app_instance",
-        "add_machine_deb_package",
-        "remove_machine_deb_package",
-        "add_machine_program_instance",
-        "remove_machine_program_instance",
-        "add_machine_principal_user",
-        "remove_machine_principal_user"
-      ]
-    }]
+    "backend": "zentral.core.stores.backends.panther",
+    "endpoint_url": "https://logs.example.runpanther.net/http/00000000-0000-0000-0000-000000000000",
+    "bearer_token": "00000000-0000-0000-0000-000000000000",
 }
 ```
 

docs/index.md

@@ -39,6 +39,7 @@ You can deploy it on your machine with [Docker](./deployment/docker-compose), or
 * [DataDog](https://www.datadoghq.com/)
 * [Elasticsearch](https://www.elastic.co/products/elasticsearch)
 * [OpenSearch](https://opensearch.org/)
+* [Panther](https://panther.com/)
 * [Snowflake](https://www.snowflake.com/en/)
 * [Splunk](https://www.splunk.com/en_us/software/features-comparison-chart.html)
 * [sumo logic](https://www.sumologic.com/)

ee/zentral/core/stores/backends/datadog.py

@@ -1,5 +1,5 @@
 from datetime import datetime
-import json
+from kombu.utils import json
 import logging
 import re
 import requests

ee/zentral/core/stores/backends/panther/__init__.py

@@ -0,0 +1,56 @@
+import gzip
+from kombu.utils import json
+import requests
+from zentral.core.exceptions import ImproperlyConfigured
+from zentral.core.stores.backends.base import BaseEventStore
+
+
+class EventStore(BaseEventStore):
+    max_batch_size = 100
+
+    def __init__(self, config_d):
+        super().__init__(config_d)
+        endpoint_url = config_d.get("endpoint_url")
+        if endpoint_url:
+            self.endpoint_url = endpoint_url
+        else:
+            raise ImproperlyConfigured("Missing or empty endpoint_url")
+        bearer_token = config_d.get("bearer_token")
+        if not bearer_token:
+            raise ImproperlyConfigured("Missing or empty bearer_token")
+        self.session = requests.Session()
+        self.session.headers.update({
+            'Authorization': f'Bearer {bearer_token}',
+            'Content-Encoding': 'gzip'
+        })
+
+    def _serialize_event(self, event):
+        if not isinstance(event, dict):
+            event_d = event.serialize()
+        else:
+            event_d = event
+        p_event_d = event_d.pop("_zentral")
+        p_event_d["payload"] = event_d
+        return p_event_d
+
+    def store(self, event):
+        p_event_d = self._serialize_event(event)
+        data = gzip.compress(json.dumps(p_event_d).encode("utf-8"))
+        response = self.session.post(self.endpoint_url, data=data)
+        response.raise_for_status()
+
+    def bulk_store(self, events):
+        if self.batch_size < 2:
+            raise RuntimeError("bulk_store is not available when batch_size < 2")
+        event_keys = []
+        payload = b""
+        for event in events:
+            serialized_event = self._serialize_event(event)
+            event_keys.append((serialized_event["id"], serialized_event["index"]))
+            if payload:
+                payload += b"\n"
+            payload += json.dumps(serialized_event).encode("utf-8")
+        data = gzip.compress(payload)
+        response = self.session.post(self.endpoint_url, data=data)
+        response.raise_for_status()
+        return event_keys