python client logs integration

integrations/observability/python_client/README.md

@@ -0,0 +1,77 @@
+# Python Client Integration
+The next integration contains instructions and tutorial of setting up python opensearch client and logging applicative telemetry into opensearch.
+
+## Logging with OpenSearch in Python: 
+
+Logging is an important aspect of software development,OpenSearch, is a robust and scalable solution, stands out for storing and analyzing logs efficiently.
+This guide walks you through integrating OpenSearch as a storage and analytics into component used in your Python project for effective logging.
+
+### Install Python Libraries
+Install the OpenSearch Python client to interact with OpenSearch:
+
+```bash
+pip install opensearch-py
+```
+See additional documentation [here](https://opensearch.org/docs/latest/clients/python-low-level/).
+
+## Integrating OpenSearch with Your Python Project
+
+### Step 1: Import the OpenSearch Client
+In your Python project, import the necessary module:
+
+```python
+from opensearchpy import OpenSearch
+```
+
+### Step 2: Establish a Connection
+Create a connection to your OpenSearch cluster:
+
+```python
+os = OpenSearch([{'host': 'opensearch_host', 'port': 9200}])
+```
+
+### Step 3: Indexing Logs
+Index your logs into OpenSearch:
+
+```python
+log_entry = {
+    'timestamp': '2024-02-05T12:00:00',
+    'level': 'info',
+    'message': 'Your log message here.',
+    'source': 'your_python_project'
+}
+
+index_name = 'index_name'
+
+os.index(index=index_name, body=log_entry)
+```
+
+### Step 4: Querying Logs
+Retrieve logs using OpenSearch's powerful search capabilities:
+
+```python
+query = {
+    'query': {
+        'match': {'level': 'error'}
+    }
+}
+
+result = os.search(index=index_name, body=query)
+print(result)
+```
+
+## Best Practices for Effective Logging
+
+1. **Descriptive Log Messages**: Include clear and detailed information.
+2. **Appropriate Log Levels**: Use different levels (INFO, DEBUG, ERROR) to categorize log messages.
+3. **Timestamps**: Always include timestamps for chronological analysis.
+4. **Contextual Information**: Add details like module, function, or user IDs.
+5. **Avoid Redundant Logging**: Balance between sufficient information and avoiding overload.
+6. **Secure Sensitive Information**: Do not log sensitive data in plain text.
+7. **Structured Logging**: Use JSON for consistent log formats.
+
+## Advanced Features of OpenSearch
+
+- **Index Patterns and Mappings**: Optimize log data structure for better analysis and retrieval.
+- **Visualization with Dashboards (Kibana)**: Create interactive dashboards for real-time log insights.
+

integrations/observability/python_client/assets/tutorial-1.0.0.ndjson

@@ -0,0 +1,4 @@
+{"attributes":{"description":"Python client application logs tutorial for ingesting applicative log data and building a dashboard ","kibanaSavedObjectMeta":{"searchSourceJSON":"{\"query\":{\"query\":\"\",\"language\":\"kuery\"},\"filter\":[]}"},"title":"python client getting started","uiStateJSON":"{}","version":1,"visState":"{\"title\":\"python client getting started\",\"type\":\"markdown\",\"aggs\":[],\"params\":{\"fontSize\":12,\"openLinksInNewTab\":false,\"markdown\":\"# Python Client Integration\\nThe next integration contains instructions and tutorial of setting up python opensearch client and logging applicative telemetry into opensearch.\\n\\n## Logging with OpenSearch in Python: \\n\\nLogging is an important aspect of software development,OpenSearch, is a robust and scalable solution, stands out for storing and analyzing logs efficiently.\\nThis guide walks you through integrating OpenSearch as a storage and analytics into component used in your Python project for effective logging.\\n\\n### Install Python Libraries\\nInstall the OpenSearch Python client to interact with OpenSearch:\\n\\n```bash\\npip install opensearch-py\\n```\\nSee additional documentation [here](https://opensearch.org/docs/latest/clients/python-low-level/).\\n\\n## Integrating OpenSearch with Your Python Project\\n\\n### Step 1: Import the OpenSearch Client\\nIn your Python project, import the necessary module:\\n\\n```python\\nfrom opensearchpy import OpenSearch\\n```\\n\\n### Step 2: Establish a Connection\\nCreate a connection to your OpenSearch cluster:\\n\\n```python\\nos = OpenSearch([{'host': 'opensearch_host', 'port': 9200}])\\n```\\n\\n### Step 3: Indexing Logs\\nIndex your logs into OpenSearch:\\n\\n```python\\nlog_entry = {\\n    'timestamp': '2024-02-05T12:00:00',\\n    'level': 'info',\\n    'message': 'Your log message here.',\\n    'source': 'your_python_project'\\n}\\n\\nindex_name = 'index_name'\\n\\nos.index(index=index_name, body=log_entry)\\n```\\n\\n### Step 4: Querying Logs\\nRetrieve logs using OpenSearch's powerful search capabilities:\\n\\n```python\\nquery = {\\n    'query': {\\n        'match': {'level': 'error'}\\n    }\\n}\\n\\nresult = os.search(index=index_name, body=query)\\nprint(result)\\n```\\n\\n## Best Practices for Effective Logging\\n\\n1. **Descriptive Log Messages**: Include clear and detailed information.\\n2. **Appropriate Log Levels**: Use different levels (INFO, DEBUG, ERROR) to categorize log messages.\\n3. **Timestamps**: Always include timestamps for chronological analysis.\\n4. **Contextual Information**: Add details like module, function, or user IDs.\\n5. **Avoid Redundant Logging**: Balance between sufficient information and avoiding overload.\\n6. **Secure Sensitive Information**: Do not log sensitive data in plain text.\\n7. **Structured Logging**: Use JSON for consistent log formats.\\n\\n## Advanced Features of OpenSearch\\n\\n- **Index Patterns and Mappings**: Optimize log data structure for better analysis and retrieval.\\n- **Visualization with Dashboards**: Create interactive dashboards for real-time log insights.\\n\\n# How to build an Application Monitor Dashboard\\nBased on the ingested logs, lets review the process of generating an informative monitor dashboard for the applicative logs:\\n\\n## Step-by-Step Tutorial: Creating an OpenSearch Dashboard for Application Logs\\n\\n### 1. Log in to opensearch dashboards\\n- Navigate to OpenSearch Dashboards.\\n- Log in and verify the logs index was created and contains logs data\\n- Go to Discover tab, select the index name and view the data\\n\\n### 2. Create an Index Pattern\\n-  Go to 'Management' > 'Index Patterns'.\\n-  Click 'Create Index Pattern' and enter the pattern (e.g., logs-*).\\n-  Select the timestamp field (e.g., @timestamp) for time-based data.\\n-  Save the index pattern.\\n\\n### 3. Build Log Queries\\n- Go to the 'Discover' tab.\\n- Use the search bar to filter logs, e.g., `log_level:ERROR` to find all error logs.\\n- For advanced filtering, utilize the Dashboard Query Language (DQL).\\n\\n### 4. Save Your Query\\n- After refining your query, save it by clicking on the 'Save' button in the 'Discover' tab.\\n- Name your saved query for easy reference.\\n\\n### 5. Create Visualizations\\n- Go to 'Visualize' > 'Create Visualization'.\\n- Select the type of visualization you want to create (e.g., bar chart, pie chart).\\n- Choose your saved query as the data source.\\n\\n### 6. Add Buckets for Data Aggregation\\n- In the visualization settings, add buckets to aggregate your data. For example:\\n    - Use 'Date Histogram' for the X-axis to display logs over time.\\n    - Add other metrics or aggregations as needed.\\n\\n### 7. Split Series for Detailed Insights\\n- Add another bucket to split data by specific fields, such as `service.name` or `host.name`.\\n- This will allow you to see log distributions across different services or hosts.\\n\\n### 8. Customize Visualization\\n- Customize the visualization with labels, colors, and other settings to make it more informative and visually appealing.\\n\\n### 9. Save and Add to Dashboard\\n- Save the visualization with a descriptive name.\\n- Navigate to the 'Dashboard' tab and create a new dashboard.\\n- Add your saved visualizations to the dashboard by selecting them from the list.\\n\\n### 10. Finalize and Share\\n- Arrange the visualizations on the dashboard as desired.\\n- Save the dashboard with a meaningful name.\\n- Share the dashboard with your team by generating a shareable link or embedding it in your application.\\n\\n### Tips for Effective Dashboards\\n- Use different types of visualizations to present various aspects of your log data.\\n- Regularly update the time filter to ensure you're viewing the most recent logs.\\n- Take advantage of OpenSearch Dashboards' interactive features, such as drill-downs and filters, for deeper analysis.\\n\\n\"}}"},"id":"8d8e5860-3fc8-11ef-a406-af6eb3c84d77","migrationVersion":{"visualization":"7.10.0"},"references":[],"type":"visualization","updated_at":"2024-07-11T21:00:07.270Z","version":"WzM5LDVd"}
+{"attributes":{"description":"Python client application logs tutorial for ingesting applicative log data and building a dashboard ","hits":0,"kibanaSavedObjectMeta":{"searchSourceJSON":"{\"query\":{\"language\":\"kuery\",\"query\":\"\"},\"filter\":[]}"},"optionsJSON":"{\"hidePanelTitles\":false,\"useMargins\":true}","panelsJSON":"[{\"version\":\"2.15.0\",\"gridData\":{\"x\":0,\"y\":0,\"w\":24,\"h\":15,\"i\":\"03db4608-6c38-4ea1-b56b-00981b5c0956\"},\"panelIndex\":\"03db4608-6c38-4ea1-b56b-00981b5c0956\",\"embeddableConfig\":{},\"panelRefName\":\"panel_0\"}]","timeRestore":false,"title":"Python client application logs tutorial dashboard","version":1},"id":"python-getting-started-tutorial-1.0.0","migrationVersion":{"dashboard":"7.9.3"},"references":[{"id":"8d8e5860-3fc8-11ef-a406-af6eb3c84d77","name":"panel_0","type":"visualization"}],"type":"dashboard","updated_at":"2024-07-11T21:00:22.748Z","version":"WzQwLDVd"}
+{"attributes":{"fields":"[{\"count\":0,\"name\":\"_id\",\"type\":\"string\",\"esTypes\":[\"_id\"],\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":false},{\"count\":0,\"name\":\"_index\",\"type\":\"string\",\"esTypes\":[\"_index\"],\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":false},{\"count\":0,\"name\":\"_score\",\"type\":\"number\",\"scripted\":false,\"searchable\":false,\"aggregatable\":false,\"readFromDocValues\":false},{\"count\":0,\"name\":\"_source\",\"type\":\"_source\",\"esTypes\":[\"_source\"],\"scripted\":false,\"searchable\":false,\"aggregatable\":false,\"readFromDocValues\":false},{\"count\":0,\"name\":\"_type\",\"type\":\"string\",\"scripted\":false,\"searchable\":false,\"aggregatable\":false,\"readFromDocValues\":false},{\"count\":0,\"name\":\"error_code\",\"type\":\"string\",\"esTypes\":[\"keyword\"],\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"count\":0,\"name\":\"function\",\"type\":\"string\",\"esTypes\":[\"keyword\"],\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"count\":0,\"name\":\"level\",\"type\":\"string\",\"esTypes\":[\"keyword\"],\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"count\":0,\"name\":\"message\",\"type\":\"string\",\"esTypes\":[\"text\"],\"scripted\":false,\"searchable\":true,\"aggregatable\":false,\"readFromDocValues\":false},{\"count\":0,\"name\":\"module\",\"type\":\"string\",\"esTypes\":[\"keyword\"],\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"count\":0,\"name\":\"source\",\"type\":\"string\",\"esTypes\":[\"keyword\"],\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"count\":0,\"name\":\"timestamp\",\"type\":\"date\",\"esTypes\":[\"date\"],\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"count\":0,\"name\":\"user_id\",\"type\":\"string\",\"esTypes\":[\"keyword\"],\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true}]","timeFieldName":"timestamp","title":"applicative_logs*"},"id":"15746330-447b-11ef-99f2-2b48f7e65060","migrationVersion":{"index-pattern":"7.6.0"},"references":[],"type":"index-pattern","updated_at":"2024-07-17T20:28:10.338Z","version":"WzUsMV0="}
+{"exportedCount":3,"missingRefCount":0,"missingReferences":[]}

integrations/observability/python_client/data/sample.json

@@ -0,0 +1,52 @@
+[
+  {
+    "timestamp": "2024-02-05T12:00:00",
+    "level": "info",
+    "message": "Application started successfully.",
+    "source": "your_python_project",
+    "module": "main",
+    "function": "start_app"
+  },
+  {
+    "timestamp": "2024-02-05T12:01:00",
+    "level": "debug",
+    "message": "Connecting to database.",
+    "source": "your_python_project",
+    "module": "database",
+    "function": "connect"
+  },
+  {
+    "timestamp": "2024-02-05T12:02:00",
+    "level": "error",
+    "message": "Database connection failed.",
+    "source": "your_python_project",
+    "module": "database",
+    "function": "connect",
+    "error_code": "DB_CONN_FAIL"
+  },
+  {
+    "timestamp": "2024-02-05T12:03:00",
+    "level": "info",
+    "message": "Retrying database connection.",
+    "source": "your_python_project",
+    "module": "database",
+    "function": "connect"
+  },
+  {
+    "timestamp": "2024-02-05T12:04:00",
+    "level": "info",
+    "message": "Database connection established.",
+    "source": "your_python_project",
+    "module": "database",
+    "function": "connect"
+  },
+  {
+    "timestamp": "2024-02-05T12:05:00",
+    "level": "info",
+    "message": "Processing user request.",
+    "source": "your_python_project",
+    "module": "request_handler",
+    "function": "process_request",
+    "user_id": "user_123"
+  }
+]

integrations/observability/python_client/getting-started/.env

@@ -0,0 +1,14 @@
+# OpenSearch version
+OPENSEARCH_VERSION=2.15.0
+OPENSEARCH_ADMIN_PASSWORD=my_%New%_passW0rd!@#
+OPENSEARCH_INITIAL_ADMIN_PASSWORD=my_%New%_passW0rd!@#
+
+# OpenSearch Node1
+OPENSEARCH_PORT=9200
+OPENSEARCH_HOST=opensearch
+OPENSEARCH_ADDR=${OPENSEARCH_HOST}:${OPENSEARCH_PORT}
+
+# OpenSearch Dashboard
+OPENSEARCH_DASHBOARD_PORT=5601
+OPENSEARCH_DASHBOARD_HOST=opensearch-dashboards
+OPENSEARCH_DASHBOARD_ADDR=${OPENSEARCH_DASHBOARD_HOST}:${OPENSEARCH_DASHBOARD_PORT}

integrations/observability/python_client/getting-started/Getting-Started.md

@@ -0,0 +1,134 @@
+# Python Client Integration
+The next integration contains instructions and tutorial of setting up python opensearch client and logging applicative telemetry into opensearch.
+
+## Logging with OpenSearch in Python: 
+
+Logging is an important aspect of software development,OpenSearch, is a robust and scalable solution, stands out for storing and analyzing logs efficiently.
+This guide walks you through integrating OpenSearch as a storage and analytics into component used in your Python project for effective logging.
+
+### Install Python Libraries
+Install the OpenSearch Python client to interact with OpenSearch:
+
+```bash
+pip install opensearch-py
+```
+See additional documentation [here](https://opensearch.org/docs/latest/clients/python-low-level/).
+
+## Integrating OpenSearch with Your Python Project
+
+### Step 1: Import the OpenSearch Client
+In your Python project, import the necessary module:
+
+```python
+from opensearchpy import OpenSearch
+```
+
+### Step 2: Establish a Connection
+Create a connection to your OpenSearch cluster:
+
+```python
+os = OpenSearch([{'host': 'opensearch_host', 'port': 9200}])
+```
+
+### Step 3: Indexing Logs
+Index your logs into OpenSearch:
+
+```python
+log_entry = {
+    'timestamp': '2024-02-05T12:00:00',
+    'level': 'info',
+    'message': 'Your log message here.',
+    'source': 'your_python_project'
+}
+
+index_name = 'index_name'
+
+os.index(index=index_name, body=log_entry)
+```
+
+### Step 4: Querying Logs
+Retrieve logs using OpenSearch's powerful search capabilities:
+
+```python
+query = {
+    'query': {
+        'match': {'level': 'error'}
+    }
+}
+
+result = os.search(index=index_name, body=query)
+print(result)
+```
+
+## Best Practices for Effective Logging
+
+1. **Descriptive Log Messages**: Include clear and detailed information.
+2. **Appropriate Log Levels**: Use different levels (INFO, DEBUG, ERROR) to categorize log messages.
+3. **Timestamps**: Always include timestamps for chronological analysis.
+4. **Contextual Information**: Add details like module, function, or user IDs.
+5. **Avoid Redundant Logging**: Balance between sufficient information and avoiding overload.
+6. **Secure Sensitive Information**: Do not log sensitive data in plain text.
+7. **Structured Logging**: Use JSON for consistent log formats.
+
+## Advanced Features of OpenSearch
+
+- **Index Patterns and Mappings**: Optimize log data structure for better analysis and retrieval.
+- **Visualization with Dashboards**: Create interactive dashboards for real-time log insights.
+
+# How to build an Application Monitor Dashboard
+Based on the ingested logs, lets review the process of generating an informative monitor dashboard for the applicative logs:
+
+## Step-by-Step Tutorial: Creating an OpenSearch Dashboard for Application Logs
+
+### 1. Log in to opensearch dashboards
+- Navigate to OpenSearch Dashboards.
+- Log in and verify the logs index was created and contains logs data
+- Go to Discover tab, select the index name and view the data
+
+### 2. Create an Index Pattern
+-  Go to 'Management' > 'Index Patterns'.
+-  Click 'Create Index Pattern' and enter the pattern (e.g., logs-*).
+-  Select the timestamp field (e.g., @timestamp) for time-based data.
+-  Save the index pattern.
+
+### 3. Build Log Queries
+- Go to the 'Discover' tab.
+- Use the search bar to filter logs, e.g., `log_level:ERROR` to find all error logs.
+- For advanced filtering, utilize the Dashboard Query Language (DQL).
+
+### 4. Save Your Query
+- After refining your query, save it by clicking on the 'Save' button in the 'Discover' tab.
+- Name your saved query for easy reference.
+
+### 5. Create Visualizations
+- Go to 'Visualize' > 'Create Visualization'.
+- Select the type of visualization you want to create (e.g., bar chart, pie chart).
+- Choose your saved query as the data source.
+
+### 6. Add Buckets for Data Aggregation
+- In the visualization settings, add buckets to aggregate your data. For example:
+    - Use 'Date Histogram' for the X-axis to display logs over time.
+    - Add other metrics or aggregations as needed.
+
+### 7. Split Series for Detailed Insights
+- Add another bucket to split data by specific fields, such as `service.name` or `host.name`.
+- This will allow you to see log distributions across different services or hosts.
+
+### 8. Customize Visualization
+- Customize the visualization with labels, colors, and other settings to make it more informative and visually appealing.
+
+### 9. Save and Add to Dashboard
+- Save the visualization with a descriptive name.
+- Navigate to the 'Dashboard' tab and create a new dashboard.
+- Add your saved visualizations to the dashboard by selecting them from the list.
+
+### 10. Finalize and Share
+- Arrange the visualizations on the dashboard as desired.
+- Save the dashboard with a meaningful name.
+- Share the dashboard with your team by generating a shareable link or embedding it in your application.
+
+### Tips for Effective Dashboards
+- Use different types of visualizations to present various aspects of your log data.
+- Regularly update the time filter to ensure you're viewing the most recent logs.
+- Take advantage of OpenSearch Dashboards' interactive features, such as drill-downs and filters, for deeper analysis.
+