diff --git a/Neural Networks/Model Performance Monitoring System/## Readme.md b/Neural Networks/Model Performance Monitoring System/## Readme.md new file mode 100644 index 00000000..b54af3b6 --- /dev/null +++ b/Neural Networks/Model Performance Monitoring System/## Readme.md @@ -0,0 +1,46 @@ +## Model Performance Monitoring System + +### Introduction + +The Model Performance Monitoring System is designed to ensure the continuous accuracy and reliability of deployed machine learning models. By monitoring key performance metrics and promptly alerting administrators of any significant changes or degradations in performance, this system helps maintain the quality of service provided by machine learning applications. + +### Motivation + +The motivation behind this system is rooted in the need to maintain model integrity and effectiveness in production environments. Ensuring that deployed models consistently meet performance standards is critical for providing reliable and accurate services to users. + +### Key Components + +**1. Model Performance Metrics** + +The system monitors the following key performance metrics: + +Accuracy +Precision +Recall +F1-score +Area Under the ROC Curve (AUC-ROC) +Root Mean Squared Error (RMSE) +Mean Absolute Error (MAE) + +**2. Alerting Mechanism** + +Administrators are alerted via email when the performance of a deployed model falls below acceptable thresholds for any of the monitored metrics. This immediate notification enables administrators to take timely actions to address issues and maintain model quality. + + + +### Requirements + +To use the Model Performance Monitoring System, the following requirements must be met: +- Python 3.x +- Libraries: pandas, scikit-learn, matplotlib, smtplib +- Installation and Configuration +- Clone the repository to your local machine. +- Install the required Python libraries using `pip install -r requirements.txt`. +Update the email credentials (sender_email, receiver_email, password) in Model_Performance_Monitoring_System.ipynb with your email details. +- Adjust the performance thresholds (threshold) in the code to align with your acceptable performance criteria. + +### Usage +- Load your deployed machine learning model and data. +- Periodically run the Model_Performance_Monitoring_System.ipynb script to evaluate model performance and send alerts if necessary. +Example +- **An example dataset and code implementation are provided in the Model_Performance_Monitoring_System.ipynb script for reference. Customize the script according to your specific model and data.** \ No newline at end of file diff --git a/Neural Networks/Model Performance Monitoring System/Model_Performance_Monitoring_System.ipynb b/Neural Networks/Model Performance Monitoring System/Model_Performance_Monitoring_System.ipynb new file mode 100644 index 00000000..4fb98f81 --- /dev/null +++ b/Neural Networks/Model Performance Monitoring System/Model_Performance_Monitoring_System.ipynb @@ -0,0 +1,274 @@ +{ + "nbformat": 4, + "nbformat_minor": 0, + "metadata": { + "colab": { + "provenance": [] + }, + "kernelspec": { + "name": "python3", + "display_name": "Python 3" + }, + "language_info": { + "name": "python" + } + }, + "cells": [ + { + "cell_type": "markdown", + "source": [ + "### Installing Necessary Libraries" + ], + "metadata": { + "id": "PZTfQJa_-k9Q" + } + }, + { + "cell_type": "code", + "execution_count": 10, + "metadata": { + "id": "7ZO1TbA-9g3e" + }, + "outputs": [], + "source": [ + "import pandas as pd\n", + "from sklearn.datasets import load_iris\n", + "from sklearn.model_selection import train_test_split\n", + "from sklearn.linear_model import LogisticRegression\n", + "from sklearn.metrics import accuracy_score\n", + "import smtplib\n", + "from email.mime.text import MIMEText\n", + "from email.mime.multipart import MIMEMultipart\n", + "import matplotlib.pyplot as plt" + ] + }, + { + "cell_type": "markdown", + "source": [ + "### Loading the Dataset" + ], + "metadata": { + "id": "BECLF8e_-uoq" + } + }, + { + "cell_type": "code", + "source": [ + "# Load the Iris dataset\n", + "iris = load_iris()\n", + "iris_df = pd.DataFrame(data=iris.data, columns=iris.feature_names)\n", + "iris_df['target'] = iris.target\n", + "\n", + "# Split the dataset into features and target\n", + "X = iris_df.drop('target', axis=1)\n", + "y = iris_df['target']\n", + "\n", + "# Split data into training and testing sets\n", + "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)" + ], + "metadata": { + "id": "s_Jytk379kUi" + }, + "execution_count": 11, + "outputs": [] + }, + { + "cell_type": "markdown", + "source": [ + "### Training the Dataset" + ], + "metadata": { + "id": "tEPuPWqv-1It" + } + }, + { + "cell_type": "code", + "source": [ + "# Train a logistic regression model\n", + "model = LogisticRegression()\n", + "model.fit(X_train, y_train)\n", + "\n", + "# Make predictions on the test set\n", + "y_pred = model.predict(X_test)" + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "collapsed": true, + "id": "HrsRNshX9wso", + "outputId": "be549de0-dcca-4316-b28f-2a08933991e2" + }, + "execution_count": 12, + "outputs": [ + { + "output_type": "stream", + "name": "stderr", + "text": [ + "/usr/local/lib/python3.10/dist-packages/sklearn/linear_model/_logistic.py:458: ConvergenceWarning: lbfgs failed to converge (status=1):\n", + "STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.\n", + "\n", + "Increase the number of iterations (max_iter) or scale the data as shown in:\n", + " https://scikit-learn.org/stable/modules/preprocessing.html\n", + "Please also refer to the documentation for alternative solver options:\n", + " https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression\n", + " n_iter_i = _check_optimize_result(\n" + ] + } + ] + }, + { + "cell_type": "markdown", + "source": [ + "### Calculating Accuracy" + ], + "metadata": { + "id": "sGhW_Rjd-6Di" + } + }, + { + "cell_type": "code", + "source": [ + "# Calculate accuracy\n", + "accuracy = accuracy_score(y_test, y_pred)\n", + "print(f\"Accuracy: {accuracy}\")" + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "614pOywp93nR", + "outputId": "c49ba69f-26ae-4319-b4c9-78ce5fe80f46" + }, + "execution_count": 13, + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": [ + "Accuracy: 1.0\n" + ] + } + ] + }, + { + "cell_type": "markdown", + "source": [ + "### Function to send email alert if accuracy falls below threshold" + ], + "metadata": { + "id": "ZOvpPn3w-9c7" + } + }, + { + "cell_type": "code", + "source": [ + "# Function to send email alert if accuracy falls below threshold\n", + "def send_alert(accuracy, threshold=0.8):\n", + " if accuracy < threshold:\n", + " sender_email = \"your_email@gmail.com\"\n", + " receiver_email = \"admin_email@gmail.com\"\n", + " password = \"your_email_password\"\n", + "\n", + " message = MIMEMultipart()\n", + " message['From'] = sender_email\n", + " message['To'] = receiver_email\n", + " message['Subject'] = \"Model Performance Alert\"\n", + "\n", + " body = f\"\"\"\n", + " Model accuracy has degraded: {accuracy}\n", + " \"\"\"\n", + "\n", + " message.attach(MIMEText(body, 'plain'))\n", + "\n", + " with smtplib.SMTP_SSL('smtp.gmail.com', 465) as server:\n", + " server.login(sender_email, password)\n", + " server.sendmail(sender_email, receiver_email, message.as_string())\n", + "\n", + "# Send alert if accuracy falls below threshold\n", + "threshold = 0.8\n", + "send_alert(accuracy, threshold)" + ], + "metadata": { + "id": "cqk2MSFH9_Mu" + }, + "execution_count": 14, + "outputs": [] + }, + { + "cell_type": "code", + "source": [ + "from sklearn.metrics import classification_report,confusion_matrix\n", + "print(\"Performance Metrics: \")\n", + "y_pred = (y_pred > 0.5)\n", + "metrics=classification_report (y_test,y_pred)\n", + "print (metrics)\n", + "cm=confusion_matrix(y_pred,y_pred)\n", + "print(cm)\n", + "plt.imshow(cm, cmap='binary')" + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 802 + }, + "id": "yfkZi8HT_-nQ", + "outputId": "64b64605-815e-4d60-d45c-d5f05a1aa2ab" + }, + "execution_count": 29, + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": [ + "Performance Metrics: \n", + " precision recall f1-score support\n", + "\n", + " 0 1.00 1.00 1.00 10\n", + " 1 0.45 1.00 0.62 9\n", + " 2 0.00 0.00 0.00 11\n", + "\n", + " accuracy 0.63 30\n", + " macro avg 0.48 0.67 0.54 30\n", + "weighted avg 0.47 0.63 0.52 30\n", + "\n", + "[[10 0]\n", + " [ 0 20]]\n" + ] + }, + { + "output_type": "stream", + "name": "stderr", + "text": [ + "/usr/local/lib/python3.10/dist-packages/sklearn/metrics/_classification.py:1344: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in labels with no predicted samples. Use `zero_division` parameter to control this behavior.\n", + " Support beyond term:`binary` targets is achieved by treating :term:`multiclass`\n", + "/usr/local/lib/python3.10/dist-packages/sklearn/metrics/_classification.py:1344: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in labels with no predicted samples. Use `zero_division` parameter to control this behavior.\n", + " Support beyond term:`binary` targets is achieved by treating :term:`multiclass`\n", + "/usr/local/lib/python3.10/dist-packages/sklearn/metrics/_classification.py:1344: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in labels with no predicted samples. Use `zero_division` parameter to control this behavior.\n", + " Support beyond term:`binary` targets is achieved by treating :term:`multiclass`\n" + ] + }, + { + "output_type": "execute_result", + "data": { + "text/plain": [ + "" + ] + }, + "metadata": {}, + "execution_count": 29 + }, + { + "output_type": "display_data", + "data": { + "text/plain": [ + "
" + ], + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAcgAAAGiCAYAAABjzlbWAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAAAt4UlEQVR4nO3de1zUdb7H8TegDNo6oIsy0OLdNE3FdCHcWm0lwTwePds5aVkiD8OT3cNM2ZP33bzkMU/F5mZeT6bmHtPaXNIo6lSkrZdNyzziUl4HbzEjWKjwPX/0cLZZvqAgAyav5+Pxe8h85/P7zuf3e8C8nZnfb35BxhgjAADgJ7i+GwAA4EpEQAIAYEFAAgBgQUACAGBBQAIAYEFAAgBgQUACAGBBQAIAYEFAAgBgQUACAGARsIA8deqURo4cKafTqYiICI0ZM0bFxcVVrtO/f38FBQX5Lffff79fzYEDBzR48GA1bdpUrVq10oQJE3T+/PlAbQYAoIFqFKiJR44cqaNHj2rz5s06d+6c0tLSNHbsWL366qtVrpeenq4ZM2b4bjdt2tT3c1lZmQYPHiyXy6WPP/5YR48e1ahRo9S4cWM9/fTTgdoUAEADFBSILyvfs2ePunbtqk8//VR9+vSRJGVnZ+v222/XoUOHFBMTY12vf//+iouL04IFC6z3//nPf9Y//dM/6ciRI4qKipIkLVy4UBMnTtTx48cVGhpa25sCAGigAvIKMi8vTxEREb5wlKSkpCQFBwdry5Yt+pd/+ZdK1125cqVeeeUVuVwuDRkyRJMnT/a9iszLy1P37t194ShJycnJGjdunD7//HP16tXLOmdpaalKS0t9t8vLy3Xq1Cn99Kc/VVBQ0OVuLgCgjhljdPr0acXExCg4ODCfFgYkIN1ut1q1auX/QI0aqUWLFnK73ZWud/fdd6tNmzaKiYnRZ599pokTJ2rv3r1at26db94fhqMk3+2q5p01a5amT59e080BAFyhDh48qJ/97GcBmbtaATlp0iTNmTOnypo9e/bUuJmxY8f6fu7evbuio6M1YMAA7d+/Xx06dKjxvJmZmcrIyPDd9ng8at26tR5//HE5HI4azwtcyTIzM+u7BSBgvF6vYmNj1axZs4A9RrUCcvz48Ro9enSVNe3bt5fL5dKxY8f8xs+fP69Tp07J5XJd8uMlJCRIkvLz89WhQwe5XC5t3brVr6awsFCSqpzX4XBYg7CyceBq4HQ667sFIOAC+TFZtQKyZcuWatmy5UXrEhMTVVRUpG3btql3796SpHfffVfl5eW+0LsUO3fulCRFR0f75v3d736nY8eO+d7C3bx5s5xOp7p27VqdTQEAoEoB+WTz+uuvV0pKitLT07V161Z99NFHeuihhzRixAjfEayHDx9Wly5dfK8I9+/fr5kzZ2rbtm366quv9MYbb2jUqFH65S9/qR49ekiSBg4cqK5du+ree+/VX//6V7399tt66qmn9OCDD/JKEABQqwL2RQErV65Uly5dNGDAAN1+++26+eab9dJLL/nuP3funPbu3aszZ85IkkJDQ/XOO+9o4MCB6tKli8aPH6877rhDb775pm+dkJAQ/elPf1JISIgSExN1zz33aNSoUX7nTQIAUBsC9kUBLVq0qPJLAdq2basfnoIZGxur999//6LztmnTRhs3bqyVHgEAqAzfxQoAgAUBCQCABQEJAIAFAQkAgAUBCQCABQEJAIAFAQkAgAUBCQCABQEJAIAFAQkAgAUBCQCABQEJAIAFAQkAgAUBCQCABQEJAIAFAQkAgAUBCQCABQEJAIAFAQkAgAUBCQCABQEJAIAFAQkAgAUBCQCABQEJAIAFAQkAgAUBCQCABQEJAIAFAQkAgAUBCQCABQEJAIAFAQkAgAUBCQCABQEJAIAFAQkAgAUBCQCARcAC8tSpUxo5cqScTqciIiI0ZswYFRcXV1n/8MMPq3PnzmrSpIlat26tRx55RB6Px68uKCiowrJ69epAbQYAoIFqFKiJR44cqaNHj2rz5s06d+6c0tLSNHbsWL366qvW+iNHjujIkSOaN2+eunbtqq+//lr333+/jhw5oj/+8Y9+tUuXLlVKSorvdkRERKA2AwDQQAUkIPfs2aPs7Gx9+umn6tOnjyTp+eef1+2336558+YpJiamwjo33HCD/ud//sd3u0OHDvrd736ne+65R+fPn1ejRn9vNSIiQi6XKxCtAwAgKUBvsebl5SkiIsIXjpKUlJSk4OBgbdmy5ZLn8Xg8cjqdfuEoSQ8++KAiIyMVHx+vJUuWyBhT5TylpaXyer1+CwAAVQnIK0i3261WrVr5P1CjRmrRooXcbvclzXHixAnNnDlTY8eO9RufMWOGfvWrX6lp06batGmTHnjgARUXF+uRRx6pdK5Zs2Zp+vTp1d8QAECDVa1XkJMmTbIeJPPD5csvv7zsprxerwYPHqyuXbtq2rRpfvdNnjxZv/jFL9SrVy9NnDhRTz75pJ555pkq58vMzJTH4/EtBw8evOweAQBXt2q9ghw/frxGjx5dZU379u3lcrl07Ngxv/Hz58/r1KlTF/3s8PTp00pJSVGzZs30+uuvq3HjxlXWJyQkaObMmSotLZXD4bDWOByOSu8DAMCmWgHZsmVLtWzZ8qJ1iYmJKioq0rZt29S7d29J0rvvvqvy8nIlJCRUup7X61VycrIcDofeeOMNhYWFXfSxdu7cqebNmxOAAIBaFZDPIK+//nqlpKQoPT1dCxcu1Llz5/TQQw9pxIgRviNYDx8+rAEDBmjFihWKj4+X1+vVwIEDdebMGb3yyit+B9O0bNlSISEhevPNN1VYWKibbrpJYWFh2rx5s55++mk98cQTgdgMAEADFrDzIFeuXKmHHnpIAwYMUHBwsO644w4999xzvvvPnTunvXv36syZM5Kk7du3+45w7dixo99cBQUFatu2rRo3bqysrCw9/vjjMsaoY8eOmj9/vtLT0wO1GQCABirIXOwciauQ1+tVeHi4Jk2axFuzuGr94wFuwNXkwvP4hdMBA4HvYgUAwIKABADAgoAEAMCCgAQAwIKABADAgoAEAMCCgAQAwIKABADAgoAEAMCCgAQAwIKABADAgoAEAMCCgAQAwIKABADAgoAEAMCCgAQAwIKABADAgoAEAMCCgAQAwIKABADAgoAEAMCCgAQAwIKABADAgoAEAMCCgAQAwIKABADAgoAEAMCCgAQAwIKABADAgoAEAMCCgAQAwIKABADAgoAEAMCCgAQAwIKABADAok4CMisrS23btlVYWJgSEhK0devWKuvXrl2rLl26KCwsTN27d9fGjRv97jfGaMqUKYqOjlaTJk2UlJSkffv2BXITAAANTMADcs2aNcrIyNDUqVO1fft29ezZU8nJyTp27Ji1/uOPP9Zdd92lMWPGaMeOHRo2bJiGDRum3bt3+2rmzp2r5557TgsXLtSWLVt0zTXXKDk5Wd99912gNwcA0EAEGWNMIB8gISFBP//5z/XCCy9IksrLyxUbG6uHH35YkyZNqlA/fPhwlZSU6E9/+pNv7KabblJcXJwWLlwoY4xiYmI0fvx4PfHEE5Ikj8ejqKgoLVu2TCNGjLhoT16vV+Hh4Zo0aZIcDkctbSlwZZk2bVp9twAEzIXncY/HI6fTGZDHCOgryLNnz2rbtm1KSkr6+wMGByspKUl5eXnWdfLy8vzqJSk5OdlXX1BQILfb7VcTHh6uhISESucsLS2V1+v1WwAAqEpAA/LEiRMqKytTVFSU33hUVJTcbrd1HbfbXWX9hX+rM+esWbMUHh7uW2JjY2u0PQCAhqNBHMWamZkpj8fjWw4ePFjfLQEArnABDcjIyEiFhISosLDQb7ywsFAul8u6jsvlqrL+wr/VmdPhcMjpdPotAABUJaABGRoaqt69eysnJ8c3Vl5erpycHCUmJlrXSUxM9KuXpM2bN/vq27VrJ5fL5Vfj9Xq1ZcuWSucEAKC6GgX6ATIyMpSamqo+ffooPj5eCxYsUElJidLS0iRJo0aN0rXXXqtZs2ZJkh599FH169dP//mf/6nBgwdr9erV+stf/qKXXnpJkhQUFKTHHntMv/3tb9WpUye1a9dOkydPVkxMjIYNGxbozQEANBABD8jhw4fr+PHjmjJlitxut+Li4pSdne07yObAgQMKDv77C9m+ffvq1Vdf1VNPPaXf/OY36tSpk9avX68bbrjBV/Pkk0+qpKREY8eOVVFRkW6++WZlZ2crLCws0JsDAGggAn4e5JWI8yDREHAeJK5mP/rzIAEA+LEiIAEAsCAgAQCwICABALAgIAEAsCAgAQCwICABALAgIAEAsCAgAQCwICABALAgIAEAsCAgAQCwICABALAgIAEAsCAgAQCwICABALAgIAEAsCAgAQCwICABALAgIAEAsCAgAQCwICABALAgIAEAsCAgAQCwICABALAgIAEAsCAgAQCwICABALAgIAEAsCAgAQCwICABALAgIAEAsCAgAQCwICABALAgIAEAsCAgAQCwqJOAzMrKUtu2bRUWFqaEhARt3bq10tpFixbplltuUfPmzdW8eXMlJSVVqB89erSCgoL8lpSUlEBvBgCgAQl4QK5Zs0YZGRmaOnWqtm/frp49eyo5OVnHjh2z1ufm5uquu+7Se++9p7y8PMXGxmrgwIE6fPiwX11KSoqOHj3qW1atWhXoTQEANCABD8j58+crPT1daWlp6tq1qxYuXKimTZtqyZIl1vqVK1fqgQceUFxcnLp06aKXX35Z5eXlysnJ8atzOBxyuVy+pXnz5pX2UFpaKq/X67cAAFCVgAbk2bNntW3bNiUlJf39AYODlZSUpLy8vEua48yZMzp37pxatGjhN56bm6tWrVqpc+fOGjdunE6ePFnpHLNmzVJ4eLhviY2NrdkGAQAajIAG5IkTJ1RWVqaoqCi/8aioKLnd7kuaY+LEiYqJifEL2ZSUFK1YsUI5OTmaM2eO3n//fQ0aNEhlZWXWOTIzM+XxeHzLwYMHa75RAIAGoVF9N1CV2bNna/Xq1crNzVVYWJhvfMSIEb6fu3fvrh49eqhDhw7Kzc3VgAEDKszjcDjkcDjqpGcAwNUhoK8gIyMjFRISosLCQr/xwsJCuVyuKtedN2+eZs+erU2bNqlHjx5V1rZv316RkZHKz8+/7J4BAJACHJChoaHq3bu33wE2Fw64SUxMrHS9uXPnaubMmcrOzlafPn0u+jiHDh3SyZMnFR0dXSt9AwAQ8KNYMzIytGjRIi1fvlx79uzRuHHjVFJSorS0NEnSqFGjlJmZ6aufM2eOJk+erCVLlqht27Zyu91yu90qLi6WJBUXF2vChAn65JNP9NVXXyknJ0dDhw5Vx44dlZycHOjNAQA0EAH/DHL48OE6fvy4pkyZIrfbrbi4OGVnZ/sO3Dlw4ICCg/+e0y+++KLOnj2rf/3Xf/WbZ+rUqZo2bZpCQkL02Wefafny5SoqKlJMTIwGDhyomTNn8jkjAKDWBBljTH03Ude8Xq/Cw8M1adIkQhVXrWnTptV3C0DAXHge93g8cjqdAXkMvosVAAALAhIAAAsCEgAACwISAAALAhIAAAsCEgAACwISAAALAhIAAAsCEgAACwISAAALAhIAAAsCEgAACwISAAALAhIAAAsCEgAACwISAAALAhIAAAsCEgAACwISAAALAhIAAAsCEgAACwISAAALAhIAAAsCEgAACwISAAALAhIAAAsCEgAACwISAAALAhIAAAsCEgAACwISAAALAhIAAAsCEgAACwISAAALAhIAAIs6CcisrCy1bdtWYWFhSkhI0NatWyutXbZsmYKCgvyWsLAwvxpjjKZMmaLo6Gg1adJESUlJ2rdvX6A3AwDQgAQ8INesWaOMjAxNnTpV27dvV8+ePZWcnKxjx45Vuo7T6dTRo0d9y9dff+13/9y5c/Xcc89p4cKF2rJli6655holJyfru+++C/TmAAAaiIAH5Pz585Wenq60tDR17dpVCxcuVNOmTbVkyZJK1wkKCpLL5fItUVFRvvuMMVqwYIGeeuopDR06VD169NCKFSt05MgRrV+/3jpfaWmpvF6v3wIAQFUaBXLys2fPatu2bcrMzPSNBQcHKykpSXl5eZWuV1xcrDZt2qi8vFw33nijnn76aXXr1k2SVFBQILfbraSkJF99eHi4EhISlJeXpxEjRlSYb9asWZo+fXqF8czMTDmdzsvZROCKFRQUVN8tAD9qAX0FeeLECZWVlfm9ApSkqKgoud1u6zqdO3fWkiVLtGHDBr3yyisqLy9X3759dejQIUnyrVedOTMzM+XxeHzLwYMHL3fTAABXuYC+gqyJxMREJSYm+m737dtX119/vf7whz9o5syZNZrT4XDI4XDUVosAgAYgoK8gIyMjFRISosLCQr/xwsJCuVyuS5qjcePG6tWrl/Lz8yXJt97lzAkAwMUENCBDQ0PVu3dv5eTk+MbKy8uVk5Pj9yqxKmVlZdq1a5eio6MlSe3atZPL5fKb0+v1asuWLZc8JwAAFxPwt1gzMjKUmpqqPn36KD4+XgsWLFBJSYnS0tIkSaNGjdK1116rWbNmSZJmzJihm266SR07dlRRUZGeeeYZff3117rvvvskfX/gwWOPPabf/va36tSpk9q1a6fJkycrJiZGw4YNC/TmAAAaiIAH5PDhw3X8+HFNmTJFbrdbcXFxys7O9h1kc+DAAQUH//2F7DfffKP09HS53W41b95cvXv31scff6yuXbv6ap588kmVlJRo7NixKioq0s0336zs7OwKXygAAEBNBRljTH03Ude8Xq/Cw8Pl8Xg4zQNXLU7zQEMQyOdxvosVAAALAhIAAAsCEgAACwISAAALAhIAAAsCEgAACwISAAALAhIAAAsCEgAACwISAAALAhIAAAsCEgAACwISAAALAhIAAAsCEgAACwISAAALAhIAAAsCEgAACwISAAALAhIAAAsCEgAACwISAAALAhIAAAsCEgAACwISAAALAhIAAAsCEgAACwISAAALAhIAAAsCEgAACwISAAALAhIAAAsCEgAACwISAAALAhIAAIs6CcisrCy1bdtWYWFhSkhI0NatWyut7d+/v4KCgiosgwcP9tWMHj26wv0pKSl1sSkAgAaiUaAfYM2aNcrIyNDChQuVkJCgBQsWKDk5WXv37lWrVq0q1K9bt05nz5713T558qR69uypf/u3f/OrS0lJ0dKlS323HQ5H4DYCANDgBPwV5Pz585Wenq60tDR17dpVCxcuVNOmTbVkyRJrfYsWLeRyuXzL5s2b1bRp0woB6XA4/OqaN28e6E0BADQgAQ3Is2fPatu2bUpKSvr7AwYHKykpSXl5eZc0x+LFizVixAhdc801fuO5ublq1aqVOnfurHHjxunkyZOVzlFaWiqv1+u3AABQlYAG5IkTJ1RWVqaoqCi/8aioKLnd7ouuv3XrVu3evVv33Xef33hKSopWrFihnJwczZkzR++//74GDRqksrIy6zyzZs1SeHi4b4mNja35RgEAGoSAfwZ5ORYvXqzu3bsrPj7eb3zEiBG+n7t3764ePXqoQ4cOys3N1YABAyrMk5mZqYyMDN9tr9dLSAIAqhTQV5CRkZEKCQlRYWGh33hhYaFcLleV65aUlGj16tUaM2bMRR+nffv2ioyMVH5+vvV+h8Mhp9PptwAAUJWABmRoaKh69+6tnJwc31h5eblycnKUmJhY5bpr165VaWmp7rnnnos+zqFDh3Ty5ElFR0dfds8AAEh1cBRrRkaGFi1apOXLl2vPnj0aN26cSkpKlJaWJkkaNWqUMjMzK6y3ePFiDRs2TD/96U/9xouLizVhwgR98skn+uqrr5STk6OhQ4eqY8eOSk5ODvTmAAAaiIB/Bjl8+HAdP35cU6ZMkdvtVlxcnLKzs30H7hw4cEDBwf45vXfvXn344YfatGlThflCQkL02Wefafny5SoqKlJMTIwGDhyomTNnci4kAKDWBBljTH03Ude8Xq/Cw8Pl8Xj4PBJXraCgoPpuAQi4QD6P812sAABYEJAAAFgQkAAAWBCQAABYEJAAAFgQkAAAWBCQAABYEJAAAFgQkAAAWBCQAABYEJAAAFgQkAAAWBCQAABYEJAAAFgQkAAAWBCQAABYEJAAAFgQkAAAWBCQAABYEJAAAFgQkAAAWBCQAABYEJAAAFgQkAAAWBCQAABYEJAAAFgQkAAAWBCQAABYEJAAAFgQkAAAWBCQAABYEJAAAFgQkAAAWBCQAABYEJAAAFgENCA/+OADDRkyRDExMQoKCtL69esvuk5ubq5uvPFGORwOdezYUcuWLatQk5WVpbZt2yosLEwJCQnaunVr7TcPAGjQAhqQJSUl6tmzp7Kysi6pvqCgQIMHD9att96qnTt36rHHHtN9992nt99+21ezZs0aZWRkaOrUqdq+fbt69uyp5ORkHTt2LFCbAQBogIKMMaZOHigoSK+//rqGDRtWac3EiRP11ltvaffu3b6xESNGqKioSNnZ2ZKkhIQE/fznP9cLL7wgSSovL1dsbKwefvhhTZo06ZJ68Xq9Cg8Pl8fjkdPprPlGAVewoKCg+m4BCLhAPo9fUZ9B5uXlKSkpyW8sOTlZeXl5kqSzZ89q27ZtfjXBwcFKSkry1diUlpbK6/X6LQAAVOWKCki3262oqCi/saioKHm9Xn377bc6ceKEysrKrDVut7vSeWfNmqXw8HDfEhsbG5D+AQBXjysqIAMlMzNTHo/Htxw8eLC+WwIAXOEa1XcDP+RyuVRYWOg3VlhYKKfTqSZNmigkJEQhISHWGpfLVem8DodDDocjID0DAK5OV9QryMTEROXk5PiNbd68WYmJiZKk0NBQ9e7d26+mvLxcOTk5vhoAAGpDQAOyuLhYO3fu1M6dOyV9fxrHzp07deDAAUnfv/U5atQoX/3999+vv/3tb3ryySf15Zdf6ve//71ee+01Pf74476ajIwMLVq0SMuXL9eePXs0btw4lZSUKC0tLZCbAgBoaEwAvffee0ZShSU1NdUYY0xqaqrp169fhXXi4uJMaGioad++vVm6dGmFeZ9//nnTunVrExoaauLj480nn3xSrb48Ho+RZDweTw23DLjy2f72WFiutiWQz+N1dh7klYTzINEQcB4kGoIGcx4kAABXCgISAAALAhIAAAsCEgAACwISAAALAhIAAAsCEgAACwISAAALAhIAAAsCEgAACwISAAALAhIAAAsCEgAACwISAAALAhIAAAsCEgAACwISAAALAhIAAAsCEgAACwISAAALAhIAAAsCEgAACwISAAALAhIAAAsCEgAACwISAAALAhIAAAsCEgAACwISAAALAhIAAAsCEgAACwISAAALAhIAAAsCEgAACwISAACLgAbkBx98oCFDhigmJkZBQUFav359lfXr1q3TbbfdppYtW8rpdCoxMVFvv/22X820adMUFBTkt3Tp0iWAWwEAaIgCGpAlJSXq2bOnsrKyLqn+gw8+0G233aaNGzdq27ZtuvXWWzVkyBDt2LHDr65bt246evSob/nwww8D0T4AoAFrFMjJBw0apEGDBl1y/YIFC/xuP/3009qwYYPefPNN9erVyzfeqFEjuVyu2moTAIAKrujPIMvLy3X69Gm1aNHCb3zfvn2KiYlR+/btNXLkSB04cKDKeUpLS+X1ev0WAACqckUH5Lx581RcXKw777zTN5aQkKBly5YpOztbL774ogoKCnTLLbfo9OnTlc4za9YshYeH+5bY2Ni6aB8A8CMWZIwxdfJAQUF6/fXXNWzYsEuqf/XVV5Wenq4NGzYoKSmp0rqioiK1adNG8+fP15gxY6w1paWlKi0t9d32er2KjY2Vx+OR0+ms1nYAPxZBQUH13QIQcIF8Hg/oZ5A1tXr1at13331au3ZtleEoSREREbruuuuUn59faY3D4ZDD4ajtNgEAV7Er7i3WVatWKS0tTatWrdLgwYMvWl9cXKz9+/crOjq6DroDADQUAX0FWVxc7PfKrqCgQDt37lSLFi3UunVrZWZm6vDhw1qxYoWk799WTU1N1X/9138pISFBbrdbktSkSROFh4dLkp544gkNGTJEbdq00ZEjRzR16lSFhITorrvuCuSmAAAaGhNA7733npFUYUlNTTXGGJOammr69evnq+/Xr1+V9cYYM3z4cBMdHW1CQ0PNtddea4YPH27y8/Or1ZfH4zGSjMfjqYWtBK5Mtr8lFparbQnk83idHaRzJfF6vQoPD+cgHVzVOEgHDUEgn8evuM8gAQC4EhCQAABYEJAAAFgQkAAAWBCQAABYEJAAAFgQkAAAWBCQAABYEJAAAFgQkAAAWBCQAABYEJAAAFgQkAAAWBCQAABYEJAAAFgQkAAAWBCQAABYEJAAAFgQkAAAWBCQAABYEJAAAFgQkAAAWBCQAABYEJAAAFgQkAAAWBCQAABYEJAAAFgQkAAAWBCQAABYEJAAAFgQkAAAWBCQAABYEJAAAFgQkAAAWBCQAABYEJAAAFgENCA/+OADDRkyRDExMQoKCtL69eurrM/NzVVQUFCFxe12+9VlZWWpbdu2CgsLU0JCgrZu3RrArQAANEQBDciSkhL17NlTWVlZ1Vpv7969Onr0qG9p1aqV7741a9YoIyNDU6dO1fbt29WzZ08lJyfr2LFjtd0+AKABaxTIyQcNGqRBgwZVe71WrVopIiLCet/8+fOVnp6utLQ0SdLChQv11ltvacmSJZo0aZJ1ndLSUpWWlvpuezweSZLX6612bwCAK4cxJmBzBzQgayouLk6lpaW64YYbNG3aNP3iF7+QJJ09e1bbtm1TZmamrzY4OFhJSUnKy8urdL5Zs2Zp+vTpFcZjY2Nrv3kAQJ05efKkwsPDAzL3FRWQ0dHRWrhwofr06aPS0lK9/PLL6t+/v7Zs2aIbb7xRJ06cUFlZmaKiovzWi4qK0pdfflnpvJmZmcrIyPDdLioqUps2bXTgwIGA7dhA8Hq9io2N1cGDB+V0Ouu7nUv2Y+1b+vH2Tt91i77rnsfjUevWrdWiRYuAPcYVFZCdO3dW586dfbf79u2r/fv369lnn9V///d/13heh8Mhh8NRYTw8PPxH90shSU6nk77r2I+1d/quW/Rd94KDA3cozRV/mkd8fLzy8/MlSZGRkQoJCVFhYaFfTWFhoVwuV320BwC4Sl3xAblz505FR0dLkkJDQ9W7d2/l5OT47i8vL1dOTo4SExPrq0UAwFUooG+xFhcX+179SVJBQYF27typFi1aqHXr1srMzNThw4e1YsUKSdKCBQvUrl07devWTd99951efvllvfvuu9q0aZNvjoyMDKWmpqpPnz6Kj4/XggULVFJS4juq9VI4HA5NnTrV+rbrlYy+696PtXf6rlv0XffqovcgE8BjZHNzc3XrrbdWGE9NTdWyZcs0evRoffXVV8rNzZUkzZ07Vy+99JIOHz6spk2bqkePHpoyZUqFOV544QU988wzcrvdiouL03PPPaeEhIRAbQYAoAEKaEACAPBjdcV/BgkAQH0gIAEAsCAgAQCwICABALC4KgPy1KlTGjlypJxOpyIiIjRmzBgVFxdXuU7//v0rXGbr/vvv96s5cOCABg8erKZNm6pVq1aaMGGCzp8/X6+9nzp1Sg8//LA6d+6sJk2aqHXr1nrkkUd8X8h+ge0yYqtXr65xn9W95NjatWvVpUsXhYWFqXv37tq4caPf/cYYTZkyRdHR0WrSpImSkpK0b9++GvdXG30vWrRIt9xyi5o3b67mzZsrKSmpQv3o0aMr7NeUlJR67XvZsmUVegoLC/Orqav9Xd3ebX+HQUFBGjx4sK8m0Pu8upfpk74/Yv/GG2+Uw+FQx44dtWzZsgo1dXGZvur2vm7dOt12221q2bKlnE6nEhMT9fbbb/vVTJs2rcL+7tKlS732XWeXRjRXoZSUFNOzZ0/zySefmP/93/81HTt2NHfddVeV6/Tr18+kp6ebo0eP+haPx+O7//z58+aGG24wSUlJZseOHWbjxo0mMjLSZGZm1mvvu3btMr/+9a/NG2+8YfLz801OTo7p1KmTueOOO/zqJJmlS5f6bd+3335box5Xr15tQkNDzZIlS8znn39u0tPTTUREhCksLLTWf/TRRyYkJMTMnTvXfPHFF+app54yjRs3Nrt27fLVzJ4924SHh5v169ebv/71r+af//mfTbt27WrcY230fffdd5usrCyzY8cOs2fPHjN69GgTHh5uDh065KtJTU01KSkpfvv11KlTtdZzTfpeunSpcTqdfj253W6/mrrY3zXp/eTJk359796924SEhJilS5f6agK9zzdu3Gj+4z/+w6xbt85IMq+//nqV9X/7299M06ZNTUZGhvniiy/M888/b0JCQkx2dravprr7oa56f/TRR82cOXPM1q1bzf/93/+ZzMxM07hxY7N9+3ZfzdSpU023bt389vfx48frte/33nvPSDJ79+7166usrMxXUxv7/KoLyC+++MJIMp9++qlv7M9//rMJCgoyhw8frnS9fv36mUcffbTS+zdu3GiCg4P9nmhefPFF43Q6TWlpab32/o9ee+01Exoaas6dO+cbu5RfuksVHx9vHnzwQd/tsrIyExMTY2bNmmWtv/POO83gwYP9xhISEsy///u/G2OMKS8vNy6XyzzzzDO++4uKiozD4TCrVq2qlZ5r0vc/On/+vGnWrJlZvny5byw1NdUMHTq01nq0qW7fS5cuNeHh4ZXOV1f725jL3+fPPvusadasmSkuLvaN1cU+v+BS/m6efPJJ061bN7+x4cOHm+TkZN/ty90PNVHTv/muXbua6dOn+25PnTrV9OzZs/Yau4jqBOQ333xTaU1t7POr7i3WvLw8RUREqE+fPr6xpKQkBQcHa8uWLVWuu3LlSkVGRuqGG25QZmamzpw54zdv9+7d/a4kkpycLK/Xq88//7zee/8hj8cjp9OpRo38vyjpwQcfVGRkpOLj47VkyZIaXUftwiXHkpKSfGMXu+RYXl6eX730/b67UF9QUCC32+1XEx4eroSEhCovYxbovv/RmTNndO7cuQpXD8jNzVWrVq3UuXNnjRs3TidPnqyVni+n7+LiYrVp00axsbEaOnSo3+9oXezvy+n9hxYvXqwRI0bommuu8RsP5D6vrov9ftfGfqgr5eXlOn36dIXf8X379ikmJkbt27fXyJEjdeDAgXrq0F9cXJyio6N122236aOPPvKN19Y+v6Ku5lEb3G63WrVq5TfWqFEjtWjRosL70z909913q02bNoqJidFnn32miRMnau/evVq3bp1vXttlti7cV5+9/9CJEyc0c+ZMjR071m98xowZ+tWvfqWmTZtq06ZNeuCBB1RcXKxHHnmkWj3W5JJjle27C9t04d+qai5XTS+V9kMTJ05UTEyM3x9dSkqKfv3rX6tdu3bav3+/fvOb32jQoEHKy8tTSEhIvfTduXNnLVmyRD169JDH49G8efPUt29fff755/rZz35WJ/u7pr3/0NatW7V7924tXrzYbzzQ+7y6Kvv99nq9+vbbb/XNN99c9u9eXZk3b56Ki4t15513+sYSEhK0bNkyde7cWUePHtX06dN1yy23aPfu3WrWrFm99BmoSyP+ox9NQE6aNElz5sypsmbPnj01nv+HgdK9e3dFR0drwIAB2r9/vzp06FDjeaXA936B1+vV4MGD1bVrV02bNs3vvsmTJ/t+7tWrl0pKSvTMM89UOyAbqtmzZ2v16tXKzc31O+BlxIgRvp+7d++uHj16qEOHDsrNzdWAAQPqo1UlJib6fXl/3759df311+sPf/iDZs6cWS891cTixYvVvXt3xcfH+41fifv8avDqq69q+vTp2rBhg99/1AcNGuT7uUePHkpISFCbNm302muvacyYMfXRasAujfiPfjQBOX78eI0ePbrKmvbt28vlcunYsWN+4+fPn9epU6eqdUmsC9/tmp+frw4dOsjlclU4AurCZbcuNm9d9H769GmlpKSoWbNmev3119W4ceMq6xMSEjRz5kyVlpZW68t+a3LJMZfLVWX9hX8LCwt9V265cDsuLu6Se6vtvi+YN2+eZs+erXfeeUc9evSosrZ9+/aKjIxUfn5+rTxZ18Yl3ho3bqxevXr5LhxQF/tburzeS0pKtHr1as2YMeOij1Pb+7y6Kvv9djqdatKkiUJCQq74y/StXr1a9913n9auXVvh7eJ/FBERoeuuu87vQhRXgvj4eH344YeSau/SiD+azyBbtmypLl26VLmEhoYqMTFRRUVF2rZtm2/dd999V+Xl5dX6QvOdO3dKku8JJDExUbt27fILsM2bN8vpdKpr16712rvX69XAgQMVGhqqN954o8Ih/ZVtX/Pmzav9Tfg1ueRYYmKiX730/b67UN+uXTu5XC6/Gq/Xqy1bttTaZcxqeqm0uXPnaubMmcrOzvb7bLgyhw4d0smTJ/2Cpz76/qGysjLt2rXL11Nd7O/L7X3t2rUqLS3VPffcc9HHqe19Xl0X+/2+0i/Tt2rVKqWlpWnVqlV+p9NUpri4WPv376+3/V2ZgFwa8ZIP5/kRSUlJMb169TJbtmwxH374oenUqZPfqRKHDh0ynTt3Nlu2bDHGGJOfn29mzJhh/vKXv5iCggKzYcMG0759e/PLX/7St86F0zwGDhxodu7cabKzs03Lli0DcppHdXr3eDwmISHBdO/e3eTn5/sd8nz+/HljjDFvvPGGWbRokdm1a5fZt2+f+f3vf2+aNm1qpkyZUqMeV69ebRwOh1m2bJn54osvzNixY01ERITvCN97773XTJo0yVf/0UcfmUaNGpl58+aZPXv2mKlTp1pP84iIiDAbNmwwn332mRk6dGhATvOoTt+zZ882oaGh5o9//KPffj19+rQxxpjTp0+bJ554wuTl5ZmCggLzzjvvmBtvvNF06tTJfPfdd/XW9/Tp083bb79t9u/fb7Zt22ZGjBhhwsLCzOeff+63bYHe3zXp/YKbb77ZDB8+vMJ4Xezz06dPmx07dpgdO3YYSWb+/Plmx44d5uuvvzbGGDNp0iRz7733+uovnOYxYcIEs2fPHpOVlWU9zaOq/VBbqtv7ypUrTaNGjUxWVpbf73hRUZGvZvz48SY3N9cUFBSYjz76yCQlJZnIyEhz7Nixeuv72WefNevXrzf79u0zu3btMo8++qgJDg4277zzjq+mNvb5VRmQJ0+eNHfddZf5yU9+YpxOp0lLS/M9qRljTEFBgZFk3nvvPWOMMQcOHDC//OUvTYsWLYzD4TAdO3Y0EyZM8DsP0hhjvvrqKzNo0CDTpEkTExkZacaPH+93KkV99H7hcGfbUlBQYIz5/lSRuLg485Of/MRcc801pmfPnmbhwoV+5wxV1/PPP29at25tQkNDTXx8vPnkk0989/Xr18+kpqb61b/22mvmuuuuM6GhoaZbt27mrbfe8ru/vLzcTJ482URFRRmHw2EGDBhg9u7dW+P+aqPvNm3aWPfr1KlTjTHGnDlzxgwcONC0bNnSNG7c2LRp08akp6fX+pNedft+7LHHfLVRUVHm9ttv9zuvzZi629/V7d0YY7788ksjyWzatKnCXHWxzyv7m7rQZ2pqqunXr1+FdeLi4kxoaKhp376933mbF1S1H+qr9379+lVZb8z3p6xER0eb0NBQc+2115rhw4eb/Pz8eu17zpw5pkOHDiYsLMy0aNHC9O/f37z77rsV5r3cfc7lrgAAsPjRfAYJAEBdIiABALAgIAEAsCAgAQCwICABALAgIAEAsCAgAQCwICABALAgIAEAsCAgAQCwICABALD4f75zwMEVYTRSAAAAAElFTkSuQmCC\n" + }, + "metadata": {} + } + ] + } + ] +} \ No newline at end of file