Docs

.github/workflows/docs.yaml

@@ -0,0 +1,24 @@
+name: docs
+on:
+  push:
+    branches:
+      - main
+      - docs
+
+permissions:
+  contents: write
+
+jobs:
+  deploy-docs:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v3
+      - uses: actions/setup-python@v4
+        with:
+          python-version: 3.x
+      - uses: actions/cache@v3
+        with:
+          key: ${{ github.ref }}
+          path: .cache
+      - run: pip install mkdocs-material
+      - run: mkdocs gh-deploy --force

.gitignore

@@ -160,3 +160,5 @@ cython_debug/
 #.idea/
 
 node_modules
+
+.DS_Store

.vscode/settings.json

@@ -3,5 +3,15 @@
     "tests"
   ],
   "python.testing.unittestEnabled": false,
-  "python.testing.pytestEnabled": true
+  "python.testing.pytestEnabled": true,
+  "yaml.schemas": {
+    "https://squidfunk.github.io/mkdocs-material/schema.json": "mkdocs.yml"
+  },
+  "yaml.customTags": [
+    "!ENV scalar",
+    "!ENV sequence",
+    "tag:yaml.org,2002:python/name:materialx.emoji.to_svg",
+    "tag:yaml.org,2002:python/name:materialx.emoji.twemoji",
+    "tag:yaml.org,2002:python/name:pymdownx.superfences.fence_code_format"
+  ]
 }

README.md

@@ -26,69 +26,39 @@
   <p align="center">
     Python task scheduler with a user-friendly web UI
     <br />
-    <a href="https://github.com/lucafaggianelli/mario-pype"><strong>Explore the docs »</strong></a>
+    <a href="https://lucafaggianelli.github.io/mario-pype/"><strong>Official website »</strong></a>
     <br />
     <br />
-    <a href="https://github.com/lucafaggianelli/mario-pype">View Demo</a>
-    ·
     <a href="https://github.com/lucafaggianelli/mario-pype/issues">Report Bug</a>
     ·
     <a href="https://github.com/lucafaggianelli/mario-pype/issues">Request Feature</a>
   </p>
 </div>
 
 
-
-<!-- TABLE OF CONTENTS -->
-<details>
-  <summary>Table of Contents</summary>
-  <ol>
-    <li>
-      <a href="#about-the-project">About The Project</a>
-      <ul>
-        <li><a href="#built-with">Built With</a></li>
-      </ul>
-    </li>
-    <li>
-      <a href="#getting-started">Getting Started</a>
-      <ul>
-        <li><a href="#prerequisites">Prerequisites</a></li>
-        <li><a href="#installation">Installation</a></li>
-      </ul>
-    </li>
-    <li><a href="#usage">Usage</a></li>
-    <li><a href="#roadmap">Roadmap</a></li>
-    <li><a href="#contributing">Contributing</a></li>
-    <li><a href="#license">License</a></li>
-    <li><a href="#contact">Contact</a></li>
-    <li><a href="#acknowledgments">Acknowledgments</a></li>
-  </ol>
-</details>
-
-
-
 <!-- ABOUT THE PROJECT -->
 ## About The Project
 
-![Mario Pype Screen Shot](docs/screenshot.png)
-
 Mario Pype is a simple task scheduler for Python with a web UI and a REST API,
 if you need to run and monitor recurring python scripts then it's
 the right tool for you!
 
+![Mario Pype Screen Shot](docs/assets/images/screenshot.png)
+
 > This project is at its beginning, so it can be shaped and improved with
   your feedback and help!
   If you like it, star it 🌟! If you want a feature or find a bug, open an issue.
 
-Features:
-- ⏰ Task scheduing based on APScheduler (supports Interval, Cron and Date triggers)
-- 💻 Built-in Web interface, no HTML/JS/CSS coding required
-- 👩‍💻🐍 Pipelines and tasks are defined in pure Python
-- 👊 Pipelines can be run manually from the web UI
-- 🔍 Debug your tasks exploring logs and output data
-- 📩 Monitor tasks executions and get alerted if something goes wrong
-- 💣 Use the REST API for advanced integrations
-- ✨ A lot more features are coming, check the [open issues](https://github.com/lucafaggianelli/mario-pype/issues)!
+## Features
+* ⏰ Task scheduling based on [APScheduler](https://github.com/agronholm/apscheduler) (supports Interval, Cron and Date triggers)
+* 💻 Built-in Web interface, no HTML/JS/CSS coding required
+* 👩‍💻🐍 Pipelines and tasks are defined in pure Python
+* 🎛️ Pipelines can be parametrized via [Pydantic](https://docs.pydantic.dev/)
+* 👉 Pipelines can be run manually from the web UI
+* 🔐 Secured via OAuth2
+* 🔍 Debug each run exploring logs and output data
+* 📩 Monitor the pipelines and get alerted if something goes wrong
+* 💣 Use the REST API for advanced integrations
 
 When you shouldn't use it:
 - you need a lot of scalability and you want to run on a distributed system
@@ -110,6 +80,9 @@ When you shouldn't use it:
 <!-- GETTING STARTED -->
 ## Getting Started
 
+Check the 👉 [official website](https://lucafaggianelli.github.io/mario-pype/)
+for more detailed info!
+
 ### Prerequisites
 
 To run Mario Pype you only need Python (v3.8 or later), if you don't have it installed yet, go
@@ -160,7 +133,8 @@ from apscheduler.triggers.interval import IntervalTrigger
 from mario import Mario, task, get_logger, Pipeline, Trigger
 
 
-dummy_pipeline = Pipeline(
+sales_pipeline = Pipeline(
+    id="sales_pipeline",
     tasks = [fetch_raw_sales_data],
     triggers = [
         Trigger(
@@ -176,7 +150,7 @@ dummy_pipeline = Pipeline(
 A *Task* is the base block in Mario Pype and it's just a Python function that
 performs an action.
 
-This is the Task `fetch_raw_sales_data` used in the `DummyPipeline` pipeline ... it doesn't do much,
+This is the Task `fetch_raw_sales_data` used in the `sales_pipeline` pipeline ... it doesn't do much,
 but it showcase the basics:
 
 ```py

docs/assets/stylesheets/extra.css

@@ -0,0 +1,13 @@
+[data-md-toggle="search"]:not(:checked) ~ .md-header .md-search__form::after {
+  position: absolute;
+  top: .3rem;
+  right: .3rem;
+  display: block;
+  padding: .1rem .4rem;
+  color: #ffffffb3;
+  font-weight: bold;
+  font-size: .8rem;
+  border: .05rem solid #ffffff3b;
+  border-radius: .1rem;
+  content: "/";
+}

docs/configuration.md

@@ -0,0 +1,187 @@
+MarioPype is configurable via environmental variables, a YAML file
+or even better via a combination of the 2.
+
+!!! info "Why a hybrid configuration?"
+
+    An entire configuration can be quite large so storing it as environmental
+    variables can be quite hard to maintain, moreover some parts of the
+    configuration should be stored together with the code as they are part
+    of the system and some parts of it are secret so you need env vars
+
+Create a configuration file in the root of your project named `mario.config.yaml`
+(or `mario.config.yml` if you prefer) and set the values you need, you should
+commit this file to the git repo:
+
+```yaml title="mario.config.yaml"
+frontend_url: https://pipelines.example.com
+
+auth:
+  client_id: $GOOGLE_CLIENT_ID
+  client_secret: $GOOGLE_CLIENT_SECRET
+  server_metadata_url: https://accounts.google.com/.well-known/openid-configuration
+
+notifications:
+  - pipeline_status:
+      - failed
+    channels:
+      - $GMAIL_ACCOUNT
+      - $MSTEAMS_WEBHOOK
+```
+
+Now define the secrets as environmental variables in a `.env` file,
+in your shell or in your hosting environment.
+
+
+!!! tip
+
+    By default, MarioPype will load any `.env` found in your project root.
+
+!!! Warning
+
+    You shouldn't commit the `.env` file as it contains secrets!
+
+```shell title=".env"
+# Auth
+GOOGLE_CLIENT_ID="ABC123"
+GOOGLE_CLIENT_SECRET="DEF456"
+
+# Notifications
+GMAIL_ACCOUNT=mailto://myuser:[email protected]
+MSTEAMS_WEBHOOK=msteams://TokenA/TokenB/TokenC/
+```
+
+## System
+
+!!! tip
+
+    If you're running MarioPype locally, in most cases you don't need to change
+    these settings
+
+### `database_url`
+
+The Sqlite DB URI, by default `sqlite:///./mario.db`
+
+### `server_url`
+
+The URL of the backend, be default `http://localhost:8000`.
+
+Change it if running in production.
+
+### `frontend_url`
+
+The URL of the frontend, by default is the same as the backend,
+change it if the frontend is served at a different URL, for example
+during the frontend development.
+
+## Notifications
+
+MarioPype can send notifications after a pipeline has run based on the status
+of the run itself (success, failure, etc.).
+
+The notifications configuration can be defined in the YAML
+file as a list of [`NotificationRule`](#notificationrule)s:
+
+```yaml title="mario.config.yaml"
+notifications:
+  # Send notifications only if the pipelines failed
+  - pipeline_status:
+      - failed
+    channels:
+      # Send them to my gmail address (from my address itself)
+      # Better to use an env var here
+      - mailto://myuser:[email protected]
+  # Send notifications only if the pipelines succeeded or was cancelled
+  - pipeline_status:
+      - completed
+      - cancelled
+    channels:
+      # Send them to a MS Teams channel
+      # Better to use an env var here
+      - msteams://mychanneltoken
+```
+
+### `NotificationRule`
+
+A notification rule defines when to send notifications and to whom.
+
+#### `pipeline_status`
+
+A list of 1 or more pipeline run status among:
+
+  * `completed`
+  * `failed`
+  * `cancelled`
+
+#### `channels`
+
+A list of 1 or more recipients where to send the notifications.
+
+A channel is an *Apprise* URI string that defines an email address or a MS Teams
+channel, for example:
+
+* **Email** mailto://myuser:[email protected]
+* **MS Teams** msteams://TokenA/TokenB/TokenC/
+* **AWS SES** ses://user@domain/AccessKeyID/AccessSecretKey/RegionName/email1/
+
+Behind the scene MarioPype uses [Apprise](https://github.com/caronc/apprise),
+a library to send notifications to many notification providers, so check their
+docs for a full list of the available channels.
+
+## Authentication
+
+MarioPype has a buil-in and ready-to-use authentication system
+based on OAuth providers, so you can use your corporate auth system
+or Google, Github, etc.
+
+To enable the auth system you just need to configure it.
+
+!!! info "Good to know"
+
+    The auth system is based on [Authlib](https://authlib.org/)
+
+### `AuthSettings`
+
+Options available
+
+#### `client_id`
+
+An OAuth app client ID
+
+#### `client_secret`
+
+An OAuth app client secret
+
+#### `server_metadata_url`
+
+This a special URL that contains information about the OAuth provider
+specific endpoints. If your provider doesn't have this URL or you don't
+know it, you need to fill up the values for the other URLs: `access_token_url`,
+`authorize_url` and `jwks_uri`.
+
+Here a table of well known Metadata URLs:
+
+| Provider | URL |
+| -------- | --- |
+| Google | https://accounts.google.com/.well-known/openid-configuration |
+
+#### `access_token_url`
+
+#### `authorize_url`
+
+#### `jwks_uri`
+
+#### `client_kwargs`
+
+Additional values to pass to the OAuth client during the auth
+process, for example the scope:
+
+```yaml
+auth:
+  client_kwargs:
+    scope: openid email profile
+```
+
+#### `secret_key`
+
+Secret key used in the backend middleware, this has a dummy default value,
+but in production you should define a decent value.