Skip to content

Commit

Permalink
Removed celery info (it is now in a dedicated branch)
Browse files Browse the repository at this point in the history
  • Loading branch information
JMorado committed Aug 31, 2023
1 parent 4d24cee commit 7f5502c
Showing 1 changed file with 0 additions and 67 deletions.
67 changes: 0 additions & 67 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -54,76 +54,9 @@ There are a few environment variables that need to be set so that the applicatio

# Pipeline architectures

## Celery-based pipeline
![Celery-based pipeline architecture](/images/celery_pipeline.png "Celery-based pipeline architecture")

## Thread-based pipeline
![Thread-based pipeline architecture](/images/thread_pipeline.png "Thread-based pipeline architecture")

# How to use the celery-based pipeline

## 1. Start RabbitMQ locally (Optional)

Set up a local instance of RabbitMQ using Docker:

```bash
docker pull rabbitmq:3-management
docker run --rm -it -p 15672:15672 -p 5672:5672 rabbitmq:3-management
```

The rabbitMQ management interface can be access on the url http://localhost:15672


NOTE: If RabbitMQ is run locally, set the BROKER_URL env variable to amqp://guest:guest@localhost. The default username and password are uest.


## 2. Start a celery worker

Start a celery worker:

```bash
python -m celery -A main worker --loglevel=INFO -n ETLPipeline@%h
```

Alternatively, run the customisable `start_celery_worker.sh` script:

```bash
./start_celery_worker.sh
```

## 4. Run the ETL data pipeline

Run one of the examples in the examples directory, e.g.:

```bash
python -u examples/example_celery.py
```

## 4. Monitor a Celery cluster with Flower (Optional)

Install Flower using pip:

```bash
pip install flower
```

Launch the Flower server at specified port (default is 5555, so `--port=5555` can be ommited):

```
python -m celery -A main flower --port=5555
```

Alternatively, run Flower via docker:

```
docker run -p 5555:5555 mher/flower
```

Access Flower on the url http://localhost:5555/




## Filesystems

### Credentials to access the object store (.json file)
Expand Down

2 comments on commit 7f5502c

@github-actions
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Coverage

Coverage Report
FileStmtsMissCoverMissing
/opt/hostedtoolcache/Python/3.10.12/x64/lib/python3.10/site-packages/dpypeline/akita
   core.py1361360%2–366
   event_handler.py41410%3–215
   factory.py34340%3–174
   queue_events.py1218431%18, 22–23, 27, 40–58, 68–75, 86, 91, 102, 113–117, 121–126, 132–140, 144–155, 159–162, 178–185, 190–194, 206–215, 227–230, 240, 262–271, 276
/opt/hostedtoolcache/Python/3.10.12/x64/lib/python3.10/site-packages/dpypeline/celery
   __init__.py110%2
   app.py440%2–7
   config.py990%2–38
/opt/hostedtoolcache/Python/3.10.12/x64/lib/python3.10/site-packages/dpypeline/cli
   argument_parser.py770%2–29
   main_cli.py32320%2–71
   yaml_constructors.py66660%2–130
   yaml_executers.py15150%2–24
   yaml_loader.py18180%2–44
/opt/hostedtoolcache/Python/3.10.12/x64/lib/python3.10/site-packages/dpypeline/etl_pipeline
   basic_pipeline.py12120%2–34
   celery_pipeline.py21210%2–84
   core.py52983%103, 115–123, 172, 178
   decorators.py24240%2–61
/opt/hostedtoolcache/Python/3.10.12/x64/lib/python3.10/site-packages/dpypeline/event_consumer
   __init__.py110%2
   consumer_parallel.py61610%2–209
   consumer_serial.py21210%2–54
   core.py43430%2–130
/opt/hostedtoolcache/Python/3.10.12/x64/lib/python3.10/site-packages/dpypeline/filesystems
   __init__.py110%2
   object_store.py90900%2–383
TOTAL85973015% 

Tests Skipped Failures Errors Time
11 0 💤 6 ❌ 0 🔥 0.532s ⏱️

@github-actions
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Coverage

Coverage Report
FileStmtsMissCoverMissing
/opt/hostedtoolcache/Python/3.9.17/x64/lib/python3.9/site-packages/dpypeline/akita
   core.py1361360%2–366
   directory_state.py433616%26–112
   event_handler.py41410%3–215
   factory.py34340%3–174
   queue_events.py1218431%18, 22–23, 27, 40–58, 68–75, 86, 91, 102, 113–117, 121–126, 132–140, 144–155, 159–162, 178–185, 190–194, 206–215, 227–230, 240, 262–271, 276
/opt/hostedtoolcache/Python/3.9.17/x64/lib/python3.9/site-packages/dpypeline/celery
   __init__.py110%2
   app.py440%2–7
   config.py880%2–38
/opt/hostedtoolcache/Python/3.9.17/x64/lib/python3.9/site-packages/dpypeline/cli
   argument_parser.py770%2–29
   main_cli.py32320%2–71
   yaml_constructors.py66660%2–130
   yaml_executers.py15150%2–24
   yaml_loader.py18180%2–44
/opt/hostedtoolcache/Python/3.9.17/x64/lib/python3.9/site-packages/dpypeline/etl_pipeline
   basic_pipeline.py12120%2–34
   celery_pipeline.py21210%2–84
   core.py522846%47–55, 83–85, 102–105, 115–123, 131, 136, 151–154, 171–174, 178
   decorators.py24240%2–61
/opt/hostedtoolcache/Python/3.9.17/x64/lib/python3.9/site-packages/dpypeline/event_consumer
   __init__.py110%2
   consumer_parallel.py60600%2–209
   consumer_serial.py20200%2–54
   core.py43430%2–130
/opt/hostedtoolcache/Python/3.9.17/x64/lib/python3.9/site-packages/dpypeline/filesystems
   __init__.py110%2
   object_store.py90900%2–383
TOTAL8567829% 

Tests Skipped Failures Errors Time
1 0 💤 0 ❌ 1 🔥 0.705s ⏱️

Please sign in to comment.