Refactor/distributed-tests (#493)

* remove a merge conflict statement that was missed * add pytest coverage library and add sample_index coverage * run fix style and add module header * add tests for encryption modules * add unit tests for util_sampling * run fix-style and fix typo * create directory for context managers and fix issue with an encryption test * add a context manager for spinning up/down the redis server * fix issue with path in one test * rework CONFIG functionality for testing * refactor config fixture so it doesn't depend on redis server to be started * split CONFIG fixtures into rabbit and redis configs, run fix-style * add unit tests for broker.py * add unit tests for the Config object * update CHANGELOG * make CONFIG fixtures more flexible for tests * add tests for results_backend.py * fix lint issues for most recent changes * fix filename issue in setup.cfg and move celeryadapter tests to integration suite * add ssl filepaths to mysql config object * add unit tests for configfile.py * add tests for the utils.py file in config/ * create utilities file and constants file * move create_dir function to utils.py * add tests for merlin/examples/generator.py * run fix-style and update changelog * add a 'pip freeze' call in github workflow to view reqs versions * re-delete the old config test files * fix tests/bugs introduced by merging in develop * add a unit test file for the dumper module * begin work on server tests and modular fixtures * start work on tests for RedisConfig * add tests for RedisConfig object * add tests for RedisUsers class * change server fixtures to use redis config files * add tests for AppYaml class * final cleanup of server_utils * fix lint issues * parametrize setup examples tests * sort example output * ensure directory is changed back on no outdir test * sort the specs in examples output * fix lint issues * start writing tests for server config * add pytest coverage library and add sample_index coverage * run fix style and add module header * add tests for encryption modules * add unit tests for util_sampling * run fix-style and fix typo * create directory for context managers and fix issue with an encryption test * add a context manager for spinning up/down the redis server * fix issue with path in one test * rework CONFIG functionality for testing * refactor config fixture so it doesn't depend on redis server to be started * split CONFIG fixtures into rabbit and redis configs, run fix-style * add unit tests for broker.py * add unit tests for the Config object * update CHANGELOG * make CONFIG fixtures more flexible for tests * add tests for results_backend.py * fix lint issues for most recent changes * fix filename issue in setup.cfg and move celeryadapter tests to integration suite * add ssl filepaths to mysql config object * add unit tests for configfile.py * add tests for the utils.py file in config/ * create utilities file and constants file * move create_dir function to utils.py * add tests for merlin/examples/generator.py * run fix-style and update changelog * fix tests/bugs introduced by merging in develop * add a unit test file for the dumper module * begin work on server tests and modular fixtures * start work on tests for RedisConfig * add tests for RedisConfig object * add tests for RedisUsers class * change server fixtures to use redis config files * add tests for AppYaml class * final cleanup of server_utils * fix lint issues * parametrize setup examples tests * sort example output * ensure directory is changed back on no outdir test * sort the specs in examples output * fix lint issues * start writing tests for server config * bake in LC_ALL env variable setting for server cmds * add tests for parse_redis_output * fix issue with scope of fixture after rebase * run fix-style * Include celerymanager and update celeryadapter to check the status of celery workers. * Fixed issue where the update status was outside of if statement for checking workers * Include worker status stop and add template for merlin restart * Added comment to the CeleryManager init * Increment db_num instead of being fixed * Added other subprocess parameters and created a linking system for redis to store env dict * Implemented stopping of celery workers and restarting workers properly * Update stopped to stalled for when the worker doesn't respond to restart * Working merlin manager run but start and stop not working properly * Made fix for subprocess to start new shell and fixed manager start and stop * Added comments and update changelog * Include style fixes * Fix style for black * Revert launch_job script that was edited when doing automated lint * Move importing of CONFIG to be within redis_connection due to error of config not being created yet * Added space to fix style * Revert launch_jobs.py: * Update import of all merlin.config to be in the function * suggested changes plus beginning work on monitor/manager collab * move managers to their own folder and fix ssl problems * final PR touch ups * Fix lint style changes * Fixed issue with context manager * Reset file that was incorrect changed * Check for ssl cert before applying to Redis connection * Comment out Active tests for celerymanager * split up create_server_config and write tests for it * add tests for config_merlin_server function * Fix lint issue with unused import after commenting out Active celery tests * Fixed style for import * add tests for pull_server_config * add tests for pull_server_image * finish writing tests for server_config.py * Fixed kwargs being modified when making a copy for saving to redis worker args. * add tests for server_commands.py * run fix-style * update README for testing directory * update the temp_output_directory to include python version * mock the open.write to try to fix github CI * ensure config dir is created * update CHANGELOG * add print of exception to OSError catch in pull_server_image * change name of config_file in test that's failing * Added password check and omit if a password doesn't exist * update CHANGELOG * change testing log level to debug * add debug statement for redis_connection * change debug log to info so github ci will display it * attempt to fix password missing from Namespace error * run checks for all necessary configurations * convert stop-workers tests to pytest format * update github wf and comment out stop-workers tests in definitions.py * add missing key to GH wf file * fix invalid syntax in definitions.py * comment out stop_workers tests * playing with new caches for workflow CI * fix yaml syntax error * fix typo for getting runner os * fix test and add python version to CI cache * add in common-setup step again with caches this time * run fix-style * update CHANGELOG * fix remaining style issues * run without caches to compare execution time of test suite * allow redis config to not use ssl * remove stop-workers and query-workers tests from definitions.py * create helper_funcs file with common testing functions * move query-workers to pytest and add base class w/ stop-workers tests * update CHANGELOG * final changes for the stop-workers & query-workers tests * run fix-style * move stop and query workers tests to the same file * run fix-style * go back to original cache setup * try new cache for singularity install * fix syntax issue in github workflow * attempt to fix singularity cache * remove ls statement that breaks workflow * revert back to no common setup * remove unnecessary dependency * update github actions versions to use latest * update action versions that didn't save * run fix-style * move distributed test suite actions back to v2 * add 'merlin run' tests and port existing ones to pytest * update CHANGELOG * add aliased fixture types for typehinting * add tests for the purge command * update CHANGELOG * update run command tests to use conditions when appropriate * start work on adding workflow tests * create function and class scoped config fixtures * add Tuple fixture type * get e2e test of feature_demo workflow running * add check for proper variable substitution in e2e test * generalize functionality to run workflows * add create_testing_dir fixture * port chord error workflow to pytest * create dataclasses to house common fixtures and reduce fixture import requirements * fix lint issues * remove hard requirement of Annotated type for python 3.7 and 3.8 * remove distributed test CI and add unit test CI * fix typo in fixture_types and fix lint issues * run fix-style * add check for python2 before adding that condition check * convert local run test to use StepFinishedFilesCount condition * update CHANGELOG.md * fix problem created by merge conflict when mergin develop * remove manager functionality from this PR * update README for test suite * change SIGTERM to SIGKILL * update Makefile to include new changes to test suite --------- Co-authored-by: Ryan Lee <[email protected]> Co-authored-by: Ryan Lee <[email protected]>
LLNL · Nov 13, 2024 · e1655df · e1655df
1 parent 9f6d2ef
commit e1655df
Show file tree

Hide file tree

Showing 35 changed files with 3,199 additions and 660 deletions.
diff --git a/.github/workflows/push-pr_workflow.yml b/.github/workflows/push-pr_workflow.yml
@@ -10,7 +10,7 @@ jobs:
 
     steps:
     - name: Checkout code
-      uses: actions/checkout@v2
+      uses: actions/checkout@v4
       with:
         fetch-depth: 0 # Checkout the whole history, in case the target is way far behind
 
@@ -40,14 +40,14 @@ jobs:
       MAX_COMPLEXITY: 15
 
     steps:
-    - uses: actions/checkout@v2
+    - uses: actions/checkout@v4
     - name: Set up Python
-      uses: actions/setup-python@v2
+      uses: actions/setup-python@v5
       with:
         python-version: '3.x'
 
     - name: Check cache
-      uses: actions/cache@v2
+      uses: actions/cache@v4
       with:
         path: ~/.cache/pip
         key: ${{ hashFiles('requirements/release.txt') }}-${{ hashFiles('requirements/dev.txt') }}
@@ -95,14 +95,14 @@ jobs:
         python-version: ['3.7', '3.8', '3.9', '3.10', '3.11']
 
     steps:
-    - uses: actions/checkout@v2
+    - uses: actions/checkout@v4
     - name: Set up Python ${{ matrix.python-version }}
-      uses: actions/setup-python@v2
+      uses: actions/setup-python@v5
       with:
         python-version: ${{ matrix.python-version }}
 
     - name: Check cache
-      uses: actions/cache@v2
+      uses: actions/cache@v4
       with:
         path: ${{ env.pythonLocation }}
         key: ${{ env.pythonLocation }}-${{ hashFiles('requirements/release.txt') }}-${{ hashFiles('requirements/dev.txt') }}
@@ -112,8 +112,7 @@ jobs:
         python3 -m pip install --upgrade pip
         if [ -f requirements.txt ]; then pip install -r requirements.txt; fi
         pip3 install -r requirements/dev.txt
-        pip freeze
-    
+
     - name: Install singularity
       run: |
         sudo apt-get update && sudo apt-get install -y \
@@ -145,48 +144,31 @@ jobs:
         merlin example feature_demo
         pip3 install -r feature_demo/requirements.txt
 
-    - name: Run pytest over unit test suite
-      run: |
-        python3 -m pytest -v --order-scope=module tests/unit/
-
     - name: Run integration test suite for local tests
       run: |
         python3 tests/integration/run_tests.py --verbose --local
 
-  Distributed-test-suite:
+  Unit-tests:
     runs-on: ubuntu-latest
-    services:
-      # rabbitmq:
-      #   image: rabbitmq:latest
-      #   ports:
-      #     - 5672:5672
-      #   options: --health-cmd "rabbitmqctl node_health_check" --health-interval 10s --health-timeout 5s --health-retries 5
-      # Label used to access the service container
-      redis:
-        # Docker Hub image
-        image: redis
-        # Set health checks to wait until redis has started
-        options: >-
-          --health-cmd "redis-cli ping"
-          --health-interval 10s
-          --health-timeout 5s
-          --health-retries 5
-        ports:
-          - 6379:6379
+    env: 
+      GO_VERSION: 1.18.1
+      SINGULARITY_VERSION: 3.9.9
+      OS: linux
+      ARCH: amd64
 
     strategy:
       matrix:
         python-version: ['3.7', '3.8', '3.9', '3.10', '3.11']
 
     steps:
-    - uses: actions/checkout@v2
+    - uses: actions/checkout@v4
     - name: Set up Python ${{ matrix.python-version }}
-      uses: actions/setup-python@v2
+      uses: actions/setup-python@v5
       with:
         python-version: ${{ matrix.python-version }}
 
     - name: Check cache
-      uses: actions/cache@v2
+      uses: actions/cache@v4
       with:
         path: ${{ env.pythonLocation }}
         key: ${{ env.pythonLocation }}-${{ hashFiles('requirements/release.txt') }}-${{ hashFiles('requirements/dev.txt') }}
@@ -197,33 +179,104 @@ jobs:
         if [ -f requirements.txt ]; then pip install -r requirements.txt; fi
         pip3 install -r requirements/dev.txt
 
-    - name: Install merlin and setup redis as the broker
+    - name: Install singularity
+      run: |
+        sudo apt-get update && sudo apt-get install -y \
+          build-essential \
+          libssl-dev \
+          uuid-dev \
+          libgpgme11-dev \
+          squashfs-tools \
+          libseccomp-dev \
+          pkg-config
+        wget https://go.dev/dl/go$GO_VERSION.$OS-$ARCH.tar.gz
+        sudo tar -C /usr/local -xzf go$GO_VERSION.$OS-$ARCH.tar.gz
+        rm go$GO_VERSION.$OS-$ARCH.tar.gz
+        export PATH=$PATH:/usr/local/go/bin
+        wget https://github.com/sylabs/singularity/releases/download/v$SINGULARITY_VERSION/singularity-ce-$SINGULARITY_VERSION.tar.gz
+        tar -xzf singularity-ce-$SINGULARITY_VERSION.tar.gz
+        cd singularity-ce-$SINGULARITY_VERSION
+        ./mconfig && \
+          make -C ./builddir && \
+          sudo make -C ./builddir install
+
+    - name: Install merlin to run unit tests
       run: |
         pip3 install -e .
-        merlin config --broker redis
+        merlin config
 
     - name: Install CLI task dependencies generated from the 'feature demo' workflow
       run: |
         merlin example feature_demo
         pip3 install -r feature_demo/requirements.txt
 
+    - name: Run pytest over unit test suite
+      run: |
+        python3 -m pytest -v --order-scope=module tests/unit/
+
+  Integration-tests:
+    runs-on: ubuntu-latest
+    env: 
+      GO_VERSION: 1.18.1
+      SINGULARITY_VERSION: 3.9.9
+      OS: linux
+      ARCH: amd64
+
+    strategy:
+      matrix:
+        python-version: ['3.7', '3.8', '3.9', '3.10', '3.11']
+
+    steps:
+    - uses: actions/checkout@v4
+    - name: Set up Python ${{ matrix.python-version }}
+      uses: actions/setup-python@v5
+      with:
+        python-version: ${{ matrix.python-version }}
+
+    - name: Check cache
+      uses: actions/cache@v4
+      with:
+        path: ${{ env.pythonLocation }}
+        key: ${{ env.pythonLocation }}-${{ hashFiles('requirements/release.txt') }}-${{ hashFiles('requirements/dev.txt') }}
+
+    - name: Install dependencies
+      run: |
+        python3 -m pip install --upgrade pip
+        if [ -f requirements.txt ]; then pip install -r requirements.txt; fi
+        pip3 install -r requirements/dev.txt
+
+    - name: Install merlin
+      run: |
+        pip3 install -e .
+        merlin config
+
+    - name: Install singularity
+      run: |
+        sudo apt-get update && sudo apt-get install -y \
+          build-essential \
+          libssl-dev \
+          uuid-dev \
+          libgpgme11-dev \
+          squashfs-tools \
+          libseccomp-dev \
+          pkg-config
+        wget https://go.dev/dl/go$GO_VERSION.$OS-$ARCH.tar.gz
+        sudo tar -C /usr/local -xzf go$GO_VERSION.$OS-$ARCH.tar.gz
+        rm go$GO_VERSION.$OS-$ARCH.tar.gz
+        export PATH=$PATH:/usr/local/go/bin
+        wget https://github.com/sylabs/singularity/releases/download/v$SINGULARITY_VERSION/singularity-ce-$SINGULARITY_VERSION.tar.gz
+        tar -xzf singularity-ce-$SINGULARITY_VERSION.tar.gz
+        cd singularity-ce-$SINGULARITY_VERSION
+        ./mconfig && \
+          make -C ./builddir && \
+          sudo make -C ./builddir install
+
+    - name: Install CLI task dependencies generated from the 'feature demo' workflow
+      run: |
+        merlin example feature_demo
+        pip3 install -r feature_demo/requirements.txt
+
+    # TODO remove the --ignore statement once those tests are fixed
     - name: Run integration test suite for distributed tests
-      env:
-        REDIS_HOST: redis
-        REDIS_PORT: 6379
-      run: |
-        python3 tests/integration/run_tests.py --verbose --distributed
-
-    # - name: Setup rabbitmq config
-    #   run: |
-    #     merlin config --test rabbitmq
-
-    # - name: Run integration test suite for rabbitmq
-    #   env:
-    #     AMQP_URL: amqp://localhost:${{ job.services.rabbitmq.ports[5672] }}
-    #     RABBITMQ_USER: Jimmy_Space
-    #     RABBITMQ_PASS: Alexander_Rules
-    #   ports:
-    #     - ${{ job.services.rabbitmq.ports['5672'] }}
-    #   run: |
-    #     python3 tests/integration/run_tests.py --verbose --ids 31 32
+      run: |
+        python3 -m pytest -v --ignore tests/integration/test_celeryadapter.py tests/integration/
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -4,6 +4,21 @@ All notable changes to Merlin will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [Unreleased]
+### Added
+- Added additional tests for the `merlin run` and `merlin purge` commands
+- Aliased types to represent different types of pytest fixtures
+- New test condition `StepFinishedFilesCount` to help search for `MERLIN_FINISHED` files in output workspaces
+- Added "Unit-tests" GitHub action to run the unit test suite
+- Added `CeleryTaskManager` context manager to the test suite to ensure tasks are safely purged from queues if tests fail
+- Added `command-tests`, `workflow-tests`, and `integration-tests` to the Makefile
+
+### Changed
+- Ported all distributed tests of the integration test suite to pytest
+  - There is now a `commands/` directory and a `workflows/` directory under the integration suite to house these tests
+  - Removed the "Distributed-tests" GitHub action as these tests will now be run under "Integration-tests"
+- Removed `e2e-distributed*` definitions from the Makefile
+
 ## [1.12.2]
 ### Added
 - Conflict handler option to the `dict_deep_merge` function in `utils.py`

diff --git a/Makefile b/Makefile
@@ -34,12 +34,13 @@ include config.mk
 .PHONY : install-workflow-deps
 .PHONY : install-dev
 .PHONY : unit-tests
+.PHONY : command-tests
+.PHONY : workflow-tests
+.PHONY : integration-tests
 .PHONY : e2e-tests
 .PHONY : e2e-tests-diagnostic
 .PHONY : e2e-tests-local
 .PHONY : e2e-tests-local-diagnostic
-.PHONY : e2e-tests-distributed
-.PHONY : e2e-tests-distributed-diagnostic
 .PHONY : tests
 .PHONY : check-flake8
 .PHONY : check-black
@@ -89,6 +90,18 @@ unit-tests:
 	. $(VENV)/bin/activate; \
 	$(PYTHON) -m pytest -v --order-scope=module $(UNIT); \
 
+command-tests:
+	. $(VENV)/bin/activate; \
+	$(PYTHON) -m pytest -v $(TEST)/integration/commands/; \
+
+
+workflow-tests:
+	. $(VENV)/bin/activate; \
+	$(PYTHON) -m pytest -v $(TEST)/integration/workflows/; \
+
+
+integration-tests: command-tests workflow-tests
+
 
 # run CLI tests - these require an active install of merlin in a venv
 e2e-tests:
@@ -111,18 +124,8 @@ e2e-tests-local-diagnostic:
 	$(PYTHON) $(TEST)/integration/run_tests.py --local --verbose
 
 
-e2e-tests-distributed:
-	. $(VENV)/bin/activate; \
-	$(PYTHON) $(TEST)/integration/run_tests.py --distributed; \
-
-
-e2e-tests-distributed-diagnostic:
-	. $(VENV)/bin/activate; \
-	$(PYTHON) $(TEST)/integration/run_tests.py --distributed --verbose
-
-
 # run unit and CLI tests
-tests: unit-tests e2e-tests
+tests: unit-tests integration-tests e2e-tests
 
 
 check-flake8:

diff --git a/merlin/celery.py b/merlin/celery.py
@@ -114,9 +114,11 @@ def route_for_task(name, args, kwargs, options, task=None, **kw):  # pylint: dis
     BROKER_URI = None
     RESULTS_BACKEND_URI = None
 
+app_name = "merlin_test_app" if os.getenv("CELERY_ENV") == "test" else "merlin"
+
 # initialize app with essential properties
 app: Celery = patch_celery().Celery(
-    "merlin",
+    app_name,
     broker=BROKER_URI,
     backend=RESULTS_BACKEND_URI,
     broker_use_ssl=BROKER_SSL,

diff --git a/merlin/examples/dev_workflows/multiple_workers.yaml b/merlin/examples/dev_workflows/multiple_workers.yaml
@@ -46,11 +46,11 @@ merlin:
     resources:
         workers:
             step_1_merlin_test_worker:
-                args: -l INFO
+                args: -l INFO --concurrency 1
                 steps: [step_1]
             step_2_merlin_test_worker:
-                args: -l INFO
+                args: -l INFO --concurrency 1
                 steps: [step_2]
             other_merlin_test_worker:
-                args: -l INFO
+                args: -l INFO --concurrency 1
                 steps: [step_3, step_4]
diff --git a/tests/README.md b/tests/README.md
@@ -4,16 +4,20 @@ This directory utilizes pytest to create and run our test suite.
 
 This directory is organized like so:
 - `conftest.py` - The script containing common fixtures for our tests
+- `constants.py` - Constant values to be used throughout the test suite.
+- `fixture_data_classes.py` - Dataclasses to help group pytest fixtures together, reducing the required number of imports.
+- `fixture_types.py` - Aliases for type hinting fixtures.
 - `context_managers/` - The directory containing context managers used for testing
     - `celery_workers_manager.py` - A context manager used to manage celery workers for integration testing
     - `server_manager.py` - A context manager used to manage the redis server used for integration testing
 - `fixtures/` - The directory containing specific test module fixtures
     - `<test_module_name>.py` - Fixtures for specific test modules
 - `integration/` - The directory containing integration tests
-    <!-- - `test_*.py` - The actual test scripts to run -->
     - `definitions.py` - The test definitions
     - `run_tests.py` - The script to run the tests defined in `definitions.py`
     - `conditions.py` - The conditions to test against
+    - `commands/` - The directory containing tests for commands of the Merlin library.
+    - `workflows/` The directory containing tests for entire workflow runs.
 - `unit/` - The directory containing unit tests
     - `test_*.py` - The actual test scripts to run