Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Version/1.12.2 #492

Merged
merged 68 commits into from
Oct 28, 2024
Merged
Changes from 1 commit
Commits
Show all changes
68 commits
Select commit Hold shift + click to select a range
aeaef79
fix default worker bug with all steps
bgunnar5 May 4, 2023
ad75ed6
bugfix/worker-all-steps (#419)
bgunnar5 May 4, 2023
f7508dd
version bump and requirements fix
bgunnar5 May 4, 2023
c8c6a44
release/1.10.1 (#420)
bgunnar5 May 4, 2023
98e06f6
Bugfix/filename-special-vars (#425)
bgunnar5 May 25, 2023
b8dd2b2
Create dependabot-changelog-updater.yml
bgunnar5 May 26, 2023
4c40423
testing outputs of modifying changelog
bgunnar5 May 26, 2023
2d231fb
delete dependabot-changelog-updater
bgunnar5 May 26, 2023
6b142d9
feature/pdf-docs (#427)
bgunnar5 Jun 28, 2023
e3e1a30
bugfix/openfoam_singularity_issues (#426)
bgunnar5 Jun 28, 2023
12ff3d7
bugfix/output-path-substitution (#430)
bgunnar5 Jul 11, 2023
5c69c0b
bugfix/scheduler-permission-error (#436)
bgunnar5 Aug 4, 2023
c01f635
Release/1.10.2 (#437)
bgunnar5 Aug 4, 2023
7888493
resolve CHANGELOG conflict
bgunnar5 Aug 4, 2023
9c52ba2
resolve develop to main merge issues (#439)
bgunnar5 Aug 5, 2023
d63661c
resolve merge conflicts
bgunnar5 Aug 5, 2023
77d09c3
merge v1.10.2 into this branch
bgunnar5 Aug 14, 2023
b0f4d86
dependabot/certifi-requests-pygments (#441)
bgunnar5 Aug 17, 2023
c641c5c
bugfix/server-pip-redis-conf (#443)
bgunnar5 Aug 18, 2023
970a06f
bump to version 1.10.3 (#444)
bgunnar5 Aug 18, 2023
0384463
Merge branch 'main' into develop
bgunnar5 Aug 18, 2023
1977f1b
Merge remote-tracking branch 'origin/develop' into develop
bgunnar5 Aug 18, 2023
7155c6d
merge v1.10.3 to local main branch
bgunnar5 Aug 22, 2023
1b0fa95
trying to merge origin main
bgunnar5 Aug 22, 2023
d8bfbdb
bugfix/sphinx-5.3.0-requirement (#446)
bgunnar5 Aug 22, 2023
dabbd3f
Merge remote-tracking branch 'upstream-main/main'
bgunnar5 Aug 25, 2023
015e88b
Merge remote-tracking branch 'origin'
bgunnar5 Aug 25, 2023
ea715db
remove github text that was causing errors
bgunnar5 Aug 25, 2023
8241bfe
feature/vlauncher (#447)
bgunnar5 Sep 28, 2023
50d0fb6
release/1.11.0 (#448)
bgunnar5 Sep 28, 2023
066bea1
resolve merge conflicts
bgunnar5 Oct 5, 2023
99d5659
bugfix/skewed-sample-hierarchy (#450)
bgunnar5 Oct 9, 2023
593dbcd
bugfix/lsf-gpu-typo (#453)
bgunnar5 Oct 23, 2023
f994f96
release/1.11.1 (#454)
bgunnar5 Oct 23, 2023
89093dd
resolve merge conflict
bgunnar5 Oct 23, 2023
8d75482
pull changes from upstream main
bgunnar5 Oct 23, 2023
5dc8206
Add Pytest Fixtures to Test Suite (#456)
bgunnar5 Nov 2, 2023
38651f2
Bugfix for WEAVE CI (#457)
bgunnar5 Nov 15, 2023
b9afbdb
bugfix/monitor-shutdown (#452)
bgunnar5 Dec 12, 2023
642f925
Add the missing restart keyword to the specification docs. (#459)
koning Jan 19, 2024
40930c2
docs/conversion-to-mkdocs (#460)
bgunnar5 Jan 25, 2024
f36c58d
openfoam tutorial modifications (#463)
xorJane Feb 14, 2024
320d12f
feature/revamped status (#464)
bgunnar5 Feb 14, 2024
d111565
release/1.12.0 (#465)
bgunnar5 Feb 14, 2024
801d0bf
resolve merge conflicts with version bump
bgunnar5 Feb 14, 2024
acb1820
Merge remote-tracking branch 'upstream-main/main' into main
bgunnar5 Feb 15, 2024
9390448
feature/retry_priority (#468)
bgunnar5 Feb 22, 2024
de5367a
docs/server-cross-node (#470)
bgunnar5 Apr 10, 2024
cdad6e5
bugfix/initial-status-issues (#471)
bgunnar5 Apr 15, 2024
51d6787
release/1.12.1 (#472)
bgunnar5 Apr 15, 2024
4a07800
resolve merge conflicts with main
bgunnar5 Apr 16, 2024
bba68ab
fix a lint issue that somehow slipped through the cracks
bgunnar5 Apr 16, 2024
eace86f
Fix filenames for OpenFoam tutorial (#475)
xorJane Apr 30, 2024
f476a98
bugfix/deep-merge-existing-keys (#476)
bgunnar5 May 10, 2024
dd96092
Improved Info (#477)
lucpeterson May 16, 2024
12f2398
Target is in source's history (#478)
lucpeterson May 16, 2024
897312f
Fix link to merlin banner image (#479)
lucpeterson May 16, 2024
297d9d5
bugfix/status_nested_workspace (#480)
bgunnar5 May 22, 2024
4b4fdee
bugfix/celery-chord-error (#481)
bgunnar5 Jun 5, 2024
f29aabb
release/1.12.2b1 (#482)
bgunnar5 Jun 5, 2024
35c2039
resolve merge conflicts
bgunnar5 Jun 5, 2024
0f6bebf
bugfix/flux-nodes (#484)
bgunnar5 Jun 10, 2024
831bc40
bugfix/flux-nodes-prior-versions (#487)
bgunnar5 Jun 11, 2024
e919ee8
Change Task ID to directory path (#486)
lucpeterson Jun 12, 2024
773ef35
Adding Several New Unit Tests (#490)
bgunnar5 Oct 24, 2024
5bf7516
Merge remote-tracking branch 'upstream-main/main' into main
bgunnar5 Oct 28, 2024
dc0747f
release/1.12.2 (#491)
bgunnar5 Oct 28, 2024
9f6d2ef
resolve merge conflicts
bgunnar5 Oct 28, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
bugfix/status_nested_workspace (#480)
* remove a merge conflict statement that was missed

* have status ignore nested workspaces and modify merge rules

* update CHANGELOG

* fixed issue with escape sequences in ascii art

* apply Luc's suggestion

* add setuptools as a requirement since python 3.12 doesn't have it natively

* modify unit tests for status to use pytest rather than unittest

* update CHANGELOG

* add fixtures for status testing and add nested workflow test

* update CHANGELOG
  • Loading branch information
bgunnar5 authored May 22, 2024
commit 297d9d5382f26e680699b4c902383b3ff644058d
6 changes: 6 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -13,15 +13,21 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- Added tests for the `dict_deep_merge` function
- Pytest-mock as a dependency for the test suite (necessary for using mocks and fixtures in the same test)
- New github action test to make sure target branch has been merged into the source first, so we know histories are ok
- Check in the status commands to make sure we're not pulling statuses from nested workspaces
- Added `setuptools` as a requirement for python 3.12 to recognize the `pkg_resources` library

### Changed
- `merlin info` is cleaner and gives python package info
- merlin version now prints with every banner message
- Applying filters for `merlin detailed-status` will now log debug statements instead of warnings
- Modified the unit tests for the `merlin status` command to use pytest rather than unittest
- Added fixtures for `merlin status` tests that copy the workspace to a temporary directory so you can see exactly what's run in a test

### Fixed
- Bugfix for output of `merlin example openfoam_wf_singularity`
- A bug with the CHANGELOG detection test when the target branch isn't in the ci runner history
- Link to Merlin banner in readme
- Issue with escape sequences in ascii art (caught by python 3.12)


## [1.12.1]
8 changes: 4 additions & 4 deletions merlin/ascii_art.py
Original file line number Diff line number Diff line change
@@ -96,11 +96,11 @@


__ __ _ _
| \/ | | (_)
| \ / | ___ _ __| |_ _ __
| |\/| |/ _ \ '__| | | '_ \
| \\/ | | (_)
| \\ / | ___ _ __| |_ _ __
| |\\/| |/ _ \\ '__| | | '_ \\
| | | | __/ | | | | | | |
|_| |_|\___|_| |_|_|_| |_|
|_| |_|\\___|_| |_|_|_| |_|

Machine Learning for HPC Workflows

31 changes: 18 additions & 13 deletions merlin/study/status.py
Original file line number Diff line number Diff line change
@@ -344,7 +344,8 @@ def get_step_statuses(self, step_workspace: str, started_step_name: str) -> Dict
Given a step workspace and the name of the step, read in all the statuses
for the step and return them in a dict.

:param `step_workspace`: The path to the step we're going to read statuses from
:param step_workspace: The path to the step we're going to read statuses from
:param started_step_name: The name of the step that we're gathering statuses for
:returns: A dict of statuses for the given step
"""
step_statuses = {}
@@ -354,7 +355,12 @@ def get_step_statuses(self, step_workspace: str, started_step_name: str) -> Dict

# Traverse the step workspace and look for MERLIN_STATUS files
LOG.debug(f"Traversing '{step_workspace}' to find MERLIN_STATUS.json files...")
for root, _, _ in os.walk(step_workspace):
for root, dirs, _ in os.walk(step_workspace, topdown=True):
# Look for nested workspaces and skip them
timestamp_regex = r"\d{8}-\d{6}$"
curr_dir = os.path.split(root)[1]
dirs[:] = [d for d in dirs if not re.search(timestamp_regex, curr_dir)]

# Search for a status file
status_filepath = os.path.join(root, "MERLIN_STATUS.json")
matching_files = glob(status_filepath)
@@ -869,8 +875,7 @@ def apply_filters(self):
if matches_found == self.args.max_tasks:
break
else:
# If our filters aren't a match for this task then delete it
LOG.warning(f"No matching filter for '{sub_step_workspace}'.")
LOG.debug(f"No matching filter for '{sub_step_workspace}'.")

# If we've hit the limit set by args.max_tasks, break out of the outer loop
if matches_found == self.args.max_tasks:
@@ -1121,7 +1126,7 @@ def status_conflict_handler(*args, **kwargs) -> Any: # pylint: disable=W0613

There are currently 4 rules:
- string-concatenate: take the two conflicting values and concatenate them in a string
- use-initial-and-log-warning: use the value from dict_a and log a warning message
- use-dict_b-and-log-debug: use the value from dict_b and log a debug message
- use-longest-time: use the longest time between the two conflicting values
- use-max: use the larger integer between the two conflicting values

@@ -1136,8 +1141,8 @@ def status_conflict_handler(*args, **kwargs) -> Any: # pylint: disable=W0613
merge_rules = {
"task_queue": "string-concatenate",
"worker_name": "string-concatenate",
"status": "use-initial-and-log-warning",
"return_code": "use-initial-and-log-warning",
"status": "use-dict_b-and-log-debug",
"return_code": "use-dict_b-and-log-debug",
"elapsed_time": "use-longest-time",
"run_time": "use-longest-time",
"restarts": "use-max",
@@ -1150,13 +1155,13 @@ def status_conflict_handler(*args, **kwargs) -> Any: # pylint: disable=W0613

# params = self.spec.get_parameters()
# for token in params.parameters:
# merge_rules[token] = "use-initial-and-log-warning"
# merge_rules[token] = "use-dict_b-and-log-debug"

# Set parameter token key rules (commented for loop would be better but it's
# only possible if this conflict handler is contained within Status object; however,
# since this function needs to be imported outside of this file we can't do that)
if path is not None and "parameters" in path:
merge_rules[key] = "use-initial-and-log-warning"
merge_rules[key] = "use-dict_b-and-log-debug"

try:
merge_rule = merge_rules[key]
@@ -1168,13 +1173,13 @@ def status_conflict_handler(*args, **kwargs) -> Any: # pylint: disable=W0613

if merge_rule == "string-concatenate":
merge_val = f"{dict_a_val}, {dict_b_val}"
elif merge_rule == "use-initial-and-log-warning":
LOG.warning(
f"Conflict at key '{key}' while merging status files. Defaulting to initial value. "
elif merge_rule == "use-dict_b-and-log-debug":
LOG.debug(
f"Conflict at key '{key}' while merging status files. Using the updated value. "
"This could lead to incorrect status information, you may want to re-run in debug mode and "
"check the files in the output directory for this task."
)
merge_val = dict_a_val
merge_val = dict_b_val
elif merge_rule == "use-longest-time":
if dict_a_val == "--:--:--":
merge_val = dict_b_val
1 change: 1 addition & 0 deletions requirements/release.txt
Original file line number Diff line number Diff line change
@@ -9,5 +9,6 @@ numpy
parse
psutil>=5.1.0
pyyaml>=5.1.2
setuptools
tabulate
redis>=4.3.4
115 changes: 112 additions & 3 deletions tests/fixtures/status.py
Original file line number Diff line number Diff line change
@@ -4,17 +4,23 @@
"""

import os
import shutil
from argparse import Namespace
from pathlib import Path

import pytest
import yaml

from tests.unit.study.status_test_files import status_test_variables

@pytest.fixture(scope="class")

@pytest.fixture(scope="session")
def status_testing_dir(temp_output_dir: str) -> str:
"""
A pytest fixture to set up a temporary directory to write files to for testing status.

:param temp_output_dir: The path to the temporary output directory we'll be using for this test run
:returns: The path to the temporary testing directory for status testing
"""
testing_dir = f"{temp_output_dir}/status_testing/"
if not os.path.exists(testing_dir):
@@ -23,15 +29,118 @@ def status_testing_dir(temp_output_dir: str) -> str:
return testing_dir


@pytest.fixture(scope="class")
@pytest.fixture(scope="session")
def status_empty_file(status_testing_dir: str) -> str: # pylint: disable=W0621
"""
A pytest fixture to create an empty status file.

:param status_testing_dir: A pytest fixture that defines a path to the the output directory we'll write to
:param status_testing_dir: A pytest fixture that defines a path to the the output
directory we'll write to
:returns: The path to the empty status file
"""
empty_file = Path(f"{status_testing_dir}/empty_status.json")
if not empty_file.exists():
empty_file.touch()

return empty_file


@pytest.fixture(scope="session")
def status_spec_path(status_testing_dir: str) -> str: # pylint: disable=W0621
"""
Copy the test spec to the temp directory and modify the OUTPUT_PATH in the spec
to point to the temp location.

:param status_testing_dir: A pytest fixture that defines a path to the the output
directory we'll write to
:returns: The path to the spec file
"""
test_spec = f"{os.path.dirname(__file__)}/../unit/study/status_test_files/status_test_spec.yaml"
spec_in_temp_dir = f"{status_testing_dir}/status_test_spec.yaml"
shutil.copy(test_spec, spec_in_temp_dir) # copy test spec to temp directory

# Modify the OUTPUT_PATH variable to point to the temp directory
with open(spec_in_temp_dir, "r") as spec_file:
spec_contents = yaml.load(spec_file, yaml.Loader)
spec_contents["env"]["variables"]["OUTPUT_PATH"] = status_testing_dir
with open(spec_in_temp_dir, "w") as spec_file:
yaml.dump(spec_contents, spec_file)

return spec_in_temp_dir


def set_sample_path(output_workspace: str):
"""
A pytest fixture to set the path to the samples file in the test spec.

:param output_workspace: The workspace that we'll pull the spec file to update from
"""
temp_merlin_info_path = f"{output_workspace}/merlin_info"
expanded_spec_path = f"{temp_merlin_info_path}/status_test_spec.expanded.yaml"

# Read in the contents of the expanded spec
with open(expanded_spec_path, "r") as expanded_file:
expanded_contents = yaml.load(expanded_file, yaml.Loader)

# Modify the samples file path
expanded_contents["merlin"]["samples"]["file"] = f"{temp_merlin_info_path}/samples.csv"

# Write the new contents to the expanded spec
with open(expanded_spec_path, "w") as expanded_file:
yaml.dump(expanded_contents, expanded_file)


@pytest.fixture(scope="session")
def status_output_workspace(status_testing_dir: str) -> str: # pylint: disable=W0621
"""
A pytest fixture to copy the test output workspace for status to the temporary
status testing directory.

:param status_testing_dir: A pytest fixture that defines a path to the the output
directory we'll write to
:returns: The path to the output workspace in the temp status testing directory
"""
output_workspace = f"{status_testing_dir}/{status_test_variables.VALID_WORKSPACE}"
shutil.copytree(status_test_variables.VALID_WORKSPACE_PATH, output_workspace) # copy over the files
set_sample_path(output_workspace) # set the path to the samples file in the expanded yaml
return output_workspace


@pytest.fixture(scope="function")
def status_args():
"""
A pytest fixture to set up a namespace with all the arguments necessary for
the Status object.

:returns: The namespace with necessary arguments for the Status object
"""
return Namespace(
subparsers="status",
level="INFO",
detailed=False,
output_path=None,
task_server="celery",
cb_help=False,
dump=None,
no_prompts=True, # We'll set this to True here since it's easier to test this way
)


@pytest.fixture(scope="session")
def status_nested_workspace(status_testing_dir: str) -> str: # pylint: disable=W0621
"""
Create an output workspace that contains another output workspace within one of its
steps. In this case it will copy the status test workspace then within the 'just_samples'
step we'll copy the status test workspace again but with a different name.

:param status_testing_dir: A pytest fixture that defines a path to the the output
directory we'll write to
:returns: The path to the top level workspace
"""
top_level_workspace = f"{status_testing_dir}/status_test_study_nested_20240520-163524"
nested_workspace = f"{top_level_workspace}/just_samples/nested_workspace_20240520-163524"
shutil.copytree(status_test_variables.VALID_WORKSPACE_PATH, top_level_workspace) # copy over the top level workspace
shutil.copytree(status_test_variables.VALID_WORKSPACE_PATH, nested_workspace) # copy over the nested workspace
set_sample_path(top_level_workspace) # set the path to the samples file in the expanded yaml of the top level workspace
set_sample_path(nested_workspace) # set the path to the samples file in the expanded yaml of the nested workspace
return top_level_workspace
Loading
Loading