Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature end to end (Actually CI implementation) #79

Merged
merged 49 commits into from
Oct 18, 2023

Conversation

jeipollack
Copy link
Contributor

@jeipollack jeipollack commented Oct 5, 2023

This PR includes a Github Actions workflow defined in ci.yml, which runs some unit tests for some methods in io.py module.

There are also two validation tests for train.py and metrics.py, respectively. The baseline used for validation are the outputs of the main branch run over two cycles and epochs with a specific random seed used for initialisation of tensor flow. These can't be run as unit tests because they require GPUs which are scarcely available when launching the CI.

To validate the training results, a mock run of the refactored code is launched using the same configuration as the main branch. Results are stored in the pytest fixture tmp_path_factory. The checkpoint weights and final psf models are read in and compared with the baseline results and rms is computed. The validation test asserts for each weight that the error < 1.e-9.

To validate the metrics, there are four tests to evaluate the metrics: low-res pixel reconstruction metric, optical path difference metric, monochromatic metric, and shape metrics produced by the refactored code with respect to the metrics computed by the main branch again ran over exactly the same configuration. The weights are read in and the psf model is regenerated and evaluated according to each metric. The tolerance level varies in this case where low res the tol is set to 1e-7 whereas the rest is 1e.-9. Bc I saw that the difference was ~4.8e-8 for the low-res pixel reconstruction.

I merged changes in the last PR to this branch. I will work on the end-to-end test in another branch.

Unused modules were removed for the release.

This PR resolves resolves issue #47

@jeipollack jeipollack self-assigned this Oct 13, 2023
@jeipollack jeipollack added the enhancement New feature or request label Oct 13, 2023
@jeipollack jeipollack added this to the Version 0.0.1 milestone Oct 13, 2023
@jeipollack jeipollack changed the title Draft:Feature end to end Feature end to end (Actually CI implementation) Oct 13, 2023
Copy link
Member

@sfarrens sfarrens left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @jeipollack looks good overall. I think there are some files inside src/wf-psf that don't really belong there, but it should be fairly easy to fix.

.github/workflows/ci.yml Show resolved Hide resolved
.github/workflows/ci.yml Show resolved Hide resolved
pyproject.toml Show resolved Hide resolved
requirements.txt Outdated Show resolved Hide resolved
.github/workflows/ci.yml Outdated Show resolved Hide resolved
@sfarrens sfarrens merged commit 425cee7 into dummy_main Oct 18, 2023
1 check passed
@sfarrens sfarrens deleted the feature_end_to_end branch October 18, 2023 14:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants