-
Notifications
You must be signed in to change notification settings - Fork 9
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Feature end to end (Actually CI implementation) #79
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
jeipollack
changed the title
Draft:Feature end to end
Feature end to end (Actually CI implementation)
Oct 13, 2023
sfarrens
reviewed
Oct 16, 2023
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hi @jeipollack looks good overall. I think there are some files inside src/wf-psf
that don't really belong there, but it should be fairly easy to fix.
src/wf_psf/tests/data/validation/end_to_end/config/configs.yaml
Outdated
Show resolved
Hide resolved
sfarrens
approved these changes
Oct 18, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR includes a Github Actions workflow defined in
ci.yml
, which runs some unit tests for some methods inio.py
module.There are also two validation tests for train.py and metrics.py, respectively. The baseline used for validation are the outputs of the main branch run over two cycles and epochs with a specific random seed used for initialisation of tensor flow. These can't be run as unit tests because they require GPUs which are scarcely available when launching the CI.
To validate the training results, a mock run of the refactored code is launched using the same configuration as the main branch. Results are stored in the pytest fixture tmp_path_factory. The checkpoint weights and final psf models are read in and compared with the baseline results and rms is computed. The validation test asserts for each weight that the error < 1.e-9.
To validate the metrics, there are four tests to evaluate the metrics: low-res pixel reconstruction metric, optical path difference metric, monochromatic metric, and shape metrics produced by the refactored code with respect to the metrics computed by the main branch again ran over exactly the same configuration. The weights are read in and the psf model is regenerated and evaluated according to each metric. The tolerance level varies in this case where low res the tol is set to 1e-7 whereas the rest is 1e.-9. Bc I saw that the difference was ~4.8e-8 for the low-res pixel reconstruction.
I merged changes in the last PR to this branch. I will work on the end-to-end test in another branch.
Unused modules were removed for the release.
This PR resolves resolves issue #47