NPI-3685 Streamline SP3 incorrect timerange unit test #71
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
… from 637K to 57K, and saving roughly 1-2 seconds in parsing time
…. Reset pyfakefs in various unit tests to avoid stale data impacting tests
…by dropping to a single SV
… as updated name was a bit misleading
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR attempts to speed up one of our slower unit tests, which parses a full day worth of SP3 data.
The number of sats in the test file has been dropped from 30 to 1.
Test run time is still slow, but perhaps 1-2 seconds faster than it was.
After a bit of investigation, it appears the main bottleneck in reading SP3 files, is the fact each epoch is parsed into a temporary
DataFrame, before being concatenated into the outputDataFrame. So, without significantly overhauling the relevant code, number of epochs is likely to be the main determiner of speed.Also added:
ContextTimer, which can be used as a context manager, to time function executions.reset()in various unit tests, to better guard against data from previous test runs impacting tests.