Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DM-45263: Add new tap_schema module #90

Merged
merged 31 commits into from
Oct 16, 2024
Merged

DM-45263: Add new tap_schema module #90

merged 31 commits into from
Oct 16, 2024

Conversation

JeremyMcCormick
Copy link
Collaborator

@JeremyMcCormick JeremyMcCormick commented Jul 18, 2024

This provides the initial implementation for a tap_schema module, eventually intended to deprecate and replace the functionality of the Felis tap module. It provides the new load-tap-schema command in the Felis CLI, which uses the new module to update a TAP_SCHEMA database. The internal representation of TAP_SCHEMA was changed to use a Felis file, found in the felis.schemas package.

Additional Changes

  • Added a few utility functions to felis.datamodel.Schema for looking up objects
  • Added methods for loading a Schema from a file or lsst.resources.ResourcePath
    • Added lsst-resources to project requirements
  • Added a test.utils module with some useful test utility functions
  • Updated project configuration to include YAML files from the felis.schemas package, which currently includes a standard TAP_SCHEMA representation
  • Added a Makefile target for installing project dependencies
  • Added the LSST pipelines site to intersphinx dependencies and removed trailing spaces from links (causes problems for some sites)

Relatively complete test coverageis provided for the new module. Some functions marked as not covered actually are tested from loading schema files from tests/data. The tap module, which this is designed to replace, remains minimally covered in testing.

No breaking changes were intended to be introduced to the API or the CLI by this update. The felis load-tap and init-tap commands should still work exactly the same as before and can be used side-by-side with the new module for comparison.


  • Ran in Jenkins CI
  • Added a news fragment

@JeremyMcCormick JeremyMcCormick marked this pull request as draft July 18, 2024 23:46
@JeremyMcCormick JeremyMcCormick force-pushed the tickets/DM-45263 branch 2 times, most recently from 626f499 to fd25d5a Compare July 18, 2024 23:49
Copy link

codecov bot commented Jul 18, 2024

Codecov Report

Attention: Patch coverage is 92.76944% with 53 lines in your changes missing coverage. Please review.

Project coverage is 92.02%. Comparing base (b34c43b) to head (d49165d).
Report is 32 commits behind head on main.

Files with missing lines Patch % Lines
python/felis/db/utils.py 52.63% 15 Missing and 3 partials ⚠️
python/felis/tap_schema.py 93.65% 9 Missing and 4 partials ⚠️
tests/test_tap_schema_postgres.py 87.50% 8 Missing and 2 partials ⚠️
python/felis/datamodel.py 90.56% 4 Missing and 1 partial ⚠️
python/felis/cli.py 91.66% 1 Missing and 1 partial ⚠️
python/felis/tests/utils.py 93.75% 1 Missing and 1 partial ⚠️
tests/test_metadata.py 96.00% 0 Missing and 2 partials ⚠️
tests/test_tap_schema.py 99.50% 0 Missing and 1 partial ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main      #90      +/-   ##
==========================================
+ Coverage   91.54%   92.02%   +0.48%     
==========================================
  Files          16       20       +4     
  Lines        1881     2534     +653     
  Branches      410      519     +109     
==========================================
+ Hits         1722     2332     +610     
- Misses         86      120      +34     
- Partials       73       82       +9     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@JeremyMcCormick JeremyMcCormick force-pushed the tickets/DM-45263 branch 5 times, most recently from 3bc98e3 to fe43632 Compare July 19, 2024 01:16
@JeremyMcCormick JeremyMcCormick changed the title DM-45263: Add new module for managing TAP_SCHEMA DM-45263: Add new tap_schema module Jul 19, 2024
@JeremyMcCormick JeremyMcCormick force-pushed the tickets/DM-45263 branch 8 times, most recently from 12e6c6e to d50ab43 Compare August 21, 2024 20:05
@JeremyMcCormick JeremyMcCormick force-pushed the tickets/DM-45263 branch 5 times, most recently from fda50b2 to 8b3334f Compare August 21, 2024 22:19
@JeremyMcCormick JeremyMcCormick force-pushed the tickets/DM-45263 branch 6 times, most recently from 672bd5f to baed0d4 Compare August 29, 2024 20:46
This makes internal YAML files available as resources when using
ResourcePath from lsst-resources.
@JeremyMcCormick JeremyMcCormick force-pushed the tickets/DM-45263 branch 5 times, most recently from 2e978e7 to 54fab96 Compare October 4, 2024 16:50
Implement loading of Felis schema data into TAP_SCHEMA using new
TableManager and DataLoader classes.

Initialization of the TAP_SCHEMA database is not supported directly
but can be done using the standard 'felis create' command using the
appropriate schema file which defines a compatible database.
This Felis file defines a TAP_SCHEMA database with nonstandard names
for the tables, postfixed with "11", and an alternate name for the
schema itself of "tap_schema11". These alternate names are used to test
functionality of the tap_schema TableManager class.
Use testing.postgresql to setup a temporary database for testing the
functionality of the tap_schema module.
Several of these checks have been put into the data model instead and
should not be necessary to repeat here.
@JeremyMcCormick JeremyMcCormick merged commit 4171af5 into main Oct 16, 2024
15 checks passed
@JeremyMcCormick JeremyMcCormick deleted the tickets/DM-45263 branch October 16, 2024 22:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants