Division title importer/exporter #1765
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This was referenced Feb 14, 2024
Populate the php setting from an env var.
ajparsons
force-pushed
the
division-namer
branch
from
2cdd02e to
00020cb
Compare
|
I've added a venv creation process to Added some packages to the packages file for future docker use but these are all present on the server anyway so should be fine. |
ajparsons
force-pushed
the
division-namer
branch
from
00020cb to
f80d569
Compare
|
Issues above resolved, I've also just quickly updated the packages now we're not stuck on an old pandas version because of 3.7 - 4905383 |
dracos
approved these changes
Mar 21, 2024
- Update existing data to be 'Manual' when it has a yes_text.
- Export parquet dumps of divisions and votes - Ingest manual and automated updates of titles from remote sources Requirements.txt until server python is updated
- IN - division titles from Parliament where they exist - OUT - a dump of the divisions table (for twfy-votes).
ajparsons
force-pushed
the
division-namer
branch
from
4905383 to
2407762
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR adds
division_io.py- a click CLI to handle importing and exporting division data.This handles bulk importing updates to division titles (e.g. from Parliament's commons votes api), and exporting parquet tables of divisions as a basic api to feed twfy-votes.
There is an adjustment to the division table schema - adding a
title_priorityfield. This is to keep track of the origin of the current title, and stop manual updates to titles being overridden by automated ones. Existing titles with ayes_textare retrospectively updated in priority.For the moment, this uses a requirements.txt file for the needed python packages. When the server is updated, I'll do a new PR to tidy up the python-tooling in general. Unsure for the moment if this needs the morningupdate commands to run in a venv, or if system packages can just be updated in the short term.