Skip to content
This repository has been archived by the owner on Sep 24, 2024. It is now read-only.

Integrate the archiver tools into Source.get() and the main workflow #22

Open
mjia8 opened this issue Jul 24, 2023 · 1 comment
Open
Assignees

Comments

@mjia8
Copy link
Collaborator

mjia8 commented Jul 24, 2023

No description provided.

@mjia8 mjia8 assigned mjia8, AaditT and jacquelinechernen and unassigned mjia8 Jul 31, 2023
@xgz2
Copy link
Collaborator

xgz2 commented Aug 1, 2023

Problem statement
The new functionality that we implemented (WFS download, Hash Comparing, crosswalk mapping guessing) and that Ethan included (archiving) currently live in separate files (lib/wfs_download.js, archiver/archive.js, lib/schema_matching.js). We would like to integrate these functionalities into the main workflow of the tool to accomplish our goals including enabling WFS download, checking hashes to prevent restoring existing files, easing the crosswalk mapping process, and aggressively archiving data and metadata.

Solution
Refer to the PRD for the specifics of what our goals and the final product should look like.
Use the following functions to accomplish our goals in the get() function of lib/source.js

  • downloadWfsDataInXmlFormat() in wfs_download.js
  • multiple functions in archiver.js, see the readme here for more information about how to do the archiving
  • multiple functions in archiver.js to do hash comparisons and store the data only if the oldest version is older than a certain amount of time or the data has a different hash

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants