Initialize Duckdb Without Huggingface

[ryanfchase]

Overview

We need to create a version of our app that initializes the map without loading Huggingface, since we are in the process of migrating to a Socrata API data ingestion

Developer Info

We are NOT merging this into develop. This is meant to be a staging branch for the Blank Map feature ONLY

• I understand

Action Items

• designate a feature branch, e.g. staging-feature-blank-map)
• manually define our requests table on application load (see createRequestsTable)
• configure the data population to use a config variable to determine its source (see proposed constant, DATA_SOURCE)
• (optional) only create table once, ensure only 1 table exists (as opposed to a table for every year)

Resources/Instructions

Notes

• currently we rely on parquet files (e.g. 2024.parquet) from HuggingFace to define the requests table
  • obtain the table columns and field types and use them to define the requests table

Resources

• Duckdb Docs: CREATE TABLE

Code References

• createRequestsTable
  • path: components > Map > index.jsx::MapContainer, L69
  • note: SQL on L77 will need to modified. We will not be using datasetFileName and instead we'll simply define the table inline
• CONSTANTS.js
  • path: components > common > CONSTANTS.js
• proposed object: DATA_SOURCE
  • HUGGING_FACE: load the data from HuggingFace, (this is the usual way we load the application)
  • SOCRATA: load the data directly from Socrata API call (this is yet to be implemented)

export const DATA_SOURCE = {
  'HUGGING_FACE': 0,
  'SOCRATA': 1,
};

• updateHfDataset.py
  • path: scripts > updateHfDataset.py
  • note: use this as a reference to how we are handling timestampformat