Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add utility functions #8

Open
PaulKalho opened this issue Nov 4, 2024 · 0 comments
Open

Add utility functions #8

PaulKalho opened this issue Nov 4, 2024 · 0 comments
Assignees

Comments

@PaulKalho
Copy link
Member

PaulKalho commented Nov 4, 2024

Corresponding to #5 we currently support 3 types of inputs.

These are the following:

env
db_table
file

For each we need to add utility functions to the sdk to access the data using the information provided/defined within the cbc.yaml

E.g.

env => read/parse env file, throw if not set
db_table => connect to db, read, write, ...
file => connect to s3, read, write, ...

Important: As we are using Apache Spark to process our data we should definetly use their methods of handling data. It is important to take a closer look at these. (e.g. RDDs, how can we make them accessible to the sdks user very easily)

We could also add utility functions for testing input/outputs (connections etc.) before the actual workload starts.

@PaulKalho PaulKalho self-assigned this Nov 13, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant