You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Corresponding to #5 we currently support 3 types of inputs.
These are the following:
env db_table file
For each we need to add utility functions to the sdk to access the data using the information provided/defined within the cbc.yaml
E.g.
env => read/parse env file, throw if not set
db_table => connect to db, read, write, ...
file => connect to s3, read, write, ...
Important: As we are using Apache Spark to process our data we should definetly use their methods of handling data. It is important to take a closer look at these. (e.g. RDDs, how can we make them accessible to the sdks user very easily)
We could also add utility functions for testing input/outputs (connections etc.) before the actual workload starts.
The text was updated successfully, but these errors were encountered:
Corresponding to #5 we currently support 3 types of inputs.
These are the following:
env
db_table
file
For each we need to add utility functions to the sdk to access the data using the information provided/defined within the
cbc.yaml
E.g.
env => read/parse env file, throw if not set
db_table => connect to db, read, write, ...
file => connect to s3, read, write, ...
Important: As we are using Apache Spark to process our data we should definetly use their methods of handling data. It is important to take a closer look at these. (e.g. RDDs, how can we make them accessible to the sdks user very easily)
We could also add utility functions for testing input/outputs (connections etc.) before the actual workload starts.
The text was updated successfully, but these errors were encountered: