Homework 1: Weather Forecast
Build a full-stack distributed system that enables users to query and visualize weather data from NEXRAD.
Following are the functionalities that our system must provide:
- A web based front-end that users can use to access our services.
- Aunthentication system to login to our system
- Detect storms
- Perform storm clustering
- Perform weather forecasts.
The components of the architecture and their functionalities are as follows:
- UI: Acts as the interface for the users to interact with our system.
- API Gateway: An API gateway is an API management tool that sits between a client and a collection of backend services. An API gateway acts as a reverse proxy to accept all application programming interface (API) calls, aggregate the various services required to fulfill them, and return the appropriate result.
- Data Ingestor: Microservice that accepts user inputs - Date, Time, and NEXRAD station name - and outputs a URL corresponding to file containing weather data.
- Registry Service: A central microservice that registers every action performed by the system and logs it. Also, it makes these logs queriable by users.
- Forecast Service: Microservice that gives a weather data file, analyses it and detects a storm using characteristics such as reflectivity. This can be done using existing C++ libraries or approaches such as Connected Component Analysis. This Microservice also groups all the storm events detected into spatial clusters using a Density-based clustering algorithm. It can be achieved using existing C++ libraries. Further, it determines whether or not to run weather forecasts and performs actual weather forecasts based on the trigger by forecast trigger service.
For all the microservices following are the common tasks.
- Decide on programming language
- Figure out the build framework and how to containerize it.
- Decide on communication to and from this service.
- Come up with API input and output formats.
Following lists specific tasks related to each microservice.
- Create a draft of the User Interface and decide the library or framework to be used.
- Decide how to work with Authentication
- Advanced track: using - http://arm-doe.github.io/pyart/ to plot weather data.
- The User Interface is a web application that is the point of entry for any user who wishes to use our service.
- The web app has been developed using JavaScript's popular library React.
- This web app provides an interface for the user to fetch weather data by providing the desired date, time, and radar station.
- We have used Google's OAuth for user authentication. This allows users to fetch their previous search logs.
- Once the user provides valid input and hits the Find button, the app communicates with our API gateway to retrieve the relevant data and display it on our UI.
- Here is an outline diagram of what our web app looks like.
- Decide on the database used and finalize database design.
- Come up with APIs to be used to expose data available in the database.
- Exposes API to query user details and log details.
- Exposes API to log user activity.
- API exposed can be consumed via HTTP Rest
- Integrated with Postgres as backend database.
- Understand how API gateways work and their functionalities.
- Figure out what are the available options and decide which suits our use case.
- How to configure and deploy it in a containerized fashion
- Exposes API to consume downstream services like forecaster, ingestor
- Logs user activity using registry microservices
- API exposed can be consumed via HTTP Rest
- Integrated with Google OAuth to validate user tokens.
- Figure out the URL formation to access weather data given user input and validate the formed URL.
- Determine how to pass this information along to storm detecter microservice
- The Data Ingestor microservice will fetch data from Nexrad's AWS S3 Bucket according to the user's query.
- It will then validate the query and download first four scans of the specified hour of the date.
- Downloads the desired data, and plots it accordingly and downloads the plot in the server side file system.
- Below is the visual of the service:
.png)
- Understand how to decide whether to run forecasts or not.
- Understand how to mock these decisions.
- Understand how to group storm events into spatial clusters using a Density-based clustering algorithm.
- Basic track: Understand the output data format and how to mock it
- Understand how to run weather forecasts and what the output will look like.
- Mock the output for weather forecast data.
All the microservices use REST protocol to communicate with each other. All the message exchanges take place in JSON format.
Cloud Events was explored to maintain standard of communication. However, to maintain simplicity in the first project, it was decided to go ahead with a simpler data model which only represents the core data being communicated - corresponds to the "data" attribute in cloud event.
Message queue-based communication was also explored RMQ.
Note: All the apis must include the following headers:
id_token: google authentication token
name: username ACC. to google
email: user's gmail id
Provides the weather data
Request:
{
"year": 2022,
"month": 2,
"day": 2,
"startTime": 1643807400279,
"endTime": 1643807400279,
"radarStation": "KAKQ"
}
Response:
{
"imageUrl": "url-of-plot",
"stormExists": True/False,
"weatherForecast": {
"min_temp": 10,
"max_temp": 50,
"humidity": 100,
"pressure": 15,
"weather_description": "clear"
}
}
api provides user details
sample usage: http://localhost:8082/getUser?userId=madhkr
Request parameters:
userId: userId for whom details is requested for.
Response:
{
"userId": "madhkr",
"userEmail": "[email protected]",
"name": "madhavan",
"timestamp": "2022-01-31",
"active": true
}
api provides a list of user activity logs
sample usage: http://localhost:8082/getLogs?userId=madhkr
Request parameters:
userId: userId for whom details is requested for.
Response:
[
{
"id": "4864b6ef-c3ef-4c6c-a03f-db255a7ed8ab",
"userId": "madhkr",
"serviceId": "registry",
"action": "addUser",
"timestamp": "2022-01-31",
"comments": "successfully added user",
"status": 0
},
{
"id": "91acdc6f-a7b3-4940-8e11-d7220166e0fa",
"userId": "madhkr",
"serviceId": "registry",
"action": "getUser",
"timestamp": "2022-01-31",
"comments": "successfully queried user details",
"status": 0
},
{
"id": "87ae0e08-4c71-4f7c-bba4-afee1633727f",
"userId": "madhkr",
"serviceId": "registry",
"action": "getAppLogs",
"timestamp": "2022-01-31",
"comments": "successfully queried logs",
"status": 0
}
]
returns activity log for a specified user
Request parameters:
userId: userId for whom log is requested
Response:
[
{
"id": "4864b6ef-c3ef-4c6c-a03f-db255a7ed8ab",
"userId": "madhkr",
"serviceId": "registry",
"action": "addUser",
"timestamp": "2022-01-31",
"comments": "successfully added user",
"status": 0
},
{
"id": "91acdc6f-a7b3-4940-8e11-d7220166e0fa",
"userId": "madhkr",
"serviceId": "registry",
"action": "getUser",
"timestamp": "2022-01-31",
"comments": "successfully queried user details",
"status": 0
},
{
"id": "87ae0e08-4c71-4f7c-bba4-afee1633727f",
"userId": "madhkr",
"serviceId": "registry",
"action": "getAppLogs",
"timestamp": "2022-01-31",
"comments": "successfully queried logs",
"status": 0
}
]
Adds a user activity log to database
Request:
{
"id": "sample-id",
"userId": "madhkr1",
"serviceId": "sample-service-id",
"action": "sample-action",
"timestamp": 1643534040000,
"status": 1,
"comments": "sample-comments"
}
Response:
{
"message": "successfully added applog",
"status": "201 CREATED"
}
Adds a user to database
Request:
{
"userId": "madhkr",
"userEmail": "[email protected]",
"timestamp": "1643538067000"
"name": "Madhavan"
}
Response:
{
"message": "successfully added user",
"status": "201 CREATED"
}
API gets the data from the S3 bucket for further processing
sample usage: http://localhost:8000/getdata?year=2000&month=03&day=31&starttime=17&endtime=19&radar=KTLX&id=292
Request parameters:
year : year for the data requested
month : month for the data requested
day : day for the data requested
starttime : starttime for the data requested
endtime : endtime for the data requested
radar : radar (station) for the data requested
id : id for the data requested
Response:
{ "data": "id.png" }
API gets the filename of the image
Request
sample usage: http://localhost:8000/getimage
Request parameters:
filename : name of the file
Response:
The output will be the id.png file
API returns an OK message
sample usage: http://localhost:8000/health
Request parameters:
No request parameters
Response:
{
"Health": OK
}
API returns a random boolean value to detect storm clustering
sample usage: http://localhost:8001/stormclustering
Request parameters:
No request parameters
Response:
{
"storm_detected": true
}
Response will switch between true and false
API returns weather data
Request
sample usage: http://localhost:8001/forecast
Request parameters:
No request parameters
Response:
{
"weather data": {
"min_temp": 60,
"max_temp": 108,
"pressure": 0,
"humidity": 160
}
}
API returns an OK message
sample usage: http://localhost:8001/health
Request parameters:
No request parameters
Response:
{
"Health": OK
}
- Installing java 11 on mac, windows and linux
- Installing maven on mac, windows, and linux
- Installing docker on mac, windows, and linux
- Installing python3 on mac, windows, and linux
- Installing npm on mac, windows, and linux
docker network create elves-network
We are using postgres as our database to store user details and logs of user activities.
Postgres can be set up in a docker container with following steps:
pull the docker image
docker pull postgres
run the docker container
docker run --net elves-network --name elves-postgres -p 5001:5432 -e POSTGRES_PASSWORD=cloudelves -d postgres
Next, we must set up the registry microservice, which is responsible for storing user details and logs of user activities.
To set up the registry first clone the CloudElves repository and checkout Homework1-release-registry branch by running the below commands:
git clone https://github.com/airavata-courses/CloudElves.git
cd CloudElves
git checkout Homework1-release-registry
Then, run the following command to build a runnable jar and docker container from it.
Note: Following commands needs to be executed from the root directory of project (same level as pom.xml)
mvn clean package -Dmaven.test.skip
this will create a runnable jar in the target/ folder
create the docker image with the following command:
docker build -t cloud-elves/registry .
The docker image can be executed with the following command
docker run -d -e db.host=elves-postgres -e db.port=5432 -p 8080:8080 --net elves-network --name registry cloud-elves/registry
PS: above command will run registry microservice on port 8080, you can verify by hitting the health endpoint
git clone https://github.com/airavata-courses/CloudElves.git
cd CloudElves
git checkout Homework1-release-ingestor
docker build -t cloud-elves/ingestor .
The docker image can be executed with the following command
docker run --rm -d --net elves-network -p 8000:8000 --name ingestor cloud-elves/ingestor
PS: above command will run ingestor microservice on port 8000, you can verify by hitting the health endpoint
git clone https://github.com/airavata-courses/CloudElves.git
cd CloudElves
git checkout Homework1-release-forecast
docker build -t cloud-elves/forecast .
The docker image can be executed with the following command
docker run --rm -d --net elves-network -p 8001:8001 --name forecast cloud-elves/forecast
PS: above command will run forecast microservice on port 8001, you can verify by hitting the health endpoint
Next, we must set up the gateway microservice, which is responsible for exposing user-facing APIS.
To set up the gateway first clone the CloudElves repository and checkout Homework1-release-gateway branch by running the below commands:
git clone https://github.com/airavata-courses/CloudElves.git
cd CloudElves
git checkout Homework1-release-gateway
Then, run the following command to build a runnable jar and docker container from it.
Note: Following commands needs to be executed from the root directory of the project (same level as pom.xml)
mvn clean package
this will create a runnable jar in the target/ folder
create the docker image with the following command:
docker build -t cloud-elves/gateway .
The docker image can be executed with the following command
docker run --rm -d --net elves-network -e forecaster.host=forecast -p 8082:8082 --name gateway cloud-elves/gateway
PS: above command will run gateway microservice on port 8082, you can verify by hitting the health endpoint
Finally, we can deploy the UI with the following commands
To set up the UI which is a React App first clone the CloudElves repository and checkout Homework1-release-ui branch by running the below commands:
git clone https://github.com/airavata-courses/CloudElves.git
cd CloudElves
git checkout Homework1-release-ui
create the docker image with the following command:
docker build -t cloud-elves/ui .
The docker image can be executed with the following command:
docker run -it -d --net elves-network --rm -p 3001:3001 --name ui cloud-elves/ui
PS: above command will run cloud weather app on port 3001, you can verify by hitting the UI endpoint