diff --git a/README.md b/README.md new file mode 100644 index 0000000..68ef4b8 --- /dev/null +++ b/README.md @@ -0,0 +1,93 @@ +# Satellite Data Download and Geoserver Upload + +The main goal of this project is to facilitate the process of downloading satellite data packages and uploading them to a Geoserver instance. This tool allows users to easily retrieve data from sources like CHIRPS and ERA5, process it, and then publish the data in a Geoserver workspace for visualization and analysis. + +# Satellite Data ETL + +This component extracts data from external satellite sources (CHIRPS and ERA5), processes it, and uploads the processed data to a Geoserver workspace. Its primary function is to streamline the download, validation, and publishing of satellite data. + +## Features + +- Built using Python +- Supports data download from CHIRPS and ERA5 +- Automates Geoserver upload of processed data +- Supports Python >= 3.10 + +### Prerequisites + +- Python >= 3.10 +- Geoserver credentials and URL + +### Project Structure + +- `downloadedData/:` Directory for storing downloaded raw data. +- `output/:` Folder where processed data is saved. +- `shapefiles/:` Contains shapefiles for different regions. For example, NICARAGUA/ includes Nicaragua shapefiles (.shp, .dbf, .prj, etc.). +- `src/:` Main source code folder. +- `src/geoserver_conexion/:` Handles connections to Geoserver. + +# Instalation + +To use ETL we must install a set of requirements, which are in a text file, for this process we recommend to create a virtual environment, this in order not to install these requirements in the entire operating system. + +1. Clone the repository + +```sh +git https://github.com/CIAT-DAPA/aclimate_etl_import_satellite_data +``` + +2. Create a virtual environment + +```sh +python -m venv venv +``` + +3. Activate the virtual environment + +- Linux + +```sh +source env/bin/activate +``` + +- windows + +```sh +env\Scripts\activate.bat +``` + +4. Install the required packages + +```sh +pip install -r requirements.txt +``` + +### Required Environment Variables + +Make sure to set up the following environment variables to connect with your Geoserver instance: + +- GEO_USER: Your Geoserver username. +- EO_PASS: Your Geoserver password. +- GEO_URL: The URL of your Geoserver instance. + +### Execution + +Use the `main.py` script located inside the src/download_process/ directory to start the data download and upload process. + +Command-line Parameters +The script accepts several command-line arguments to control the process: + +`-o` or `--outputs`: Path where processed data will be saved. +`-s` or `--startDate`: Start date for downloading data, formatted as YYYY-MM. +`-e` or `--endDate`: End date for downloading data, formatted as YYYY-MM. +`-c` or `--country`: Country for which the data is being cut. +`-d` or `--download`: Path where raw data will be downloaded. +`-w` or `--workspace`: Geoserver workspace where the data will be uploaded. + +## Usage examples + +To download data for Nicaragua and upload it to the Geoserver: + +```bash +python src/download_process/main.py -o output/ -s 2024-07 -e 2024-09 -c NICARAGUA -d downloadedData/ -w +``` diff --git a/src/download_forecast/__init__.py b/src/download_process/__init__.py similarity index 100% rename from src/download_forecast/__init__.py rename to src/download_process/__init__.py diff --git a/src/download_forecast/chirps_data.py b/src/download_process/chirps_data.py similarity index 100% rename from src/download_forecast/chirps_data.py rename to src/download_process/chirps_data.py diff --git a/src/download_forecast/era5_data.py b/src/download_process/era5_data.py similarity index 100% rename from src/download_forecast/era5_data.py rename to src/download_process/era5_data.py diff --git a/src/download_forecast/geoserver_conexion/__init__.py b/src/download_process/geoserver_conexion/__init__.py similarity index 100% rename from src/download_forecast/geoserver_conexion/__init__.py rename to src/download_process/geoserver_conexion/__init__.py diff --git a/src/download_forecast/geoserver_conexion/geoserver.py b/src/download_process/geoserver_conexion/geoserver.py similarity index 100% rename from src/download_forecast/geoserver_conexion/geoserver.py rename to src/download_process/geoserver_conexion/geoserver.py diff --git a/src/download_forecast/geoserver_conexion/properties/indexer.properties b/src/download_process/geoserver_conexion/properties/indexer.properties similarity index 100% rename from src/download_forecast/geoserver_conexion/properties/indexer.properties rename to src/download_process/geoserver_conexion/properties/indexer.properties diff --git a/src/download_forecast/geoserver_conexion/properties/timeregex.properties b/src/download_process/geoserver_conexion/properties/timeregex.properties similarity index 100% rename from src/download_forecast/geoserver_conexion/properties/timeregex.properties rename to src/download_process/geoserver_conexion/properties/timeregex.properties diff --git a/src/download_forecast/geoserver_conexion/tool.py b/src/download_process/geoserver_conexion/tool.py similarity index 100% rename from src/download_forecast/geoserver_conexion/tool.py rename to src/download_process/geoserver_conexion/tool.py diff --git a/src/download_forecast/geoserver_upload.py b/src/download_process/geoserver_upload.py similarity index 100% rename from src/download_forecast/geoserver_upload.py rename to src/download_process/geoserver_upload.py diff --git a/src/download_forecast/main.py b/src/download_process/main.py similarity index 100% rename from src/download_forecast/main.py rename to src/download_process/main.py diff --git a/src/download_forecast/tools.py b/src/download_process/tools.py similarity index 100% rename from src/download_forecast/tools.py rename to src/download_process/tools.py