feat: simplifying dev environment and required infra/services

.gitignore

@@ -5,6 +5,8 @@
 .tmpdirs
 .pytest_cache
 /.env
+dev/data/
+venv/
 
 syntax: glob
 __pycache__

README.md

@@ -146,3 +146,80 @@ This also allow auto reloading.
 To run tests, run `yarn build` (or `poetry run frontend-build`) at least once
 (that build is used for static serving tests), activate the virtualenv,
 then run `pytest`.
+
+### Setting up a Development Environment with Docker Compose
+
+To quickly set up your local infrastructure for development on your laptop using Docker containers, follow these three simple steps (assuming that you already have Docker and Docker-compose pre-installed):
+
+```bash
+# clone the repository
+git clone [email protected]:NLNOG/irrexplorer.git`
+# go to the dev directory
+cd dev/
+# start the development environment
+docker-compose -f docker-up -d
+```
+
+After executing these steps, you will have all the necessary services up and running for development:
+```bash
+% docker ps
+CONTAINER ID   IMAGE             COMMAND                  CREATED          STATUS                    PORTS                                            NAMES
+9b2ee198962f   dev-irrexplorer   "/app/irrexplorer/in…"   34 minutes ago   Up 33 minutes (healthy)   0.0.0.0:8000->8000/tcp                           irrexplorer
+3b08fb5ba0c8   dev-irrd          "/app/irrd/init"         34 minutes ago   Up 33 minutes (healthy)   0.0.0.0:8043->8043/tcp, 0.0.0.0:8080->8080/tcp   irrd
+67284918c7ea   postgres          "docker-entrypoint.s…"   34 minutes ago   Up 34 minutes (healthy)   5432/tcp                                         postgres_irre
+c689fad6d0e9   postgres          "docker-entrypoint.s…"   34 minutes ago   Up 34 minutes (healthy)   5432/tcp                                         postgres_irrd
+36414b39a412   redis             "docker-entrypoint.s…"   34 minutes ago   Up 34 minutes (healthy)   6379/tcp                                         redis_irrd
+```
+
+> You can can access IRR explorer web interface at http://localhost:8000/ and the API at http://localhost:8000/api/.
+
+> IRRd web interface available at http://localhost:8080/.
+
+> If you need to include or remove Routing Registries, check the file `irrd.yaml`/`sources`.
+
+If you need to access the containers, use the following command:
+```bash
+docker exec -it <container_name> /bin/sh
+```
+
+Since uvicorn is running in the foreground, you can access the logs with:
+```bash
+docker logs -f irrexplorer
+```
+
+Uvicorn supports auto-reloading, making it easy to check changes. If a complete reload is required, simply run:
+```bash
+docker container restart irrexplorer
+```
+
+ Tail container logs:
+```bash
+docker-compose logs -f --tail 100
+```
+
+
+#### To streamline your development workflow, we have included several convenient `Make` commands:
+
+`make clean`: Removes all unused Docker objects (containers, networks, volumes, and images) using `docker system prune --all`.
+
+`make start`: Starts the Docker containers defined in the `docker-compose.yml` file using `docker-compose up -d`.
+
+`make stop`: Stops and removes the Docker containers defined in the docker-compose.yml file using docker-compose down.
+
+`make rebuild`: Executes the stop, clean, and start commands in sequence to rebuild the development environment.
+
+`make dump_irrd_db`: Dumps the irrd database from the `postgres_irrd` container to a file named `irrdb.sql` using `docker exec` and `pg_dump`.
+
+`make dump_irre_db`: Dumps the irrexplorer database from the `postgres_irre` container to a file named `irrexplorer_db.sql` using `docker exec` and `pg_dump`.
+
+These commands provide a convenient way to manage the development environment, including starting, stopping, rebuilding, and dumping the databases.
+
+#### The infrastructure includes the following components:
+
+- PostgreSQL running for IRR Explorer using credentials specified in the docker-compose.yml;
+- PostgreSQL running for IRRd using credentials specified in the docker-compose.yml;
+- Redis used by IRRd;
+- IRRd caching registry data, keeping it updated using NRTM, and serving this data to IRR Explorer;
+- **Finally, IRR Explorer itself!**
+
+Happy coding!

dev/.env

@@ -0,0 +1,4 @@
+DATABASE_URL=postgresql://irrexplorer:irrexplorer@postgres_irre:5432/irrexplorer?keepalives=1&keepalives_idle=5
+IRRD_ENDPOINT=http://irrd:8080/graphql/
+DEBUG=False
+REGISTROBR_URL=https://ftp.registro.br/pub/numeracao/origin/nicbr-asn-blk-latest.txt

dev/Dockerfiles/irrd4

@@ -0,0 +1,28 @@
+# description="IRRd4 Docker Image"
+#
+# Dockerfile for the IRRd4 application.
+#
+# This Dockerfile sets up a Python 3.12.3 environment and installs the necessary dependencies for running the IRRd4 application. It copies an init script to the container, updates the package lists, installs build tools and other required packages, installs the IRRd4 package, and sets up the necessary directories and permissions.
+#
+# The resulting Docker image can be used to run the IRRd4 application in a containerized environment.
+
+FROM python:3.12.3-bookworm
+
+ARG IRRD_VERSION=4.4.4
+
+COPY init-irrd.sh /app/irrd/init
+
+RUN apt-get update \
+    && apt-get install -y --no-install-recommends build-essential gcc gnupg dumb-init cron
+
+RUN pip install --no-cache-dir -U pip psycopg2 \
+    && pip install --no-cache-dir irrd==$IRRD_VERSION
+
+RUN apt-get clean autoclean \
+    && apt-get autoremove -y \
+    && rm -rf /var/lib/apt/lists/*
+
+RUN mkdir /var/run/irrd
+RUN chown daemon:daemon /var/run/irrd
+
+CMD ["/app/irrd/init"]

dev/Dockerfiles/irrexplorer

@@ -0,0 +1,23 @@
+# description="IRR explorer Docker Image"
+# Dockerfile for the IRR explorer application.
+#
+# This Dockerfile sets up a Python 3.12.3 environment with the necessary dependencies to run the IRR explorer application. It installs Git, Node.js, and npm, and then uses pip and yarn to install additional dependencies.
+#
+# The resulting Docker image can be used to install/run the IRR explorer application in a containerized environment.
+
+FROM python:3.12.3-bookworm
+
+WORKDIR /app/irrexplorer
+
+RUN apt-get update \
+    && apt-get install -y --no-install-recommends git nodejs npm 
+
+RUN pip install --no-cache-dir -U pip
+
+RUN npm install -g yarn
+
+RUN apt-get clean autoclean \
+    && apt-get autoremove -y \
+    && rm -rf /var/lib/apt/lists/*
+
+CMD ["/app/irrexplorer/init"]

dev/Makefile

@@ -0,0 +1,23 @@
+# Makefile for the IRR explorer development environment
+.PHONY: clean
+clean:
+	docker system prune --all
+
+.PHONY: start
+start:
+	docker-compose -f docker-compose.yml up -d
+
+.PHONY: stop
+stop:
+	docker-compose -f docker-compose.yml down
+
+.PHONY: rebuild
+rebuild: stop clean start
+
+.PHONY: dump_irrd_db
+dump_irrd_db:
+	docker exec postgres_irrd pg_dump -d irrd -U irrd -w > irrdb.sql
+
+.PHONY: dump_irre_db
+dump_irre_db:
+	docker exec postgres_irre pg_dump -d irrexplorer -U irrexplorer -w > irrexplorer_db.sql

dev/docker-compose.yml

@@ -0,0 +1,95 @@
+---
+# This Docker Compose configuration sets up a development environment for the IRRd and IRRExplorer applications. It includes the following services:
+#
+# - irrd: The IRRd (Internet Routing Registry Daemon) service, which is built from the Dockerfile located in the Dockerfiles/irrd4 directory. It mounts the irrd.yaml  configuration file and a volume for the PID file. It exposes ports 8080 and 8043 and has a health check that checks the / endpoint.
+# - postgres_irrd: A PostgreSQL database service for the IRRd application, with the database name, user, and password configured via environment variables. It mounts a volume for the database data and has a health check that checks the database connection.
+# - redis_irrd: A Redis service for the IRRd application, with a health check that checks the connection.
+# - irrexplorer: The IRRExplorer service, which is built from the Dockerfile located in the Dockerfiles/irrexplorer directory. It mounts the parent directory as the application code, the init-irrexplorer.sh script, and the .env file. It exposes port 8000 and has a health check that checks the / endpoint.
+# - postgres_irre: A PostgreSQL database service for the IRR explorer application, with the database name, user, and password configured via environment variables. It mounts a volume for the database data and has a health check that checks the database connection.
+#
+# The services are configured to depend on each other, with the irrd and irrexplorer services depending on the respective PostgreSQL and Redis services being healthy.
+services:
+  irrd:
+    build:
+      context: .
+      dockerfile: Dockerfiles/irrd4
+    container_name: irrd
+    volumes:
+      - ./irrd.yaml:/etc/irrd.yaml:ro
+      - ./data/irrd_pid:/var/run/irrd
+    ports:
+      - 8080:8080
+      - 8043:8043
+    healthcheck:
+      test: [ "CMD-SHELL", "curl -LI localhost:8080" ]
+      interval: 15s
+      timeout: 15s
+      retries: 5
+    depends_on:
+      postgres_irrd:
+        condition: service_healthy
+      redis_irrd:
+        condition: service_healthy
+
+  postgres_irrd:
+    image: postgres
+    container_name: postgres_irrd
+    environment:
+      - "PGUSER=irrd"
+      - "POSTGRES_USER=irrd"
+      - "POSTGRES_PASSWORD=irrd"
+      - "POSTGRES_DB=irrd"
+    volumes:
+      - ./data/postgresql_irrd:/var/lib/postgresql/data
+    healthcheck:
+      test: [ "CMD-SHELL", "pg_isready -d irrd" ]
+      interval: 15s
+      timeout: 15s
+      retries: 5
+
+  redis_irrd:
+    image: redis
+    container_name: redis_irrd
+    healthcheck:
+      test: [ "CMD-SHELL", "redis-cli get nil" ]
+      interval: 60s
+      timeout: 5s
+      retries: 5
+
+  irrexplorer:
+    build:
+      context: .
+      dockerfile: Dockerfiles/irrexplorer
+    container_name: irrexplorer
+    ports:
+      - 8000:8000
+    volumes:
+      - ../:/app/irrexplorer
+      - ./init-irrexplorer.sh:/app/irrexplorer/init:ro
+      - ./.env:/app/irrexplorer/.env:ro
+    healthcheck:
+      test: [ "CMD-SHELL", "curl -LI localhost:8000" ]
+      interval: 15s
+      timeout: 15s
+      retries: 5
+    depends_on:
+      postgres_irre:
+        condition: service_healthy
+      irrd:
+        condition: service_healthy
+
+  postgres_irre:
+    image: postgres
+    container_name: postgres_irre
+    environment:
+      - "PGUSER=irrexplorer"
+      - "POSTGRES_USER=irrexplorer"
+      - "POSTGRES_PASSWORD=irrexplorer"
+      - "POSTGRES_DB=irrexplorer"
+    volumes:
+      - ./data/postgresql_irre:/var/lib/postgresql/data
+    healthcheck:
+      test: [ "CMD-SHELL", "pg_isready -d irrexplorer" ]
+      interval: 15s
+      timeout: 15s
+      retries: 5

dev/init-irrd.sh

@@ -0,0 +1,10 @@
+#!/bin/sh
+# This script performs the following actions:
+# 1. Removes any existing PID files for the IRRD service from the /var/run/irrd directory.
+# 2. Upgrades the IRRD database using the configuration file located at /etc/irrd.yaml.
+# 3. Starts the IRRD service in the foreground using the configuration file located at /etc/irrd.yaml.
+
+rm /var/run/irrd/* >/dev/null 2>&1 || true
+
+irrd_database_upgrade --config=/etc/irrd.yaml
+exec /usr/local/bin/irrd --foreground --config=/etc/irrd.yaml

dev/init-irrexplorer.sh

@@ -0,0 +1,28 @@
+#!/bin/sh
+# This script is responsible for setting up and running the IRRExplorer application.
+# 
+# It performs the following tasks:
+# - Installs the required Python dependencies using Poetry
+# - Runs the frontend installation and build commands
+# - Runs the database migrations using Alembic
+# - Schedules a cron job to periodically import data
+# - Starts the IRRExplorer application using the Uvicorn server
+# 
+# The script should be executed as part of the deployment process for the IRRExplorer application.
+
+pip install --no-cache-dir poetry
+poetry install
+poetry run frontend-install
+poetry run frontend-build
+poetry run alembic upgrade head
+
+/bin/echo "0 */3 * * * /usr/local/bin/poetry run import-data >> /var/log/cron.log 2>&1" >/etc/cron.d/poetry_import_data
+/bin/echo "" >>/etc/cron.d/poetry_import_data
+/bin/chmod 0644 /etc/cron.d/poetry_import_data
+/usr/bin/crontab /etc/cron.d/poetry_import_data
+/bin/touch /var/log/cron.log
+/usr/sbin/service cron start
+
+poetry run import-data&
+
+exec poetry run uvicorn irrexplorer.app:app --host 0.0.0.0 --port 8000 --workers 4 --reload