From 47c6b4be1f2970c3a5d2ee9d20a9285922e2f55d Mon Sep 17 00:00:00 2001 From: Pouriya Date: Mon, 22 Apr 2024 23:26:59 +0200 Subject: [PATCH] update README --- README.md | 108 +++++++++++++++++++++++++++++++++++++++--------------- 1 file changed, 78 insertions(+), 30 deletions(-) diff --git a/README.md b/README.md index 9d24bed..e0577a9 100644 --- a/README.md +++ b/README.md @@ -6,33 +6,25 @@ --- -**nginwho** is a lightweight, efficient and extremely fast `nginx` log parser that stores the logs into a **sqlite3** database for further analysis and actions. +**nginwho** is a lightweight, efficient and extremely fast program offering three main features at its core: -Additionally, it has the ability (`--show-real-ips` flag) to continuously parse **Cloudflare CIDRs** through their APIs so that nginx can leverage it to get the real IP addresses of requests hitting your servers. +1. **nginx** log parser: Stores nginx logs into a **sqlite3** database for further analysis and actions +2. Restore **Cloudflare** original visitor IP: Continuously parses **Cloudflare CIDRs** (`IPv4` and `IPv6`) through their **API**s so that nginx can leverage it to restore the original IP address of visitors +3. **Block** untrusted requests: Uses **nftables** to block HTTP and HTTPS requests coming from unknown IP addresses -## How it works - -By default, **nginwho** reads `nginx` logs from `/var/log/nginx/access.log` and stores the parsed results in `/var/log/nginwho.db` unless overridden by the [available flags](#flags). - -> [!IMPORTANT] -> nginwho only supports the default nginx log format - -Using the `--show-real-ips` flag requires **root privileges** and leads to fetching the Cloudflare CIDRs every six hours and storing them in `/etc/nginx/nginwho`. - -Inside your nginx configuration add this line: - -```text -include /etc/nginx/nginwho; -``` +Table of contents: -So that the fetched CIDRs could be loaded into your configuration. - -> [!IMPORTANT] -> The `--show-real-ips:true` option causes a **soft reload** (nginx -s reload) at 3AM **only if there are CIDR changes** in comparison to the last fetch. Specifying the reload time will be available through a flag in the future. +- [nginwho](#nginwho) + - [Usage](#usage) + - [Flags](#flags) + - [How it works](#how-it-works) + - [nginx Log Parser](#nginx-log-parser) + - [Restore Cloudflare Original Visitor IP](#restore-cloudflare-original-visitor-ip) + - [Block Untrusted Requests](#block-untrusted-requests) ## Usage -1. Download the executable file from this URL: +1. Download **nginwho** from this URL: ```bash wget https://github.com/pouriyajamshidi/nginwho/releases/latest/download/nginwho @@ -61,14 +53,14 @@ So that the fetched CIDRs could be loaded into your configuration. --dbPath:/var/log/nginwho.db \ --omit-referrer:thegraynode.io - # If you want to get real IP addresses of the visitors coming from Cloudflare (replace thegraynode.io with your domain): + # If you want to get real IP addresses of the visitors coming from Cloudflare: nginwho --logPath:/var/log/nginx/access.log \ --dbPath:/var/log/nginwho.db \ --omit-referrer:thegraynode.io \ --show-real-ips:true ``` -4. Optionally, use the [accompanying systemd](https://github.com/pouriyajamshidi/nginwho/blob/master/nginwho.service) service to run `nginwho` in the background and for the program to survive system reboots: +4. Optionally, use the [accompanying systemd](https://github.com/pouriyajamshidi/nginwho/blob/master/nginwho.service) to run **nginwho** as a service in the background and for the it to survive system reboots: ```bash sudo cp nginwho.service /etc/systemd/system/nginwho.service @@ -81,11 +73,67 @@ So that the fetched CIDRs could be loaded into your configuration. Here are the available flags: ```text ---help, -h : show help ---version, -v : Display version and quit ---dbPath, : Path to SQLite database to log reports (default: /var/log/nginwho.db) ---logPath, : Path to nginx access logs (default: /var/log/nginx/access.log) ---interval : Refresh interval in seconds (default: 10) ---omit-referrer : omit a specific referrer from being logged (default: "") ---show-real-ips : Show real IP of visitors by getting Cloudflare CIDRs to include in nginx config. Updates every three hours. (default: false) + --help, -h : Show help + --version, -v : Display version and quit + --dbPath, : Path to SQLite database to log reports (default: /var/log/nginwho.db) + --logPath, : Path to nginx access logs (default: /var/log/nginx/access.log) + --interval : Refresh interval in seconds (default: 10) + --omit-referrer : Omit a specific referrer from being logged (default: none) + --analyze-nginx-logs : Whether to analyze nginx logs or not. (default: true) + --show-real-ips : Show real IP of visitors by getting Cloudflare CIDRs to include in nginx config. + Self-updates every six hours (default: false) + --block-untrusted-cidrs : Block untrusted IP addresses using nftables. Only allows Cloudflare CIDRs (default: false) ``` + +## How it works + +Let's see how nginwho works in a somewhat detailed yet short fashion. + +### nginx Log Parser + +For the first feature and by default, **nginwho** reads `nginx` logs from `/var/log/nginx/access.log` and stores the parsed results in a **sqlite3** database located in `/var/log/nginwho.db` unless overridden by the [available flags](#flags). + +> [!WARNING] +> nginwho only supports the default nginx log format for now + +### Restore Cloudflare Original Visitor IP + +The second feature, `--show-real-ips` flag fetches **Cloudflare CIDRs** (`IPv4` and `IPv6`) every _six hours_ through their **API**s and writes the result to a file located in `/etc/nginx/nginwho`. + +It is worthwhile to mention that **nginwho** leverages the `etag` field in Cloudflare's API response, so, if the newly fetched `etag` is the same as the current one, the `/etc/nginx/nginwho` file will not be overwritten. + +If the `/etc/nginx/nginwho` file has changed or this is a fresh run, **nginwho** schedules the **nginx** service to be soft reloaded (`nginx -s reload`) at 3 AM. + +> [!IMPORTANT] +> The `--show-real-ips` flag requires **root privileges**. + +For `--show-real-ips` flag to work, you need to alter your nginx configuration add this line: + +```text +include /etc/nginx/nginwho; +``` + +So that nginx knows how to restore original visitor IP addresses. + +### Block Untrusted Requests + +The third feature, `--block-untrusted-cidrs` flag periodically gets Cloudflare CIDRs, either through: + +1. Cloudflare APIs when used in conjunction with `--show-real-ips` flag +2. or the `/etc/nginx/nginwho` file when the `--show-real-ips` flag is not specified + +The fetched CIDRs will be checked against your existing **nftables** rules and if necessary, the required rules will be created and added through _nftables JSON API_. + +There will be a bunch of tests and pre-checks done before applying any policies. These checks include: + +1. Existence of Cloudflare IPv4 CIDRs nftables Set (`Cloudflare_IPv4`) +1. Existence of Cloudflare IPv6 CIDRs nftables Set (`Cloudflare_IPv6`) +1. Existence of nftables `nginwho` chain (`prerouting` hook) +1. Existence of nftables `input` chain +1. Existence of **drop** policy inside `nginwho` chain for untrusted IP addresses on port **80** and **443** +1. Existence of **accept** policy inside `input` chain for trusted IP addresses on port **80** and **443** + +**nginwho** only creates the necessary changes for **nftables**, if no changes are required, no action will be taken. For instance, if a CIDR gets added or removed, only that part of **nftables** configuration will be changed and the rest remain unchanged. + +> [!IMPORTANT] +> Since playing with **nftables** could result in blocking yourself out, **nginwho** requires you to have some basic policies in place, in specific, having an `inet filter` table. If you do not have it, **nginwho** will detect that and shows you how to create one.