Merge branch 'develop' of https://github.com/LibreQoE/LibreQoS into d…

…evelop
LibreQoE · Oct 18, 2023 · c94d825 · c94d825
2 parents 1f714d6 + e8d8641
commit c94d825
Show file tree

Hide file tree

Showing 99 changed files with 5,404 additions and 1,191 deletions.
diff --git a/.gitignore b/.gitignore
@@ -56,6 +56,14 @@ src/webusers.toml
 src/lqusers.toml
 src/dist
 src/rust/lqos_anonymous_stats_server/anonymous.sqlite
+src/rust/long_term_stats/license_server/lqkeys.bin
+src/rust/long_term_stats/lts_node/lqkeys.bin
+src/rust/long_term_stats/pgdb/.env
+src/rust/long_term_stats/site_build/node_modules
+src/rust/long_term_stats/site_build/output
+src/rust/long_term_stats/site_build/package-lock.json
+src/rust/long_term_stats/wasm_pipe/staging
+src/rust/long_term_stats/lts_node/deploy.sh
 
 # Ignore Rust build artifacts
 src/rust/target
@@ -105,3 +113,11 @@ tramp
 # virtual environments
 .venv
 venv
+src/integrationUISPbandwidths.template.csv
+src/lts_keys.bin
+src/network
+src/network.json.good
+src/network.pdf
+src/ShapedDevices.csv.good
+.gitignore
+src/rust/lqosd/lts_keys.bin
diff --git a/.readthedocs.yaml b/.readthedocs.yaml
@@ -0,0 +1,64 @@
+# Read the Docs configuration file for Sphinx projects
+
+# See https://docs.readthedocs.io/en/stable/config-file/v2.html for details
+
+
+# Required
+
+version: 2
+
+
+# Set the OS, Python version and other tools you might need
+
+build:
+
+  os: ubuntu-22.04
+
+  tools:
+
+    python: "3.11"
+
+    # You can also specify other tool versions:
+
+    # nodejs: "20"
+
+    # rust: "1.70"
+
+    # golang: "1.20"
+
+
+# Build documentation in the "docs/" directory with Sphinx
+
+sphinx:
+
+  configuration: conf.py
+
+  # You can configure Sphinx to use a different builder, for instance use the dirhtml builder for simpler URLs
+
+  # builder: "dirhtml"
+
+  # Fail on all warnings to avoid broken references
+
+  # fail_on_warning: true
+
+
+# Optionally build your docs in additional formats such as PDF and ePub
+
+# formats:
+
+#    - pdf
+
+#    - epub
+
+
+# Optional but recommended, declare the Python requirements required
+
+# to build your documentation
+
+# See https://docs.readthedocs.io/en/stable/guides/reproducible-builds.html
+
+python:
+
+   install:
+
+   - requirements: requirements.txt
diff --git a/README.md b/README.md
@@ -15,104 +15,12 @@ Learn more about [Equinix Metal here](https://deploy.equinix.com/metal/).
 
 Please support the continued development of LibreQoS by sponsoring us via [GitHub Sponsors](https://github.com/sponsors/LibreQoE) or [Patreon](https://patreon.com/libreqos).
 
-## Documentation
+## Documentation / Get Started / System Requirements
 
-[Docs](https://libreqos.readthedocs.io)
+[ReadTheDocs](https://libreqos.readthedocs.io/en/latest/)
 
 ## Matrix Chat
 
 Our Matrix chat channel is available at [https://matrix.to/#/#libreqos:matrix.org](https://matrix.to/#/#libreqos:matrix.org).
 
 <img alt="LibreQoS" src="https://user-images.githubusercontent.com/22501920/223866474-603e1112-e2e6-4c67-93e4-44c17b1b7c43.png"></a>
-
-## Features
-
-### Flexible Hierarchical Shaping / Back-Haul Congestion Mitigation
-
-<img src="https://raw.githubusercontent.com/LibreQoE/LibreQoS/main/docs/nestedHTB2.png" width="350"></img>
-
-Starting in version v1.1+, operators can map their network hierarchy in LibreQoS. This enables both simple network hierarchies (Site>AP>Client) as well as much more complex ones (Site>Site>Micro-PoP>AP>Site>AP>Client). This can be used to ensure that a given site’s peak bandwidth will not exceed the capacity of its back-haul links (back-haul congestion control). Operators can support more users on the same network equipment with LibreQoS than with competing QoE solutions which only shape by AP and Client.
-
-### CAKE
-
-CAKE is the product of nearly a decade of development efforts to improve on fq\_codel. With the diffserv\_4 parameter enabled – CAKE groups traffic in to Bulk, Best Effort, Video, and Voice. This means that without having to fine-tune traffic priorities as you would with DPI products – CAKE automatically ensures your clients’ OS update downloads will not disrupt their zoom calls. It allows for multiple video conferences to operate on the same connection which might otherwise “fight” for upload bandwidth causing call disruptions. With work-from-home, remote learning, and tele-medicine becoming increasingly common – minimizing video call disruptions can save jobs, keep students engaged, and help ensure equitable access to medical care.
-
-### XDP
-
-Fast, multi-CPU queueing leveraging xdp-cpumap-tc and cpumap-pping. Currently tested in the real world past 11 Gbps (so far) with just 30% CPU use on a 16 core Intel Xeon Gold 6254. It's likely capable of 30Gbps or more.
-
-### Graphing
-
-You can graph bandwidth and TCP RTT by client and node (Site, AP, etc), using InfluxDB.
-
-### CRM Integrations
-
-## Server Recommendations
-It is most cost-effective to buy a used server with specifications matching your unique requirements, as laid out in the System Requirements section below.
-For those who do not have the time to do that, here are some off-the-shelf options to consider:
-* 1 Gbps | [Supermicro SuperServer E100-9W-L](https://www.thinkmate.com/system/superserver-e100-9w-l)
-* 10 Gbps | [Supermicro SuperServer 510T-ML (Choose E-2388G)](https://www.thinkmate.com/system/superserver-510t-ml)
-* 20 Gbps | [Dell R450 Config](https://www.dell.com/en-us/shop/servers-storage-and-networking/poweredge-r450-rack-server/spd/poweredge-r450/pe_r450_15127_vi_vp?configurationid=a7663c54-6e4a-4c96-9a21-bc5a69d637ba)
-
-The [AsRock 1U4LW-B6502L2T](https://www.thinkmate.com/system/asrock-1u4lw-b6502l2t/635744) can be a great lower-cost option as well.
-
-## System Requirements
-### VM or physical server
-* For VMs, NIC passthrough is required for optimal throughput and latency (XDP vs generic XDP). Using Virtio / bridging is much slower than NIC passthrough. Virtio / bridging should not be used for large amounts of traffic.
-
-### CPU
-* 2 or more CPU cores
-* A CPU with solid [single-thread performance](https://www.cpubenchmark.net/singleThread.html#server-thread) within your budget. Queuing is very CPU-intensive, and requires high single-thread performance.
-
-Single-thread CPU performance will determine the max throughput of a single HTB (cpu core), and in turn, what max speed plan you can offer customers.
-
-| Customer Max Plan   | Passmark Single-Thread   |
-| --------------------| ------------------------ |
-| 100 Mbps            | 1000                     |
-| 250 Mbps            | 1500                     |
-| 500 Mbps            | 2000                     |
-| 1 Gbps              | 2500                     |
-| 2 Gbps              | 3000                     |
-
-Below is a table of approximate aggregate throughput capacity, assuming a a CPU with a [single thread](https://www.cpubenchmark.net/singleThread.html#server-thread) performance of 2700 or greater:
-
-| Aggregate Throughput    | CPU Cores     |
-| ------------------------| ------------- |
-| 500 Mbps                | 2             |
-| 1 Gbps                  | 4             |
-| 5 Gbps                  | 6             |
-| 10 Gbps                 | 8             |
-| 20 Gbps                 | 16            |
-| 50 Gbps*                | 32            |
-
-(* Estimated)
-
-So for example, an ISP delivering 1Gbps service plans with 10Gbps aggregate throughput would choose a CPU with a 2500+ single-thread score and 8 cores, such as the Intel Xeon E-2388G @ 3.20GHz.
-
-### Memory
-* Minimum RAM = 2 + (0.002 x Subscriber Count) GB
-* Recommended RAM:
-
-| Subscribers   | RAM           |
-| ------------- | ------------- |
-| 100           | 4 GB          |
-| 1,000         | 8 GB          |
-| 5,000         | 16 GB         |
-| 10,000*       | 18 GB         |
-| 50,000*       | 24 GB         |
-
-(* Estimated)
-
-### Network Interface Requirements
-* One management network interface completely separate from the traffic shaping interfaces. Usually this would be the Ethernet interface built in to the motherboard.
-* Dedicated Network Interface Card for Shaping Interfaces
-  * NIC must have 2 or more interfaces for traffic shaping.
-  * NIC must have multiple TX/RX transmit queues. [Here's how to check from the command line](https://serverfault.com/questions/772380/how-to-tell-if-nic-has-multiqueue-enabled).
-  * Known supported cards:
-    * [NVIDIA Mellanox MCX512A-ACAT](https://www.fs.com/products/119649.html)
-    * NVIDIA Mellanox MCX416A-CCAT
-    * [Intel X710](https://www.fs.com/products/75600.html)
-    * Intel X520
-
-## Get Started
-- [ReadTheDocs](https://libreqos.readthedocs.io/en/develop/docs/SystemRequirements/Compute.html)
diff --git a/conf.py b/conf.py
@@ -7,7 +7,7 @@
 # https://www.sphinx-doc.org/en/master/usage/configuration.html#project-information
 
 project = "LibreQoE"
-copyright = "2023, LibreQoE"
+copyright = "2023, LibreQoE, LLC"
 author = "Zach Biles"
 
 # -- General configuration ---------------------------------------------------

diff --git a/docs/Quickstart/quickstart-libreqos-1.4.md b/docs/Quickstart/quickstart-libreqos-1.4.md
@@ -18,8 +18,8 @@ Run ```sudo crontab -e``` and remove any entries pertaining to LibreQoS from v1.
 Use the deb package from the [latest v1.4 release](https://github.com/LibreQoE/LibreQoS/releases/).
 
 ```shell
-sudo echo "deb http://stats.libreqos.io/ubuntu jammy main" > /etc/apt/sources.list.d/libreqos.list
-wget -O - -q http://stats.libreqos.io/repo.asc | apt-key add -
+sudo echo "deb http://stats.libreqos.io/ubuntu jammy main" | sudo tee -a /etc/apt/sources.list.d/libreqos.list
+sudo wget -O - -q http://stats.libreqos.io/repo.asc | sudo apt-key add -
 apt-get update
 apt-get install libreqos
 ```

diff --git a/docs/SystemRequirements/Compute.md b/docs/SystemRequirements/Compute.md
@@ -11,10 +11,11 @@ Single-thread CPU performance will determine the max throughput of a single HTB
 | Customer Max Plan   | Passmark Single-Thread   |
 | --------------------| ------------------------ |
 | 100 Mbps            | 1000                     |
-| 250 Mbps            | 1500                     |
-| 500 Mbps            | 2000                     |
-| 1 Gbps              | 2500                     |
+| 250 Mbps            | 1250                     |
+| 500 Mbps            | 1500                     |
+| 1 Gbps              | 2000                     |
 | 2 Gbps              | 3000                     |
+| 4 Gbps              | 4000                     |
 
 Below is a table of approximate aggregate throughput capacity, assuming a a CPU with a [single thread](https://www.cpubenchmark.net/singleThread.html#server-thread) performance of 2700 or greater:
 
@@ -48,8 +49,12 @@ So for example, an ISP delivering 1Gbps service plans with 10Gbps aggregate thro
 ### Server Recommendations
 It is most cost-effective to buy a used server with specifications matching your unique requirements, as laid out in the System Requirements section below.
 For those who do not have the time to do that, here are some off-the-shelf options to consider:
-* 1 Gbps | [Supermicro SuperServer E100-9W-L](https://www.thinkmate.com/system/superserver-e100-9w-l)
-* 10 Gbps | [Supermicro SuperServer 510T-ML (Choose E-2388G)](https://www.thinkmate.com/system/superserver-510t-ml)
-* 20 Gbps | [Dell R450 Config](https://www.dell.com/en-us/shop/servers-storage-and-networking/poweredge-r450-rack-server/spd/poweredge-r450/pe_r450_15127_vi_vp?configurationid=a7663c54-6e4a-4c96-9a21-bc5a69d637ba)
 
-The [AsRock 1U4LW-B6502L2T](https://www.thinkmate.com/system/asrock-1u4lw-b6502l2t/635744) can be a great lower-cost option as well.
+|   Aggregate   | 100Mbps Plans |  1Gbps Plans  |  4Gbps Plans  |
+| ------------- | ------------- | ------------- | ------------- |
+| 1 Gbps Total  |       A       |               |               |
+| 10 Gbps Total |               |       B       |       C       |
+
+* A | [Supermicro SuperServer E100-9W-L](https://www.thinkmate.com/system/superserver-e100-9w-l)
+* B | [Supermicro SuperServer 510T-ML](https://www.thinkmate.com/system/superserver-510t-ml) (Select E-2388G)
+* C | [Supermicro AS-1015A-MT](https://store.supermicro.com/us_en/as-1015a-mt.html) (Ryzen 9 7700X, 2x16GB DDR5 4800MHz ECC, 1xSupermicro 10-Gigabit XL710+ X557)
diff --git a/docs/SystemRequirements/Networking.md b/docs/SystemRequirements/Networking.md
@@ -6,5 +6,5 @@
   * Known supported cards:
     * [NVIDIA Mellanox MCX512A-ACAT](https://www.fs.com/products/119649.html)
     * NVIDIA Mellanox MCX416A-CCAT
-    * [Intel X710](https://www.fs.com/products/75600.html)
+    * [Intel X710](https://www.fs.com/products/75600.html) * Note - possible i40e driver issue with XDP Redirect for high throughput 10G+
     * Intel X520
diff --git a/docs/TechnicalDocs/complex-install.md b/docs/TechnicalDocs/complex-install.md
@@ -18,7 +18,7 @@ By specifying `libreqos` at the end, git will ensure the folder name is lowercas
 You need to have a few packages from `apt` installed:
 
 ```shell
-sudo apt-get install -y python3-pip clang gcc gcc-multilib llvm libelf-dev git nano graphviz curl screen llvm pkg-config linux-tools-common linux-tools-`uname -r` libbpf-dev
+sudo apt-get install -y python3-pip clang gcc gcc-multilib llvm libelf-dev git nano graphviz curl screen llvm pkg-config linux-tools-common linux-tools-`uname -r` libbpf-dev libssl-dev
 ```
 
 Then you need to install some Python dependencies:

diff --git a/docs/TechnicalDocs/integrations.md b/docs/TechnicalDocs/integrations.md
@@ -20,6 +20,8 @@ You have the option to run integrationUISP.py automatically on boot and every 30
 
 First, set the relevant parameters for Splynx (splynx_api_key, splynx_api_secret, etc.) in ispConfig.py.
 
+The Splynx Integration uses Basic authentication. For using this type of authentication, please make sure you enable [Unsecure access](https://splynx.docs.apiary.io/#introduction/authentication) in your Splynx API key settings. Also the Splynx API key should be granted access to the necessary permissions.
+
 To test the Splynx Integration, use
 
 ```shell

diff --git a/docs/TechnicalDocs/troubleshooting.md b/docs/TechnicalDocs/troubleshooting.md
@@ -17,3 +17,20 @@ Node manager and scheduler are dependent on the `lqos.service` being in a health
 ### RTNETLINK answers: Invalid argument
 
 This tends to show up when the MQ qdisc cannot be added correctly to the NIC interface. This would suggest the NIC has insufficient RX/TX queues. Please make sure you are using the [recommended NICs](../SystemRequirements/Networking.md).
+
+### InfluxDB "Failed to update bandwidth graphs"
+
+The scheduler (scheduler.py) runs the InfluxDB integration within a try/except statement. If it fails to update InfluxDB, it will report "Failed to update bandwidth graphs".
+To find the exact cause of the failure, please run ```python3 graphInfluxDB.py``` which will provde more specific errors.
+
+### All customer IPs are listed under Unknown IPs, rather than Shaped Devices in GUI
+```
+cd /opt/libreqos/src
+sudo systemctl stop lqos_scheduler
+sudo python3 LibreQoS.py
+```
+
+The console output from running LibreQoS.py directly provides more specific errors regarding issues with ShapedDevices.csv and network.json
+Once you have identified the error and fixed ShapedDevices.csv and/or Network.json, please then run
+
+```sudo systemctl start lqos_scheduler```
diff --git a/src/INTEGRATION_HOWTO.md b/src/INTEGRATION_HOWTO.md
@@ -100,23 +100,23 @@ Let's attach some access points and point-of-presence sites:
 ```python
 net.addRawNode(NetworkNode(id="AP_A", displayName="AP_A", parentId="Site_1", type=NodeType.ap, download=500, upload=500))
 net.addRawNode(NetworkNode(id="Site_3", displayName="Site_3", parentId="Site_1", type=NodeType.site, download=500, upload=500))
-net.addRawNode(NetworkNode(id="PoP_5", displayName="PoP_5", parentId="Site_3", type=NodeType.site, download=200, upload=200))        
-net.addRawNode(NetworkNode(id="AP_9", displayName="AP_9", parentId="PoP_5", type=NodeType.ap, download=120, upload=120))
-net.addRawNode(NetworkNode(id="PoP_6", displayName="PoP_6", parentId="PoP_5", type=NodeType.site, download=60, upload=60))
-net.addRawNode(NetworkNode(id="AP_11", displayName="AP_11", parentId="PoP_6", type=NodeType.ap, download=30, upload=30))
-net.addRawNode(NetworkNode(id="PoP_1", displayName="PoP_1", parentId="Site_2", type=NodeType.site, download=200, upload=200))
-net.addRawNode(NetworkNode(id="AP_7", displayName="AP_7", parentId="PoP_1", type=NodeType.ap, download=100, upload=100))
+net.addRawNode(NetworkNode(id="Site_5", displayName="Site_5", parentId="Site_3", type=NodeType.site, download=200, upload=200))        
+net.addRawNode(NetworkNode(id="AP_9", displayName="AP_9", parentId="Site_5", type=NodeType.ap, download=120, upload=120))
+net.addRawNode(NetworkNode(id="Site_6", displayName="Site_6", parentId="Site_5", type=NodeType.site, download=60, upload=60))
+net.addRawNode(NetworkNode(id="AP_11", displayName="AP_11", parentId="Site_6", type=NodeType.ap, download=30, upload=30))
+net.addRawNode(NetworkNode(id="Site_4", displayName="Site_4", parentId="Site_2", type=NodeType.site, download=200, upload=200))
+net.addRawNode(NetworkNode(id="AP_7", displayName="AP_7", parentId="Site_4", type=NodeType.ap, download=100, upload=100))
 net.addRawNode(NetworkNode(id="AP_1", displayName="AP_1", parentId="Site_2", type=NodeType.ap, download=150, upload=150))
 ```
 
-When you attach a customer, you can specify a tree entry (e.g. `PoP_5`) as a parent:
+When you attach a customer, you can specify a tree entry (e.g. `Site_5`) as a parent:
 
 ```python
 # Add the customer
 customer = NetworkNode(
     id="Unique Customer ID",
     displayName="The Doe Family",
-    parentId="PoP_5",
+    parentId="Site_5",
     type=NodeType.client,
     download=100, # Download is in Mbit/second
     upload=20, # Upload is in Mbit/second
@@ -146,7 +146,7 @@ net.createShapedDevices() # Create the `ShapedDevices.csv` file.
 
 You can also add a call to `net.plotNetworkGraph(False)` (use `True` to also include every customer; this can make for a HUGE file) to create a PDF file (currently named `network.pdf.pdf`) displaying your topology. The example shown here looks like this:
 
-![](testdata/sample_layout.png)
+![](testdata/network_new.png)
 
 ## Longest Prefix Match Tip
 You could theoretically throttle all unknown IPs until they are associated with a client. For example, you could limit every unknown to 1.5x0.5 with single entry in ShapedDevices.csv, until you associate them with an account. IPs need to be non-exact matches. So you can't have two 192.168.1.1 entries, but you can have a 192.168.1.0/24 subnet and a 192.168.1.2/32 - they aren't duplicates, and the LPM search is smart enough to pick the most exact match.