diff --git a/README.md b/README.md index 438c3d4..3f0a9ed 100644 --- a/README.md +++ b/README.md @@ -2,39 +2,49 @@ # foxsi-4matter Code for the FOXSI-4 formatter. -This formatter replaces older FOXSI formatters with a [Raspberry Pi](https://www.raspberrypi.org) processor unit. A peripheral SPMU-001 board (based on the Xilinx Spartan 7) for [SpaceWire](https://www.star-dundee.com/wp-content/star_uploads/general/SpaceWire-Users-Guide.pdf) communication support. +This formatter replaces older FOXSI formatters with a [Raspberry Pi](https://www.raspberrypi.org) processor unit. There's also an SPMU-001 board (based on the Xilinx Spartan 7) for [SpaceWire](https://www.star-dundee.com/wp-content/star_uploads/general/SpaceWire-Users-Guide.pdf) communication support, and a [Housekeeping board](https://github.com/foxsi/foxsi4-hk) for commanding the onboard power supply. ## Setting up -See [PISETUP.md](PISETUP.md) for instructions on booting and configuring the Raspberry Pi. - -### Dependencies -- [Boost](https://www.boost.org/) - - boost::asio - - boost::bind -- [nlohmann JSON](https://github.com/nlohmann/json) -- [Doxygen](https://doxygen.nl/) -- [googletest](https://github.com/google/googletest) -- [concurrentqueue](https://github.com/cameron314/concurrentqueue) (included) -- [spdlog](https://github.com/gabime/spdlog) (included) - -### Examples -You can find a few examples in the [examples folder](examples). They are, unfortunately, out of date. +If you are setting up a fresh Raspbeery Pi, see [PISETUP.md](PISETUP.md) for instructions. If you are just trying to get this software running, stay where you are (this README). ## Network configuration The physical system is laid out like this: ![The Formatter is connected to many systems. Sorry, an image would be handy.](doc/assets/formatter_layout.svg "Formatter physical interfaces") +There are a lot of things that plug into the Formatter. The role of this software is to pipe data between all these interfaces. That involves +1. configuring the software to have permission to access all these ports on startup, +2. then handling all the idiosyncrasies of packet formatting for all the connected systems, so that the actual data can pass between interfaces without extraneous headers/wrappers coming along for the ride. + +This software owes its life to the [`foxsi4-commands`](https://github.com/foxsi/foxsi4-commands/tree/main) repository, which serves tells the code about the diagram above. Specifically, the top-level file [`foxsi4-commands/systems.json`](https://github.com/foxsi/foxsi4-commands/blob/main/systems.json) defines which interfaces are available to the Formatter and how they are constrained—what IP addresses and port numbers and maximum packet sizes and data rates and timeouts should be used for some given interface. For downlink and uplink commands, `systems.json` is the "contract" between the GSE and Formatter that defines their common interface. + +> [!NOTE] +> For [v1.2.1](https://github.com/foxsi/foxsi-4matter/releases/tag/v1.2.1) and earlier, there is a nuisance in [foxsi4-commands/systems.json](https://github.com/foxsi/foxsi4-commands/blob/main/systems.json) for Formater use. The Formatter will run and transmit downlink data to the Ethernet endpoint defined by [`gse.ethernet_interface.address`](https://github.com/foxsi/foxsi4-commands/blob/77d61f94183432e3a4fdff0bf0356e8361575445/systems.json#L5-L11). But the GSE will listen to [`gse.ethernet_interface.mcast_group`](https://github.com/foxsi/foxsi4-commands/blob/77d61f94183432e3a4fdff0bf0356e8361575445/systems.json#L5-L11). +> +> So you'll need to replace the **Formatter's** GSE configuration in `systems.json` with this: +> ```json +> "ethernet_interface": { +> "protocol": "udp", +> "address": "224.1.1.118", +> "port": 9999, +> "max_payload_bytes": 2000 +> }, +> ``` +> This is fixed after v1.2.2, and the Formatter will try to use `gse.ethernet_interface.mcast_group`, but if that doesn't exist, it will fall back to `gse.ethernet_interface.address`. + ## How to build +Here's how you can build this software from scratch to run on the Raspberry Pi. + In the following, "your computer" is a laptop or desktop connected to the Raspberry Pi, which actually runs the software. -Decide whether you need to build the software on your local computer in addition to building on the Raspberry Pi. If you make local edits and want to check that everything compiles, or if you want to run certain tests, or if you want to use the `foxsimile` emulator, you should build locally. If all you need to do is forward code to the Raspberry Pi and build it there, skip to [building on the Raspberry Pi to run](#building-on-the-raspberry-pi-to-run). +Decide whether you need to build the software on your computer *in addition* to building on the Raspberry Pi. If you make local edits and want to check that everything compiles, or if you want to run certain tests, or if you want to use the `foxsimile` emulator, you should build on your machine. If all you need to do is forward code to the Raspberry Pi and build it there, skip to [building on the Raspberry Pi to run](#building-on-the-raspberry-pi-to-run). ### Installing dependencies -If this is your first time building on your computer, you may need to install two dependencies. If you are setting up the Raspberry Pi, see [PISETUP.md](PISETUP.md) for instructions on installing dependencies. +If this is your first time building on your computer, you may need to install two dependencies. If you are setting up a new Raspberry Pi, see [PISETUP.md](PISETUP.md) for instructions on installing dependencies. #### macOS ```bash +brew install cmake brew install nlohmann-json brew install boost brew install doxygen @@ -42,7 +52,7 @@ brew install googletest ``` (if you want to check if they are already installed, you can run `brew info ` to check). -I have used `boost` v1.83.0 and v1.80.0, and `nlohmann-hson` v3.11.3. I expect other 1.8x versions of `boost` and other 3.1x versions of `nlohmann-json` to also work, but have not tested. +I have used `boost` v1.83.0 and v1.80.0, and `nlohmann-hson` v3.11.3. I expect other `boost` versions 1.7x and up, and other 3.1x versions of `nlohmann-json` to also work, but have not tested. ### Building on the Raspberry Pi to run The flight Raspberry Pi already has dependencies installed. If you are setting up a brand new Raspberry Pi, see [PISETUP.md](PISETUP.md). Otherwise, choose instructions for either @@ -52,7 +62,7 @@ The flight Raspberry Pi already has dependencies installed. If you are setting u #### Set up from a release Select the release you want to build on the [`foxsi-4matter` tags page](https://github.com/foxsi/foxsi-4matter/tags). On the page for a specific release ([v0.0.8, as an example](https://github.com/foxsi/foxsi-4matter/releases/tag/v0.0.8)) download the .zip or tar.gz archive to your computer. -Unzip the archive you downloaded, then open a terminal inside that folder (which should be named `foxsi-4matter-x.y.z`, where `x.y.z` is the version number you downloaded). From that terminal, run: +Unzip the archive you downloaded, then open a terminal *inside* that folder (which should be named `foxsi-4matter-x.y.z`, where `x.y.z` is the version number you downloaded). From that terminal, run: ```bash git clone https://github.com/foxsi/foxsi4-commands.git ``` @@ -66,6 +76,8 @@ cmake --build . You can omit this local build step if you just want to build on the Raspberry Pi. #### Set up from latest `main` +I try to keep the latest `main` on GitHub versioned anyway. But this way you can use `git clone` instead of downloading zips. + Open a terminal in a folder where you want this code, and run ```bash git clone --recursive https://github.com/foxsi/foxsi-4matter.git @@ -80,94 +92,74 @@ cmake --build . ``` #### Building on Raspberry Pi -Now that you have code stored on your computer, you can push the code base to the Raspberry Pi over an Ethernet connection. First, `ssh` into the Raspberry Pi: +Now that you have code stored on your computer, you can push the code to the Raspberry Pi over an Ethernet connection. First, `ssh` into the Raspberry Pi: ``` ssh foxsi@192.168.1.8 password:... ``` -If you have issues `ssh`ing, make sure you can ping the IP `192.168.1.8` and that your local subnetwork includes that address. +The password for the flight Pi is `four`. -Once inside the Pi, create a new folder to hold your build and enter it: +If you have issues `ssh`ing, make sure you can `ping` the IP `192.168.1.8` and that your local subnetwork includes that address. + +Now you can prepare to build new code on the Formatter. Before building any new code, make sure to stop and power off any detector systems onboard. Then stop running the Formatter service: ```bash -mkdir your-folder-name -cd your-folder-name +sudo systemctl stop formatter.service ``` +The Formatter service will resume running after a reboot. + +> [!NOTE] This will overwrite the existing `formatter` binary in the Formatter. +> If you want to save a copy of it first, do the following to save a copy of the binary in whatever directory you are currently in on your machine: +> ```bash +> scp foxsi@192.168.1.8:foxsi-4matter/bin/formatter . +> ``` + + From another terminal on your computer, navigate to the downloaded `foxsi-4matter` code, which should be in a folder called either `foxsi-4matter` (for a build from `main`) or `foxsi-4matter-x.y.z` (for a build from a release). Then run this command (it's long) to push the code into `your-folder-name` in the Raspberry Pi: ```bash -rsync -av --exclude=build --exclude=bin --exclude=doc --exclude=log ../foxsi-4matter-x.y.z foxsi@192.168.1.8:/path/to/your-folder-name +rsync -av --exclude=build --exclude=bin --exclude=doc --exclude=log --exclude=foxsi4-commands ../foxsi-4matter-x.y.z foxsi@192.168.1.8:/home/foxsi ``` -but replace `-x.y.z` with the version number, or omit it entirely for a build from `main`; and replace `/path/to/your-folder-name` with the real path to the folder you created. +but replace `-x.y.z` with the version number, or omit it entirely for a build from `main`. -Now, go back to your other terminal that is still `ssh`'d inside the Raspberry Pi, and create a `build` folder inside `your-folder-name/`: +Now, go back to your other terminal that is still `ssh`'d inside the Raspberry Pi, and create a new build folder: ```bash -mkdir build +cd foxsi-4matter +rm -r build/* cd build cmake .. cmake --build . -j4 ``` -The Raspberry Pi build process takes a couple minutes. This will create several binaries in `your-folder-name/bin/` that you can run. - -## How to use - -First, command the power board to turn on/off the desired systems. Then start software on detector readout systems. +The Raspberry Pi build process takes a couple minutes. This will create several binaries in `foxsi-4matter/bin/` that you can run. The main program that runs in flight is `foxsi-4matter/bin/formatter`. -### Starting remote software -#### CdTe DE -As of Dec 16 2023, the DE software no longer needs to be started manually. You can ignore this section. -```bash -ssh de -``` -(on the GSE computer) then navigate to the DE main software: -```bash -cd CdTe_DE/production/run/ -``` -and run the DE software: +You should have stopped executing `formatter` via `systemctl` above. The new binary you just built will be run after a reboot. Or, if you want to see all the `cout` output as it happens, you can run it now like this: ```bash -./main_CdTeDE configuration_spmu001.xml +./bin/formatter --verbose --config foxsi4-commands/systems.json ``` -For detail, go to the [DE github repository](https://github.com/foxsi/CdTe_DE). +## How to use +Once the Formatter software starts running (either on boot, or because you have manually started running something) it will try to poll all the detector systems for housekeeping data. These systems may or may not be connected or powered on. -The DE will store raw data for each connected detector in `~/CdTe_DE/production/run/data/`. Note that the DE timestamp is disconnected from any all clock time. +You can use the GSE to command onboard systems on/off. Then start software on each detector readout systems. Refer to the FOXSI-4 -#### CMOS -The CMOS readout software automatically starts running on boot. When you start this Formatter software it will command any connected CMOS to start collecting and saving data. +For detail on the CdTe DE software, go to the [DE github repository](https://github.com/foxsi/CdTe_DE). ### Detector mapping -The DE identifies connected canisters (in the raw data recordings) on the SpaceWire port they are connected to. The power system requires a unique byte to be sent to turn on/off each system. For the flight configuration, this is the DE nomenclature for each detector and power code: +The DE identifies connected canisters (in the raw data recordings) on the SpaceWire port they are connected to. The power system requires a unique byte to be sent to turn on/off each system. For the FOXSI-4 flight configuration, this is the DE nomenclature for each detector and power code: | System (by focal plane position) | Detector | DE number | Power board byte | | -------------------------------- | --------- | --------- | ---------------- | | Canister 2 | no2021_06 | `det4` | `0x06` | -| Canister 3 | no2021_07 | `det2` | `0x05` | -| Canister 4 | no2021_05 | `det3` | `0x04` | +| Canister 3 | no2021_07 | `det2` | `0x04` | +| Canister 4 | no2021_05 | `det3` | `0x05` | | Canister 5 | no2022_01 | `det1` | `0x03` | | DE | --- | --- | `0x00` | | CMOS 1 | `0010` | --- | `0x08` | | CMOS 2 | `0002` | --- | `0x07` | | Timepix | --- | --- | `0x01` | -| SAAS | --- | --- | `0x01` | +| SAAS | --- | --- | `0x02` | ### Starting the Formatter software -As of Nov 28 2023, the Formatter software also needs to be started manually. Do this: -```bash -ssh formatter -``` -then navigate to the Formatter software directory: -```bash -cd foxsi-4matter -``` -and then run the formatter software: -```bash -./bin/formatter --verbose --config foxsi4-commands/systems.json -``` -When the Formatter runs, it will locally record log files (describing Formatter behavior, not saving detector data) to `~/foxsi-4matter/log/`. You can stop the Formatter with ctrl-C. - -On startup the Formatter will walk through each system and try to set it up. No bias voltage will be applied to CdTe. If it cannot talk to a specific system, it will ignore it for data readout (this behavior will be different for flight). - -#### Automatically running on boot -The flight Raspberry Pi will automatically run the Formatter software after booting, and restart it if it crashes. This is managed by Linux `systemd`, with the Formatter software running as a "service." You can interact with this service using these commands: +The Formatter will try to run the binary `/home/foxsi/foxsi4-matter/bin/formatter` after boot, and again every 10 seconds after that binary stops running. This is managed by [[`systemctl`](https://www.freedesktop.org/software/systemd/man/latest/systemctl.html)], which is a powerful Linux utility for running *services*. You can interact with the Formatter service using these commands: ```bash sudo systemctl stop formatter.service # stop running the Formatter service @@ -179,6 +171,15 @@ sudo systemctl status formatter.service # report the current status of th The last command, querying the `status` of the running Formatter service, will tell you if it is running still or has stopped. It will also print some of the Formatter's recent output. Note that while the Formatter service may still be running, the main loop may have effectively stopped due to subsystem disconnects. If you query the status multiple times and always see identical printout, the service may no longer be running correctly. +When the service is running, if you need to debug you will need to find the correct `~/foxsi-4matter/log/` file to collect evidence. This can be inconvenient to track down since Unixtime changes with every reboot. + +A helpful debugging workflow is to stop the Formatter service, then launch it manually so you can see the stdout output: +```bash +sudo systemctl stop formatter.service +./bin/formatter --verbose --config foxsi4-commands/systems.json +``` +You can stop the Formatter with ctrl-C. + ## Downlink data The Formatter transmits data over the UDP interface defined in `foxsi4-commands/systems.json`'s `gse` field. A complete raw data frame from an onboard system may be larger than the maximum downlink packet size, in which case it will be fragmented. A given downlink packet has the following header: @@ -189,7 +190,7 @@ The Formatter transmits data over the UDP interface defined in `foxsi4-commands/ | `1-2` | 2 bytes | `n` | Number of packets in this frame. | | `3-4` | 2 bytes | `i` | Index of this packet in the frame. **This is 1-indexed.** | | `5` | 1 bytes | `data` | An identifier of data type stored in the packet. | -| `6-7` | 2 bytes | `reserved` | Reserved. | +| `6-7` | 2 bytes | `frame_counter` | Identifier used to associate packets with the complete frame they originally came from. Planned addition in v1.3.0. | The raw data payload is concatenated after the 8-byte header. @@ -205,6 +206,7 @@ For the `system` field, the ID value is taken from `foxsi4-commands/systems.json | `0x12` | `temp` | Temperature data | | `0x13` | `stat` | System status data | | `0x14` | `err` | System error data | +| `0x20` | `ping` | Software status (added in v1.3.0) | | `0x30` | `reply` | Forwarded reply to command | | `0xff` | `none` | No data type | @@ -233,6 +235,8 @@ Additionally, an ASCII-formatted file called `catch.log` is produced to log any ## Common issues 1. If you run the formatter software, kill it, then try to run it immediately again, you will get an error containing `connect: Address already in use`. This is because the kernel retains control of TCP sockets for a while (~1 minute) after you close them to allow any messages still on the wire to come through. Just wait a moment and try running again. Of course, it is possible (depending on your configuration) that another process on the same machine actually is using the IP address you want as well. +2. Unix time resets to a similar value on every reboot. Do not trust any timestamps you get out of this software. +3. If you never clear the `~/foxsi-4matter/log/` folder, you can completely fill the disk and get a read-only filesystem. Then you will need to edit `fstab` manually and with a racing heart while the whole flare campaign waits for you. This happened at PFRR during the FOXSI-4 launch campaign. ## Documentation If you enjoyed this measly little README, you're going to *love* the rest of the documentation! You can view these docs in a web browser (they are HTML), and until they are hosted somewhere, you will need these instructions to build them: diff --git a/doc/assets/formatter_layout.pdf b/doc/assets/formatter_layout.pdf new file mode 100644 index 0000000..1dc0296 Binary files /dev/null and b/doc/assets/formatter_layout.pdf differ diff --git a/doc/assets/software_layers.pdf b/doc/assets/software_layers.pdf new file mode 100644 index 0000000..3679b35 Binary files /dev/null and b/doc/assets/software_layers.pdf differ diff --git a/doc/breathe/source/index.rst b/doc/breathe/source/index.rst index 2e27a7f..28a6ee7 100644 --- a/doc/breathe/source/index.rst +++ b/doc/breathe/source/index.rst @@ -41,7 +41,7 @@ Some default parameters are defined in ``Parameters.h``, and the bulk of configu Here is the network configuration: -.. image:: ../../assets/formatter_layout.svg +.. image:: ../../assets/formatter_layout.pdf An overview of the system configuration data can be found here: @@ -65,7 +65,9 @@ Class reference The software is abstracted like this, with lower layers constraining and configuring the behavior of higher layers: -.. image:: ../../assets/software_layers.svg +.. image:: ../../assets/software_layers.pdf + +An element in brackets like ``[SystemManager]`` means there's an array of them. .. toctree:: :maxdepth: 2 diff --git a/util/README.md b/util/README.md index 8531abd..0e34649 100644 --- a/util/README.md +++ b/util/README.md @@ -1,18 +1,20 @@ # util -This folder includes some high-level utilities for Formatter actions. +This folder includes some convenient utilities for Formatter actions. -## Contents: +A typical workflow is to pull from GitHub, modify some code on your laptop, try to build it, and if that works, push it over an Ethernet cable to the Formatter using [update_formatter.sh](update_formatter.sh). Then you can build the code on the Formatter (Raspberry Pi) and test it out. Maybe you find Ethernet connection issues! Then you can use some of the Python scripts to send dummy packets and validate Ethernet interfaces. The files in this `util/` folder help with that development workflow. + +## Shell scripts +1. [update_formatter.sh](update_formatter.sh) will push code you have on a computer over an Ethernet connection to the Formatter. A typical workflow is to write some code This script will not push build, bin, doc, or log folders over to the Formatter. It uses `rsync`. +2. [build_doc.sh](build_doc.sh) is used to build all the autodoc-type documentation (from docstrings in the code). This builds both PDF (LaTeX) and HTML docs in `foxsi-4matter/doc/breathe/build/`. +3. [copy_remote_logs.sh](copy_remote_logs.sh) will copy all the log fils on the Formatter into the `foxsi-4matter/log/formatter/` folder on this machine. +4. [delete_logs.sh](delete_logs.sh) deletes the logs in the `log/` folder. This doesn't do anything remotely—if you want to delete logs from the Formatter instead of your laptop, you'll need to `ssh` and run it there. +5. [assign_all_loopbacks.sh](assign_all_loopbacks.sh) will add all loopback IP addresses on your laptop. So you can send loopback not just from 127.0.0.1, but up to 127.0.0.255. This is used for mock testing in the `foxsimile` emulator. If you're doing stuff with `foxsimile` you'll need to run this first. + +## Python scripts +A lot of these have to do with network testing between the GSE and Formatter. They contain hardcoded IP addresses/ports. These may or may not match the configuration in foxsi4-commands/systems.json. 1. [spipower.py](spipower.py) is a Python script to control (via command line) the MAX7317 serial-to-parallel chip on the power distribution board. You can use this to enable/disable power output from the board to specific subsystems. 2. [monitortemp.py](monitortemp.py) is a Python script that prints the Raspberry Pi's CPU temperature to the command line at 5-second intervals. 3. [udpecho.py](udpecho.py) is a Python script that echoes received UDP packets back to the sender. -4. [udpgarbage.py](udpgarbage.py) is a Python script that continuously transmits roughly 18 Mbps of garbage data to the ground station. - -## [spipower.py](spipower.py) - -This script is intended to be used with the Power Distribution board developed at UMN. To use, connect Raspberry Pi SPI0 (GPIOs 8-11, pins 24, 21, 19, 23) to the Power Distribution board. Run this script and follow the command prompt to turn systems on and off. - -## [monitortemp.py](monitortemp.py) - -This script prints out the CPU temperature every 5 seconds. \ No newline at end of file +4. [udpgarbage.py](udpgarbage.py) is a Python script that continuously transmits roughly 18 Mbps of garbage data to the ground station IP address. \ No newline at end of file diff --git a/util/build_doc.sh b/util/build_doc.sh index 432cf44..6b601a3 100644 --- a/util/build_doc.sh +++ b/util/build_doc.sh @@ -6,6 +6,8 @@ echo "running doxygen..." doxygen echo "running sphinx..." source doc/env/bin/activate -sphinx-build -M html doc/breathe/source doc/breathe/build +cd doc/breathe +sphinx-build -M html source build +sphinx-build -M latexpdf source build echo "opening docs..." open doc/breathe/build/html/index.html \ No newline at end of file diff --git a/util/udpgarbage.py b/util/udpgarbage.py index 27e1419..1188d23 100644 --- a/util/udpgarbage.py +++ b/util/udpgarbage.py @@ -3,8 +3,8 @@ sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM, socket.IPPROTO_UDP) # multicast group address and port -mcast_grp = '224.1.1.0' -mcast_port = 3000 +mcast_grp = '224.1.1.118' +mcast_port = 9999 while True: # make some random data to transmit