RAST Toolset

Table of Contents

• How To Work With This Repository
• Improving RAST with SDN (WIP)
• Improved RAST Simulator and Load Tester Components (PADS 2024)
• Automated Estimator Pipeline (SIMUtools 2023)
• RAST Simulator and Load Tester Components (MASCOTS 2022)
• Locust Performance Testing Infrastructure (MASCOTS 2020)
• References

How To Work With This Repository

Within this repository, you’ll discover the source code, detailed instructions, and, where applicable, the results generated from my research on the RAST approach that was originally introduced at the MASCOTS 2022 conference.

💡

To effectively navigate the historical states of this repository corresponding to the specific papers, I’ve organized them into dedicated branches. For instance, the "MASCOTS2022" Branch represents the exact state of the repository at the time of crafting and presenting the paper submitted for review and later published at MASCOTS 2022. Use the Table of Contents for easier navigation and searching for the desired conference information.

Improving RAST with SDN (WIP)

Last change from: 05.01.2024

Currently, I am working on improving the prediction quality of RAST by enriching the request logs with network metrics recorded by an SDN controller. For my experiments, I use containernet (fork of mininet) as the network emulator and pox as the SDN controller. I built a containernet topology that encapsulates two different kinds of experiments:

1. Running an experiment using the TeaStore benchmarking application (our [TeaStore Fork](https://github.com/jtpgames/TeaStore)).

2. Running an experiment with a simulated version of TeaStore.

Within an experiment a load test is used to generate request logs and network metrics.

Preparations:

• Clone pox in a directory called `pox. Use the halosaur branch.

• Clone the locust_scripts repository in a directory called locust_scripts.

• Clone our TeaStore Fork repository to your local machine.

• Build the TeaStore application and the associated Docker images by following the instructions provided by the developers. You can find these instructions in the GET_STARTED.md file within the cloned repository.

• Both directories should be on the same directory level, e.g., ~/pox and ~/locust_scripts.

Recommended workflow (for terminal users):

• This session will be used to start the Load Test. Navigate to the locust_scripts folder.

• sudo -s to gain root access (required for containernet).

• source activate_venv.sh

• (python mininet/teastore_topo.py --help) to show help message.

• python mininet/teastore_topo.py to run one workload intensity,

• python mininet/teastore_topo.py -a to run all,

• -s uses the simulation for the experiment, e.g., python mininet/teastore_topo.py -s -a

The relevant files after the experiment are:

• switch_flow_stats_{date}.json: contains the observed bytes and packets per second per SDN switch.

• TeaStore kieker logs (needs to be downloaded from the /logs endpoint of TeaStore): contains the kieker logs of the real TeaStore application. Needs to be converted using our Kieker_ETL project to be usable for RAST.

• Simulators/teastore_simulation.log: contains the request logs of our simulator including received requests and their processing times, already in a usable format for RAST.

• locust_log.log: contains the logs of the load tester, requests and their response times.

These files serve as input to our different analysis tools in the Regression-Analysis_Workload-Characterization project.

Improved RAST Simulator and Load Tester Components (PADS 2024)

Moved to Repo

Automated Estimator Pipeline (SIMUtools 2023)

Last update from: 03.06.2023

DOI: https://doi.org/10.1007/978-3-031-57523-5_2

The [Automated Estimator Pipline](https://github.com/jtpgames/Automated-Estimator-Pipeline) by Adrian Liermann was developed as part of his master’s thesis. In his thesis, he implements and evaluates improvements to our RAST approach, in particular, improvements to the training process. In the RAST approach, the following components are of particular interest to the training process:

• Log Transformer

• Predictive Model Creator

The Automated Estimator Pipeline is an implementation of these improved components wrapped in a CLI tool.

RAST Simulator and Load Tester Components (MASCOTS 2022)

Last update from: 25.06.2023

In our [latest publication at MASCOTS 2022](https://www.doi.org/10.1109/MASCOTS56607.2022.00015), we explain our extension of this project.

• We provide a regression model learned from log files using our RAST approach;

• we improved the Simulator so that it uses the regression model to simulate processing times of the System-Under-Evaluation (SUE);

• we built a [mininet](http://mininet.org/) topology to ease the process of reproducibly launching experiments. We modelled the link parameters based on the infrastructure of the SUE;

• we implemented additional ancillary Python scripts to help analyse the log files.

Instructions to launch an experiment

Quick start

ℹ️	Tested with Python 3.8.10 on Ubuntu 20.04 LTS Python 3.9.12 on macOS 12.6.1

• Clone the MASCOTS2022 branch of this repository

• run cd <the_directory_you_cloned_the_repository>

• create a python virtual environment in a directory called venv, e.g., python3 -m venv venv

• activate virtual environment with source activate_venv.sh

• run pip install -r requirements.txt

• install mininet using the command sudo apt-get install mininet (in case you need additional help, consult the [mininet documentation](http://mininet.org/download/)

• run cd mininet

• execute ./start_mininet.sh

• if mininet successfully started the experiment, you will notice a series of log files that were created:

  • ARS_simulation.log (means that the Simulator is running)

  • locust-parameter-variation.log (means that the load tester is generating the alarm device workload)

  • locust.log (means that the load tester is generating the background workload)

• in another terminal, run cd <the_directory_you_cloned_the_repository>

• run tail -f locust-parameter-variation.log

• the experiment may run for a couple of hours. The experiment ends, once the locust-parameter-variation.log prints the message "Finished performance test. System failed at …​"

• after that, you can analyse the log files. The locust-parameter-variation.log is of particular interest as it contains the measured average and maximum response times depending on the number of simulated alarm devices.

Locust Performance Testing Infrastructure (MASCOTS 2020)

Last update from: 25.06.2023

In [1], we introduced a generic performance testing infrastructure and used it in an industrial case study. Our idea is to have decoupled components, Python scripts in our case, that together allow to:

1. reproducible execute a load testing tool with a set of parameters for a particular experiment,

2. evaluate the performance measurements assisted by visualizations or automatic evaluators.

Generally, we have four types of components in our infrastructure:

• Executors: execute a particular Load Tester as long as the Load Tester provides a CLI or an API;

• Load Testers: execute the load test, parametrized with values given by an Executor. Have to output a logfile containing the response times;

• Evaluators: postprocess the logfile and for example plot the response times;

• Systems under Test (SUTs): Target systems we want to test. Usually, the target systems will be external systems, e.g., web servers. In our case, we build software that simulates the behavior of a real system, in order to provide the means for others to roughly reproduce our experiments.

More details about our generic performance testing infrastructure can be found in our paper [1].

This repository contains the aforementioned Python scripts:

• Executors:

  • executor.py: executes Locust with a set of parameters;

  • locust-parameter-variation.py: executes Locust and keeps increasing the load. This is similar to Locust’s [Step Load Mode](https://docs.locust.io/en/stable/running-locust-in-step-load-mode.html), however, our approach increases the number of clients for as long as the ARS complies with real-time requirements in order to find the saturation point of the ARS.

• Load Testers:

  • locust_tester.py: contains specific code for Locust to perform the actual performance test. For demonstration purposes, this script tests ARS_simulation.py. Outputs a locust_log.log;

  • locust_multiple_requests: an enhanced version of locust_tester that sends additional requests to generate more load.

  • locust_teastore.py: performs load testing against TeaStore, or our simulated TeaStore.

• Evaluators:

  • loadtest_plotter.py: reads the locust_log.log, plots response times, and additional metrics to better visualize, if the real-time requirements of the EN 50136 are met.

• SUTs

  • Alarm Receiving Software Simulation (ARS_simulation.py): simulates an industrial ARS based on data measured in the production environment of the GS company group.

  • TeaStore (teastore_simulation.py): simulates TeaStore based on a predictive model generated in a lab environment.

Instructions to reproduce results in our paper

Quick start

• Clone the MASCOTS2020 branch of this repository;

• run pip3 install -r requirements.txt;

• In the file ARS_simulation.py make sure that the constant MASCOTS2020 is set to True.

• open two terminal shells:

  1. run python3 ARS_simulation.py in one of them;

  2. run python3 executor.py. in the other.

• to stop the test, terminate the executor.py script;

• run python3 loadtest_plotter.py, pass the locust_log.log and see the results. :)

Details

Using the performance testing infrastructure available in this repository, we conducted performance tests in a real-world alarm system provided by the GS company. To provide a way to reproduce our results without the particular alarm system, we build a software simulating the Alarm Receiving Software. The simulation model uses variables, we identified as relevant and also performed some measurements in the production environment, to initialize the variables correctly.

To reproduce our results, follow the steps in the Section "Quick start". The scripts are already preconfigured, to simulate a realistic workload, inject faults, and automatically recover from them. The recovery is performed after the time, the real fault management mechanism requires.

If you follow the steps and, for example, let the test run for about an hour, you will get similar results to the ones you can find in the Folder "Tests under Fault".

Results after running our scripts for about an hour:

---

Keep in mind that we use a simulated ARS here; in our paper we present measurements performed with a real system, thus the results reproduced with the code here are slightly different.

Nonetheless, the overall observations we made in our paper, are in fact reproducible.

---

Instructions on how to adapt our performance testing infrastructure to other uses

After cloning the repository, take a look at the locust_tester.py. This is, basically, an ordinary [Locust script](https://docs.locust.io/en/stable/writing-a-locustfile.html) that sends request to the target system and measures the response time, when the response arrives. Our locust_tester.py is special, because:

• we implemented a [custom client](https://docs.locust.io/en/stable/testing-other-systems.html) instead of using the default;

• we additionally log the response times to a logfile instead of using the [.csv files](https://docs.locust.io/en/stable/retrieving-stats.html) Locust provides.

So, write a performance test using Locust, following the instructions of the Locust developers on how to write a Locust script. The only thing to keep in mind is, that your Locust script has to output the measured response times to a logfile in the same way our script does it. Use logger.info("Response time %s ms", total_time) to log the response times.

When you have your Locust script ready, execute it with python3 executor.py, pass the path to your script as argument, and when you want to finish the load test, terminate it with Ctrl + C.

Use python3 executor.py --help to get additional information.

Example call:

% python3 executor.py locust_scripts/locust_tester.py

After that, plot your results:

% python3 loadtest_plotter.py
Path to the logfile: locust_log.log

References

• [1](https://www.doi.org/10.1007/978-3-030-68110-4_9)

• [teastore_fork](https://github.com/jtpgames/TeaStore)

• [simulator_repo](https://github.com/jtpgames/Simulators)

• [datalore_notebook](https://datalore.jetbrains.com/notebook/6K6VkECuLMtN5t5nSYg6WK/TVGp1egwDQlwI19astdVlM)