Skip to content

Commit

Permalink
Merge pull request #157 from ls1intum/develop
Browse files Browse the repository at this point in the history
Develop
  • Loading branch information
valentin-boehm authored Mar 2, 2024
2 parents 4ffcb3c + f6961b7 commit f27c5f7
Show file tree
Hide file tree
Showing 9 changed files with 42 additions and 0 deletions.
42 changes: 42 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -254,10 +254,18 @@ Each user has an ID. The ID is used to determine which users will participate in
- If you specify the exact users that should participate in the simulation, you need to enter the IDs of the users.
- See [Creating a Simulation](#creating-a-simulation) for more information.

The following image shows the form for adding users with patterns:

![Create-Users](docs/create-pattern.png)

When you add users with patterns, you can additionally check the box `Create users on Artemis`. You will then be able to provide patterns for first name, last name, and email. The tool will try to create the users on Artemis with the specified patterns.
This is useful if your Artemis instance does not yet have the necessary amount of test users registered. This feature is only available if an Admin user is available for the respective Artemis server.
Please note that editing and deleting users in the Benchmarking Tool does not affect the users on Artemis.

The following image shows the form for creating users in Artemis via a pattern:

![Create-Users-Artemis](docs/create-on-artemis.png)

When you add users through a CSV file, the file must have the following format:

```
Expand All @@ -275,6 +283,10 @@ id,username,password
2,user2,password2
```

This video explains the User Management in more detail:

https://github.com/ls1intum/Artemis-Benchmarking/assets/129070641/ace00d70-3c3a-490a-83c0-44e1eb46f321

**IMPORTANT**
The Benchmarking Tool stores the credentials of the Artemis users in the database. Therefore, it is important to keep the database secure. The credentials are stored in plain text and can be accessed by anyone who has access to the database. The Benchmarking Tool should only be used in a secure environment.
**The database must only be accessible for people who are authorized to see the Artemis users' credentials. Only credentials of test users should be stored in the Benchmarking Tool.**
Expand Down Expand Up @@ -307,6 +319,10 @@ Explanation of the form fields:
- If you want to simulate against the Artemis Production instance and chose a mode that requires Instructor rights, you can optionally specify the necessary credentials. _They will be stored in the database!_ Please refer to the [Artemis User Management](#artemis-user-management) section for more information. This step is optional and can be done later.
- `Number of commits and pushes`: The range of pushes that users will perform per programming exercise. The tool will randomly choose a number of pushes for each user in the range.

The following image shows the form for creating a simulation:

![Create-Simulation](docs/create-form.png)

### Running a Simulation

To run a simulation, find the simulation in the list of simulations and click the "Start Run" button. The new run will be added to the list of runs of the simulation.
Expand All @@ -328,6 +344,10 @@ During the simulation, the tool will perform the following steps for each studen
While the simulation is running, you can see some log messages indicating the progress of the simulation. They will also show if something went wrong.
If the tool is connected to Prometheus, you can see the workload data of the configured Prometheus targets during the simulation.

The log message for a successful run will look similar to this:

![Log messages](docs/logs.png)

If a fatal error occurs, the run will be stopped and marked as failed. You will be able to see the cause of the error in the log messages. If an error occurs for a single student, the simulation will continue for the other students.

You can click on "Abort" to cancel the simulation run. The tool will stop the simulation run as soon as possible. Please note that this might leave the exam in an inconsistent state.
Expand All @@ -350,10 +370,24 @@ The results consist of the number of requests and average response times for the

The results are displayed in the detail view of the run. The metrics are also available in a per-minute resolution. If enough data is available, you can see the metrics in line-charts over time.

The following image shows the simulation results in the category `Push`, first as a table and then as a line-chart:

![Push-Table](docs/push-table.png)

![Push-Chart](docs/push-chart.png)

<a id="ci-status"></a>
If the simulation is running against an Artemis instance that uses the Integrated Code Lifecycle setup, the tool will fetch the CI status after the simulation. The CI status is displayed in the detail view of the run.
It includes the number of builds and the average builds per minute.

The following image shows the CI status:

![CI-Status](docs/ci-status.png)

The following video shows the process of creating, executing and evaluating simulations:

https://github.com/ls1intum/Artemis-Benchmarking/assets/129070641/aa694de3-1c7b-4815-a3e6-45e55ceaa0f8

### Scheduling Simulations

The Benchmarking Tool can be used to schedule simulations. This is useful if you want to run simulations at a specific time or at regular intervals.
Expand All @@ -372,6 +406,14 @@ In the form that opens, you can specify the following fields:
After creating the schedule, it will be added to the list of schedules for the simulation.
You can edit or delete the schedule by clicking on the respective icons.

The schedule dialog looks like this:

![Schedule-Dialog](docs/schedule-dialog.png)

This video shows how to create schedules:

https://github.com/ls1intum/Artemis-Benchmarking/assets/129070641/cd0e62dd-a8c3-4fb2-8482-262fc5d9b1a8

**Subscribing to a Schedule**
Click on the Bell icon of the respective schedule to subscribe to it. You can specify your email address here.
After subscribing, you will receive an email notification containing a summary of the simulation results after every scheduled run.
Expand Down
Binary file added docs/ci-status.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/create-form.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/create-on-artemis.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/create-pattern.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/logs.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/push-chart.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/push-table.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/schedule-dialog.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.

0 comments on commit f27c5f7

Please sign in to comment.