Introducing Scenario Simulator Failure Analysis

[ahmeddesokyebrahim]

Introduction

Scenario simulator is a scenario testing framework for Autoware that enables Autoware developers to write scenarios at once and run them in various kinds of simulator.
Scenario simulator test can be used to make sure Autoware is functioning properly in specific scenarios.
Autoware evaluator utilizes scenario simulator to test Autoware using set of defined scenarios, on weekly cadence.

Challenge

When it comes to a scenario that was passing then it fails between two weeks runs, investigation should take place to know which commit introduced this failure to start fixing it. This investigation takes lots of time and effort.

Proposed Solution

I am excited to share with you scenario simulator failure evaluation tool.
The tool aims to help Autoware developers find the commit(s) that made a specific scenario fail within a specific time period.

Briefly ...

• The tool requires from the developer to provide the following:
  • .repos file which can be the one associated in Autoware Evaluator with a failing scenario
  • scenario_file_path which is the path of the failing scenario
  • date_to_start_searching which is the date you would like to tool to start evaluation process by
  • date_to_stop_searching which can be the date you know this specific scenario was passing in Autoware Evaluator.-
• The tool ....
  • checks out different commits that were pushed during this period (between start and stop searching dates)
  • compiles Autoware
  • runs scenario simulator
  • checks if the scenario is passing or not
  • keeps iterating within the specified dates
• When the tool comes to a combination of commits that is compiling Autoware successfully and passing with the scenario simulator, it stops the searching process and creates a .repos file for you with all commits that made this scenario fail so that you can start debugging from this point and a visualization for the executed evaluation process to make it easier for the user to narrow down the debugging process.

How to use

• The tool assumes you already have Autoware installed and can use it. In addition, the tool requires from the user to have some prerequisites that you can check here.
• The tool supports both calling the script with passing needed arguments from the command line or setting these arguments in the class constructor. You can follow the how to use section in the tool repository.

Expected Output

When a failing scenario becomes passing in one iteration, the tool organizes the output for the user in the following files

• $scenario_name$_failed_commits.repos
  This is the .repos file that includes the commit ids that you use to start debugging autoware for that failing scenario. That means it includes the commit just before the successful trial.

• last_changed_repo.txt
  This file includes the repo and commit id that are lately changed for the successful trial. That means, this is the commit that the tool just checked out, then the scenario passed.

• README.md
  This is the mermaid visualization file that you can use to visualize the output of the evaluation process.

Visualization

The visualization provided by scenario simulator failure evaluation tool is a simple representation of the different Autoware repositories and commits, indicating at the time of the scenario became successful which commit was checked out in different repositories, the commit that lastly checked-out that made the scenario successful, and the commit after it that is highly suspected to be the commit causing this failure.

• You can check here a sample output for the visualization.
• This section explains how to understand the visualization output.

How the tool works ?

• This section lists some of the functionalities that tool is actually doing for you after you invoke it from command line.
• As well, this flowchart explains in more details how the tool works.

How to make a quick test ?

In this section, you can find some tips about how to make a quick test with the needed files.

Call for Testers

We would like to invite everyone to participate in testing this tool and let us know about any issues, comments, thoughts, improvements about the tool.