2024-11-18 ODD working group weekly meeting (214)

[Igata-ctrl]

Attendee

• [Place holders]

• Tran Huu Nhat Huy (Tokyo Institute of Technology, student)
• David Walmroth (PIX Moving)
• @mitsudome-r Ryohsuke Mitsudome (AWF TSC/TIER IV)
• @Igata-ctrl Hiroshi IGATA (TIER IV)

Agenda/Minutes

• On October 20, 2024, 297/340 "public road bus" and "pull out" scenarios succeeded on Autoware.universe main branch.

  • 111/142 public road bus scenarios were successful. Identical to the results of recent weeks.

    • UC-23-test-bikeway1 : Drivable area looks appropriately enhanced, but EGO stops for "Outside (of) Drivable Area" and Bicycle stops against Ego vehicle though "isBLIND:true" is set.
    • In addition to the above, UC-23-004-0001-With2Obstacles-ShalunRS and 30 other scenarios (including parameter variations) failed due to system errors (Simulation Health Check, Time Out etc.).

  • 186/198 pull out scenarios were successful. 12 scenarios failed.

    • UC-v2-F-15-00003_001_case01 : 1/4 case failed as Ego was not able to back up as intended.(It failed even after "act_ego nostop" condition was released.)
    • UC-v2-F-15-00006_001_case(01|02|03)_cmn_general : Failed -> StoryboardElementStateCondition has been modified to reflect the original use case intention of "reach near max speed within 200m" but the "act_ego_speed_check" was not met as EGO accelerates only up to 15kph though the max speed is set higher and the lane speed limit is 60kph (9 parameter cases).
    • UC-v2-F-15-00007_001_case04_cmn_general : Last week's error was solved, but 2/36 case failed with "TimeOut" and "InternalError" message="attempting to request a message part outside the valid range"

• On October 20 2024, 79/96 Dense Urban scenarios were successful on Autoware.universe scenario-dev branch. 230 scenarios failed. From this week three modified scenarios for reference design working group have been added and they all succeeded.

  • Test-SafetyPool-map1 : Has been successful for a month, but failed this time. (A simple lane change did not happen.)
  • Test-roundabout1_Shalun : Ego cannot avoid collision with the NPC in the round about. -> It looks like NPC is not detected (as the red bounding box does not appear in Rviz screen).
    • Above two scenarios succeed in the test on October 3 with the build in May 2024.
  • Test-dense-urban_ITRI(2|3|4) : EGO cannot use the space in the opposite lane when it passes/overtakes NPC/Pedestrians.
  • Test_MRM_Fault_Injection1 : MRM not conducted upon the fault injection.
  • (UC-NTR-001-0003_case2 : all 3 cases succeeded without violating excess deceleration threshold.)
  • UC-NTR-001-0005 : All 3 cases failed for "Ego moved before Npc takeover finished.
  • (UC-NTR-001-0008 : All 3 case succeeded without any Ego's deceleration overshoot (<0.8*Vslow).)
  • (UC-NTR-002-0003_case2 : All 3 cases succeeded.)
  • UC-NTR-004-0001 : EGO did not avoid the NPCs coming in the wrong direction.
  • UC-NTR-005-0001 : All 3 case succeeded without any Ego's deceleration overshoot (<0.8*Vslow).
  • UC-VRU-002-0001_case(1|2) : Failed with the excess deceleration.
  • UC-VRU-002-0005 : Failed with the excess deceleration.
    • Above two scenarios succeed in the test on October 3 with the build in May 2024.

• On October 27, 2024, 295/340 "public road bus" and "pull out" scenarios succeeded on Autoware.universe main branch.

  • 110/142 public road bus scenarios were successful.

    • UC-23-test-bikeway1 : Drivable area looks appropriately enhanced, but EGO stops for "Outside (of) Drivable Area" and Bicycle stops against Ego vehicle though "isBLIND:true" is set.
    • In addition to the above, UC-23-004-0001-With2Obstacles-ShalunRS and 30 other scenarios (including parameter variations) failed due to system errors (Simulation Health Check, Time Out etc.).
    • UC-PB-002-0002-Hsinchu : 1/4 case failed with a system error. -> Succeeded in a simple retry.

  • 186/198 pull out scenarios were successful. 12 scenarios failed.

    • UC-v2-F-15-00003_001_case01 : 1/4 case failed as Ego was not able to back up as intended.(It failed even after "act_ego nostop" condition was released.)
    • UC-v2-F-15-00006_001_case(01|02|03)_cmn_general : Failed -> StoryboardElementStateCondition has been modified to reflect the original use case intention of "reach near max speed within 200m" but the "act_ego_speed_check" was not met as EGO accelerates only up to 15kph though the max speed is set higher and the lane speed limit is 60kph (9 parameter cases).
    • UC-v2-F-15-00007_001_case04_cmn_general : Last week's error was solved, but 2/36 case failed with "TimeOut" and "InternalError" message="attempting to request a message part outside the valid range"

• On October 27 2024, 75/96 Dense Urban scenarios were successful on Autoware.universe scenario-dev branch.

  • Test-SafetyPool-map1 : Has been successful for a month, but failed this time. (A simple lane change did not happen.)
  • Test-roundabout1_Shalun : Ego cannot avoid collision with the NPC in the round about. -> It looks like NPC is not detected (as the red bounding box does not appear in Rviz screen).
    • Above two scenarios succeed in the test on October 3 with the build in May 2024.
  • Test-dense-urban_ITRI(2|3|4) : EGO cannot use the space in the opposite lane when it passes/overtakes NPC/Pedestrians.
  • Test_MRM_Fault_Injection1 : MRM not conducted upon the fault injection.
  • UC-NTR-001-0003_case2 : 1/3 case failed with excess deceleration.
  • UC-NTR-001-0005 : All 3 cases failed for "Ego moved before Npc takeover finished.
  • (UC-NTR-001-0008 : All 3 case succeeded without any Ego's deceleration overshoot (<0.8*Vslow).)
  • UC-NTR-002-0001_case1 : 1/3 case failed with TimeOut.
  • (UC-NTR-002-0003_case2 : All 3 cases succeeded.)
  • UC-NTR-004-0001 : EGO did not avoid the NPCs coming in the wrong direction.
  • UC-NTR-005-0001 : All 3 case succeeded without any Ego's deceleration overshoot (<0.8*Vslow).
  • UC-VRU-001-0001_case1 : 1/2 case failed with excess deceleration.
  • UC-VRU-002-0001_case(1|2) : Failed with the excess deceleration.
  • UC-VRU-002-0005 : Failed with the excess deceleration.
    • Above two scenarios succeed in the test on October 3 with the main build in May 2024.

• On November 3, 2024, 294/340 "public road bus" and "pull out" scenarios succeeded on Autoware.universe main branch.

  • 109/142 public road bus scenarios were successful.

    • UC-23-test-bikeway1 : Drivable area looks appropriately enhanced, but EGO stops for "Outside (of) Drivable Area" and Bicycle stops against Ego vehicle though "isBLIND:true" is set.
    • UC-LD-001-0010-1-Gebze : 1/3 case failed with unnecessary stop. -> Succeeded after deleting redundant Ego initial speed ($vel).
    • UC-PB-001-0006-ITRI : 1/3 case failed with a system error (Simulator waited for the Autoware state to transition to WaitingForRoute, but time is up. The current Autoware state is INITIALIZING). -> No action taken.
    • In addition to the above, UC-23-004-0001-With2Obstacles-ShalunRS and 30 other scenarios (including parameter variations) failed due to system errors (Simulation Health Check, Time Out etc.).

  • 185/198 pull out scenarios were successful. 13 scenarios failed.

    • UC-v2-F-15-00002_001_case03 : 1/6 case failed with Ego unnecessary stop before reaching its goal ($Ve is set to 4.2m/s).
    • UC-v2-F-15-00003_001_case01 : 1/4 case failed as Ego was not able to back up as intended.(It failed even after "act_ego nostop" condition was released.)
    • UC-v2-F-15-00003_001_case03 : All 4/4 cases succeeded.
    • UC-v2-F-15-00003_002_case01 : All 4/4 cases succeeded.
    • UC-v2-F-15-00006_001_case(01|02|03)_cmn_general : Failed -> StoryboardElementStateCondition has been modified to reflect the original use case intention of "reach near max speed within 200m" but the "act_ego_speed_check" was not met as EGO accelerates only up to 15kph though the max speed is set higher and the lane speed limit is 60kph (9 parameter cases).
    • UC-v2-F-15-00007_001_case04_cmn_general : 1/36 case failed with "TimeOut".

• On November 3, the build for Dense Urban suite with scenario-dev branch failed. No simulation result obtained.

• On November 10, 2024, 298/340 "public road bus" and "pull out" scenarios succeeded on Autoware.universe main branch.

  • 110/142 public road bus scenarios were successful.

    • UC-23-004-0001-With2Obstacles-ShalunRS was replaced with UC-23-004-0001-LimitedLOS-ITRI to avoid system errors.
    • UC-23-test-bikeway1 : Drivable area looks appropriately enhanced, but EGO stops for "Outside (of) Drivable Area" and Bicycle stops against Ego vehicle though "isBLIND:true" is set.
    • In addition to the above, 30 other scenarios (including parameter variations) failed due to system errors (Simulation Health Check, Time Out etc.).

  • 188/198 pull out scenarios were successful. 10 scenarios failed.

    • UC-v2-F-15-00003_001_case01 : 1/4 case failed as Ego was not able to back up as intended.(It failed even after "act_ego nostop" condition was released.)
    • UC-v2-F-15-00006_001_case(01|02|03)_cmn_general : 3/4 cases failed for each of the scenario cases -> StoryboardElementStateCondition has been modified to reflect the original use case intention of "reach near max speed within 200m" but the "act_ego_speed_check" was not met as EGO accelerates only up to 15kph though the max speed is set higher and the lane speed limit is 60kph (9 parameter cases).

• On November 11, 77/96 Dense Urban scenarios were successful on Autoware.universe scenario-dev branch after a retry by @mitsudome-r.

  • Test-SafetyPool-map1 : Has been successful for a month, but failed this time. (A simple lane change did not happen.)
  • Test_FollowTrajectoryAction : 1/4 case failed with unnecessary Ego stop. Succeed after explicitly setting Ego target speed to 4.2 m/s.
  • Test-roundabout1_Shalun : Ego cannot avoid collision with the NPC in the round about. -> It looks like NPC is not detected (as the red bounding box does not appear in Rviz screen).
    • Above two scenarios succeed in the test on October 3 with the build in May 2024.
  • Test-dense-urban_ITRI(2|3|4) : EGO cannot use the space in the opposite lane when it passes/overtakes NPC/Pedestrians.
  • Test_MRM_Fault_Injection1 : MRM not conducted upon the fault injection.
  • UC-NTR-001-0005 : All 3 cases failed for "Ego moved before Npc takeover finished.
  • UC-NTR-004-0001 : Both 2 cases failed for EGO not avoiding the NPCs coming in the wrong direction.
  • UC-VRU-002-0001_case(1|2) : All failed with the excess deceleration.
  • UC-VRU-002-0005 : Failed with the excess deceleration.

• On November 17, 2024, 295/340 "public road bus" and "pull out" scenarios succeeded on Autoware.universe main branch.

  • 110/142 public road bus scenarios were successful.

    • UC-LD-001-0002-Gebze : 1/3 case failed with time out. -> Succeeded with a simple retry.
    • UC-23-test-bikeway1 : Drivable area looks appropriately enhanced, but EGO stops for "Outside (of) Drivable Area" and Bicycle stops against Ego vehicle though "isBLIND:true" is set.
    • 30 other scenarios failed. All of these are on Shalun map and were succeeding on and before August 18, 2024. Currently, these scenarios are failing only by putting the Ego destination in or beyond the intersection ahead. @mitsudome-r is kindly investigating if Autoware.universe was modified in regards to handling of the intersections or the traffic signals after August 18.

  • 183/198 pull out scenarios were successful. 15 scenarios failed.

    • UC-v2-F-15-00002_001_case01 : 1/6 case failed as the expected back-up did not happen.
    • UC-v2-F-15-00002_001_case04 : 1/6 case failed with unnecessary stop of Ego. -> Looks OK after explicitly setting Ego max speed to 4.2m/s (15km/h).
    • UC-v2-F-15-00003_001_case01 : 1/4 case failed as Ego was not able to back up as intended. (It failed even after "act_ego nostop" condition was released.)
    • UC-v2-F-15-00006_001_case(01|02|03)_cmn_general : 3/4 cases failed for each of the scenario cases -> StoryboardElementStateCondition has been modified to reflect the original use case intention of "reach near max speed within 200m" but the "act_ego_speed_check" was not met as EGO accelerates only up to 15kph though the max speed is set higher and the lane speed limit is 60kph (9 parameter cases).
    • UC-v2-F-15-00007_001_case04_cmn_general : 1/36 failed with unnecessary stop of Ego. -> Ego max speed was already to 4.2m/s (15km/h) and succeeded in a retry.
    • UC-v2-F-15-00102_003_case01_cmn_general : 1/2 failed with unnecessary stop of Ego. -> Ego max speed was already to 4.2m/s (15km/h) and succeeded in a retry.
    • UC-v2-F-15-01001_001_case01_cmn_general : Filed. The playback looked successful but some of the success conditions "(Is the /planning/planning_evaluator/metrics_modified_goal_{longitudinal | lateral | yaw}_deviation_mean (= ) is lessThan 1.0?)" were not met.

• On November 17, 77/96 Dense Urban scenarios were successful on Autoware.universe scenario-dev branch.

  • Test-SafetyPool-map1 : Has been successful for a month, but failed this time. (A simple lane change did not happen.)
  • Test-roundabout1_Shalun was ported onto ITRI map as Test-roundabout2_ITRI and succeeded.
  • Test-dense-urban_ITRI(2|3|4) : EGO cannot use the space in the opposite lane when it passes/overtakes NPC/Pedestrians.
  • Test_MRM_Fault_Injection1 : MRM not conducted upon the fault injection.
  • UC-NTR-001-0003_case2 : 1/3 case failed but succeeded in a retry.
  • UC-NTR-001-0005 : All 3 cases failed for "Ego moved before Npc takeover finished.
  • UC-NTR-002-0003_case1 : 1/3 case failed with the excess deceleration.
  • UC-NTR-004-0001 : Both 2 cases failed for EGO not avoiding the NPCs coming in the wrong direction.
  • UC-VRU-002-0001_case(1|2) : All failed with the excess deceleration.
  • UC-VRU-002-0005 : Failed with the excess deceleration.

• As for the new scenario suite (catalog) for LSV, it seems that almost all the scenario files are just changed with names. May need to ask Daniel-san to revert all the titles to the original one and take the following time-consuming process to duplicate and modify the copied scenarios for LSV/Reference Design use.

  • Open the original scenario with the scenario editor in Evaluator.
  • [Export yaml]
  • [+ New Scenario]
  • [Import yaml]
  • Give the new name like "RefLSV-UC-A-01-0001" in the [Title] column in the scenario editor
  • [Save] the scenario
  • Deselect the obsolete scenarios from the [Catalogs]Urban-Dense - [SUITES] LSV for Ref Design #63636 and add new ones.

• [recap of the discussions until last week]

• Had a Safety Pool/AWF meeting regarding the existing issues.

• Discussed with Daniel whether the scenarios are appropriately copied to LSV catalog instead of modifying the original. -> @Igata-ctrl to double check it by next week.

• Preliminary discussion with TIER IV Evaluator team revealed that the import of OpenSCENARIO 1.x files will be supported hopefully by the end of 2024. Currently, Scenario Simulator v2 uses the proprietary t4v2 format in yaml to better handle the parameter variations. This nee import function enables the use of out side scenario editor like AVL SCENIUS™️ Scenario Designer and help importing scenarios from Safety Pool database, though the issue of map format and the lack of destinations may remain.

• Discussion to create Evaluator improvement wish list.

  • Embed illustrations/animation in the Scenario Editor description pane or enumeration. (Armağan)
  • Integration with Vector Map Editor. (Igata) -> Clarify if some of this function is already available.
  • Create a discussion to collect needs/wants for Evaluator from larger audience.

• Discussed the improvement requests for Evaluator.

  • Will continue discussion next week, Armağan's initial ides included embedding the use case definition (slides) and the scenario (NPC behavior) preview by the time-slider in the scenario editor screen, etc.
  • @Igata-ctrl explained that Evaluator is not an OSS but a product from TIER IV, so the development team has a long wish list form TIER IV and OEM projects.
  • May need to clarify the content of the wish list from AWF before discussing the priority with TIER IV CI/CD team. @Igata-ctrl to try to collect recent situation with the team.

• Regarding the needs for calculating the accurate RSS safety distance from Vf(Npc) and Vr(Ego), TIER IV Mapping and Localization team will be able to add the functionality using UserDefinedValueCondition like UC-v2-F-15-01001_001_case01_cmn_general case. It may take more than two weeks depending on their resource margin/availability. -> @brkay54 to kindly submit an issue regarding this request mentioning Takeuchi-san of TIER IV (@kosuke55) who is working on UserDefinedValueCondition.

• For the current scenario suite for weekly execution above, the RSS definition for minimum safety distance and 2.499m/s^2 for maximum comfortable deceleration have been applied.

• @Igata-ctrl got the information in Simulation working group meeting that the resource allocated to Scenario Simulation v2 on the cloud may not be sufficient and it can make the simulation result not trustworthy enough. We may need to analyze the Simulation Log file to see if there are many waring messages in there when we get unexpected simulation result.

• @Igata-ctrl to investigate if we can use real time speed of Ego and Npc to calculate RSS Safe Distance more accurately.

• Discussion (RSS distance and comfortable deceleration) pages have been added to the Use cases for Dense Urban Area 2024 presentation.

• Reached a consensus to use RSS definition for minimum safety distance and 2.499m/s^2 for maximum comfortable deceleration for low speed shuttle.

• Discussion of the condition/threshold setting so that we can better track the Autoware maturity through scenario simulation.

• After the discussion, it turned out "min_acc" parameter should better be reverted to original value of -1.5 (this week).

• Current discussion with Safety Pool

  • [Map] Successfully converted a sample OpenDRIVE map from Safety Pool database to Lanelet2, but we have following issues. Though Mohammad (Deepen.ai/AWF Safety Assurance WG lead) proposed in May TSC that Autoware should consider accepting OpenDRIVE maps in a direct fashion as it solves all these issues, given that we may need to map-less autonomy, it may not worth switching map format from now.
    • Speed limit, lane change flag and LHT/RHT are lost through the conversion.
    • The converted Lanelet2 map contains a lot of unnecessary points on line strings. (e.g. 16,000points on 1 km one way straight road)
  • [Scenario] Autoware requires EGO's destination in each scenario, while all of the Safety Pool scenarios do not have the destination. Though Safety Pool once suggested to run Autoware in the “roaming” mode which does not require the destination in the scenario, it sounds unrealistic for Autoware to operate without destination as the destination is almost always an important part of the use case/scenario intention. (e.g. turning left/right).
    • In the Safety Pool Webinar, they demonstrated Autoware running with Safety Pool scenario, but they say that it requires some special settings for it and it is not scalable.
  • [Scenario file format] Though both Safety Pool and Autoware (Scenario Simulator v2) use OpenSCENARIO v1.x (a.k.a. OpenSCENARIO XML) standard, Autoware (Scenario Simulator v2) uses ”.yaml” file to better handle the parameters, while Safety Pool uses “.xosc” file, which is the default of the OpenSCENRIO standard. If the number of scenarios is not too big, we might be able to convert them from .xosc to .yaml format manually. (Currently, no automated converter is available.) If the scenario simulator v2 is installed on-premise, it will accept the .xosc file in a direct fashion.
  • [Scenario availability] After signing up for the Safety Pool database, the number of the accessible/visible scenarios are limited. 32,588 scenarios are visible but most of the scenarios are marked with "access restricted" and only 1957 scenarios are accessible. The total number of the scenario which Safety Pool claims is one million. Still waiting for an answer from Safety Pool

Action Item

Documents

• The bus ODD use case list which was assigned to the members is here.

• The bus ODD use case list has been also uploaded here to make it visible to anyone on the net.

• The ODD working group shared document folder is here. [Restored!]

• Discussions and Q&As in AWF Discord ODD WG channel are also encouraged.

Tools

• [Autoware Evaluator (CI/CD pipeline)]

  • Cloud based DevOps (integration of the development tools including the scenario editor and the scenario simulator below)
  • A product from TIER IV and offered for the official Autoware Foundation projects like Cargo Delivery and Public Road Bus, etc.
  • The user guide is available TIER IV document site
  • As it consumes AWS resource, the (batch) weekly execution of the scenario simulation is managed by the Software/ODD WG leads

• [Scenario Simulator V2 (Scenario testing framework)]

  • Stand alone scenario simulation tool
  • An OSS from TIER IV freely available for any Autoware developer/researcher
  • The documentation of the Scenario Testing Framework (open sourced from Tier IV) is on GitHub

• The GUI Scenario editor

  • Web based GUI scenario editor freely available from TIER IV
  • You can create and export scenarios with this web interface
  • The user guide is available TIER IV document site

• [TIER IV account]

  • The working group members who are interested in creating and testing scenarios are advised to create a free TIER IV account here.
  • Once you have created your account, please let @Igata-ctrl know the (long) User ID which appears on your login page. After @Igata-ctrl registered you to the AWF group in the Evaluator (CI/CD pipeline), you can go to AWF Autoware Evaluator page to see the simulation results, create/edit scenarios, etc.
  • If you already have a TIER IV account, your 4-digit User ID continues to work, so you do not need to register to TIER IV account again.

Administrative

• The two meetings on December 25th (Christmas day) and January 1st (New Year day) have been cancelled.

• Please check the ODD WG wiki page.

• Recurring weekly meetings have been scheduled. Please check Autoware Foundation events calendar and add this calendar to your own Google calendar by clicking the right bottom button and/or add your contact to ODD WG invitation group to receive invitations for future meetings.

• ODD WG meetings are held weekly in the following single time slot.

  • 7:00am, Monday (PST) / 6:00am, Monday (PDT) US Pacific Time
  • 10:00am, Monday (EST) / 9:00am, Monday (EDT) US Eastern Time
  • 4:00pm, Monday (CEST) / 3:00pm, Monday (CEDT) Poland time
  • 5:00pm, Monday (TRT) Turkey time
  • 10:00pm, Monday (CST) Taiwan time
  • 11:00pm, Monday (JST) Japan Time