Questions about VIO results and custom VIOs

[TakShimoda]

Hello, thank you for this library.

I have been running COVINS(G) on ROS noetic on Ubuntu 20.04 with no problem with the provided VIO's: ORB-SLAM3 Mono-Inertial, SVO-PRO and VINS-FUSION, all with the ROS versions. I installed exactly as the instructions have said. I have run tests on the EuRoC Machine Hall and Vicon Rooms. I have been trying to expand experiments with this library but have been encountering a couple of problems.

I was trying to run COVINS(G) on this challenging dataset collected by 2D robots where the robots cover a very large distance on campus grounds. I was trying to run on the "campus hybrid" dataset which consists of 8 robots, with all ORB-SLAM3 as the front-end. I encountered drift on some of the trajectories which may be the limitation of ORB-SLAM3 or calibration issues (although I copied calibration data from the set). Upon looking at the ORB-SLAM3 repository, one of the authors recommended stereo-inertial for 2D robots due to lack of IMU excitation. So I uncommented Lines 123-130 of CMakeLists.txt in the ORB-SLAM3 ROS folder in covins, rebuilt the library, and ran the experiments again, and still encountered drift again. Outside of this issue, I have a couple of issues currently with COVINS(G) with ORB-SLAM3:

1. I tried running individual robots on this dataset with ORB-SLAM3 stereoi with covins back-end. One thing I noticed is when I ran roslaunch ORB_SLAM3 launch_ros_euroc.launch (which I edited for the calibration/ORB-SLAM3 stereoi/image topics), the ORB-SLAM3 viewer pops up, which doesn't happen for monoi in COVINS. The one thing I noticed is that for some trajectories, the ORB-SLAM3 trajectory on the viewer will look fairly accurate, but the trajectory on COVINS (in both rviz/KF output files) have drift. Here is an example where this is the output from COVINS:
   
   This has drift, but you can see at the same time the ORB-SLAM3 viewer has no drift and has a fairly accurate trajectory:
   
   I just want to ask what is the possible discrepancy between the ORB-SLAM3 library being used in COVINS and the result that COVINS is getting? For other test runs, I've had cases where they're the same (both good trajectories or both poor trajectories) as well, which should be expected for all robots.

2. Relating to 1, I was thinking, especially with the ORB-SLAM viewer showing up, that COVINS was only built for ORB-SLAM3 mono-inertial, as you mention here that the instructions to interface a custom VIO shows an example for ORB-SLAM3 mono-inertial. Does this mean I have to make some changes for stereo-inertial as well? It did run fine, just with poor performance (which may just be things I need to tune). Also, I'm a little confused what I have to do to interface a custom VIO. It says that modifications are made to ORB-SLAM3 in covins/covins_comm/include/covins/covins_base/typedefs_base.hpp but I can't really interpret anything from it, or know how to derive a communicator class to pass data. Can you possibly provide more information on how to do this?

3. While running the multi-robot case on the mentioned dataset, I encountered a fatal error on the covins backend node which caused it to crash. Here's a screenshot:
   
   A frontend agent also had an error of map reset, since ORB-SLAM3 resets the map but COVINS doesn't:
   
   I was wondering if the backend crashing simply had to do with the map reset and if so, should I simply skip to the point it happens, or just not use this sequence which causes ORB-SLAM3 to reset the map? I noticed the backend error mentions KF-65535 so I thought it was an overflow error or triggered by a certain condition.

Thanks.

[manthan99]

Hi

Regarding point 3, yes the back-end was not designed to handle the map-resets of ORB_SLAM3 and thus it will fail everytime a map-reset is performed. You can maybe try using some other front-end like VINS where the map resets usually do not occur.

Regarding points 1 and 2, I have personally never tried running ORB_SLAM3 Stereo-inertial with the COVINS backend. I will need to look into the codebase of ORB-SLAM3 before being able to give an answer. Again maybe you can try running VINS Fusion (stereo-inertial) with COVINS-G backend for a quick solution (If not constrained by using ORB_SLAM3)

Best
Manthan

[manthan99]

#26 Could be something useful for you as well

[TakShimoda]

Hello manthan99,

Thank you for your reply. For now, I don't absolutely need ORB-SLAM3, but it would be ideal to get stereo-inertial running as intended. Right now as I mentioned above, it does run, and sometimes it runs well, but I want to make sure everything is working fine, such as sending only the left keyframe to the communicator.

I just have a question about the specific changes I have to make. I know I have to implement the run() and convertToMsg() functions from frontend_wrapper but I'm confused as to where I use this? Following the ORB_SLAM3 example, the commands are:

roscore
rosrun covins_backend covins_backend_node
roslaunch ORB_SLAM3 launch_ros_euroc.launch
<rosbag play>

When this process runs, the frontend_wrapper isn't involved from checking the nodes, and checking the source code of ros_mono_inertial.cc, everything was the same as from the original ORB-SLAM3 code except for the line to include config_comm.hpp.

I'm wondering when the run() and convertToMsg() functions are called in this process and how I can change it for stereo to make sure only keyframes from one camera are sent to the communicator.

Regards,

[manthan99]

Hi

You will need to look into the CPP code to see where the changes are done. For example some changes are done in the following files. There could be more of the changes so just look for all the changes which have "COVINS_MOD" in the file.

https://github.com/VIS4ROB-lab/covins/blob/master/orb_slam3/src/KeyFrame.cc
https://github.com/VIS4ROB-lab/covins/blob/master/orb_slam3/src/LocalMapping.cc
https://github.com/VIS4ROB-lab/covins/blob/master/orb_slam3/src/Tracking.cc

I hope this is helpful. I will need to look more into details to provide a better answer.

Best
Manthan

[TakShimoda]

Thanks Manthan. Thanks for the help.

This might be a question that needs more insight, but if I was to try to implement a nodelet for the frontend so that publisher-subscriber message passing for the images and imu data could be handled with C++ pointers instead of over TCP, would I just modify the ROS code such as ros_stereo_inertial as the subscriber/publisher and image parsing code is written there?

Thanks,