Creating a 3D thermal map using LSD SLAM

[akshay-iyer]

Hello, I am trying to create a 3D thermal map as per the paper: https://pdfs.semanticscholar.org/fe3a/9b89c0adcacb95581ad2e528e86b7d18c1f0.pdf
I am using an RGB camera Orbbec Astra and a FLIR E40 and they have been calibrated (intrinsic and extrinsic)
The process to create a corresponding thermal map is simple. For a keyframe: consider a pixel, get its 3D coordinates in the RGB camera frame using the inverse depth map.
Using the extrinsic parameters, get the point in the thermal camera frame and reproject the point into the thermal image using its intrinsic matrix. The value of the image will give us the thermal value of the pixel of the image.

Now, I have been looking into the LSD SLAM-core code and used Doxygen as well to understand it better. However, I'm unable to determine where and how do I start making changes to the code to get this thing working.

Any help would really be appreciated as I have a deadline for the same in a month.

Thanks

[bespoke-code]

Hi Akshay,

To jump-start your code, have a look at the RGB pointcloud implementation in PR #146 (work by Alexander Koumis) or the one in Kevin George's Ubuntu 16.04 fork.

You would have to match a corresponding thermal image to each RGB image in a similar way as the one used in the RGB implementation I mentioned. You can then use your proposed method to create the thermal depth map.

If you want to use the RGB pointcloud output and save the thermal property of each pixel, you would have to add each pixel's thermal value/s by modifying the InputPointDense class, which serves as a point representation within a KeyFrame. I think it won't be a big issue to implement an additional rqt_reconfigure checkbox option to switch between RGB view and Thermal view inside the lsd_slam_viewer node.

I hope this clarifies the task for you a bit. Good luck and happy coding!