scene render_to_image() and render_to_depth_image() not the same transform

[jerome-godbout-lychens]

Hi,

I try to capture the current image and the depth to perform some 2D detection and bring it back to 3D. When I raycast back to 3D, the unproject work for UV coordinate but I get a wrong depth. I did inspect the depth map that result from the render_to_depth_image() and what stroke me is that the object position is not the same as the actual scene image. The zoom is not even the same, the position either. Can somebody shed some light on this and how it work?

The strip down example use a cube at the origin and camera with look at, nothing else, the capture is start automatically after 2s. I did try to capture depth before image or image before depth, same results. I did try to add delay between the capture, same results. How is the depth image generated? do I need something special or is the scene look-at not considered somehow??? Both image have the right dimension compare to the window, they just not display the same scene. I also try to invert UV coordinate from the depth map just in case: self._capture_depth[:, [0, 1]] = self._capture_depth[:, [1, 0]] but without much results. Seem like it always use the origine as the top right corder of the image and doesn't care where the camera is point to.

The normalized depth image from render_to_depth_image():

The actual scene image from render_to_image():

The demo code used for this (reduce set to minimal example):
demo_bug.py.txt

Is their any way to get the depth map of the actual rendered scene?

I also try to replace the cube with a pyramide to see if I can better understand what is going on, just seems off for some odd reason:
use this into the def add_cube(self): instead
mesh_box = o3d.geometry.TriangleMesh.create_cone(radius=1.5, height=3.0, resolution=4, split=1, create_uv_map=False)

you can also add the axis into the init:
self._scene.scene.show_axes(True)

Any help or tips on what is going on with that would be appreciated thanks,

I did try to use the pyramide and axes and superpose the captured image and depth: and this is the results:

[jerome-godbout-lychens]

I think I have figure out, there is no tranform, it's just the wrong image scale, I'm on a Macbook, seems like the render to image use the pixel density but the depth map doesn't. Resulting into a depth map 1/2 the height and 1/2 the width of the render. So the render to image and the render to depth image have a different way of processing the screen dpi and pixel density. The render to image is right but the render to image depth is not resulting in only a 1/4 of the actual surface.