Projection of color images onto a mesh

[Grimwork]

Hi,

I have seen in your README.md that you present a use case where you project an image's color onto a mesh.

And i wanted to know how did you do that ?
Did you programmatically add the color of each pixel on each point ?

[marip8]

• transform the mesh into camera coordinates
• project mesh into camera using camera matrix
• for each projected vertex
  • convert floating point projected vertex to integers to create u-v coordinates
  • index into color image at projected vertex u-v coordinate to get color for vertex
  • add color info to original mesh vertex

Here's a snippet of some Python psuedocode for doing this

# Get transform from camera to mesh (using TF)
camera_to_mesh = ...

# Extract camera matrix from ROS camera info message
camera_matrix = np.array(camera_info.P).reshape((3, 4))
  
# Put mesh vertices into homogenous coordinates (i.e., [x, y, z, 1])
verts = np.vstack([mesh_verts.T, np.ones((mesh_verts.shape[0],))])

# Project the mesh vertices into the camera frame
projected_verts = camera_matrix @ camera_to_mesh @ verts
projected_verts /= projected_verts[2, :]

# Convert to u-v coordinates
vert_coords = projected_verts[:2, :].T.astype(int)

# Extract the colors from the image
colors = image[vertex_coords[:, 1], vertex_coords[:, 0]]