Constructing pose graph of images from legacy reconstruction system by setting fragment size = 1

[marioblue5]

Hello all!

I want to construct a pose graph from RGB-D images, and I've been trying to utilize Open3D for this. I utilize the legacy reconstruction system to make, register, and refine fragments which are derived from just one image. I do this in hopes of finding a pose for each image/camera. This is my current script:

import json
import numpy as np
import os
from tqdm import tqdm

# Input file paths
optimized_posegraph_path = "C:/Users/molina.mario/open3d_data/extract/LoungeRGBDImages/scene/global_registration.json"
color_image_path = "C:\\Users\\molina.mario\\open3d_data\\extract\\LoungeRGBDImages\\color\\"
depth_image_path = "C:\\Users\\molina.mario\\open3d_data\\extract\\LoungeRGBDImages\\depth\\"

# Example camera intrinsics
camera_intrinsics = {
    "fl_x": 525.0,
    "fl_y": 525.0,
    "cx": 319.5,
    "cy": 239.5,
    "w": 640,
    "h": 480,
    "camera_model": "OPENCV",
}

# Read the optimized pose graph
with open(optimized_posegraph_path, 'r') as f:
    posegraph = json.load(f)

# Initialize frames list
frames = []

# Iterate through nodes to construct frame data
for node in tqdm(posegraph['nodes'], desc="Processing nodes"):
    # Extract and reshape the pose matrix
    pose_array = node['pose']
    transform_matrix = np.array(pose_array).reshape((4, 4), order='F')

    # Construct file paths based on node IDs
    node_id = posegraph['nodes'].index(node)
    color_file = os.path.join(color_image_path, f"{node_id+1:06d}.png")
    depth_file = os.path.join(depth_image_path, f"{node_id+1:06d}.png")

    frame_data = {
        "file_path": color_file,
        "transform_matrix": transform_matrix.tolist(),
        "depth_file_path": depth_file
    }
    frames.append(frame_data)

# Combine camera intrinsics and frames into final output
output_data = camera_intrinsics.copy()
output_data["frames"] = frames

# Write to a JSON file
output_json_path = os.path.join(os.path.dirname(optimized_posegraph_path), "transforms.json")
with open(output_json_path, 'w') as f:
    json.dump(output_data, f, indent=4)

print(f"Converted JSON saved to: {output_json_path}")

My main issue is that the pose graphs constructed are rather inaccurate, specifically their rotations/angles. I'm using the first 50 images of the Stanford Lounge dataset for proof of concept. The reconstructions from open3D come out just fine when using the pose graph, but when I try and fit it into NeRFStudio for Gaussian Splatting the quality falls off. One thing I'm unsure if is if I'm exporting the posegraph correctly from Open3D to NeRFStudio's format.

The data convention/format for NeRFStudio is:

        // [+X0 +Y0 +Z0 X]
        // [+X1 +Y1 +Z1 Y]
        // [+X2 +Y2 +Z2 Z]
        // [0.0 0.0 0.0 1]

While Open3D is just a 16 element 1D array:

"pose" : 
			[
				0.99975739114952911,
				-0.020041879285794945,
				-0.0091368439068721646,
				0.0,
				0.019354763322122229,
				0.99736382673217883,
				-0.069934185223672354,
				0.0,
				0.010514370101415923,
				0.069740377120058164,
				0.99750976327078744,
				0.0,
				0.00090600290204298842,
				-0.15309069045490412,
				-0.024475202327008741,
				1.0
			],

My code above uses the "F" Fortran-Style "Column Major" which I believe is correct since the coordinates at the bottom match up. However, I was unable to find any documentation on the formatting of the 16-element array which open3D utilizes.

Once again, my goal is to create an accurate pose graph for RGB-D images. If anyone can give any advice or tips or documentation that would be greatly appreciated. Thanks!