Merging changes from the hilti challenge #515

ayushbaid · 2022-06-07T06:41:06Z

No description provided.

dellaert

Awesome. I flagged what I could. Would be good to have report before merging.

gtsfm/averaging/translation/averaging_1dsfm.py

gtsfm/bundle/bundle_adjustment.py

gtsfm/frontend/inlier_support_processor.py

gtsfm/loader/loader_base.py

gtsfm/two_view_estimator.py

gtsfm/utils/metrics.py

akshay-krishnan

This is going to be huge!

gtsfm/averaging/rotation/rotation_averaging_base.py

akshay-krishnan · 2022-06-08T02:43:23Z

gtsfm/averaging/rotation/shonan.py

    ) -> BetweenFactorPose3s:
        """Create between factors from the priors on relative poses."""
        between_factors = BetweenFactorPose3s()
+        noise_model = gtsam.noiseModel.Isotropic.Sigma(POSE3_DOF, PRIOR_SIGMA)


why don't we use the covariance now? (i.e address your TODO?)

another nit: the POSE3_DOF is probably overkill. It's something that will never change.

I think we did this somewhere, so you can reuse that code.

I was getting convergence issues with Shonan if I used priors from Bayesian ICP. So I was hardcoding it. What should we do now as a feature introduction to master?

Is this Shonan specific? Were you getting good results in BA or TA?

If all of them need hardcoding, we can do that in the loader. One of them needing it is weird, maybe the other hardcoded values can be changed?

I think the main problem here is not the convergence issue, but the noise model. We cannot accept a full covariance matrix in the Shonan averaging. We need to address the TODO on the next line.

This is a pending comment from an old review, please ignore if fixed:

I remember Frank suggested using an isotropic noise model with a Sigma which is an average of the diagonal elements. Not sure how that works.

gtsfm/averaging/rotation/shonan.py

gtsfm/retriever/rig_retriever.py

akshay-krishnan · 2022-06-08T05:38:24Z

gtsfm/scene_optimizer.py

+REACT_METRICS_PATH = BASE_PATH / "rtf_vis_tool" / "src" / "result_metrics"
+REACT_RESULTS_PATH = BASE_PATH / "rtf_vis_tool" / "public" / "results"


can we remove this altogether?

I think Frank was talking about some student who will pick up the JS work again. So lets keep it for the time being.

Yeah, but isnt it the same data that we save in both results_path and react_metrics_path ? Do we need this duplication?

I remember there was some need for this because the viz tool could not access the parent of the root folder.

gtsfm/scene_optimizer.py

akshay-krishnan · 2022-06-08T05:50:01Z

gtsfm/scene_optimizer.py

-
-def save_gtsfm_data(image_graph: List[Delayed], ba_input_graph: Delayed, ba_output_graph: Delayed) -> List[Delayed]:
-    """Saves the Gtsfm data before and after bundle adjustment.
+    aligned_ba_input = ba_input.align_via_Sim3_to_poses(gt_poses)


is this duplicated when we compute metrics somewhere?

Removed it. Thanks

Its aligned in a different function. We do align it in the BundleAdjustmentOptimizer::evaluate, but that aligned GtsfmData objects are not returned.

was this the reason why we were aligning ba_input in the other TODO comment? Its pretty expensive so we should only do it once I think

gtsfm/utils/metrics.py

travisdriver · 2022-06-08T12:55:58Z

Is there any way to break this PR up into smaller ones?

dellaert · 2022-06-08T15:30:47Z

Is there any way to break this PR up into smaller ones?

Yes, obviously there is, but we discussed and decided to do it this way.

akshay-krishnan

nice work! we're getting closer, I'm not going to add new comments, some of my old ones are not yet resolved.

gtsfm/averaging/rotation/rotation_averaging_base.py

akshay-krishnan · 2022-06-14T00:41:28Z

gtsfm/averaging/rotation/shonan.py

    ) -> BetweenFactorPose3s:
        """Create between factors from the priors on relative poses."""
        between_factors = BetweenFactorPose3s()
+        noise_model = gtsam.noiseModel.Isotropic.Sigma(POSE3_DOF, PRIOR_SIGMA)


Is this Shonan specific? Were you getting good results in BA or TA?

If all of them need hardcoding, we can do that in the loader. One of them needing it is weird, maybe the other hardcoded values can be changed?

gtsfm/averaging/rotation/shonan.py

akshay-krishnan · 2022-06-18T05:32:18Z

gtsfm/evaluation/eval_hilti.py

+
+# Modified for using both fastlio and gtsfm TUM files
+# If you already have matplotlib, scipy and numpy, this will need an additional
+# pip install evo==1.13.5


I am not sure. I originally put in it in the evaluation directory above gtsfm/ directory, which is currently empty, because it is not part of gtsfm.

Or we could also move it to 3rdparty.

gtsfm/frontend/detector_descriptor/superpoint.py

gtsfm/loader/hilti_loader.py

gtsfm/multi_view_optimizer.py

akshay-krishnan · 2022-06-18T05:41:28Z

gtsfm/multi_view_optimizer.py

+        # TODO(Frank): Why would we do this here? But maybe we should simply fix align_via_Sim3_to_poses
+        # ba_input_graph = dask.delayed(ba_input_graph.align_via_Sim3_to_poses)(gt_wTi_list)


could you check who consumes ba_input? and whether they expect it to be aligned to GT? is the output aligned to GT?

akshay-krishnan · 2022-06-18T05:43:10Z

gtsfm/scene_optimizer.py

+REACT_METRICS_PATH = BASE_PATH / "rtf_vis_tool" / "src" / "result_metrics"
+REACT_RESULTS_PATH = BASE_PATH / "rtf_vis_tool" / "public" / "results"


Yeah, but isnt it the same data that we save in both results_path and react_metrics_path ? Do we need this duplication?

akshay-krishnan · 2022-06-18T05:45:18Z

gtsfm/scene_optimizer.py

-
-def save_gtsfm_data(image_graph: List[Delayed], ba_input_graph: Delayed, ba_output_graph: Delayed) -> List[Delayed]:
-    """Saves the Gtsfm data before and after bundle adjustment.
+    aligned_ba_input = ba_input.align_via_Sim3_to_poses(gt_poses)


was this the reason why we were aligning ba_input in the other TODO comment? Its pretty expensive so we should only do it once I think

akshay-krishnan · 2022-06-18T19:00:28Z

@ayushbaid we can do this in another PR if we want to submit this one soon: we can remove the "augmentation" logic in TA after we have a release with borglab/gtsam#1224.

akshay-krishnan

Looks like all comments have been resolved. Just need to fix the CI now.

akshay-krishnan · 2022-06-27T07:33:23Z

gtsfm/averaging/rotation/shonan.py

    ) -> BetweenFactorPose3s:
        """Create between factors from the priors on relative poses."""
        between_factors = BetweenFactorPose3s()
+        noise_model = gtsam.noiseModel.Isotropic.Sigma(POSE3_DOF, PRIOR_SIGMA)


This is a pending comment from an old review, please ignore if fixed:

I remember Frank suggested using an isotropic noise model with a Sigma which is an average of the diagonal elements. Not sure how that works.

akshay-krishnan · 2022-07-12T00:44:01Z

gtsfm/evaluation/eval_hilti.py

+
+# Modified for using both fastlio and gtsfm TUM files
+# If you already have matplotlib, scipy and numpy, this will need an additional
+# pip install evo==1.13.5


if we are not going to use this often, no need to depend on evo. Feel free to mark this resolved.

dellaert · 2022-07-13T13:21:33Z

Still 2 CI fails?

johnwlambert · 2022-08-09T04:21:46Z

tests/loader/test_hilti_loader.py

+
+def read_fastlio_result(poses_txt_path: str) -> Tuple[List[Pose3], np.ndarray]:
+    """
+    Read FastLIO result


@ayushbaid can we update this docstring to the style guide when you get a chance?

johnwlambert · 2022-08-09T04:22:04Z

tests/loader/test_hilti_loader.py

+    return Pose3(pose_rotation, pose_translation)
+
+
+def create_initial_estimate(poses: List[Pose3]):


Can we add the return type hint here?

johnwlambert · 2022-08-09T04:22:08Z

tests/loader/test_hilti_loader.py

+EXP07_PATH = DATA_ROOT_PATH / "exp07"
+
+
+def pose_of_vector(pose_vector):


Can we add the return type hint here?

johnwlambert · 2022-08-09T04:23:01Z

gtsfm/runner/run_full_hilti_nodask.py

+        Dict[Tuple[int, int], Optional[Unit3]],
+        Dict[Tuple[int, int], np.ndarray],
+        Dict[int, Tuple[int, int]],
+    ]:


@ayushbaid do you mind adding the return type hint here?

johnwlambert · 2022-08-19T04:57:13Z

gtsfm/frontend/matcher/superglue_matcher.py

+            use_cuda (optional): Use CUDA (GPU) for inference. Defaults to True.
+            use_outdoor_model (optional): Use the outdoor weights for SuperGlue network. Defaults to True.
+            init_model_in_constructor (optional): Initialize the model with weights in the constructor instead of the
+                                                  `match` function. This setting is recommended when we are not using


I think the formatting got messed up here @ayushbaid ?

ayushbaid and others added 30 commits May 12, 2022 22:38

Changing image name format

10aa568

Fixing stack of delayed calls inside two-view-estimator

5ad2ebc

Adding script to run hilti

62251ae

Killing 2view reports

282049f

Setting higher memory limit to prevent crashes

0b1935c

fixing types

b6e6255

Using for-loop based run to cache frontend

fab40c0

cleaning up unused client

ed95438

moving model loading to init to play well with the for loop

1b3d369

Adding subsampling in hilti loader

28ef8d5

Merge branch 'feature/hilti-subsample' into hkust-cluster

f8c5665

reverting changes to deep model loading

b4a9e4b

revert to using dask

26904b1

enhancing run script

2c29c3b

fixing 2-view estimator duplicate function and cleaning up configs

8787689

changing params and adding subsampling

0591611

Adding cacher for two-view-estimator plate

abeb231

using 2-view-estimator cache

9fa1f41

Allow using old-style image numbering

d0b65aa

"fix" config to not do view_graph

a7df688

Add more logging and rename run functions

44df604

flake issues

8fde20a

Rename run -> run_ba

867984b

Add more logging

4a84c4e

Rename more run methods

237a39e

Option to turn off MFAS

432d9de

Re-center to trajectory mean

f5f5662

Allow centering on trajectory

d4d3309

More logging and run renaming

2736efc

Added more logging

b1c944d

ayushbaid requested review from dellaert and akshay-krishnan June 7, 2022 06:41

dellaert reviewed Jun 7, 2022

View reviewed changes

akshay-krishnan reviewed Jun 8, 2022

View reviewed changes

ayushbaid added 2 commits June 12, 2022 15:54

addressing review comments

c35ab71

adding back code to align visualizations

f02fd9f

ayushbaid requested review from dellaert and akshay-krishnan June 16, 2022 04:23

akshay-krishnan reviewed Jun 18, 2022

View reviewed changes

addressing comments

df2b398

akshay-krishnan approved these changes Jul 12, 2022

View reviewed changes

ayushbaid added 3 commits July 11, 2022 23:09

removing old attribute

1b7ea0d

removing hilti trajectory eval script

8af8da7

fixing astronet runner

9cfdb92

ayushbaid added 4 commits July 19, 2022 23:10

Addressing comments

39c384c

Merge branch 'master' into hkust-cluster

ece601a

adressing comments

4468b48

Fixing logging

237047a

johnwlambert reviewed Aug 9, 2022

View reviewed changes

johnwlambert reviewed Aug 19, 2022

View reviewed changes

ayushbaid mentioned this pull request Sep 19, 2022

Adding Shonan averaging related changes from the hilti branch #567

Merged

johnwlambert mentioned this pull request Sep 19, 2022

Update TranslationAveraging to return Pose(3), not Point(3) #568

Merged

ayushbaid closed this Mar 12, 2024

		REACT_METRICS_PATH = BASE_PATH / "rtf_vis_tool" / "src" / "result_metrics"
		REACT_RESULTS_PATH = BASE_PATH / "rtf_vis_tool" / "public" / "results"

		# TODO(Frank): Why would we do this here? But maybe we should simply fix align_via_Sim3_to_poses
		# ba_input_graph = dask.delayed(ba_input_graph.align_via_Sim3_to_poses)(gt_wTi_list)

		return Pose3(pose_rotation, pose_translation)


		def create_initial_estimate(poses: List[Pose3]):

		EXP07_PATH = DATA_ROOT_PATH / "exp07"


		def pose_of_vector(pose_vector):

Merging changes from the hilti challenge #515

Merging changes from the hilti challenge #515

Conversation

ayushbaid commented Jun 7, 2022

dellaert left a comment

Choose a reason for hiding this comment

akshay-krishnan left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

akshay-krishnan Jun 27, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

travisdriver commented Jun 8, 2022

dellaert commented Jun 8, 2022

akshay-krishnan left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

akshay-krishnan commented Jun 18, 2022

akshay-krishnan left a comment

Choose a reason for hiding this comment

akshay-krishnan Jun 27, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dellaert commented Jul 13, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

akshay-krishnan Jun 27, 2022 •

edited

Loading

akshay-krishnan Jun 27, 2022 •

edited

Loading