feat(deepen): Add initial lidarseg support #191

KSeangTan · 2025-01-16T10:49:33Z

Description

This PR adds features to support 3d painting (lidar segmantic segmentation) in t4 dataset, mainly, it adds:

Download lidar seg from deepen
- perception_dataset/deepen/download_annotations.py
convert the dowloaded annotations to t4 dataset in annotations/lidarseg.json and lidarseg/annotation/****.bin
- perception_dataset/t4_dataset/classes/lidarseg.py
- perception_dataset/deepen/deepen_to_t4_converter.py
- perception_dataset/deepen/segmentation/painting3d.py
- perception_dataset/t4_dataset/classes/category.py
To support lidarseg, it also adds a new attribute index to CategoryTable
- perception_dataset/t4_dataset/classes/category.py

How to test

test data

Update params in convert_deepen_to_t4_lidarseg_3d_painting_sample.yaml:
- dataset_corresponding: with correct dataset name and deepen id
Download annotations.json

poetry run python perception_dataset/deepen/download_annotations.py --config config/convert_deepen_to_t4_lidarseg_3d_painting_sample.yaml --output_dir ../lidar_semseg_sample/ --label_type paint3d

Test json data from deepen should be in the following format:

[
        {
            "dataset_id": "dummy_dataset_id",
            "file_id": "0.pcd",
            "label_type": "3d_point",
            "label_id": "none",		# Keep it for consistency with downstream tasks
            "label_category_id": "none",	# Keep it for consistency with downstream tasks
            "total_lidar_points": 173430,
            "sensor_id": "lidar",
            "stage_id": "QA",
            "paint_categories": ["car", "wall", ...],
            "lidarseg_anno_file": "lidar_seg/dummy_dataset__id_0.pcd.bin"
        },
        ...
    ]

test command

Update params in convert_deepen_to_t4_lidarseg_3d_painting_sample.yaml:
- input_anno_file
Convert annotations.json to t4 dataset

poetry run python perception_dataset/convert.py --config config/convert_deepen_to_t4_lidarseg_3d_painting_sample.yaml

Unit tests

poetry run pytest tests/deepen/test_lidarseg_deepen_to_t4_converter.py

poetry run pytest tests/t4_dataset/test_lidarseg_annotation_files_generator.py

poetry run pytest tests/t4_dataset/classes/test_category.py

poetry run pytest tests/t4_dataset/classes/test_lidarseg.py

ktro2828 · 2025-01-28T04:51:58Z

perception_dataset/deepen/download_annotations.py

+    parser.add_argument(
+        "--label_type",
+        type=str,
+        default="labels",
+        help="Annotation type to request labels, supported values: ['labels', 'paint3d']",
+    )


[NITS] choices makes clear options that we can choose.

Suggested change

parser.add_argument(

"--label_type",

type=str,

default="labels",

help="Annotation type to request labels, supported values: ['labels', 'paint3d']",

)

parser.add_argument(

"--label_type",

type=str,

default="labels",

choices=["labels", "paint3d"],

help="Annotation type to request labels, supported values: ['labels', 'paint3d']",

)

Yeah, it looks better
updated

ktro2828 · 2025-01-28T04:55:15Z

perception_dataset/deepen/segmentation/painting3d.py

+    paint_categories: List[str] = field(default_factory=list)
+    lidarseg_anno_file: str = ""


If we doesn't allow empty list and string, we shouldn't set their default values.

Suggested change

paint_categories: List[str] = field(default_factory=list)

lidarseg_anno_file: str = ""

paint_categories: List[str]

lidarseg_anno_file: str

Make sense, updated

ktro2828 · 2025-01-28T05:02:15Z

docs/t4_format_3d_detailed.md

@@ -326,6 +326,7 @@ The item "description" for the category is not implemented for now.
  - "token": [str] -- Unique record identifier.
  - "name": [str] -- Category name. The latest format is "class" (e.g. car, truck), but "category.class" format (e.g. vehicle.car) is also supported.
  - "description": [str] -- Category description. Empty string `""` for now. **(Not available)**
+  - "index": [int] -- Category index, this is added to support lidarseg.


[Q] Should we also allow None for the case category.json doesn't contain the index field, or set it 0 by default?

Let's user None instead of 0 since 0 means non-painted pointclouds in deepen by default

kminoda · 2025-01-29T23:39:47Z

@KSeangTan Would you also update tools_overview.md and t4_format_3d_detailed.md? For instance, we would need to modify the directory structure in the latter document.

kminoda · 2025-01-29T23:59:40Z

perception_dataset/deepen/deepen_to_t4_converter.py

+    def convert(self):
+        """Convert all scene annotations to T4 dataset format."""
+        if self._label_info is not None and self._label_info.label_type == LabelType.POINT_3D:
+            scenes_anno_dict = self._format_point_3d_annotations()
+        else:
+            scenes_anno_dict = self._format_annotations()


Do you think this modification has a room of improvement?
I was wondering if we can avoid this increasing complexity of the code with nested if-else among convert(), _format_point_3d_annotations(), and _format_annotations. I thought that the simplest way is to reduce the nest e.g. as

if self._label_info is None: ... elif self._label_info.label_type == LabelType.SEGMENTATION_2D: ... elif self._label_info.label_type == LabelType.POINT_3D: ... else: ...

But any reason not doing so?

I was thinking to remove the nested if-else level in this part by separating two label types here.
However, you are right, I have reformatted this part by introducing
_format_annotations instead

kminoda · 2025-01-30T00:10:51Z

perception_dataset/t4_dataset/annotation_files_generator.py

+                scene_anno_dict (Dict[int, List[Dict[str, Any]]]): [description]
+                frame_index_to_sample_token (Dict[int, str]): [description]
+                frame_index_to_sample_data_token (Dict[int, str]):

        scene_anno_dict:
        {
-            0: [
-                {
-                    "category_name" (str): category name of object,
-                    "instance_id" (str): instance id of object,
-                    "attribute_names" (List[str]): list of object attributes,
-                    "three_d_bbox": {
-                        "translation": {
-                            "x" (float): x of object location,
-                            "y" (float): y of object location,
-                            "z" (float): z of object location,
+                0: [
+                        {
+                                "category_name" (str): category name of object,
+                                "instance_id" (str): instance id of object,
+                                "attribute_names" (List[str]): list of object attributes,
+                                "three_d_bbox": {
+                                        "translation": {
+                                                "x" (float): x of object location,
+                                                "y" (float): y of object location,
+                                                "z" (float): z of object location,
+                                        },
+                                        "velocity" (Optional[Dict[str, float]]): {
+                                                "x" (float): x of object velocity,
+                                                "y" (float): y of object velocity,
+                                                "z" (float): z of object velocity,
+                                        },
+                                        "acceleration" (Optional[Dict[str, float]]): {
+                                                "x" (float): x of object acceleration,
+                                                "y" (float): y of object acceleration,
+                                                "z" (float): z of object acceleration,
+                                        },
+                                        "size": {
+                                                "width" (float): width of object size,
+                                                "length" (float): length of object size,
+                                                "height" (float): height of object size,
+                                        },
+                                        "rotation": {
+                                                "w" (float): w of object quaternion,
+                                                "x" (float): x of object quaternion,
+                                                "y" (float): y of object quaternion.
+                                                "z" (float): z of object quaternion,
+                                        },
+                                },
+                                "two_d_box": [
+                                        "x" (float): x of left top corner,
+                                        "y" (float): y of left top corner,
+                                        "w" (float): width of bbox,
+                                        "h" (float): height of bbox,
+                                ]
+                                "sensor_id": id of the camera
+                                "num_lidar_pts" (int): the number of lidar points in object,
+                                "num_radar_pts" (int): the number of radar points in object,
                        },
-                        "velocity" (Optional[Dict[str, float]]): {
-                            "x" (float): x of object velocity,
-                            "y" (float): y of object velocity,
-                            "z" (float): z of object velocity,
-                        },
-                        "acceleration" (Optional[Dict[str, float]]): {
-                            "x" (float): x of object acceleration,
-                            "y" (float): y of object acceleration,
-                            "z" (float): z of object acceleration,
-                        },
-                        "size": {
-                            "width" (float): width of object size,
-                            "length" (float): length of object size,
-                            "height" (float): height of object size,
-                        },
-                        "rotation": {
-                            "w" (float): w of object quaternion,
-                            "x" (float): x of object quaternion,
-                            "y" (float): y of object quaternion.
-                            "z" (float): z of object quaternion,
-                        },
-                    },
-                    "two_d_box": [
-                        "x" (float): x of left top corner,
-                        "y" (float): y of left top corner,
-                        "w" (float): width of bbox,
-                        "h" (float): height of bbox,
-                    ]
-                    "sensor_id": id of the camera
-                    "num_lidar_pts" (int): the number of lidar points in object,
-                    "num_radar_pts" (int): the number of radar points in object,
-                },
-                ...
-                ],
-            1: []. ...
+                        ...
+                        ],
+                1: []. ...


Seems that this is an unnecessary change.

Not sure why, but it's modified by my pre-commit automatically
I have reverted the changes

KSeangTan · 2025-01-31T09:14:28Z

@KSeangTan Would you also update tools_overview.md and t4_format_3d_detailed.md? For instance, we would need to modify the directory structure in the latter document.

Yeah, I have updated the docs about lidarseg, would you mind taking a look again? Thanks

KSeangTan · 2025-01-31T09:32:50Z

I got the following error in the CI:

Error: This request has been automatically failed because it uses a deprecated version of `actions/upload-artifact: v3`. Learn more: https://github.blog/changelog/2024-04-16-deprecation-notice-v3-of-the-artifact-actions/

Do you know any fix? @ktro2828 @kminoda

Add initial lidarseg support

563ee8e

KSeangTan self-assigned this Jan 16, 2025

kokseang added 18 commits January 17, 2025 11:50

Fix fronzen dataclass inheritance

f3e4c38

Move lidarseg to anno_dir

ba201e3

Move lidarseg to anno_dir

1f9e1fe

Move lidarseg to anno_dir

c150ba4

Fix formatting

9e8eee8

Fix formatting

5707312

Fix formatting

0ca57af

Fix conversion bug

3b30746

Fix format

10339c5

Fix format

31e43a3

Fix format

c46509e

Fix format

f09a0be

Fix category table unit test

01939ff

Fix category unit test

8ce0543

Added unittest for lidarseg conversion

2b875f2

Added unittest

c672402

Added unittest

eed7428

add test_lidarseg.py

b64fd08

KSeangTan requested review from kminoda and ktro2828 January 27, 2025 07:29

KSeangTan changed the title ~~WIP: feat: Add initial lidarseg support~~ feat: Add initial lidarseg support Jan 27, 2025

KSeangTan changed the title ~~feat: Add initial lidarseg support~~ feat(deepen): Add initial lidarseg support Jan 27, 2025

kokseang added 2 commits January 27, 2025 16:45

Update readme

2037dee

Update readme

e52bed8

ktro2828 reviewed Jan 28, 2025

View reviewed changes

kminoda reviewed Jan 30, 2025

View reviewed changes

kokseang added 8 commits January 31, 2025 16:37

Resolve comments

3709e3b

Resolve comments

1b525ef

Resolve comments

eee1c4a

Resolve comments

e3f8d1f

Resolve comments

13e9256

Resolve comments

bc9e168

Merge branch 'main' into feat/add_lidarseg_annotations

5f32f21

Update tools_overview.md

6cd5eb7

KSeangTan closed this Jan 31, 2025

KSeangTan reopened this Jan 31, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(deepen): Add initial lidarseg support #191

feat(deepen): Add initial lidarseg support #191

KSeangTan commented Jan 16, 2025 •

edited

Loading

ktro2828 Jan 28, 2025

KSeangTan Jan 31, 2025

ktro2828 Jan 28, 2025

KSeangTan Jan 31, 2025

ktro2828 Jan 28, 2025

KSeangTan Jan 31, 2025

kminoda commented Jan 29, 2025 •

edited

Loading

kminoda Jan 29, 2025

KSeangTan Jan 31, 2025 •

edited

Loading

kminoda Jan 30, 2025

KSeangTan Jan 31, 2025

KSeangTan commented Jan 31, 2025

KSeangTan commented Jan 31, 2025

		paint_categories: List[str] = field(default_factory=list)
		lidarseg_anno_file: str = ""

feat(deepen): Add initial lidarseg support #191

Are you sure you want to change the base?

feat(deepen): Add initial lidarseg support #191

Conversation

KSeangTan commented Jan 16, 2025 • edited Loading

Description

How to test

test data

test command

Unit tests

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kminoda commented Jan 29, 2025 • edited Loading

Choose a reason for hiding this comment

KSeangTan Jan 31, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

KSeangTan commented Jan 31, 2025

KSeangTan commented Jan 31, 2025

KSeangTan commented Jan 16, 2025 •

edited

Loading

kminoda commented Jan 29, 2025 •

edited

Loading

KSeangTan Jan 31, 2025 •

edited

Loading