Support for YOLO-Worldv2 models alongside currently supported YOLO-like ONNX model files

[TimC2225]

Preliminary Checks

• This issue is not a duplicate. Before opening a new issue, please search existing issues.
• This issue is not a question, bug report, or anything other than a feature request directly related to this project.

Proposal

Support for YOLO-Worldv2 models could be added to the "Custom Object Detection with YOLO-like ONNX model file" feature. I have tried using persisting models saved using the following method provided by Ultralytics, but it fails to detect any objects when launching display_zed_cam.launch.py. The PT file saved was exported as ONNX through the CLI as given in the README.

from ultralytics import YOLO

# Initialize a YOLO-World model
model = YOLO("yolov8s-worldv2.pt")  # or select yolov8m/l-worldv2.pt

# Define custom classes
model.set_classes(["person", "bus"])

# Save the model with the defined offline vocabulary
model.save("custom_yolov8s.pt")

Ultralytics' documentation for YOLO-World states that the models saved using the above method behave like any other pre-trained YOLOv8 model, so I am unsure if the saved persisting YOLO-Worldv2 models should already work with the ZED ROS2 wrapper once exported as ONNX.

Use-Case

This would allow persisting YOLO-World models to display objects detected through the ROS 2 plugin used to visualize the results of the Object Detection, hopefully making zero-shot detection with dynamic custom classes possible with the ZED stereo cameras.

Anything else?

No response

[Myzhar]

Hi @TimC2225
can you try if your model works using native ZED SDK?
You can use these examples: https://github.com/stereolabs/zed-sdk/tree/master/object%20detection/custom%20detector

[TimC2225]

From my previous experience with the detector.py example, the YOLO-Worldv2 models work using the native ZED SDK.

[TimC2225]

Could it be possible that the YOLO-Worldv2 models may be failing to work due to the optimizations performed for the graphics card? Based on the output of the logger, I have noticed that the detector.py example does not appear to put the models through any optimization before use (though this may be because .pt files are used for the models in detector.py rather than .onnx files).

.onnx model files also work somewhat with detector.py when used as the argument for weights, but stop the program from being killed correctly when trying to end the program with ctrl+c.

[Myzhar]

The ZED SDK offers two methods for supporting external detections.

1. direct inference by using ONNX file
2. external inference and custom bounding box injection

The ZED ROS 2 Wrapper only accommodates the first method, as the second would necessitate an external node to perform inference, leading to increased latency and delay.

Please test your YOLO-World v2 models with ONNX and the ZED SDK native code.

[adujardin]

@TimC2225 Exporting the YOLO-Worldv2 model will produce a compatible ONNX model with the ZED SDK. It works with both the internal ONNX mode or the standard sample with TensorRT inference

I just tested it using this snippet

# Initialize a YOLO-World model
model = YOLO("yolov8s-worldv2.pt")  # or select yolov8m/l-world.pt

# Define custom classes
model.set_classes(["person", "bus"])

# Save the model with the defined offline vocabulary
model.save("custom_yolov8s2.pt")

model.export(format="onnx")

# Load into the ZED SDK the resulting "yolov8s-worldv2.onnx" file

[TimC2225]

@adujardin I have just tried using the code shown above to export a custom model with the same custom classes (person and bus), which was then run with the cpp custom detectors. However, no bounding boxes around me or any person in frame appear for both the onnx and onnx internal samples.

I have realised that I am one version down for my ZED SDK however, with it being v4.2.1 instead of the latest v4.2.2. I will update the SDK and see if it fixes the issue.

[TimC2225]

I have updated the ZED SDK by running the installer for v4.2.2. However, I still do not get any output when running either the tensorrt_yolov5-v6-v8_onnx or tensorrt_yolov5-v6-v8_onnx_internal samples with the exported yolov8s-worldv2.onnx file.

However, I am able to see both the 2D and 3D bounding boxes when running the same samples with the standard yolov8m.onnx model and a custom trained best.onnx model downloaded after running Ultralytics' Colab notebook: https://colab.research.google.com/github/roboflow-ai/notebooks/blob/main/notebooks/train-yolov8-object-detection-on-custom-dataset.ipynb