Real-Time Inference

[Henistein]

I noticed that in all demo scripts the video is first processed in whole and then propagated, would it be possible to run a video in real-time?

Thank you in advance!

[rentainhe]

Thanks for you issue, I think you can try to use smaller SAM 2 model for faster inference, and maybe you can refer to SAM 2's official repo to see if there're some acceleration methods on this issue

[Henistein]

@rentainhe sorry, maybe I expressed myself badly, what I meant was if would be possible to run grounded sam2 end to end, like in a stream. Imagine I have a real time video stream and I wanted to segment it live while keeping the track of each object, would it be possible?

[rentainhe]

I think SAM 2 now only supports non-stream video input. I have no idea how to support streaming input at this time. Maybe we can refer to the community to see if there are some solutions for this issue.

[patrick-tssn]

@Henistein, I hope my implementation will be helpful, though these is latency.
https://github.com/patrick-tssn/Streaming-Grounded-SAM-2

[Masrur02]

Hi, @Henistein
Can your code take prompts like the grounded sam2? I mean I want to track road amd cars in a video with continuous id? Is it possible by your implementation??

TIA