End-to-End Implementation of Object Detection Pipeline in Livepeer AI Network [$2500]

[JJassonn69]

Overview

We are excited to expand the capabilities of the Livepeer AI Network by developing a robust object detection pipeline with multi-use applications. This project will enable near real-time tracking of objects, like a ball in live sports footage to power applications in sports analytics, security surveillance, and content moderation. By providing high-speed, accurate tracking, this solution will open up new possibilities for enhanced real-time insights across multiple industries.🏅

We are seeking community support to implement this pipeline within the Livepeer AI network. The solution will leverage model PekingU/rtdetr_r50vd for precise object tracking. By contributing to this pipeline, you will help expand our detection and analytics offerings, opening new possibilities for real-time data! 🚀

Required Skillset

• Proficiency in FastAPI.
• Strong experience in Python.
• Familiarity with Docker and Livepeer AI Network.
• Basic understanding of Go programming and Hugging Face models is advantageous.

Bounty Requirements

1. Implementation: Create a functional /object-detection route and pipeline within the AI-worker repository. This new pipeline should be accessible through Docker on port 8900. Also, develop the necessary code within the go-livepeer repository to integrate access to the object-detection pipeline from the ai-worker component. This includes implementing the payment logic and job routing to ensure seamless operation within the network.

2. Functionality: The pipeline must accept a video file or possibly a stream to output the processed video file with labels for the detected objects and confidence scores for them per frame. It should also include the necessary post processing steps for the video to display the labels for objects in the video. Also, ensure that users can submit AI job requests to the network in a manner consistent with other AI-Network features.

Example request:
curl -X POST "https://your-gateway-url/object-detection -F pipeline="object-detection” -F model_id="" -F [email protected]

Scope Exclusions

• None

Implementation Tips

Getting started with initial pipeline structure, refer to the HuggingFace space. You can also explore the following pull requests to see how other video and image handling pipelines were implemented:

• PR #130
• PR #223

In some cases, you might encounter dependencies conflicts and not be able to integrate the new pipeline directly into the regular AI Runner. If this occurs, you can follow the approach outlined in SAM2 PR to create a custom container for the pipeline. This approach uses the regular AI runner as the base while keeping the base container lean.

To streamline development, keep these best practices in mind:

• Use Developer Documentation: Leverage developer documentation for the worker and runner that provides tips for mocking pipelines and direct debugging, which can streamline the development process. Similarly, developer documentation for the installation of go-livepeer and the general Livepeer documentation for example usage and setup instructions offer valuable insights that can expedite your development process, including automatic scripts for orchestrators and gateways.
• Update OpenAPI Specification: Execute the runner/gen_openapi.py script to generate an updated OpenAPI specification.
• Generate Go-Livepeer Bindings: In the main repository directory, execute the make command to generate the necessary bindings, ensuring compatibility with the go-livepeer repository.
• Build Binaries: Run the make command in the main repository folder to generate Livepeer binaries. This will allow you to test your implementation and ensure it integrates smoothly.
• Create Docker Images: Build Docker images of Livepeer and test them using appropriate tools and settings to identify any edge cases or bugs. This step is crucial for ensuring robustness and reliability in your implementation.

How to Apply

1. Express Interest: Comment on this issue with a brief explanation of your experience and suitability for this task.
2. Await Review: Our team will review the applications and select a qualified candidate.
3. Get Assigned: If selected, the GitHub issue will be assigned to you.
4. Start Working: Begin the task! For questions or support, comment on the issue or join discussions in the #developer-lounge channel on our Discord server.
5. Submit Your Work: Create a pull request in the relevant repository and request a review.
6. Notify Us: Comment on this GitHub issue once your pull request is ready for review.
7. Receive Your Bounty: Upon pull request approval, we will arrange the bounty payment.
8. Earn Recognition: Your contribution will be highlighted in our project’s changelog.

We look forward to your interest and contributions to this exciting project! 💛

Warning

Please ensure the issue is assigned to you before starting work. To avoid duplication of efforts, unassigned issue submissions will not be accepted.

[RUFFY-369]

I would really love to take this one on 😉 @rickstaa @JJassonn69

[rickstaa]

I would really love to take this one on 😉 @rickstaa @JJassonn69

Great! I assigned you to this issue 🚀.

[yadavshubham01]

Hello @rickstaa @JJassonn69 ,

I'm interested in working on the object detection pipeline for the Livepeer AI Network. I have a strong background in developing high-performance APIs and data processing pipelines, with expertise in Python, FastAPI, and Docker. Additionally, I have experience integrating machine learning models for real-time applications, making me well-suited to implement the requested object detection functionality.

Here's a summary of my relevant experience:

1. Proficiency in FastAPI and Python: I’ve developed multiple scalable API services using FastAPI, focusing on efficient request handling and minimal latency.
2. Experience with Docker: I’m skilled in Dockerizing applications, ensuring compatibility across environments, and addressing dependency conflicts, which will help in creating a robust container for the object detection pipeline.
3. Familiarity with Livepeer AI and Hugging Face Models: I have a working knowledge of the Livepeer AI Network’s setup and the Hugging Face library, which will streamline the integration of PekingU/rtdetr_r50vd for object tracking.
4. Basic Understanding of Go: Although my primary expertise is in Python, I’m comfortable working with Go, allowing me to navigate the go-livepeer codebase for integration and routing updates.

I’m confident that my skills and experience align well with the project requirements, and I’m excited to contribute to expanding Livepeer's object detection and tracking capabilities. Looking forward to the opportunity!

Best regards,
Shubham Yadav

[c0d33ngr]

Hello @rickstaa @JJassonn69 I was browsing GitHub for bounties to work on and came across this repo. Are there timeframe once someone is assigned to an issue or it's just unlimited time?

[JJassonn69]

Hei, @c0d33ngr unfortunately its not available as it has already been assigned to @RUFFY-369 . You can check the for future bounties that will surely be posted here. Thank you.