Outpainting Pipeline Implementation on ai-runner Bounty [$850]

[JJassonn69]

Overview

We are excited to announce the expansion of our AI Network with the introduction of Outpainting pipeline. This feature will enable advanced image expansion and modification techniques, improving the versatility of our AI network. 🔥

We are reaching out to the community to contribute to the implementation of this this essential pipeline within the AI-worker side of the AI-Network. Your contribution will be an invaluable addition to our image editing capabilites, pushing Livepeer AI Network at the forefront of AI-media generation. 🚀

Required Skillset

• Proficiency with FastAPI.
• Strong skills in Python.
• Familiarity with Docker and Hugging Face 🤗.

Bounty Requirements

1. Implementation: Develop a functional /out-paint route and pipeline within the AI-worker repository. This new pipeline should be accessible through Docker on port 8700.
2. Functionality: The pipeline must be able to take an initial image, and text input as a prompt, return an modified image as the result. It should also allow for optional parameters like negative_prompt, controlnet_conditioning_scale, num_inference_steps, and guidance_scale to be adjustable according to the outpainting model utilized.

Example workflow

Originally posted by @stronk-dev in #56 (comment)

Scope Exclusions

• This bounty does NOT include the complete end-to-end implementation of this pipeline on the go-livepeer side, such as payment processing or job routing. These elements will be handled by the AI SPE team or in a future bounty.

Implementation Tips

To learn how to create a new pipeline, refer to the Hugging Face pipeline documentation. You can also explore the following pull requests to see how other pipelines were implemented:

• Pull Request #96
• Pull Request #103

In some cases, you may not be able to integrate the new pipeline directly into the regular AI Runner Docker container due to dependency conflicts or missing packages. If this occurs, you can follow the approach outlined in this example to create a custom container for the pipeline. This approach uses the regular AI runner as the base while keeping the base container lean.

Additionally, keep the following best practices in mind:

• Leverage Previous Work: Examine existing implementations of outpainting available in Hugging Face documentation and this space. These can serve as valuable starting points.
• Consult Developer Documentation: Our developer documentation for the worker and runner includes helpful tips for mocking pipelines and direct debugging, speeding up the development process.
• Update OpenAPI Specification: Run the runner/gen_openapi.py script to generate an updated OpenAPI specification.
• Generate Go-Livepeer Bindings: In the main repository directory, execute the make command to generate the necessary bindings, ensuring compatibility with the go-livepeer repository.

How to Apply

1. Express Interest: Comment on this issue to show your interest and explain why you are a great fit for this task.
2. Await Review: Our team will review expressions of interest and select the most suitable candidate.
3. Get Assigned: If chosen, the GitHub issue will be assigned to you.
4. Start Working: Begin the task! If you need help or guidance, comment on the issue or join discussions in the #developer-lounge channel on our Discord server.
5. Submit Your Work: Create a pull request in the relevant repository and request a review.
6. Notify Us: Comment on this GitHub issue when your pull request is ready for review.
7. Receive Your Bounty: Once your pull request is approved, we will arrange the bounty payment.
8. Earn Recognition: Your valuable contributions will be highlighted in our project's changelog.

We appreciate your interest in contributing to our project! 💛

Warning

Please wait for the issue to be assigned to you before starting work. To prevent duplication of effort, submissions for unassigned issues will not be accepted.

[stronk-dev]

Not sure if this is a requirement for good outpainting results, but just wanted to drop this image from Reddit here for the person that will be working on this bounty. Apparently generating a mosaic over the area to be repainted helps?

[rickstaa]

Not sure if this is a requirement for good outpainting results, but just wanted to drop this image from Reddit here for the person that will be working on this bounty. Apparently generating a mosaic over the area to be repainted helps?

@stronk-dev damn that image is great! I added it as an example workflow to the bounty description. Thanks for jumping in 🙏🏻❤️.

[lukiod]

will work on it after finishing the stable-audio pipeline work if someone took on in between no problem will require a time period of a week to finish the previous bounty

[aybanda]

@JJassonn69 can you assign it to me?

[rickstaa]

Hi @aybanda, thanks for your patience! We've been wrapped up with the public launch at Token2049. It’s fantastic to have you contributing to our open-source project 🚀. I’ve assigned you to the bounty now. Don't hesitate to reach out if you have any questions!

[aybanda]

@rickstaa will do my best in delivering this 👍
Thank you

[rickstaa]

Hi @aybanda, how's the progress on your bounty implementation? Is there anything we can do to assist?

Also, I came across some exciting news— the Transformers library will add support for a new model called ProPainter, which excels at both inpainting and outpainting. You can check it out here: ProPainter on GitHub.

Let me know if this might be helpful for your work!

[aybanda]

Hey @rickstaa
Thanks for checking in. I'm making good progress on the bounty implementation, will give you an update very shortly.

[aybanda]

Hey @rickstaa

I'm almost ready with the code for the image_outpainting pipeline using the ProPainter model. Here's a summary of my current implementation:

I'm using the "ruffy369/propainter" model from Hugging Face.
and handling image resizing and mask creation for outpainting.

I have a few questions:

1. We're currently using AutoPipelineForInpainting for both inpainting and outpainting tasks, manipulating the input image and mask to achieve outpainting. Is this the most effective approach with ProPainter, or do you recommend any adjustments to better handle outpainting specifically?

2. Do you have any additional recommendations or best practices for integrating ProPainter into our pipeline, especially for outpainting tasks?

Any insights or suggestions would be greatly appreciated as I prepare to push this implementation. Thank you!