Integrating of MediaPipe with OpenVINO [GSOC]

[Ashwin4514]

Hi There OpenVino Developers,

I am Ashwin Mani, working as an Entry level Software Engineer but aspiring to pursue a career in embedded AI. I have had experience working with Embedded Systems during my undergrad and an Internship at CarScan.AI where work mostly revolved around AI models benefitting Computer Vision Solutions. I was highly interested in working on this problem and further extend it as a part of my gsoc project.

I have gone through a general workflow of OpenVINO and have worked with mediapipe earlier while trying Pose Estimation examples..
Any initial tasks that could make me familiar with the problem even better would be great.

Thanks for your time

[adrianboguszewski]

@ngaloppo @Radwan-Ibrahim, could you look at this?

[ngaloppo]

Hello Ashwin, thanks for your interest!

Here's an initial task you could perform: implement a new calculator (and test) based on the following requirements:

// Calculator for normalizing and converting an ImageFrame or Matrix
// into a ov::Tensor or a GpuBuffer to a ov::RemoteTensor
//
// This calculator is designed to be used with the OpenVINOInferenceCalculator,
// as a pre-processing step for calculator inputs.
//
// IMAGE and IMAGE_GPU inputs are normalized to [-1,1] (default) or [0,1],
// specified by options (unless outputting a quantized tensor).
//
// Input:
//  One of the following tags:
//  IMAGE - ImageFrame (assumed to be 8-bit or 32-bit data).
//  IMAGE_GPU - GpuBuffer (assumed to be RGBA or RGB GL texture).
//
// Output:
//  One of the following tags:
//  TENSORS - Vector of ov::Tensors
//  TENSORS_REMOTE - Vector of ov::RemoteTensors
//
// Example use:
// node {
//   calculator: "OpenVINOConverterCalculator"
//   input_stream: "IMAGE:input_image"
//   output_stream: "TENSORS:image_tensor"
//   options: {
//     [mediapipe.OpenVINOConverterCalculatorOptions.ext] {
//       zero_center: true
//     }
//   }
// }

[Ashwin4514]

Great, Will start working on this.