Use OpenVINO as backend for Tensorflow and PyTorch for training

[redradist]

Idea

OpenVINO is a great framework for deploying neural networks, but at the same time OpenVINO has many operations needed for training the network.
It would be nice to use OpenVINO for training using Tensorflow and PyTorch ...

There is already exists package for Tensorflow inference, but it is limited to a inference only.
New PyTorch 2.0 has simplified adding a new backend with new architecture https://pytorch.org/get-started/pytorch-2.0

For me it is obvious that OpenVINO will benefit as ecosystem from integration Tensorflow/PyTorch training

Related issues

openvino_tensorflow - Pytorch integration

[devangaggarwal]

Thanks for the feedback, @redradist.
We will take this into consideration when planning for future releases of OpenVINO.
Till then, have you explored contributing to OpenVINO? You yourself can take a stab at this and see if you are able to make any progress.

[redradist]

@devangaggarwal
Yes, I am a previous team-leader (@lohika-denis-kotov) of NVIDIA plugin :)

[devangaggarwal]

With 2023.1 OV release, we have release torch.compile OV backend.

[redradist]

It is a great news !!
It it very important to use the same toolchain for deployment and for training at the same time. You can be sure that everything will work the same way in production and also you do not need any additional packages.

Great work guys !!

[redradist]

With 2023.1 OV release, we have release torch.compile OV backend. Learn more here: #19163

Is there any documentation how to used OV backend in torch.compile ?
Link that you provided is just a discussion for different topic

[yury-gorbachev]

You might want to check this one: https://docs.openvino.ai/2023.1/pytorch_2_0_torch_compile.html

[redradist]

I would suggest to use options field for passing arguments to backend as additional way to environment variables:

def compile(model: Optional[Callable] = None, *,
            fullgraph: builtins.bool = False,
            dynamic: builtins.bool = False,
            backend: Union[str, Callable] = "inductor",
            mode: Union[str, None] = None,
            options: Optional[Dict[str, Union[str, builtins.int, builtins.bool]]] = None,
            disable: builtins.bool = False) -> Callable:
    ...

[redradist]

An upstream PyTorch has already supported this options:

    if backend == "inductor":
        backend = _TorchCompileInductorWrapper(mode, options, dynamic)
    else:
        backend = _TorchCompileWrapper(backend, mode, options, dynamic)

See _TorchCompileWrapper for custom backend

[redradist]

@yury-gorbachev
Browsing the openvino repositories I noticed that https://github.com/openvinotoolkit/openvino_tensorflow is not longer maitained and developers should prefer itex (Intel Extension for Tensorflow). First I thought itex is based on OpenVINO, but looks like it has completly separate runtime !!
Why was made such decision ? As for me OpenVINO is completely suitable for such plugin like PluggableDevice for Tensorflow ...

It could be possible seamlessly for users to register OpenVINO runtime for PyTorch:

import openvino.torch

and Tensorflow:

import openvino.tensorflow

and use single toolchain for training as well as deployment

[yury-gorbachev]

@ynimmaga - could you please check? This seems reasonable to me.
cc: @ilya-lavrenov

[redradist]

@yury-gorbachev I've created PR for it #20038

[ynimmaga]

@redradist, really appreciate the suggestion and the PR. @suryasidd was already working on providing options for OpenVINO backend with torch.compile. Adding him to look at the PR and comment.