support minicpmv

[eaidova]

What does this PR do?

from optimum.intel.openvino import OVModelForVisualCausalLM
from transformers import AutoProcessor
from PIL import Image
import requests

model_id = "openbmb/MiniCPM-V-2_6"

processor = AutoProcessor.from_pretrained(model_id, trust_remote_code=True)

prompt= "<|im_start|>user\n(<image>./</image>)\nWhat is unusual on this image?<|im_end|>\n<|im_start|>assistant\n"
image = Image.open(requests.get("https://github.com/openvinotoolkit/openvino_notebooks/assets/29454499/d5fbbd1a-d484-415c-88cb-9986625b7b11", stream=True).raw).convert('RGB')

model = OVModelForVisualCausalLM.from_pretrained(model_dir, trust_remote_code=True)

inputs = processor([prompt], [image], return_tensors="pt")

result  = model.generate(**inputs, max_new_tokens=20)

print(processor.tokenizer.batch_decode(result[:, inputs["input_ids"].shape[1]:]))

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you make sure to update the documentation with your changes?
• Did you write any new necessary tests?

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[AlexKoff88]

It is not related to the PR but in general. Is there a good way to automate the prompt definition for the user without a need to figure out what is the right point for the specific model?

[eaidova]

It is not related to the PR but in general. Is there a good way to automate the prompt definition for the user without a need to figure out what is the right point for the specific model?

usually it provided by chat template

conversation = [
    {
      "role": "user",
      "content": [
          {"type": "text", "text": "Describe image"},
          {"type": "image"},
        ],
    },
]

prompt = processor.apply_chat_template(conversation, add_generation_prompt=True)

but minicpm not fully follow to standard practices for creating model it required checking model code to understand how to apply it... (but I do not speak that it is impossible)

[IlyasMoutawwakil]

Looks good ! I left a couple of comments