Allow for OpenVINO models to be stored in a model subdirectory #923

tomaarsen · 2024-10-03T15:46:05Z

What does this PR do?

This PR allows users to load a model from a subdirectory via just file_name.

My goal:

from optimum.intel import OVModelForFeatureExtraction

model_id = "sentence-transformers-testing/stsb-bert-tiny-openvino"
model = OVModelForFeatureExtraction.from_pretrained(model_id, file_name="openvino/openvino_model.xml")
print(model)

Where only the modeling files are in the openvino subfolder.
Result:

No OpenVINO files were found for sentence-transformers-testing/stsb-bert-tiny-openvino, setting `export=True` to convert the model to the OpenVINO IR. Don't forget to save the resulting model with `.save_pretrained()`
Framework not specified. Using pt to export the model.
model.safetensors: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████| 17.5M/17.5M [00:01<00:00, 15.3MB/s]
tokenizer_config.json: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████| 1.50k/1.50k [00:00<?, ?B/s]
vocab.txt: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████| 232k/232k [00:00<00:00, 1.36MB/s]
tokenizer.json: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████| 712k/712k [00:00<00:00, 2.71MB/s]
special_tokens_map.json: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████| 732/732 [00:00<?, ?B/s]
...

I.e. it starts exporting because it doesn't find any openvino file anywhere (because it only looks in the root directory).

For reference, this is exactly how I do it for ONNX via Optimum, and it works well there.

An alternative

So I tried using subfolder:

from optimum.intel import OVModelForFeatureExtraction

model_id = "sentence-transformers-testing/stsb-bert-tiny-openvino"
model = OVModelForFeatureExtraction.from_pretrained(model_id, file_name="openvino_model.xml", subfolder="openvino")
print(model)

But then the rest of the files outside of the subfolder can't be read, and I get:

Could not infer whether the model was already converted or not to the OpenVINO IR, keeping `export=False`.
The library name could not be automatically inferred. If using the command-line, please provide the argument --library {transformers,diffusers,timm,sentence_transformers}. Example: `--library diffusers`.
Traceback (most recent call last):
  File "c:\code\sentence-transformers\demo_export_onnx.py", line 54, in <module>
    model = OVModelForFeatureExtraction.from_pretrained(model_id, file_name="openvino_model.xml", subfolder="openvino")
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\tom\.conda\envs\sentence-transformers\Lib\site-packages\optimum\intel\openvino\modeling_base.py", line 373, in from_pretrained
    return super().from_pretrained(
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\tom\.conda\envs\sentence-transformers\Lib\site-packages\optimum\modeling_base.py", line 381, in from_pretrained
    library_name = TasksManager.infer_library_from_model(model_id, subfolder, revision, cache_dir, token=token)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\tom\.conda\envs\sentence-transformers\Lib\site-packages\optimum\exporters\tasks.py", line 1901, in infer_library_from_model
    library_name = cls._infer_library_from_model_name_or_path(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\tom\.conda\envs\sentence-transformers\Lib\site-packages\optimum\exporters\tasks.py", line 1863, in _infer_library_from_model_name_or_path       
    raise ValueError(
ValueError: The library name could not be automatically inferred. If using the command-line, please provide the argument --library {transformers,diffusers,timm,sentence_transformers}. Example: `--library diffusers`.

I'm also pretty sure that I can't specify the library anywhere, so this is a dead end. Ideally, I want it to work equivalently to ONNX anyways.

The fix

The simple fix that I've applied for now is also look for openvino files in subdirectories. After all, if there's an exported openvino file anywhere, you probably want to inform the user rather than re-export a model.

After the fix, I get this:

from optimum.intel import OVModelForFeatureExtraction

model_id = "sentence-transformers-testing/stsb-bert-tiny-openvino"
model = OVModelForFeatureExtraction.from_pretrained(model_id, file_name="openvino/openvino_model.xml")
print(model)

Compiling the model to CPU ...
<optimum.intel.openvino.modeling.OVModelForFeatureExtraction object at 0x0000024A2057FD10>

Before submitting

This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
Did you make sure to update the documentation with your changes?
Did you write any new necessary tests?

Note

This issue will likely prevent me from including UKPLab/sentence-transformers#2712 into the next Sentence Transformers release, sadly. That PR adds OpenVINO and ONNX backends to Sentence Transformers, but currently it's not possible to load saved OpenVINO files due to this issue.

P.s. I have no plans to further update sentence-transformers-testing/stsb-bert-tiny-openvino, I'll be using it in my own tests as well.

cc @echarlaix @helena-intel

Tom Aarsen

HuggingFaceDocBuilderDev · 2024-10-03T15:52:29Z

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

tomaarsen · 2024-10-03T16:13:08Z

P.s. the original problematic snippet:

from optimum.intel import OVModelForFeatureExtraction

model_id = "sentence-transformers-testing/stsb-bert-tiny-openvino"
model = OVModelForFeatureExtraction.from_pretrained(model_id, file_name="openvino/openvino_model.xml")
print(model)

does work with optimum-intel 1.18.3, it's related to the new "setting export=True when needed".

Tom Aarsen

echarlaix · 2024-10-07T16:59:23Z

Thanks a lot for the PR @tomaarsen ! We are using the subfolder argument for such cases so you're correct the second alternative you shared is supposed to work :

from optimum.intel import OVModelForFeatureExtraction

model_id = "sentence-transformers-testing/stsb-bert-tiny-openvino"
model = OVModelForFeatureExtraction.from_pretrained(model_id, file_name="openvino_model.xml", subfolder="openvino")

The issue here is that the model's configuration is missing from the subfolder (which is something currently expected on the optimum side https://github.com/huggingface/optimum/blob/049b00f61c9bb17bd2b20a3b77d04cc4c0f20d86/optimum/modeling_base.py#L383) if too constraining we could remove this need and check in the parent folder for the configuration when not present in the specified subfolder. In the meantime adding the model's config directly in the subfolder should fix this

tomaarsen · 2024-10-07T17:44:51Z

I'm hesitant to duplicate configuration, so for now my workaround is to restrict optimum-intel to <1.19.0 and loading with "file_name": "openvino/openvino_model.xml". That should allow me to bring out a Sentence Transformers release, one that should keep working nicely if

from optimum.intel import OVModelForFeatureExtraction

model_id = "sentence-transformers-testing/stsb-bert-tiny-openvino"
model = OVModelForFeatureExtraction.from_pretrained(model_id, file_name="openvino/openvino_model.xml")
print(model)

works again in the future. Are you planning on that? Then it'll have feature parity with ONNX - I'd rather keep the same structure for both ONNX and OV in Sentence Transformers.

Tom Aarsen

… test

…nto openvino_in_subdirectory

echarlaix · 2024-10-09T09:05:05Z

@tomaarsen I'll open a PR in optimum to remove the need for the configuration to be in the subfolder, in the meantime for you to be able to use file_name you can modify this line directly

optimum-intel/optimum/intel/openvino/modeling_base.py

Line 442 in 785e1a6

pattern=r"(.*)?openvino(.*)?\_model.xml",

tomaarsen · 2024-10-09T09:20:33Z

@echarlaix I don't think I'm following - how could I modify that line? Do you mean in this PR? Or will you expose this pattern argument so that I can modify it from Sentence Transformers?

echarlaix · 2024-10-09T10:01:54Z

optimum/intel/utils/modeling_utils.py

-        files = [Path(p) for p in repo_files if re.match(pattern, str(p)) and str(p.parent) == subfolder]
+        files = [Path(p) for p in repo_files if re.match(pattern, str(p))]


in this case subfolder is not used at all which is an issue (will result in issue when subfolder is not defined correctly)

this modification shouldn't be needed with huggingface/optimum#2044

echarlaix · 2024-10-09T10:06:49Z

@echarlaix I don't think I'm following - how could I modify that line? Do you mean in this PR? Or will you expose this pattern argument so that I can modify it from Sentence Transformers?

It was in the case where you were using file_name but I see that's not the case anymore (I was suggesting to modify subfolder accordingly)

echarlaix · 2024-10-09T10:07:43Z

optimum/intel/openvino/modeling_base.py

@@ -439,7 +439,7 @@ def from_pretrained(

            ov_files = _find_files_matching_pattern(
                model_dir,
-                pattern=r"(.*)?openvino(.*)?\_model.xml",
+                pattern=r"(.*)?openvino(.*)?\_model.xml$",


good catch !

As a bit of extra context: The HfApi().snapshot_download with local_files creates a .cache folder with every file in the remote repository appended with .metadata. The resulting openvino_model.xml.metadata was also matched by the pattern.

tomaarsen · 2024-10-09T13:12:24Z

@echarlaix
If I modify my code such that it uses subfolder="openvino", file_name="openvino_model.xml" and subfolder="onnx", file_name="model.onnx", and I use huggingface/optimum#2044 (at commit f17cf1d53dedf7fbf849605bcf00775753e961f2) & https://github.com/huggingface/optimum-intel at main, then:

ONNX works 🎉
OpenVINO works 🎉

But only if I use exactly main from optimum-intel and fix from optimum.
Should I close this PR? It won't be necessary anymore. Although perhaps I can make a new PR for adding the $ in the regex patterns?

I'll await the optimum and optimum-intel releases. Once they're made, then I can also release Sentence Transformers with the new backends.

Tom Aarsen

echarlaix · 2024-10-09T13:40:29Z

Great to hear!

Should I close this PR? It won't be necessary anymore. Although perhaps I can make a new PR for adding the $ in the regex patterns?

A PR for the regex pattern + the test you added would be great !

I'll await the optimum and optimum-intel releases. Once they're made, then I can also release Sentence Transformers with the new backends.

One last thing needs to be fixed for the onnx export to be compatible with transformers v4.45 and we can do the release. Will let you know when ready (hopefully today)

tomaarsen · 2024-10-09T13:44:45Z

Sounds good!

tomaarsen · 2024-10-09T14:10:13Z

See #931. Apologies for the delay.

Tom Aarsen

tomaarsen added 2 commits October 3, 2024 17:08

Allow for OpenVINO models to be stored in a model subdirectory

736dd9d

Add test case

d231f3b

Run 'make style'

7f1bfbb

Use "" as a default subfolder instead of None

b40b42f

tomaarsen added 2 commits October 8, 2024 18:01

Add $ to pattern to prevent openvino_model.xml.metadata match; update…

fac5a3d

… test

Merge branch 'main' of https://github.com/huggingface/optimum-intel i…

75c74f2

…nto openvino_in_subdirectory

echarlaix reviewed Oct 9, 2024

View reviewed changes

tomaarsen mentioned this pull request Oct 9, 2024

remove the need for the config to be in the subfolder huggingface/optimum#2044

Merged

tomaarsen closed this Oct 9, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Allow for OpenVINO models to be stored in a model subdirectory #923

Allow for OpenVINO models to be stored in a model subdirectory #923

tomaarsen commented Oct 3, 2024 •

edited

Loading

HuggingFaceDocBuilderDev commented Oct 3, 2024

tomaarsen commented Oct 3, 2024

echarlaix commented Oct 7, 2024 •

edited

Loading

tomaarsen commented Oct 7, 2024

echarlaix commented Oct 9, 2024

tomaarsen commented Oct 9, 2024

echarlaix Oct 9, 2024 •

edited

Loading

echarlaix Oct 9, 2024

echarlaix commented Oct 9, 2024

echarlaix Oct 9, 2024

tomaarsen Oct 9, 2024

tomaarsen commented Oct 9, 2024 •

edited

Loading

echarlaix commented Oct 9, 2024

tomaarsen commented Oct 9, 2024

tomaarsen commented Oct 9, 2024

		files = [Path(p) for p in repo_files if re.match(pattern, str(p)) and str(p.parent) == subfolder]
		files = [Path(p) for p in repo_files if re.match(pattern, str(p))]

Allow for OpenVINO models to be stored in a model subdirectory #923

Allow for OpenVINO models to be stored in a model subdirectory #923

Conversation

tomaarsen commented Oct 3, 2024 • edited Loading

What does this PR do?

My goal:

An alternative

The fix

Before submitting

Note

HuggingFaceDocBuilderDev commented Oct 3, 2024

tomaarsen commented Oct 3, 2024

echarlaix commented Oct 7, 2024 • edited Loading

tomaarsen commented Oct 7, 2024

echarlaix commented Oct 9, 2024

tomaarsen commented Oct 9, 2024

echarlaix Oct 9, 2024 • edited Loading

Choose a reason for hiding this comment

echarlaix Oct 9, 2024

Choose a reason for hiding this comment

echarlaix commented Oct 9, 2024

echarlaix Oct 9, 2024

Choose a reason for hiding this comment

tomaarsen Oct 9, 2024

Choose a reason for hiding this comment

tomaarsen commented Oct 9, 2024 • edited Loading

echarlaix commented Oct 9, 2024

tomaarsen commented Oct 9, 2024

tomaarsen commented Oct 9, 2024

tomaarsen commented Oct 3, 2024 •

edited

Loading

echarlaix commented Oct 7, 2024 •

edited

Loading

echarlaix Oct 9, 2024 •

edited

Loading

tomaarsen commented Oct 9, 2024 •

edited

Loading