sd3 pipeline support #916

eaidova · 2024-09-26T10:20:08Z

What does this PR do?

Fixes # (issue)

Before submitting

This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
Did you make sure to update the documentation with your changes?
Did you write any new necessary tests?

HuggingFaceDocBuilderDev · 2024-09-26T10:26:08Z

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

IlyasMoutawwakil · 2024-10-10T09:44:04Z

The changes to be made in this PR after the refactorization are very simple and mostly will make it more readable:

no need to copy paste modeling from diffusers anymore, you just subclass (OVDiffusionPipeline, StabeldDiffusion3Pipeline) and it should be good in most cases.
adding tests is as simple as adding the architecture to the task specific tests, I see you haven't added tests yet tho...

I also suggest you take a look at the original refactorization work in huggingface/optimum#2021, I explain there issues with the previous approach and why the refactorization is not only that but also fixing many issues with the previous design (reproducibility of results & numeric consistency with diffusers), which weren't tested before or simply ignored for some architectures/tasks.

optimum/exporters/openvino/convert.py

optimum/intel/utils/dummy_openvino_and_diffusers_objects.py

optimum/intel/openvino/modeling_diffusion.py

IlyasMoutawwakil

LGTM, can you run styling and add tests please.

tests/openvino/utils_tests.py

IlyasMoutawwakil · 2024-10-12T11:38:06Z

it seems sd3 still doesn't support callbacks (idk if it ever will as it doesn't have a unet), need to be skipped.
it's also not as numerically consistent with diffusers, which is interesting, we can increase atol and rtol to 1e-3 and 1e-1 for this architecture for now, but worth investigating, especially if each exported component is numerically consistent.
and finally I guess output_type="latent" is different for this model, again because it uses a transformer instead of unet https://github.com/huggingface/optimum-intel/blob/main/tests/openvino/test_diffusion.py#L419-L422

IlyasMoutawwakil

will be merged once tests are updated and passed

tests/openvino/utils_tests.py

optimum/exporters/openvino/model_configs.py

optimum/exporters/openvino/model_patcher.py

eaidova · 2024-10-22T17:37:51Z

@IlyasMoutawwakil now all relevant tests are fixed, please take a look one more time

optimum/exporters/openvino/model_configs.py

optimum/intel/__init__.py

optimum/intel/openvino/modeling_diffusion.py

Co-authored-by: Ilyas Moutawwakil <[email protected]>

tests/openvino/test_diffusion.py

Co-authored-by: Ilyas Moutawwakil <[email protected]>

IlyasMoutawwakil · 2024-10-22T19:23:15Z

optimum/intel/openvino/modeling_diffusion.py

-        ov_outputs = self.request(model_inputs, share_inputs=True).to_dict()
-
+        ov_outputs = self.request(model_inputs, share_inputs=True)
+        main_out = ov_outputs[0]
        model_outputs = {}
-        for key, value in ov_outputs.items():
-            model_outputs[next(iter(key.names))] = torch.from_numpy(value)
-
-        if output_hidden_states:
-            model_outputs["hidden_states"] = []
-            for i in range(self.config.num_hidden_layers):
-                model_outputs["hidden_states"].append(model_outputs.pop(f"hidden_states.{i}"))
-            model_outputs["hidden_states"].append(model_outputs.get("last_hidden_state"))
+        model_outputs[self.model.outputs[0].get_any_name()] = torch.from_numpy(main_out)
+        if len(self.model.outputs) > 1 and "pooler_output" in self.model.outputs[1].get_any_name():
+            model_outputs["pooler_output"] = torch.from_numpy(ov_outputs[1])
+        if self.hidden_states_output_names and "last_hidden_state" not in model_outputs:
+            model_outputs["last_hidden_state"] = torch.from_numpy(ov_outputs[self.hidden_states_output_names[-1]])
+        if (
+            self.hidden_states_output_names
+            and output_hidden_states
+            or getattr(self.config, "output_hidden_states", False)
+        ):
+            hidden_states = [torch.from_numpy(ov_outputs[out_name]) for out_name in self.hidden_states_output_names]
+            model_outputs["hidden_states"] = hidden_states


not sure this complexity is needed, the original code does the same thing by seperating the torch.from_numpy and collecting the hidden states in a list if requested, the only change you'll have to do is to account for the missing self.config.num_hidden_layers by using self.config.num_layers otherwise for t5-encoder https://huggingface.co/stabilityai/stable-diffusion-3-medium-diffusers/blob/main/text_encoder_3/config.json#L21

eaidova changed the title ~~Ea/sd3 wip~~ sd3 pipeline support Sep 26, 2024

echarlaix requested a review from IlyasMoutawwakil October 2, 2024 12:32

eaidova force-pushed the ea/sd3_wip branch 5 times, most recently from a19a436 to b543d94 Compare October 7, 2024 13:17

eaidova force-pushed the ea/sd3_wip branch from b543d94 to 37a39dd Compare October 11, 2024 14:27

IlyasMoutawwakil reviewed Oct 11, 2024

View reviewed changes

optimum/exporters/openvino/convert.py Outdated Show resolved Hide resolved

IlyasMoutawwakil reviewed Oct 11, 2024

View reviewed changes

optimum/intel/utils/dummy_openvino_and_diffusers_objects.py Show resolved Hide resolved

IlyasMoutawwakil reviewed Oct 11, 2024

View reviewed changes

optimum/intel/openvino/modeling_diffusion.py Outdated Show resolved Hide resolved

IlyasMoutawwakil approved these changes Oct 11, 2024

View reviewed changes

IlyasMoutawwakil reviewed Oct 12, 2024

View reviewed changes

tests/openvino/utils_tests.py Show resolved Hide resolved

IlyasMoutawwakil requested changes Oct 12, 2024

View reviewed changes

IlyasMoutawwakil reviewed Oct 14, 2024

View reviewed changes

tests/openvino/utils_tests.py Outdated Show resolved Hide resolved

eaidova force-pushed the ea/sd3_wip branch 3 times, most recently from 7916966 to be4624d Compare October 21, 2024 08:28

eaidova force-pushed the ea/sd3_wip branch from 7c535b2 to 66d9491 Compare October 21, 2024 17:10

eaidova requested a review from IlyasMoutawwakil October 21, 2024 17:10

eaidova force-pushed the ea/sd3_wip branch from 66d9491 to f1f9e92 Compare October 22, 2024 05:02

IlyasMoutawwakil reviewed Oct 22, 2024

View reviewed changes

optimum/exporters/openvino/model_configs.py Outdated Show resolved Hide resolved

IlyasMoutawwakil reviewed Oct 22, 2024

View reviewed changes

optimum/exporters/openvino/model_configs.py Outdated Show resolved Hide resolved

IlyasMoutawwakil reviewed Oct 22, 2024

View reviewed changes

optimum/exporters/openvino/model_patcher.py Show resolved Hide resolved

eaidova force-pushed the ea/sd3_wip branch from 6fe72b5 to 5df09e1 Compare October 22, 2024 14:34

eaidova requested a review from IlyasMoutawwakil October 22, 2024 17:37

IlyasMoutawwakil reviewed Oct 22, 2024

View reviewed changes

optimum/exporters/openvino/model_configs.py Outdated Show resolved Hide resolved

IlyasMoutawwakil reviewed Oct 22, 2024

View reviewed changes

optimum/exporters/openvino/model_configs.py Outdated Show resolved Hide resolved

IlyasMoutawwakil reviewed Oct 22, 2024

View reviewed changes

optimum/intel/__init__.py Show resolved Hide resolved

IlyasMoutawwakil reviewed Oct 22, 2024

View reviewed changes

optimum/intel/openvino/modeling_diffusion.py Outdated Show resolved Hide resolved

eaidova and others added 22 commits October 22, 2024 23:01

WIP: conversion and pipeline base

c5df4c3

Support SD3

b0fc176

img2img pipeline

d2e7d04

fix model export

ba4eca4

update after migration on new pipeline style

b219211

fix inference issues

b84da62

fix missed tokenizer export

7759353

add support in quantization

dfec4b4

Update optimum/intel/openvino/modeling_diffusion.py

9b055ad

Co-authored-by: Ilyas Moutawwakil <[email protected]>

add tests

1af0185

fix tests

ddab249

update tests

96ba985

Update tests/openvino/utils_tests.py

d07ba0b

Co-authored-by: Ilyas Moutawwakil <[email protected]>

fix tests

41afd03

add export tests

09b5d40

fix cli tests

6ce4b50

use fp32 timesteps

22232d2

add flux

8da6765

fix after black update

4597641

apply review comments

4d9249d

compatibility with diffusers 0.31.0

b3a8726

apply review comments

0667320

eaidova force-pushed the ea/sd3_wip branch from 52f40af to 0667320 Compare October 22, 2024 19:11

IlyasMoutawwakil reviewed Oct 22, 2024

View reviewed changes

tests/openvino/test_diffusion.py Outdated Show resolved Hide resolved

Update tests/openvino/test_diffusion.py

163a466

Co-authored-by: Ilyas Moutawwakil <[email protected]>

IlyasMoutawwakil reviewed Oct 22, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

sd3 pipeline support #916

sd3 pipeline support #916

eaidova commented Sep 26, 2024

HuggingFaceDocBuilderDev commented Sep 26, 2024

IlyasMoutawwakil commented Oct 10, 2024 •

edited

Loading

IlyasMoutawwakil left a comment •

edited

Loading

IlyasMoutawwakil commented Oct 12, 2024

IlyasMoutawwakil left a comment

eaidova commented Oct 22, 2024

IlyasMoutawwakil Oct 22, 2024 •

edited

Loading

sd3 pipeline support #916

Are you sure you want to change the base?

sd3 pipeline support #916

Conversation

eaidova commented Sep 26, 2024

What does this PR do?

Before submitting

HuggingFaceDocBuilderDev commented Sep 26, 2024

IlyasMoutawwakil commented Oct 10, 2024 • edited Loading

IlyasMoutawwakil left a comment • edited Loading

Choose a reason for hiding this comment

IlyasMoutawwakil commented Oct 12, 2024

IlyasMoutawwakil left a comment

Choose a reason for hiding this comment

eaidova commented Oct 22, 2024

IlyasMoutawwakil Oct 22, 2024 • edited Loading

Choose a reason for hiding this comment

IlyasMoutawwakil commented Oct 10, 2024 •

edited

Loading

IlyasMoutawwakil left a comment •

edited

Loading

IlyasMoutawwakil Oct 22, 2024 •

edited

Loading