unexpected inference results for blur controlnets

[Eric-Hao]

Thank you for your excellent work!

I’ve run into an issue with the inference results and I’m hoping you can shed some light on it. When I follow the blur controlnet inference instructions in README.md:

python sd3_infer.py --model models/sd3.5_large.safetensors --controlnet_ckpt models/sd3.5_large_controlnet_blur.safetensors --controlnet_cond_image inputs/blur.png --prompt "generated ai art, a tiny, lost rubber ducky in an action shot close-up, surfing the humongous waves, inside the tube, in the style of Kelly Slater"

I’m noticing some odd white grid points in the output:

The issue persists even without the prompt, indicating that the text encoder might not be the culprit.

Additionally, I’ve noticed that the code defaults to using fp32 precision for t5xxl.safetensors, whereas the Huggingface checkpoint only provides fp16 and fp8 versions.

Any advice or tips on how to address this would be greatly appreciated.