Refactor prompt parser #3535
Refactor prompt parser #3535
Conversation
| return True | ||
|
|
||
|
|
||
| def prepare_model(): |
There was a problem hiding this comment.
why would you remove accelerate hooks? it kind of invalidates any kind of model offloading. i get you need to have te on right device, but basically killing offloading is not the way to go.
There was a problem hiding this comment.
Git blame puts that code on @Disty0 I just moved it.
There was a problem hiding this comment.
ok, i'll handle that. it should definitely not be executed that frequently - it does have significant performance implications.
There was a problem hiding this comment.
Status check on this?
If there is significant performance implications here, they exist in the main code base as well.
|
closes #2712 |
|
btw, i haven't commented on this for past few days because it looks ok, need to actually test it and i've been delaying that. |
Signed-off-by: Vladimir Mandic <[email protected]>
modules/prompt_parser_diffusers.py
Outdated
| if len(batch[i]) == 0: | ||
| return None | ||
| else: | ||
| res.append(batch[i][step]) |
There was a problem hiding this comment.
i'm getting immediate list-index-out-of-range runtime error here, triggered from diffusers_callback
Signed-off-by: Vladimir Mandic <[email protected]>
Full refactor of prompt parsing.
Moved prompt embeds inside of a class object that parses prompts at initialization and returns them when called.
Full scheduling capabilities restored
LRU cache for prompts, size adjustable from options