Cross Attention variant?

[Xynonners]

Hi there,

Sorry if this is a stupid issue but I was wondering if it would be possible to apply Ring Attention to Cross Attention? I was thinking of using RingFlashAttentionCUDAFunction directly but it seems like the transformer block itself has modifications.

Thanks

[lucidrains]

@Xynonners hey, it is a good question and i'm still thinking about how to approach it

i think it ties together with inference w/ kv caching, so i'll prob get a solution out for both around the same time

can keep this issue open until then