got 0 ops for nn.MultiheadAttention

[wangtiance]

My thop version: 0.1.1

Minimal code to replicate:

import thop
import torch.nn as nn
import torch

f = nn.MultiheadAttention(100, 1, batch_first=True)

x = torch.ones((1, 1000, 100))

result = thop.profile(f, (x,x,x))
print(result)

I got (0.0, 0). Is this module not supported yet?

[wangtiance]

Does it only support modules listed here?

pytorch-OpCounter/thop/profile.py

Line 21 in 43c064a

register_hooks = {

[CaA23187]

you are right. I guess this module doesn't supported MHSA layer yet.

[quancs]

facing the same problem

[HaoKang-Timmy]

You may use https://github.com/HaoKang-Timmy/torchanalyse or torchprofile for NLP models

[HaoKang-Timmy]

You may use https://github.com/HaoKang-Timmy/torchanalyse or torchprofile for NLP models

These two repo could profile transformers, which I have tried.

[CaA23187]

You may use https://github.com/HaoKang-Timmy/torchanalyse or torchprofile for NLP models

thank you for your reply. I'll try it!

[cooma04]

It seems thop does not account for any parameters created with nn.Parameter. For any created with nn.Parameter, it simply yields 0.