You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Following @rflamary's advice, it could be interesting to see whether MaNo can be used as a realistic unsupervised scorer.
The method was designed for Unsupervised Accuracy Estimation and shows a linear correlation with the test OOD accuracy without needing test labels or training data/labels. Hence, one could in principle use it as a realistic unsupervised scorer.
It was tailored for computer vision classification tasks with large-scale deep models, hence it is not guaranteed at all that it can be useful for other modalities
In particular, it relies on test logits obtained after an inference step of the neural network. The implementation in Skada should/could work both for shallow and deep methods and thus the original implementation should be adapted
As the unsupervised setting is particularly hard, two hyperparameters are needed in MaNo (although fixed and quite stable in the original experiments). This might also be an issue for its efficiency/versatility to other modalities and with smaller neural networks
The text was updated successfully, but these errors were encountered:
Following @rflamary's advice, it could be interesting to see whether MaNo can be used as a realistic unsupervised scorer.
The method was designed for Unsupervised Accuracy Estimation and shows a linear correlation with the test OOD accuracy without needing test labels or training data/labels. Hence, one could in principle use it as a realistic unsupervised scorer.
Disclaimer
The text was updated successfully, but these errors were encountered: