Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add MaNo Scorer to metrics.py as a realistic unsupervised scorer #288

Open
ambroiseodt opened this issue Jan 8, 2025 · 0 comments
Open

Comments

@ambroiseodt
Copy link
Contributor

Following @rflamary's advice, it could be interesting to see whether MaNo can be used as a realistic unsupervised scorer.

The method was designed for Unsupervised Accuracy Estimation and shows a linear correlation with the test OOD accuracy without needing test labels or training data/labels. Hence, one could in principle use it as a realistic unsupervised scorer.

Disclaimer

  • I am an author of MaNo (NeurIPS 2024)
  • It was tailored for computer vision classification tasks with large-scale deep models, hence it is not guaranteed at all that it can be useful for other modalities
  • In particular, it relies on test logits obtained after an inference step of the neural network. The implementation in Skada should/could work both for shallow and deep methods and thus the original implementation should be adapted
  • As the unsupervised setting is particularly hard, two hyperparameters are needed in MaNo (although fixed and quite stable in the original experiments). This might also be an issue for its efficiency/versatility to other modalities and with smaller neural networks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant