Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: Use a more flexible approach to check favorability of metrics #1063

Merged
merged 4 commits into from
Jan 9, 2025

Conversation

glemaitre
Copy link
Member

closes #1061

image

We are more flexible using a regular expression to check the score names. In addition, we take care to test first the neg_ part that would mean that negative score are therefore "higher is greater" convention.

@augustebaum augustebaum self-requested a review January 9, 2025 09:00
Copy link
Contributor

@augustebaum augustebaum left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The "neg_ means higher is better" rule makes me a bit wary of false positives.
For example, neg_accuracy gets falsely considered as higher-is-better.

@glemaitre
Copy link
Member Author

It will never happen in practice. The neg_ is only a scikit-learn convention used to negate the loss/error/deviance such that whatever metrics used in a grid-search will be maximized.

So if you are a user, you are never going to defined your own neg_ metric to make a score that lower is better.

@augustebaum
Copy link
Contributor

Sounds good.

Copy link
Contributor

@augustebaum augustebaum left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

@augustebaum augustebaum merged commit b7ab74a into probabl-ai:main Jan 9, 2025
19 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

The check for metric name and show lower/higher is better is too strict
2 participants