Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix repetition 'sunglasses' label #842

Closed

Conversation

darkasevgen
Copy link

@darkasevgen darkasevgen commented Mar 14, 2024

There was a repetition "sunglasses" at IMAGENET_CLASSES.
As far as I understand, these class names have already been corrected (for instance, 'crane' transform into "crane bird" and "construction crane", but "sunglasses" haven't been).
Reference: https://www.kaggle.com/c/imagenet-object-localization-challenge/data?select=LOC_synset_mapping.txt

The desired mapping {folder name: class name}:
n04355933 sunglass
n04356056 sunglasses, dark glasses, shades

I have added this difference so that there are 1000 unique labels.

@EIFY
Copy link
Contributor

EIFY commented Mar 31, 2024

There is another repeated label: "missile". According to huggingface the 2nd one has been changed to "projectile, missile".

@rwightman
Copy link
Collaborator

rwightman commented Jun 7, 2024

uniqueness doesn't really matter here... we and CLIP_benchmark are using the same class names and prompts as OpenAI did for their original CLIP evals. Changing that would alter the results and not be consistent with all other evals.

I'd also argue that the changes OpenAI made from default names are correct. If you look at the validation images, the curators did not understand the distinction .. both sunglass and sunglasses are sunglasses, and missiles are all missiles without much to distinguish any specific subtype correctly.

@rwightman rwightman closed this Jun 7, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants