Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update to starter colab notebook for lm-eval-harness #3

Open
sepiatone opened this issue May 23, 2024 · 2 comments
Open

Update to starter colab notebook for lm-eval-harness #3

sepiatone opened this issue May 23, 2024 · 2 comments

Comments

@sepiatone
Copy link
Contributor

There has been a refactor of lm-eval-harness a few months ago. The starter colab notebook for lm-eval-harness, linked to from this section has to be slightly updated to reflect this.

Instead of

!python lm-evaluation-harness/main.py \
    --model gpt2 \
    --tasks truthfulqa_gen \
    --device 0

we have

!lm-eval --model hf --model_args pretrained=EleutherAI/Pythia-70m --tasks truthfulqa_gen --device 0
@jas-ho
Copy link
Member

jas-ho commented May 24, 2024

thanks, I'd be glad if you submitted a PR!

@sepiatone
Copy link
Contributor Author

thanks, I'd be glad if you submitted a PR!

will do that!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants