-
Notifications
You must be signed in to change notification settings - Fork 5
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Details of LoRA of pruned models. #1
Comments
Thank you for your interests. Below is the scripts for lora ft:
You should first save the pruned model to |
Thanks alot, any idea on the GPU hours you need for this run? |
@Arnav0400 Hi,sorry for the late reply. It roughly costs one GPU day for LLaMA1-7B. |
Thanks for your reply! Did you perform a zero-shot evaluation on the LoRA fine-tuned+pruned models? This is practically very important as models undergo fine-tuning before deployment. |
@Arnav0400 Not yet. But it is easy to perform this evaluation. What you should do is to save the checkpoint of LoRA fine-tuned pruned LLMs. Using LoRA to fine-tune the pruned model is very common, especially for structured pruning methods. You can refer to my repo with Also, the fine-tuning code in our repo is not strong enough, as it just provide limited PEFT methods, I recommend you use LLM-Adapters repo, which provide more datasets and more PEFT adapters. |
Please let me know if you have the checkpoint someplace, would save some time for me. I am interested in seeing the role of peft in sparse models. |
Great work and thanks for the codebase!
I want to know the exact detailed of LoRA fine-tuning as mentioned in Table 6 of the main paper.
Also if you could point-out to the bash script to reproduce the same, would be great!
The text was updated successfully, but these errors were encountered: