-
Notifications
You must be signed in to change notification settings - Fork 2.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
CUDA errors when using models that have been imported from HF and trained with SentenceTransformers #324
Comments
How does your complete code look like? Do these examples work when you do not change anything? |
I have the same issue. As mentioned in https://github.com/UKPLab/sentence-transformers#training, However, I got the following error:
CUDA: 10.2 |
Hi @yuwon, Sadly I don't know where the error comes from. Appears some issue with CUDA/Pytorch. Maybe you can try it with a different CUDA / pytorch version? Best |
Hi @yuwon I had lots of problems like this. I've moved to developing in a docker container with an official pytorch cuda base image and never had problems again. Blog post on deving in docker here: http://braaannigan.github.io/software/2020/07/26/dev_in_docker.html |
Thanks @braaannigan. Yes, I've also tried with Pytorch's official Docker image but I also failed. |
I also encountered the same error as @yuwon ( |
Hi,
Expected behaviour: When I create a SentenceTransformer model by importing in a HF model and fine tuning it with the NLI code example, it should work when encodding text.
Actual behaviour: CUDA errors occur when trying to embed text.
The pretrained models from the SentenceTransformers package are able to embed this text without errors.
How the model was initialised:
This error still occurs even for models trained from scratch without layer freezing.
Could someone let me know what could be going wrong?
The text was updated successfully, but these errors were encountered: