RuntimeError: CUDA error when run work.sh

[LIU-FAYANG]

Hi, I'm trying to use pretrained model to parse txt to AMR graphs, I tried to run stanford core nlp server and the data preprocessing is successeful( test.txt.raw.features.preproc was successfully generated)
However when I tried to run work.sh this strange bug happened and I had no clue about why this would happen. I tried to run it on CPU and CUDA error still showed up, could you have any advice on this?
my work.sh looks like this:

python3 -u -m parser.work --test_data test.txt.raw.features.preproc --test_batch_size 6666 --load_path ckpt/ckpt.pt --beam_size 8 --alpha 0.6 --max_time_step 100 --output_suffix _test_out

Error log：

sh work.sh
To use data.metrics please install scikit-learn. See https://scikit-learn.org/stable/index.html
read from test.txt.raw.features.preproc, 2 amrs
Get 2 AMRs from test.txt.raw.features.preproc
ckpt/ckpt.pt
Traceback (most recent call last):
File "/media/ntu/volume1/home/s122md301_07/anaconda3/envs/AMR-gs/lib/python3.6/runpy.py", line 193, in _run_module_as_main
"main", mod_spec)
File "/media/ntu/volume1/home/s122md301_07/anaconda3/envs/AMR-gs/lib/python3.6/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/media/ntu/volume1/home/s122md301_07/AMR-gs/parser/work.py", line 145, in
parse_data(model, pp, another_test_data, args.test_data, test_model+args.output_suffix, args.beam_size, args.alpha, args.max_time_step)
File "/media/ntu/volume1/home/s122md301_07/AMR-gs/parser/work.py", line 68, in parse_data
res = parse_batch(model, batch, beam_size, alpha, max_time_step)
File "/media/ntu/volume1/home/s122md301_07/AMR-gs/parser/work.py", line 42, in parse_batch
beams = model.work(batch, beam_size, max_time_step)
File "/media/ntu/volume1/home/s122md301_07/AMR-gs/parser/parser.py", line 83, in work
word_repr, word_mask, probe = self.encode_step_with_bert(data['tok'], data['lem'], data['pos'], data['ner'], data['word_char'], data['bert_token'], data['token_subword_index'])
File "/media/ntu/volume1/home/s122md301_07/AMR-gs/parser/parser.py", line 65, in encode_step_with_bert
bert_embed, _ = self.bert_encoder(bert_token, token_subword_index=token_subword_index)
File "/media/ntu/volume1/home/s122md301_07/anaconda3/envs/AMR-gs/lib/python3.6/site-packages/torch/nn/modules/module.py", line 547, in call
result = self.forward(*input, **kwargs)
File "/media/ntu/volume1/home/s122md301_07/AMR-gs/parser/bert_utils.py", line 55, in forward
input_ids, attention_mask, token_type_ids)
File "/media/ntu/volume1/home/s122md301_07/anaconda3/envs/AMR-gs/lib/python3.6/site-packages/transformers/modeling_bert.py", line 627, in forward
head_mask=head_mask)
File "/media/ntu/volume1/home/s122md301_07/anaconda3/envs/AMR-gs/lib/python3.6/site-packages/torch/nn/modules/module.py", line 547, in call
result = self.forward(*input, **kwargs)
File "/media/ntu/volume1/home/s122md301_07/anaconda3/envs/AMR-gs/lib/python3.6/site-packages/transformers/modeling_bert.py", line 348, in forward
layer_outputs = layer_module(hidden_states, attention_mask, head_mask[i])
File "/media/ntu/volume1/home/s122md301_07/anaconda3/envs/AMR-gs/lib/python3.6/site-packages/torch/nn/modules/module.py", line 547, in call
result = self.forward(*input, **kwargs)
File "/media/ntu/volume1/home/s122md301_07/anaconda3/envs/AMR-gs/lib/python3.6/site-packages/transformers/modeling_bert.py", line 326, in forward
attention_outputs = self.attention(hidden_states, attention_mask, head_mask)
File "/media/ntu/volume1/home/s122md301_07/anaconda3/envs/AMR-gs/lib/python3.6/site-packages/torch/nn/modules/module.py", line 547, in call
result = self.forward(*input, **kwargs)
File "/media/ntu/volume1/home/s122md301_07/anaconda3/envs/AMR-gs/lib/python3.6/site-packages/transformers/modeling_bert.py", line 283, in forward
self_outputs = self.self(input_tensor, attention_mask, head_mask)
File "/media/ntu/volume1/home/s122md301_07/anaconda3/envs/AMR-gs/lib/python3.6/site-packages/torch/nn/modules/module.py", line 547, in call
result = self.forward(*input, **kwargs)
File "/media/ntu/volume1/home/s122md301_07/anaconda3/envs/AMR-gs/lib/python3.6/site-packages/transformers/modeling_bert.py", line 202, in forward
mixed_query_layer = self.query(hidden_states)
File "/media/ntu/volume1/home/s122md301_07/anaconda3/envs/AMR-gs/lib/python3.6/site-packages/torch/nn/modules/module.py", line 547, in call
result = self.forward(*input, **kwargs)
File "/media/ntu/volume1/home/s122md301_07/anaconda3/envs/AMR-gs/lib/python3.6/site-packages/torch/nn/modules/linear.py", line 87, in forward
return F.linear(input, self.weight, self.bias)
File "/media/ntu/volume1/home/s122md301_07/anaconda3/envs/AMR-gs/lib/python3.6/site-packages/torch/nn/functional.py", line 1371, in linear
output = input.matmul(weight.t())
RuntimeError: CUDA error: CUBLAS_STATUS_EXECUTION_FAILED when calling cublasSgemm( handle, opa, opb, m, n, k, &alpha, a, lda, b, ldb, &beta, c, ldc)