stage2 training error

[ziming-liu]

File "/home/ziliu/anaconda3/envs/semantics/lib/python3.8/site-packages/mmcv/runner/fp16_utils.py", line 186, in new_func
return old_func(*args, **kwargs)
File "/data/acentauri/user/ziliu/data/voxformer/projects/mmdet3d_plugin/voxformer/modules/deformable_cross_attention.py", line 143, in forward
index_query_per_img = mask_per_img[0].sum(-1).nonzero().squeeze(-1)
RuntimeError: CUDA error: an illegal memory access was encountered
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.

[Sxiansheng24]

Hello, I have the same problem, may I ask you to solve this problem? If so, can you provide some solutions

[ziming-liu]

Hello, I have the same problem, may I ask you to solve this problem? If so, can you provide some solutions

no, I didn't move forward.

[codeimdy]

Hello, after completing the first phase of training, how should I proceed with the second phase? Are there any additional steps required, or do I only need to enter the training command in the terminal? Directly entering the training command results in an error, as shown in the picture.