Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

2.14rc在rk3399上推理官方的yolov5u_n报错 #10585

Open
leslie2046 opened this issue Nov 6, 2024 · 13 comments
Open

2.14rc在rk3399上推理官方的yolov5u_n报错 #10585

leslie2046 opened this issue Nov 6, 2024 · 13 comments

Comments

@leslie2046
Copy link

为使您的问题得到快速解决，在建立 Issue 前，请您先通过如下方式搜索是否有相似问题:历史 issue, FAQ 文档, 官方文档

如果您没有查询到相似问题，为快速解决您的提问，建立 issue 时请提供如下细节信息：

  • 标题：简洁、精准描述您的问题，例如“最新预测库的 API 文档在哪儿 ”
  • 版本、预测库信息：
       1）Paddle Lite 版本：2.14rc
       2）Host 环境：centos 7.9
       3）运行设备环境：rk3399
       4）预测后端信息：GPU
  • 预测信息
       1）预测 API：c++
       2）预测选项信息：armv8
       3）预测库来源：
       git clone https://github.com/PaddlePaddle/Paddle-Lite.git
       git checkout v2.14-rc
       ./lite/tools/build_android.sh --with_opencl=ON --arch=armv8 --with_extra=ON --with_log=ON --with_cv=ON --with_exception=ON
  • 复现信息：如为报错，请提供剥离业务环境的可复现预测 demo，用于问题复现
  • 问题描述：请详细描述您的问题，同步贴出报错信息、日志/代码关键片段
    可以运行起来，过一会就会报错
    2024-11-06 14:05:56.148 31714-31737 Paddle-Lite com...e.lite.demo.yolov5n_detection A [F 11/ 6 14: 5:56.148 ...-Lite/lite/backends/opencl/cl_context.cc:47 AddKernel] OpenCL error with code CL_OUT_OF_RESOURCES happened in file /home/leslie/Paddle-Lite/lite/backends/opencl/cl_context.cc at line 47. Exiting.
    2024-11-06 14:05:56.149 31714-31737 libc++abi com...e.lite.demo.yolov5n_detection E terminating with uncaught exception of type paddle::lite::PaddleLiteException: Paddle-Lite C++ Exception:
    [F 11/ 6 14: 5:56.148 ...-Lite/lite/backends/opencl/cl_context.cc:47 AddKernel] OpenCL error with code CL_OUT_OF_RESOURCES happened in file /home/leslie/Paddle-Lite/lite/backends/opencl/cl_context.cc at line 47. Exiting.
    2024-11-06 14:05:56.150 31714-31737 libc com...e.lite.demo.yolov5n_detection A Fatal signal 6 (SIGABRT), code -6 in tid 31737 (GLThread 159)
@cmcamdy
Copy link
Contributor

cmcamdy commented Nov 7, 2024

可以提供下可用于复现的demo？看起来是内存不足导致的

@cmcamdy
Copy link
Contributor

cmcamdy commented Nov 7, 2024

或者试下更小的模型看下有没有相同的问题

@leslie2046
Copy link
Author

可以提供下可用于复现的demo？看起来是内存不足导致的

是发你APK还是demo代码

@cmcamdy
Copy link
Contributor

cmcamdy commented Nov 7, 2024

是发你APK还是demo代码

  • 最好是demo工程
  • 这边建议你先用参数小一点的下位模型试试能不能跑，如果正常了就说明模型太大了，如果还是报错再提供下此时的demo工程，以便这边复现分析

@leslie2046
Copy link
Author

是发你APK还是demo代码

* 最好是demo工程

* 这边建议你先用参数小一点的下位模型试试能不能跑，如果正常了就说明模型太大了，如果还是报错再提供下此时的demo工程，以便这边复现分析

好的

@leslie2046
Copy link
Author

是发你APK还是demo代码

* 最好是demo工程

* 这边建议你先用参数小一点的下位模型试试能不能跑，如果正常了就说明模型太大了，如果还是报错再提供下此时的demo工程，以便这边复现分析

现在有2个现象:

  1. 会时不时的卡住几秒后恢复
  2. 运行一会就会crash
    terminating with uncaught exception of type paddle::lite::PaddleLiteException: Paddle-Lite C++ Exception:
    [F 11/ 7 14: 7: 6.692 ...-Lite/lite/backends/opencl/cl_context.cc:47 AddKernel] OpenCL error with code CL_OUT_OF_RESOURCES happened in file /home/leslie/Paddle-Lite/lite/backends/opencl/cl_context.cc at line 47. Exiting.
    我在YOLOv5u-n（2.65M）的权重导出320/160尺寸后，以及YOLOv10-n（2.3M）的权重导出320尺寸的nb都会复现这2个问题。
    这3个nb都在工程里面。
    demo工程下载链接：http://60.174.141.59:5000/robot/APK/yolo_detection_demo.zip

@leslie2046 leslie2046 mentioned this issue Nov 7, 2024
Open
@leslie2046
Copy link
Author

leslie2046 commented Nov 7, 2024 •
edited
Loading

然后新情况，如果用YOLOv5-n（1.87M）导出的nb,长时间测试还没发现crash，但是也会时不时的卡住几秒后恢复
yolov5_n_300e_320_coco_2.14rc_gpu.zip

@cmcamdy
Copy link
Contributor

cmcamdy commented Nov 8, 2024

试试编译库的时候加上 LITE_WITH_PROFILE，看看那个算子卡了？

@leslie2046
Copy link
Author

leslie2046 commented Nov 8, 2024 •
edited
Loading

4-18330 Paddle-Lite com...e.lite.demo.yolov5n_detection I [3 11/ 8 13:48: 5.102 ...-Lite/lite/backends/opencl/cl_context.cc:59 GetKernel] --- kernel count: 1233 ---
2024-11-08 13:48:05.102 18304-18330 Paddle-Lite com...e.lite.demo.yolov5n_detection I [4 11/ 8 13:48: 5.102 ...e/kernels/opencl/concat_image_compute.cc:182 Run] concat input shape:
2024-11-08 13:48:05.102 18304-18330 Paddle-Lite com...e.lite.demo.yolov5n_detection I [4 11/ 8 13:48: 5.102 ...e/kernels/opencl/concat_image_compute.cc:184 Run] inputs [0] dims: {1600,2}
2024-11-08 13:48:05.102 18304-18330 Paddle-Lite com...e.lite.demo.yolov5n_detection I [4 11/ 8 13:48: 5.102 ...e/kernels/opencl/concat_image_compute.cc:184 Run] inputs [1] dims: {400,2}
2024-11-08 13:48:05.102 18304-18330 Paddle-Lite com...e.lite.demo.yolov5n_detection I [4 11/ 8 13:48: 5.102 ...e/kernels/opencl/concat_image_compute.cc:184 Run] inputs [2] dims: {100,2}
2024-11-08 13:48:05.102 18304-18330 Paddle-Lite com...e.lite.demo.yolov5n_detection I [4 11/ 8 13:48: 5.102 ...e/kernels/opencl/concat_image_compute.cc:188 Run] concat output shape:
2024-11-08 13:48:05.102 18304-18330 Paddle-Lite com...e.lite.demo.yolov5n_detection I [4 11/ 8 13:48: 5.102 ...e/kernels/opencl/concat_image_compute.cc:189 Run] out dims: {2100,2}
2024-11-08 13:48:05.102 18304-18330 Paddle-Lite com...e.lite.demo.yolov5n_detection I [4 11/ 8 13:48: 5.102 ...e/kernels/opencl/concat_image_compute.cc:190 Run] axis_: 0
2024-11-08 13:48:05.102 18304-18330 Paddle-Lite com...e.lite.demo.yolov5n_detection I [4 11/ 8 13:48: 5.102 ...e/kernels/opencl/concat_image_compute.cc:191 Run] flag_: 2
2024-11-08 13:48:05.102 18304-18330 Paddle-Lite com...e.lite.demo.yolov5n_detection I [4 11/ 8 13:48: 5.102 ...e/kernels/opencl/concat_image_compute.cc:192 Run] align_: false
2024-11-08 13:48:05.102 18304-18330 Paddle-Lite com...e.lite.demo.yolov5n_detection I [4 11/ 8 13:48: 5.102 ...e/kernels/opencl/concat_image_compute.cc:194 Run] opencl
2024-11-08 13:48:05.103 18304-18330 Paddle-Lite com...e.lite.demo.yolov5n_detection I [4 11/ 8 13:48: 5.102 ...e/kernels/opencl/concat_image_compute.cc:195 Run] output_image_shape(w,h): 2 2100
2024-11-08 13:48:05.103 18304-18330 Paddle-Lite com...e.lite.demo.yolov5n_detection I [4 11/ 8 13:48: 5.103 ...e/kernels/opencl/concat_image_compute.cc:197 Run] output_tensor_w: 2
2024-11-08 13:48:05.103 18304-18330 Paddle-Lite com...e.lite.demo.yolov5n_detection I [4 11/ 8 13:48: 5.103 ...e/kernels/opencl/concat_image_compute.cc:198 Run] global_work_size: 1 2 2100
2024-11-08 13:48:05.103 18304-18330 Paddle-Lite com...e.lite.demo.yolov5n_detection I [4 11/ 8 13:48: 5.103 ...e/kernels/opencl/concat_image_compute.cc:200 Run] kernel_func_name: Concat3InputAxis2
2024-11-08 13:48:05.103 18304-18330 Paddle-Lite com...e.lite.demo.yolov5n_detection I [3 11/ 8 13:48: 5.103 ...-Lite/lite/backends/opencl/cl_context.cc:59 GetKernel] --- kernel count: 1233 ---
2024-11-08 13:48:05.103 18304-18330 Paddle-Lite com...e.lite.demo.yolov5n_detection I [3 11/ 8 13:48: 5.103 ...-Lite/lite/backends/opencl/cl_context.cc:59 GetKernel] --- kernel count: 1233 ---
2024-11-08 13:48:05.104 18304-18330 Paddle-Lite com...e.lite.demo.yolov5n_detection I [3 11/ 8 13:48: 5.104 ...-Lite/lite/backends/opencl/cl_context.cc:59 GetKernel] --- kernel count: 1233 --

似乎每次都是卡在这

@leslie2046
Copy link
Author

2024-11-08 14:41:00.138 18304-18330 Paddle-Lite com...e.lite.demo.yolov5n_detection I [3 11/ 8 14:41: 0.138 ...-Lite/lite/backends/opencl/cl_context.cc:59 GetKernel] --- kernel count: 18474 ---
2024-11-08 14:41:00.138 18304-18330 Paddle-Lite com...e.lite.demo.yolov5n_detection I [3 11/ 8 14:41: 0.138 ...-Lite/lite/backends/opencl/cl_context.cc:59 GetKernel] --- kernel count: 18474 ---
2024-11-08 14:41:00.139 18304-18330 Paddle-Lite com...e.lite.demo.yolov5n_detection I [3 11/ 8 14:41: 0.139 ...-Lite/lite/backends/opencl/cl_context.cc:59 GetKernel] --- kernel count: 18474 ---
2024-11-08 14:41:00.139 18304-18330 Paddle-Lite com...e.lite.demo.yolov5n_detection I [3 11/ 8 14:41: 0.139 ...-Lite/lite/backends/opencl/cl_context.cc:59 GetKernel] --- kernel count: 18474 ---
2024-11-08 14:41:00.140 18304-18330 Paddle-Lite com...e.lite.demo.yolov5n_detection I [3 11/ 8 14:41: 0.140 ...-Lite/lite/backends/opencl/cl_context.cc:59 GetKernel] --- kernel count: 18474 ---
2024-11-08 14:41:00.140 18304-18330 Paddle-Lite com...e.lite.demo.yolov5n_detection I [3 11/ 8 14:41: 0.140 ...-Lite/lite/backends/opencl/cl_context.cc:59 GetKernel] --- kernel count: 18474 ---
2024-11-08 14:41:00.140 18304-18330 Paddle-Lite com...e.lite.demo.yolov5n_detection I [4 11/ 8 14:41: 0.140 .../kernels/opencl/reshape_image_compute.cc:71 Run] out_dims= {100,2}
2024-11-08 14:41:00.141 18304-18330 Paddle-Lite com...e.lite.demo.yolov5n_detection I [3 11/ 8 14:41: 0.141 ...-Lite/lite/backends/opencl/cl_context.cc:59 GetKernel] --- kernel count: 18474 ---
2024-11-08 14:41:00.141 18304-18330 Paddle-Lite com...e.lite.demo.yolov5n_detection I [4 11/ 8 14:41: 0.141 ...e/kernels/opencl/concat_image_compute.cc:182 Run] concat input shape:
2024-11-08 14:41:00.141 18304-18330 Paddle-Lite com...e.lite.demo.yolov5n_detection I [4 11/ 8 14:41: 0.141 ...e/kernels/opencl/concat_image_compute.cc:184 Run] inputs [0] dims: {1600,2}
2024-11-08 14:41:00.141 18304-18330 Paddle-Lite com...e.lite.demo.yolov5n_detection I [4 11/ 8 14:41: 0.141 ...e/kernels/opencl/concat_image_compute.cc:184 Run] inputs [1] dims: {400,2}
2024-11-08 14:41:00.141 18304-18330 Paddle-Lite com...e.lite.demo.yolov5n_detection I [4 11/ 8 14:41: 0.141 ...e/kernels/opencl/concat_image_compute.cc:184 Run] inputs [2] dims: {100,2}
2024-11-08 14:41:00.141 18304-18330 Paddle-Lite com...e.lite.demo.yolov5n_detection I [4 11/ 8 14:41: 0.141 ...e/kernels/opencl/concat_image_compute.cc:188 Run] concat output shape:
2024-11-08 14:41:00.141 18304-18330 Paddle-Lite com...e.lite.demo.yolov5n_detection I [4 11/ 8 14:41: 0.141 ...e/kernels/opencl/concat_image_compute.cc:189 Run] out dims: {2100,2}
2024-11-08 14:41:00.141 18304-18330 Paddle-Lite com...e.lite.demo.yolov5n_detection I [4 11/ 8 14:41: 0.141 ...e/kernels/opencl/concat_image_compute.cc:190 Run] axis_: 0
2024-11-08 14:41:00.141 18304-18330 Paddle-Lite com...e.lite.demo.yolov5n_detection I [4 11/ 8 14:41: 0.141 ...e/kernels/opencl/concat_image_compute.cc:191 Run] flag_: 2
2024-11-08 14:41:00.141 18304-18330 Paddle-Lite com...e.lite.demo.yolov5n_detection I [4 11/ 8 14:41: 0.141 ...e/kernels/opencl/concat_image_compute.cc:192 Run] align_: false
2024-11-08 14:41:00.141 18304-18330 Paddle-Lite com...e.lite.demo.yolov5n_detection I [4 11/ 8 14:41: 0.141 ...e/kernels/opencl/concat_image_compute.cc:194 Run] opencl
2024-11-08 14:41:00.141 18304-18330 Paddle-Lite com...e.lite.demo.yolov5n_detection I [4 11/ 8 14:41: 0.141 ...e/kernels/opencl/concat_image_compute.cc:195 Run] output_image_shape(w,h): 2 2100
2024-11-08 14:41:00.141 18304-18330 Paddle-Lite com...e.lite.demo.yolov5n_detection I [4 11/ 8 14:41: 0.141 ...e/kernels/opencl/concat_image_compute.cc:197 Run] output_tensor_w: 2
2024-11-08 14:41:00.141 18304-18330 Paddle-Lite com...e.lite.demo.yolov5n_detection I [4 11/ 8 14:41: 0.141 ...e/kernels/opencl/concat_image_compute.cc:198 Run] global_work_size: 1 2 2100
2024-11-08 14:41:00.141 18304-18330 Paddle-Lite com...e.lite.demo.yolov5n_detection I [4 11/ 8 14:41: 0.141 ...e/kernels/opencl/concat_image_compute.cc:200 Run] kernel_func_name: Concat3InputAxis2
2024-11-08 14:41:00.141 18304-18330 Paddle-Lite com...e.lite.demo.yolov5n_detection I [3 11/ 8 14:41: 0.141 ...-Lite/lite/backends/opencl/cl_context.cc:59 GetKernel] --- kernel count: 18474 ---
2024-11-08 14:41:00.142 18304-18330 Paddle-Lite com...e.lite.demo.yolov5n_detection I [3 11/ 8 14:41: 0.142 ...-Lite/lite/backends/opencl/cl_context.cc:59 GetKernel] --- kernel count: 18474 ---
2024-11-08 14:41:00.142 18304-18330 Paddle-Lite com...e.lite.demo.yolov5n_detection I [3 11/ 8 14:41: 0.142 ...-Lite/lite/backends/opencl/cl_context.cc:59 GetKernel] --- kernel count: 18474 ---
2024-11-08 14:41:00.143 18304-18330 Paddle-Lite com...e.lite.demo.yolov5n_detection I [3 11/ 8 14:41: 0.142 ...-Lite/lite/backends/opencl/cl_context.cc:59 GetKernel] --- kernel count: 18474 ---
2024-11-08 14:41:00.143 18304-18330 Paddle-Lite com...e.lite.demo.yolov5n_detection I [4 11/ 8 14:41: 0.143 ...e/kernels/opencl/concat_image_compute.cc:182 Run] concat input shape:
2024-11-08 14:41:00.143 18304-18330 Paddle-Lite com...e.lite.demo.yolov5n_detection I [4 11/ 8 14:41: 0.143 ...e/kernels/opencl/concat_image_compute.cc:184 Run] inputs [0] dims: {1600,1}
2024-11-08 14:41:00.143 18304-18330 Paddle-Lite com...e.lite.demo.yolov5n_detection I [4 11/ 8 14:41: 0.143 ...e/kernels/opencl/concat_image_compute.cc:184 Run] inputs [1] dims: {400,1}
2024-11-08 14:41:00.143 18304-18330 Paddle-Lite com...e.lite.demo.yolov5n_detection I [4 11/ 8 14:41: 0.143 ...e/kernels/opencl/concat_image_compute.cc:184 Run] inputs [2] dims: {100,1}
2024-11-08 14:41:00.143 18304-18330 Paddle-Lite com...e.lite.demo.yolov5n_detection I [4 11/ 8 14:41: 0.143 ...e/kernels/opencl/concat_image_compute.cc:188 Run] concat output shape:
2024-11-08 14:41:00.143 18304-18330 Paddle-Lite com...e.lite.demo.yolov5n_detection I [4 11/ 8 14:41: 0.143 ...e/kernels/opencl/concat_image_compute.cc:189 Run] out dims: {2100,1}
2024-11-08 14:41:00.143 18304-18330 Paddle-Lite com...e.lite.demo.yolov5n_detection I [4 11/ 8 14:41: 0.143 ...e/kernels/opencl/concat_image_compute.cc:190 Run] axis_: 0
2024-11-08 14:41:00.143 18304-18330 Paddle-Lite com...e.lite.demo.yolov5n_detection I [4 11/ 8 14:41: 0.143 ...e/kernels/opencl/concat_image_compute.cc:191 Run] flag_: 2
2024-11-08 14:41:00.143 18304-18330 Paddle-Lite com...e.lite.demo.yolov5n_detection I [4 11/ 8 14:41: 0.143 ...e/kernels/opencl/concat_image_compute.cc:192 Run] align_: false
2024-11-08 14:41:00.143 18304-18330 Paddle-Lite com...e.lite.demo.yolov5n_detection I [4 11/ 8 14:41: 0.143 ...e/kernels/opencl/concat_image_compute.cc:194 Run] opencl
2024-11-08 14:41:00.143 18304-18330 Paddle-Lite com...e.lite.demo.yolov5n_detection I [4 11/ 8 14:41: 0.143 ...e/kernels/opencl/concat_image_compute.cc:195 Run] output_image_shape(w,h): 1 2100
2024-11-08 14:41:00.143 18304-18330 Paddle-Lite com...e.lite.demo.yolov5n_detection I [4 11/ 8 14:41: 0.143 ...e/kernels/opencl/concat_image_compute.cc:197 Run] output_tensor_w: 1
2024-11-08 14:41:00.143 18304-18330 Paddle-Lite com...e.lite.demo.yolov5n_detection I [4 11/ 8 14:41: 0.143 ...e/kernels/opencl/concat_image_compute.cc:198 Run] global_work_size: 1 1 2100
2024-11-08 14:41:00.143 18304-18330 Paddle-Lite com...e.lite.demo.yolov5n_detection I [4 11/ 8 14:41: 0.143 ...e/kernels/opencl/concat_image_compute.cc:200 Run] kernel_func_name: Concat3InputAxis2
2024-11-08 14:41:00.143 18304-18330 Paddle-Lite com...e.lite.demo.yolov5n_detection I [3 11/ 8 14:41: 0.143 ...-Lite/lite/backends/opencl/cl_context.cc:59 GetKernel] --- kernel count: 18474 ---
2024-11-08 14:41:00.144 18304-18330 Paddle-Lite com...e.lite.demo.yolov5n_detection I [4 11/ 8 14:41: 0.144 .../kernels/opencl/reshape_image_compute.cc:71 Run] out_dims= {1,4,16,1600}
2024-11-08 14:41:00.145 18304-18330 Paddle-Lite com...e.lite.demo.yolov5n_detection I [3 11/ 8 14:41: 0.145 ...-Lite/lite/backends/opencl/cl_context.cc:59 GetKernel] --- kernel count: 18474 ---
2024-11-08 14:41:00.145 18304-18330 Paddle-Lite com...e.lite.demo.yolov5n_detection I [4 11/ 8 14:41: 0.145 .../kernels/opencl/softmax_image_compute.cc:92 Run] extend_in_dims: {1,1600,4,16}
2024-11-08 14:41:00.145 18304-18330 Paddle-Lite com...e.lite.demo.yolov5n_detection I [3 11/ 8 14:41: 0.145 ...-Lite/lite/backends/opencl/cl_context.cc:38 AddKernel] --- to get program image/matmul_kernel.cl ---
2024-11-08 14:41:00.145 18304-18330 Paddle-Lite com...e.lite.demo.yolov5n_detection I [4 11/ 8 14:41: 0.145 ...-Lite/lite/backends/opencl/cl_runtime.cc:156 GetProgram] precision_: CL_PRECISION_FP16
2024-11-08 14:41:00.145 18304-18330 Paddle-Lite com...e.lite.demo.yolov5n_detection I [4 11/ 8 14:41: 0.145 ...-Lite/lite/backends/opencl/cl_runtime.cc:157 GetProgram] OpenCL build_option: -cl-fast-relaxed-math -cl-mad-enable -DCL_DTYPE_half
2024-11-08 14:41:00.145 18304-18330 Paddle-Lite com...e.lite.demo.yolov5n_detection I [3 11/ 8 14:41: 0.145 ...-Lite/lite/backends/opencl/cl_runtime.cc:185 CheckFromCache] --- program -> image/matmul_kernel.cl -cl-fast-relaxed-math -cl-mad-enable -DCL_DTYPE_half has been built in cache ---
2024-11-08 14:41:00.145 18304-18330 Paddle-Lite com...e.lite.demo.yolov5n_detection I [3 11/ 8 14:41: 0.145 ...-Lite/lite/backends/opencl/cl_context.cc:42 AddKernel] --- end get program ---
2024-11-08 14:41:00.145 18304-18330 Paddle-Lite com...e.lite.demo.yolov5n_detection I [3 11/ 8 14:41: 0.145 ...-Lite/lite/backends/opencl/cl_context.cc:43 AddKernel] --- to create kernel: matmul_highdimx_ydim2 ---
2024-11-08 14:41:00.145 18304-18330 Paddle-Lite com...e.lite.demo.yolov5n_detection I [3 11/ 8 14:41: 0.145 ...-Lite/lite/backends/opencl/cl_context.cc:49 AddKernel] --- end create kernel ---
2024-11-08 14:41:00.145 18304-18330 Paddle-Lite com...e.lite.demo.yolov5n_detection I [3 11/ 8 14:41: 0.145 ...-Lite/lite/backends/opencl/cl_context.cc:59 GetKernel] --- kernel count: 18475 ---
2024-11-08 14:41:00.145 18304-18330 Paddle-Lite com...e.lite.demo.yolov5n_detection I [4 11/ 8 14:41: 0.145 ...e/kernels/opencl/matmul_image_compute.cc:486 SetGlobalLocalWorkSize] local_work_size[3D]: 32 8 1
2024-11-08 14:41:00.145 18304-18330 Paddle-Lite com...e.lite.demo.yolov5n_detection I [4 11/ 8 14:41: 0.145 ...e/kernels/opencl/matmul_image_compute.cc:488 SetGlobalLocalWorkSize] global_work_size[3D]: 32 400 4
2024-11-08 14:41:00.146 18304-18330 Paddle-Lite com...e.lite.demo.yolov5n_detection I [3 11/ 8 14:41: 0.146 ...-Lite/lite/backends/opencl/cl_context.cc:59 GetKernel] --- kernel count: 18475 ---
2024-11-08 14:41:00.146 18304-18330 Paddle-Lite com...e.lite.demo.yolov5n_detection I [4 11/ 8 14:41: 0.146 ...encl/squeeze_unsqueeze_buffer_compute.cc:65 Run] squeeze out dims: {1,1600,4}
2024-11-08 14:41:00.146 18304-18330 Paddle-Lite com...e.lite.demo.yolov5n_detection I [4 11/ 8 14:41: 0.146 ...encl/squeeze_unsqueeze_buffer_compute.cc:66 Run] out->memory_size():12800
2024-11-08 14:41:00.146 18304-18330 Paddle-Lite com...e.lite.demo.yolov5n_detection I [3 11/ 8 14:41: 0.146 ...-Lite/lite/backends/opencl/cl_context.cc:59 GetKernel] --- kernel count: 18475 ---
2024-11-08 14:41:00.146 18304-18330 Paddle-Lite com...e.lite.demo.yolov5n_detection I [4 11/ 8 14:41: 0.146 .../kernels/opencl/reshape_image_compute.cc:71 Run] out_dims= {1,4,40,40}
2024-11-08 14:41:00.146 18304-18330 Paddle-Lite com...e.lite.demo.yolov5n_detection I [3 11/ 8 14:41: 0.146 ...-Lite/lite/backends/opencl/cl_context.cc:59 GetKernel] --- kernel count: 18475 ---
2024-11-08 14:41:00.147 18304-18330 Paddle-Lite com...e.lite.demo.yolov5n_detection I [4 11/ 8 14:41: 0.147 .../kernels/opencl/reshape_image_compute.cc:71 Run] out_dims= {1,4,16,400}
2024-11-08 14:41:00.147 18304-18330 Paddle-Lite com...e.lite.demo.yolov5n_detection I [3 11/ 8 14:41: 0.147 ...-Lite/lite/backends/opencl/cl_context.cc:59 GetKernel] --- kernel count: 18475 ---
2024-11-08 14:41:00.148 18304-18330 Paddle-Lite com...e.lite.demo.yolov5n_detection I [4 11/ 8 14:41: 0.148 .../kernels/opencl/softmax_image_compute.cc:92 Run] extend_in_dims: {1,400,4,16}
2024-11-08 14:41:00.148 18304-18330 Paddle-Lite com...e.lite.demo.yolov5n_detection I [3 11/ 8 14:41: 0.148 ...-Lite/lite/backends/opencl/cl_context.cc:38 AddKernel] --- to get program image/matmul_kernel.cl ---
2024-11-08 14:41:00.148 18304-18330 Paddle-Lite com...e.lite.demo.yolov5n_detection I [4 11/ 8 14:41: 0.148 ...-Lite/lite/backends/opencl/cl_runtime.cc:156 GetProgram] precision_: CL_PRECISION_FP16
2024-11-08 14:41:00.148 18304-18330 Paddle-Lite com...e.lite.demo.yolov5n_detection I [4 11/ 8 14:41: 0.148 ...-Lite/lite/backends/opencl/cl_runtime.cc:157 GetProgram] OpenCL build_option: -cl-fast-relaxed-math -cl-mad-enable -DCL_DTYPE_half
2024-11-08 14:41:00.148 18304-18330 Paddle-Lite com...e.lite.demo.yolov5n_detection I [3 11/ 8 14:41: 0.148 ...-Lite/lite/backends/opencl/cl_runtime.cc:185 CheckFromCache] --- program -> image/matmul_kernel.cl -cl-fast-relaxed-math -cl-mad-enable -DCL_DTYPE_half has been built in cache ---
2024-11-08 14:41:00.148 18304-18330 Paddle-Lite com...e.lite.demo.yolov5n_detection I [3 11/ 8 14:41: 0.148 ...-Lite/lite/backends/opencl/cl_context.cc:42 AddKernel] --- end get program ---
2024-11-08 14:41:00.148 18304-18330 Paddle-Lite com...e.lite.demo.yolov5n_detection I [3 11/ 8 14:41: 0.148 ...-Lite/lite/backends/opencl/cl_context.cc:43 AddKernel] --- to create kernel: matmul_highdimx_ydim2 ---
2024-11-08 14:41:00.148 18304-18330 Paddle-Lite com...e.lite.demo.yolov5n_detection I [3 11/ 8 14:41: 0.148 ...-Lite/lite/backends/opencl/cl_context.cc:49 AddKernel] --- end create kernel ---
2024-11-08 14:41:00.148 18304-18330 Paddle-Lite com...e.lite.demo.yolov5n_detection I [3 11/ 8 14:41: 0.148 ...-Lite/lite/backends/opencl/cl_context.cc:59 GetKernel] --- kernel count: 18476 ---
2024-11-08 14:41:00.148 18304-18330 Paddle-Lite com...e.lite.demo.yolov5n_detection I [4 11/ 8 14:41: 0.148 ...e/kernels/opencl/matmul_image_compute.cc:486 SetGlobalLocalWorkSize] local_work_size[3D]: 32 8 1
2024-11-08 14:41:00.148 18304-18330 Paddle-Lite com...e.lite.demo.yolov5n_detection I [4 11/ 8 14:41: 0.148 ...e/kernels/opencl/matmul_image_compute.cc:488 SetGlobalLocalWorkSize] global_work_size[3D]: 32 104 4
2024-11-08 14:41:00.148 18304-18330 Paddle-Lite com...e.lite.demo.yolov5n_detection I [3 11/ 8 14:41: 0.148 ...-Lite/lite/backends/opencl/cl_context.cc:59 GetKernel] --- kernel count: 18476 ---
2024-11-08 14:41:00.149 18304-18330 Paddle-Lite com...e.lite.demo.yolov5n_detection I [4 11/ 8 14:41: 0.149 ...encl/squeeze_unsqueeze_buffer_compute.cc:65 Run] squeeze out dims: {1,400,4}
2024-11-08 14:41:00.149 18304-18330 Paddle-Lite com...e.lite.demo.yolov5n_detection I [4 11/ 8 14:41: 0.149 ...encl/squeeze_unsqueeze_buffer_compute.cc:66 Run] out->memory_size():3200
2024-11-08 14:41:00.149 18304-18330 Paddle-Lite com...e.lite.demo.yolov5n_detection I [3 11/ 8 14:41: 0.149 ...-Lite/lite/backends/opencl/cl_context.cc:59 GetKernel] --- kernel count: 18476 ---
2024-11-08 14:41:00.149 18304-18330 Paddle-Lite com...e.lite.demo.yolov5n_detection I [4 11/ 8 14:41: 0.149 .../kernels/opencl/reshape_image_compute.cc:71 Run] out_dims= {1,4,20,20}
2024-11-08 14:41:00.149 18304-18330 Paddle-Lite com...e.lite.demo.yolov5n_detection I [3 11/ 8 14:41: 0.149 ...-Lite/lite/backends/opencl/cl_context.cc:59 GetKernel] --- kernel count: 18476 ---
2024-11-08 14:41:00.150 18304-18330 Paddle-Lite com...e.lite.demo.yolov5n_detection I [4 11/ 8 14:41: 0.150 .../kernels/opencl/reshape_image_compute.cc:71 Run] out_dims= {1,4,16,100}
2024-11-08 14:41:00.150 18304-18330 Paddle-Lite com...e.lite.demo.yolov5n_detection I [3 11/ 8 14:41: 0.150 ...-Lite/lite/backends/opencl/cl_context.cc:59 GetKernel] --- kernel count: 18476 ---
2024-11-08 14:41:00.150 18304-18330 Paddle-Lite com...e.lite.demo.yolov5n_detection I [4 11/ 8 14:41: 0.150 .../kernels/opencl/softmax_image_compute.cc:92 Run] extend_in_dims: {1,100,4,16}
2024-11-08 14:41:00.150 18304-18330 Paddle-Lite com...e.lite.demo.yolov5n_detection I [3 11/ 8 14:41: 0.150 ...-Lite/lite/backends/opencl/cl_context.cc:38 AddKernel] --- to get program image/matmul_kernel.cl ---
2024-11-08 14:41:00.150 18304-18330 Paddle-Lite com...e.lite.demo.yolov5n_detection I [4 11/ 8 14:41: 0.150 ...-Lite/lite/backends/opencl/cl_runtime.cc:156 GetProgram] precision_: CL_PRECISION_FP16
2024-11-08 14:41:00.150 18304-18330 Paddle-Lite com...e.lite.demo.yolov5n_detection I [4 11/ 8 14:41: 0.150 ...-Lite/lite/backends/opencl/cl_runtime.cc:157 GetProgram] OpenCL build_option: -cl-fast-relaxed-math -cl-mad-enable -DCL_DTYPE_half
2024-11-08 14:41:00.150 18304-18330 Paddle-Lite com...e.lite.demo.yolov5n_detection I [3 11/ 8 14:41: 0.150 ...-Lite/lite/backends/opencl/cl_runtime.cc:185 CheckFromCache] --- program -> image/matmul_kernel.cl -cl-fast-relaxed-math -cl-mad-enable -DCL_DTYPE_half has been built in cache ---
2024-11-08 14:41:00.150 18304-18330 Paddle-Lite com...e.lite.demo.yolov5n_detection I [3 11/ 8 14:41: 0.150 ...-Lite/lite/backends/opencl/cl_context.cc:42 AddKernel] --- end get program ---
2024-11-08 14:41:00.151 18304-18330 Paddle-Lite com...e.lite.demo.yolov5n_detection I [3 11/ 8 14:41: 0.150 ...-Lite/lite/backends/opencl/cl_context.cc:43 AddKernel] --- to create kernel: matmul_highdimx_ydim2 ---
2024-11-08 14:41:00.151 18304-18330 Paddle-Lite com...e.lite.demo.yolov5n_detection A [F 11/ 8 14:41: 0.151 ...-Lite/lite/backends/opencl/cl_context.cc:47 AddKernel] OpenCL error with code CL_OUT_OF_RESOURCES happened in file /home/leslie/Paddle-Lite/lite/backends/opencl/cl_context.cc at line 47. Exiting.
--------- beginning of crash
2024-11-08 14:41:00.152 18304-18330 libc++abi com...e.lite.demo.yolov5n_detection E terminating with uncaught exception of type paddle::lite::PaddleLiteException: Paddle-Lite C++ Exception:
[F 11/ 8 14:41: 0.151 ...-Lite/lite/backends/opencl/cl_context.cc:47 AddKernel] OpenCL error with code CL_OUT_OF_RESOURCES happened in file /home/leslie/Paddle-Lite/lite/backends/opencl/cl_context.cc at line 47. Exiting.

这些是crash前的日志

@cmcamdy
Copy link
Contributor

cmcamdy commented Nov 8, 2024 •
edited
Loading

你这个是原来大模型crash的日志是吗？

@leslie2046
Copy link
Author

你这个是原来大模型crash的日志是吗？

* 看这里的代码，就是分配资源的时候不足了，https://github.com/PaddlePaddle/Paddle-Lite/blob/c8097755a303ce01a6ec3fa150c3851f88d64096/lite/backends/opencl/cl_context.cc#L45C1-L47C26

是的。是稍大些的YOLOv5u-n（2.65M）以及YOLOv10-n（2.3M）崩溃的日志，有办法解决吗

@leslie2046
Copy link
Author

@cmcamdy 关于卡顿有什么解决办法吗，什么原因导致的

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@leslie2046 @cmcamdy