-
|
oneDNN implemented most of the hard work for CNN inference, and major DL frameworks are all based on oneDNN, including OpenVINO, but OpenVINO has better performance than other DL framework, why? Tickets: 67678 |
Beta Was this translation helpful? Give feedback.
Replies: 2 comments 5 replies
-
|
Based Tickets 67678, we can answer this by comparing inference performance of some famous networks between OpenVINO and PaddlePaddle. Compile Paddle:https://www.paddlepaddle.org.cn/documentation/docs/zh/1.5/beginners_guide/install/compile/compile_Ubuntu.html 编译飞桨过程中可能会打开很多文件，Ubuntu 18.04 默认设置最多同时打开的文件数是1024（参见 ulimit -a），需要更改这个设定值。 Inference using PaddleThere is a repo for inference using Paddle git clone https://github.com/PaddlePaddle/Paddle-Inference-Demo.git
cd Paddle-Inference-Demo/c++/lib
# create paddle_inference softlink
ln -sf ~/paddle-venv/Paddle/_build/paddle_inference_install_dir paddle_inference
cd resnet50
# change compile.sh: `WITH_GPU=OFF`
./compile.sh
# run.sh will download/extract/run resnet50 model (folder resnet50)
chmod +x ./run.sh
./run.sh
# we can re-trigger the test manually with different config
./build/resnet50_test --model_file resnet50/inference.pdmodel --params_file resnet50/inference.pdiparams --repeats 10 --batch_size 10But only one core(and one thread) is used in the whole process. on average one image take 84 ms to infer on paddle resnet50. Compile and install OpenVINO:
# use virtual env to install mo's python dependency
virtualenv ov
. ./ov/bin/activate
cd ~/openvino/_build/install/tools/model_optimizer/install_prerequisites/
./install_prerequisites_caffe.sh
./install_prerequisites_onnx.sh
./install_prerequisites_tf.shInference using OpenVINOPrepare: https://github.com/openvinotoolkit/open_model_zoo/blob/master/tools/model_tools/README.md git clone [email protected]:openvinotoolkit/open_model_zoo.git
cd open_model_zoo/tools/downloader
pip install -r ./requirements-pytorch.in
pip install -r ./requirements-caffe2.in
pip install -r ./requirements-tensorflow.inDownload and convert model cd open_model_zoo/tools/downloader
./downloader.py --name resnet-50-pytorch
./converter.py --name resnet-50-pytorch --precisions FP32Inference with benchmark_app: $ ./benchmark_app -m /home/hddl/open_model_zoo/tools/downloader/public/resnet-50-pytorch/FP32/resnet-50-pytorch.xml -d CPU
Count: 2804 iterations
Duration: 60108.46 ms
Latency: 81.84 ms
Throughput: 46.65 FPSsince benchmark_app use all 4 cores(8 hyper threads), the throughput is roughly (4*1000/81.84). $ ./benchmark_app -m /home/hddl/open_model_zoo/tools/downloader/public/resnet-50-pytorch/FP32/resnet-50-pytorch.xml -d CPU -nstreams=1 -nthreads=1
Count: 741 iterations
Duration: 60143.24 ms
Latency: 80.29 ms <=================Paddle takes 84ms
Throughput: 12.32 FPSLuocheng's method:
to show performance: utils.py:82+ config.enable_profile() |
Beta Was this translation helpful? Give feedback.
-
|
for resnet50 test, adding env setting DNNL_VERBOSE=2 shows that Paddle also use channel blocked "aBcd8b" layout inference, that's also why it's performance is comparable to OpenVINO |
Beta Was this translation helpful? Give feedback.
-
MobileNetV2
Compare smaller pre-trained models: PaddlePaddle: We need install paddlepaddle python package in order to use https://github.com/PaddlePaddle/models $ pip install python/dist/paddlepaddle-2.1.1-cp36-cp36m-linux_x86_64.whlconvert pre-trained model to binary format (model & params): $ git clone https://github.com/PaddlePaddle/models.git
$ cd models/PaddleCV/image_classification
$ python ./infer.py --model=MobileNetV2 --pretrained_model=/home/hddl/paddle-venv/pre/MobileNetV2_pretrained --save_inference=True --use_gpu=False
$ tree ./MobileNetV2/
MobileNetV2/
├── model
└── params
Then we can test it with the same C++ app(since input is still of shape 1x3x224x224) $ ./build/resnet50_test -model_file=/home/hddl/paddle-venv/models/PaddleCV/image_classification/MobileNetV2/model -params_file=/home/hddl/paddle-venv/models/PaddleCV/image_classification/MobileNetV2/params --repeats 100 --batch_size=1 --warmup=10
run avg time is 14.8 ms
...OpenVino: $ ./benchmark_app -m /home/hddl/open_model_zoo/tools/downloader/public/mobilenet-v2-pytorch/FP32/mobilenet-v2-pytorch.xml -d CPU -nthreads 1 -nstreams 1 -niter=1000
Count: 1000 iterations
Duration: 12750.75 ms
Latency: 12.66 ms
Throughput: 78.43 FPSComparing DNNL_VERBOSE=2 logs, we found that paddle model has softmax but openvino model dosn't have. But we checked the code and modify infer.py so softmax is removed, but this only gives about 0.1ms improvements. so softmax isn't the key factor. if we switch to use https://github.com/openvinotoolkit/open_model_zoo/tree/master/models/public/mobilenet-v2 , than openvino would become slower |
Beta Was this translation helpful? Give feedback.
-
Root cause the performance differenceSince all framework is based on oneDNN, performance difference analysis can be done with DNNL_VERBOSE=2 logs: # OpenVINO use higher version of oneDNN, so log format is little different
python ./extract_log.py ~/logOpenVINO.txt attr-post-ops:
python ./extract_log.py ~/logPaddle.txt post_ops:we can see OpenVINO generated fewer oneDNN primitives than PaddlePaddle at the beginning of the network: OpenVINO
1: convolution jit:avx2 attr-post-ops:eltwise_clip:0:6
2: convolution jit_dw:avx2 attr-post-ops:eltwise_clip:0:6
3: convolution jit_1x1:avx2
4: convolution jit_1x1_with_dw_conv:avx2 attr-post-ops:eltwise_clip:0:6+depthwise_conv_old+eltwise_clip:0:6
PaddlePaddle
1: convolution jit:avx2 post_ops:'eltwise_bounded_relu:6;';
2: convolution jit_1x1:avx2 post_ops:'eltwise_bounded_relu:6;';
3: convolution jit_dw:avx2 post_ops:'eltwise_bounded_relu:6;';
4: convolution jit_1x1:avx2
5: convolution jit_1x1:avx2 post_ops:'eltwise_bounded_relu:6;';
6: convolution jit_dw:avx2 post_ops:'eltwise_bounded_relu:6;';search OpenVINO's code we saw we suspect that there is some transformation which explicit requires jit_1x1_with_dw_conv kernel. from mkldnn_graph_optimizer.cpp, we saw FuseConvolutionAndDWConvolution() seems to be the one who fuses convolution with another convolution, adding a log into the code we saw that this transformation is really worked once: Remove this transform from the code we got little drop in performance but still much better than PaddlePaddle,
PS: extract_log.py:import sys
src, pat = sys.argv[1:3]
lno = 1
with open(src) as file:
for line in file:
line = line.rstrip()
allfields = line.split(",")
match = False
for field in allfields:
if pat in field:
print(f'{lno}:\t{allfields[3]}\t{allfields[4]}\t{field}')
match = True
break
if not match:
print(f'{lno}:\t{allfields[3]}\t{allfields[4]}')
lno += 1 |
Beta Was this translation helpful? Give feedback.
-
Compare method revised for mobilenet v2a better way to compare performance: use the same pre-trained paddle model: prepare model$ git clone https://github.com/PaddlePaddle/PaddleClas.git
$ cd PaddleClas
$ pip install -r requirements.txt
# download MobileNetV2_pretrained.pdparam into pre folder
# export pdparam into inference model (note remove .pdparam extension)
$ python tools/export_model.py -c ppcls/configs/ImageNet/MobileNetV2/MobileNetV2.yaml -o Global.pretrained_model=../pre/MobileNetV2_pretrained
$ tree inference/
inference/
├── inference.pdiparams
├── inference.pdiparams.info
└── inference.pdmodel
# further export to OpenVINO IR
mo.py --input_model ~/paddle-venv/PaddleClas/inference/inference.pdmodel --output_dir ./mobilenet_v2_paddle -b 1 --framework=paddleperformance testOpenVINO is faster, and comparing DNNL_VERBOSE=2 logs we see that OpenVINO even do more works: a reorder at the begin of the network that convert u8 input to f32, a inner_product layer before softmax. Paddle has much more Cache miss than OpenVINOusing following command to check cache miss, Paddle shows 40+% cache-misses while OpenVINO only has 20+%. sudo perf stat -B -e cache-references,cache-misses,cycles,instructions,branches,faults,migrations,L1-dcache-load-misses,L1-dcache-loads,L1-dcache-stores,L1-icache-load-misses,LLC-loads,LLC-load-misses,LLC-stores,LLC-store-misses,LLC-prefetches /your/command/line/with/args
Performance counter stats for './benchmark_app -m /home/hddl/open_model_zoo/tools/downloader/mobilenet_v2_paddle/inference.xml -d CPU -nthreads 1 -nstreams 1 -niter=500 -nireq=1':
1,382,164,912 cache-references (22.78%)
312,128,676 cache-misses # 22.583 % of all cache refs (30.70%)
15,476,158,839 cycles (31.09%)
38,699,003,485 instructions # 2.50 insn per cycle (38.83%)
935,142,331 branches (39.10%)
21,863 faults
623 migrations
937,461,153 L1-dcache-load-misses # 6.26% of all L1-dcache hits (39.06%)
14,976,451,108 L1-dcache-loads (38.82%)
1,015,849,877 L1-dcache-stores (38.66%)
32,810,756 L1-icache-load-misses (30.69%)
80,287,697 LLC-loads (30.67%)
17,932,405 LLC-load-misses # 22.34% of all LL-cache hits (30.49%)
45,858,441 LLC-stores (14.96%)
7,744,306 LLC-store-misses (15.04%)
<not supported> LLC-prefetches
7.395575671 seconds time elapsed
7.397691000 seconds user
0.425246000 seconds sys
But we can do a simple experiment of comparing perf result between infer 1 frame and 100,200,300,... frames and check the delta: cache-misses(M) vs cache-references(M) delta increase per 100 repeats We can clearly see that OpenVINO does much better job at inference cache hit rate. Here is the test script we used: Does Paddle dynamically allocate temporary memory for each inference?Add following code into ConvMKLDNNOpKernel::ComputeFP32(), and from the log we saw:
std::cout << ctx.InputName("Input") + ctx.InputName("Filter") << ": "
<< src_memory_p->get_data_handle() << ","
<< weights_memory_p->get_data_handle() << ","
<< dst_memory_p->get_data_handle() << std::endl;after we change OpenVINO so it allocate memory dynamically cache miss rate increased dramatically(but the latency is only slightly increased by 0.5ms): So we know the relative high cache miss rate is not the key reason here. Is Paddle framework too heavy (executed too much instructions) ?By comparing Further profiling using vtune shows a considerable CPU cost is spent on std::hash & new in string constructor, call stack reveals a surprising fact, Paddle framework kernel we can workaround a few places by constructing a light-weight hash map caching the MKLDNN primitive using object address as key instead of string and it does bring dramatic improvements (about 1+ms), so it confirmed us that it's the careless design of Paddle framework causes the performance drop. OperatorWithKernel::RunImpl |
Beta Was this translation helpful? Give feedback.
-
Paddle framework is heavy for inferencesince Paddle reused same framework for both train & infer, it treats inference as same as train-forwarding pass but this limited the optimization can be done at framework level, OpenVINO, on the other hand, has no such burden, it's designed especially for inference so framework cost is much lower than Paddle. This framework level cost may be not a big deal for training since major computation dominate the whole latency, but it becomes a considerable cost when inference is done on light-weighted CNN like MobileNetV2, because in these CNNs, each convolution layer is executed extremely fast since they are depth-wise or 1x1, but framework cost keeps unchanged for all kinds of convolution. std::string is bad choice as key of mapPaddle framework uses std::string as key of map structure, even when caller actually passes a string literal as key. this requires run-time string construction and hashing operation. Performance analysis
$ sudo -E perf stat -B -e cache-references,cache-misses,cycles,instructions sleep 1
Performance counter stats for 'sleep 1':
65,532 cache-references
30,131 cache-misses # 45.979 % of all cache refs
1,329,247 cycles
1,050,063 instructions # 0.79 insn per cycle
1.001454689 seconds time elapsed
0.001199000 seconds user
0.000000000 seconds sys
$ sudo -E perf stat -B -e cache-references,cache-misses,cycles,instructions sleep 10
Performance counter stats for 'sleep 10':
77,823 cache-references
35,103 cache-misses # 45.106 % of all cache refs
1,515,698 cycles
1,050,428 instructions # 0.69 insn per cycle
10.001294553 seconds time elapsed
0.001234000 seconds user
0.000000000 seconds sys |
Beta Was this translation helpful? Give feedback.
-
|
Very impressive work! |
Beta Was this translation helpful? Give feedback.
Paddle framework is heavy for inference
since Paddle reused same framework for both train & infer, it treats inference as same as train-forwarding pass but this limited the optimization can be done at framework level, OpenVINO, on the other hand, has no such burden, it's designed especially for inference so framework cost is much lower than Paddle.
This framework level cost may be not a big deal for training since major computation dominate the whole latency, but it becomes a considerable cost when inference is done on light-weighted CNN like MobileNetV2, because in these CNNs, each convolution layer is executed extremely fast since they are depth-wise or 1x1, but framework cost keeps unc…