Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

benchdnn: graph: enhance input displace and shape rewrite for linked attribute and shapes #2354

Merged
merged 6 commits into from
Jan 22, 2025

Conversation

wzt1997
Copy link
Contributor

@wzt1997 wzt1997 commented Jan 8, 2025

Description

The PR enhances input displacement and rewrite functionality in benchdnn graph for the following aspects:

  1. Support mb rewrite on SRC1 of MatMul and scale and zp of DynamicDequantize to support SDPA patterns rewriting.
  2. Support shape rewrite for linked attributes and shapes, such as group_shape and scale/zp input of DynamicDequantize. If user provides shapes for one of the attributes or input shapes, benchdnn graph will update the other accordingly after performing some checks.
  3. Fix the data type setting for input displacement in case the primitive cannot be created, which solved two specific cases: f8_e4m3 cases as primitive creating might fail as f8_e4m3:f8_e4m3:f8_e5m2 is not supported, and bf16:int4:bf16 matmul cases where f32:int4:bf16 matmul is not supported.

For example:

# Rewrite input shape only
0:PASSED __REPRO: --graph --in-shapes=7:1x16x128x1+8:1x16x128x1 --case=/home/wangzhitao/oneDNN-src/tests/benchdnn/inputs/graph/complex_fusion/mha/sdpa-compressed-v-int8-gs32.json
# Rewrite group-shape only
1:PASSED __REPRO: --graph --op-attrs=34107656704:group_shape:1x1x128x1 --case=/home/wangzhitao/oneDNN-src/tests/benchdnn/inputs/graph/complex_fusion/mha/sdpa-compressed-k-int8-gs32.json
# Rewrite mb size
2:PASSED __REPRO: --graph --mb=10 --case=/home/wangzhitao/oneDNN-src/tests/benchdnn/inputs/graph/complex_fusion/mha/sdpa-compressed-v-int8-gs32.json

@wzt1997 wzt1997 self-assigned this Jan 8, 2025
@github-actions github-actions bot added component:graph-api Codeowner: @oneapi-src/onednn-graph component:tests Codeowner: @oneapi-src/onednn-arch labels Jan 8, 2025
@wzt1997 wzt1997 force-pushed the zhitao/enhance-shape-rewrite branch from 4af44a1 to d65e846 Compare January 8, 2025 06:09
@wzt1997 wzt1997 changed the title [WIP]benchdnn: graph: enhance mb and shape rewrite [WIP]benchdnn: graph: enhance input displace and shape rewrite for linked attribute and shapes Jan 8, 2025
@wzt1997 wzt1997 force-pushed the zhitao/enhance-shape-rewrite branch 2 times, most recently from c82a899 to b6a63b5 Compare January 10, 2025 05:56
@wzt1997 wzt1997 changed the title [WIP]benchdnn: graph: enhance input displace and shape rewrite for linked attribute and shapes benchdnn: graph: enhance input displace and shape rewrite for linked attribute and shapes Jan 10, 2025
@wzt1997 wzt1997 marked this pull request as ready for review January 10, 2025 06:50
@wzt1997 wzt1997 requested review from a team as code owners January 10, 2025 06:50
@wzt1997
Copy link
Contributor Author

wzt1997 commented Jan 10, 2025

make test
enable benchdnn_nightly
disable benchdnn_all
enable benchdnn_graph

tests/benchdnn/graph/deserialize.cpp Show resolved Hide resolved
tests/benchdnn/graph/flex_rewrite.cpp Show resolved Hide resolved
tests/benchdnn/graph/flex_rewrite.cpp Outdated Show resolved Hide resolved
tests/benchdnn/graph/flex_rewrite.cpp Show resolved Hide resolved
tests/benchdnn/graph/flex_rewrite.cpp Outdated Show resolved Hide resolved
@wzt1997 wzt1997 force-pushed the zhitao/enhance-shape-rewrite branch 2 times, most recently from 0c5a191 to 6b3be02 Compare January 14, 2025 02:04
@wzt1997 wzt1997 force-pushed the zhitao/enhance-shape-rewrite branch from 6b3be02 to fdb56d6 Compare January 14, 2025 05:40
@wzt1997 wzt1997 force-pushed the zhitao/enhance-shape-rewrite branch from fdb56d6 to 05022d1 Compare January 15, 2025 01:21
@wzt1997
Copy link
Contributor Author

wzt1997 commented Jan 16, 2025

make test
set test_scope=NIGHTLY
disable benchdnn_all
enable benchdnn_graph

@wzt1997 wzt1997 force-pushed the zhitao/enhance-shape-rewrite branch from 05022d1 to a3d94cd Compare January 17, 2025 06:07
@wzt1997 wzt1997 force-pushed the zhitao/enhance-shape-rewrite branch from a3d94cd to f8ef788 Compare January 20, 2025 01:23
@wzt1997
Copy link
Contributor Author

wzt1997 commented Jan 20, 2025

make test
set test_scope=NIGHTLY
disable benchdnn_all
enable benchdnn_graph

@wzt1997 wzt1997 force-pushed the zhitao/enhance-shape-rewrite branch from f8ef788 to c992118 Compare January 21, 2025 02:42
Comment on lines 132 to 133
else if (base_op_ref.kind_ == "ConvTranspose")
bia_dt = convert_dt(base_op_ref.in_lts_[2].get_data_type());
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the logic should be a little bit different:

else if (is_integral(dt)) {
    // Or maybe just f32 instead?
    bia_dt = convert_dt(base_op_ref.in_lts_[2].get_data_type());
} else {
    bia_dt = dt;
}

The limitation to not support s8/u8/s32 bias is uniform across the library and not limited to a specific op.

@wzt1997 wzt1997 force-pushed the zhitao/enhance-shape-rewrite branch from c992118 to 8096ced Compare January 21, 2025 05:26
@wzt1997
Copy link
Contributor Author

wzt1997 commented Jan 21, 2025

make test
set test_scope=NIGHTLY
disable benchdnn_all
enable benchdnn_graph

@TaoLv TaoLv merged commit 42d256c into main Jan 22, 2025
17 of 18 checks passed
@TaoLv TaoLv deleted the zhitao/enhance-shape-rewrite branch January 22, 2025 08:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
component:graph-api Codeowner: @oneapi-src/onednn-graph component:tests Codeowner: @oneapi-src/onednn-arch
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants