Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Fix](core) Fix wal space back pressure core and add regression test #27311

Merged
merged 10 commits into from
Nov 27, 2023
Merged

[Fix](core) Fix wal space back pressure core and add regression test #27311

merged 10 commits into from
Nov 27, 2023

Conversation

Yukang-Lian
Copy link
Collaborator

Proposed changes

Issue Number: close #xxx

Relative PR #26907

Tests will be added soon.

Further comments

If this is a relatively large or complex change, kick off the discussion at [email protected] by explaining why you chose the solution you did and what alternatives you considered, etc...

@Yukang-Lian
Copy link
Collaborator Author

run buildall

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@Yukang-Lian
Copy link
Collaborator Author

run buildall

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G' 
Tpch sf100 test result on commit 2caec72cccf933945f17faeea22101657406d2f6, data reload: false

run tpch-sf100 query with default conf and session variables
q1	4937	4650	4669	4650
q2	359	158	160	158
q3	2039	1915	1916	1915
q4	1387	1235	1240	1235
q5	4007	3941	3994	3941
q6	243	129	135	129
q7	1419	878	907	878
q8	2728	2773	2772	2772
q9	9738	9734	10063	9734
q10	3494	3555	3543	3543
q11	371	251	239	239
q12	441	299	297	297
q13	4610	3811	3839	3811
q14	317	285	282	282
q15	597	548	524	524
q16	681	585	583	583
q17	1134	966	929	929
q18	7748	7383	7432	7383
q19	1656	1707	1650	1650
q20	582	322	275	275
q21	4425	3946	3940	3940
q22	473	370	371	370
Total cold run time: 53386 ms
Total hot run time: 49238 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	4597	4568	4599	4568
q2	338	228	246	228
q3	4015	3985	3989	3985
q4	2697	2700	2697	2697
q5	9602	9597	9582	9582
q6	244	125	125	125
q7	3019	2475	2447	2447
q8	4485	4496	4483	4483
q9	13261	13138	13029	13029
q10	4097	4184	4171	4171
q11	771	700	677	677
q12	980	829	833	829
q13	4324	3636	3547	3547
q14	394	348	342	342
q15	581	518	525	518
q16	753	700	701	700
q17	3868	3868	3892	3868
q18	9403	9083	8983	8983
q19	1810	1786	1751	1751
q20	2396	2059	2026	2026
q21	8783	8370	8555	8370
q22	850	779	858	779
Total cold run time: 81268 ms
Total hot run time: 77705 ms

@doris-robot
Copy link

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 44.6 seconds
stream load tsv: 569 seconds loaded 74807831229 Bytes, about 125 MB/s
stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 34 seconds loaded 861443392 Bytes, about 24 MB/s
insert into select: 28.6 seconds inserted 10000000 Rows, about 349K ops/s
storage size: 17098943923 Bytes

@doris-robot
Copy link

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.09 seconds
stream load tsv: 568 seconds loaded 74807831229 Bytes, about 125 MB/s
stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 33 seconds loaded 861443392 Bytes, about 24 MB/s
insert into select: 28.4 seconds inserted 10000000 Rows, about 352K ops/s
storage size: 17097957666 Bytes

@Yukang-Lian Yukang-Lian force-pushed the Fix_Group_Commit_Core_2 branch from 2caec72 to 00efadc Compare November 21, 2023 07:51
@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@Yukang-Lian
Copy link
Collaborator Author

run buildall

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 36.52% (8445/23125)
Line Coverage: 28.84% (68651/238006)
Region Coverage: 27.81% (35503/127677)
Branch Coverage: 24.56% (18104/73714)
Coverage Report: http://coverage.selectdb-in.cc/coverage/00efadcd9b6c05f4e3743d1264e8f828ae974469_00efadcd9b6c05f4e3743d1264e8f828ae974469/report/index.html

@doris-robot
Copy link

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G' 
Tpch sf100 test result on commit 00efadcd9b6c05f4e3743d1264e8f828ae974469, data reload: false

run tpch-sf100 query with default conf and session variables
q1	4906	4655	4651	4651
q2	356	142	151	142
q3	2035	1895	1855	1855
q4	1383	1262	1241	1241
q5	3952	3953	4009	3953
q6	250	133	134	133
q7	1434	897	892	892
q8	2739	2786	2764	2764
q9	9679	9866	9542	9542
q10	3460	3570	3544	3544
q11	380	246	247	246
q12	439	293	293	293
q13	4588	3819	3847	3819
q14	316	303	283	283
q15	590	542	525	525
q16	666	580	574	574
q17	1123	967	939	939
q18	7784	7384	7292	7292
q19	1667	1660	1675	1660
q20	550	299	320	299
q21	4382	3953	3945	3945
q22	479	377	370	370
Total cold run time: 53158 ms
Total hot run time: 48962 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	4588	4595	4592	4592
q2	344	227	255	227
q3	3995	3981	3979	3979
q4	2705	2677	2690	2677
q5	9723	9703	9693	9693
q6	245	124	126	124
q7	2988	2502	2468	2468
q8	4444	4433	4426	4426
q9	13160	13121	13088	13088
q10	4100	4182	4183	4182
q11	802	653	712	653
q12	967	811	812	811
q13	4272	3589	3560	3560
q14	386	344	358	344
q15	578	532	529	529
q16	740	670	654	654
q17	3973	3855	3915	3855
q18	9522	8994	9087	8994
q19	1793	1762	1803	1762
q20	2406	2063	2063	2063
q21	8764	8568	8724	8568
q22	901	767	813	767
Total cold run time: 81396 ms
Total hot run time: 78016 ms

@doris-robot
Copy link

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.63 seconds
stream load tsv: 565 seconds loaded 74807831229 Bytes, about 126 MB/s
stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.9 seconds inserted 10000000 Rows, about 346K ops/s
storage size: 17098115979 Bytes

hust-hhb
hust-hhb approved these changes Nov 22, 2023
View reviewed changes
Copy link
Contributor

@hust-hhb hust-hhb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

dataroaring
dataroaring previously approved these changes Nov 25, 2023
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@Yukang-Lian Yukang-Lian force-pushed the Fix_Group_Commit_Core_2 branch from 00efadc to ba801b6 Compare November 25, 2023 15:39
@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@Yukang-Lian Yukang-Lian force-pushed the Fix_Group_Commit_Core_2 branch from ba801b6 to 04e57d7 Compare November 25, 2023 15:53
github-actions[bot]
github-actions bot reviewed Nov 25, 2023
View reviewed changes
Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

be/src/olap/wal_manager.h
@@ -17,6 +17,7 @@

#include <gen_cpp/PaloInternalService_types.h>
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

warning: 'gen_cpp/PaloInternalService_types.h' file not found [clang-diagnostic-error]

#include <gen_cpp/PaloInternalService_types.h>
         ^

@Yukang-Lian
Copy link
Collaborator Author

run buildall

@doris-robot
Copy link

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G' 
Tpch sf100 test result on commit 5709d33c6933d2b65a81946c26d1a360d483a9a4, data reload: false

run tpch-sf100 query with default conf and session variables
q1	4938	4637	4659	4637
q2	376	160	161	160
q3	2029	1933	1832	1832
q4	1392	1227	1218	1218
q5	3928	3944	3977	3944
q6	242	128	131	128
q7	1408	885	885	885
q8	2767	2791	2769	2769
q9	9701	9570	9478	9478
q10	3479	3538	3521	3521
q11	376	245	245	245
q12	446	294	291	291
q13	4532	3811	3790	3790
q14	329	290	291	290
q15	589	556	528	528
q16	661	602	585	585
q17	1130	989	947	947
q18	7722	7324	7256	7256
q19	1676	1664	1679	1664
q20	555	297	281	281
q21	4417	3938	3963	3938
q22	471	369	367	367
Total cold run time: 53164 ms
Total hot run time: 48754 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	4590	4596	4581	4581
q2	325	233	263	233
q3	3987	3993	3969	3969
q4	2678	2660	2671	2660
q5	9612	9503	9486	9486
q6	241	123	126	123
q7	3002	2473	2479	2473
q8	4441	4443	4432	4432
q9	12931	12688	12769	12688
q10	4055	4159	4215	4159
q11	773	647	644	644
q12	989	806	800	800
q13	4271	3600	3581	3581
q14	384	359	369	359
q15	593	524	527	524
q16	730	659	690	659
q17	3880	3845	3839	3839
q18	9395	8979	8887	8887
q19	1825	1784	1791	1784
q20	2387	2067	2031	2031
q21	8806	8395	8502	8395
q22	896	777	816	777
Total cold run time: 80791 ms
Total hot run time: 77084 ms

@doris-robot
Copy link

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.66 seconds
stream load tsv: 566 seconds loaded 74807831229 Bytes, about 126 MB/s
stream load json: 25 seconds loaded 2358488459 Bytes, about 89 MB/s
stream load orc: 71 seconds loaded 1101869774 Bytes, about 14 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 29.1 seconds inserted 10000000 Rows, about 343K ops/s
storage size: 17097447726 Bytes

@Yukang-Lian
Copy link
Collaborator Author

run buildall

@Yukang-Lian Yukang-Lian changed the title [Fix](core) Fix wal space back pressure core [Fix](core) Fix wal space back pressure core and add regression test Nov 26, 2023
@doris-robot
Copy link

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G' 
Tpch sf100 test result on commit cfc6af8c36ca7c99f046fbb047120ac593917d1f, data reload: false

run tpch-sf100 query with default conf and session variables
q1	4955	4643	4624	4624
q2	354	146	137	137
q3	2047	1964	1904	1904
q4	1379	1267	1238	1238
q5	3943	3961	4027	3961
q6	257	132	133	132
q7	1441	896	879	879
q8	2775	2784	2772	2772
q9	9683	9435	9430	9430
q10	3446	3501	3524	3501
q11	383	248	245	245
q12	438	285	283	283
q13	4559	3787	3832	3787
q14	333	290	281	281
q15	584	537	521	521
q16	669	592	580	580
q17	1141	967	926	926
q18	7769	7341	7260	7260
q19	1669	1674	1689	1674
q20	534	322	299	299
q21	4430	3978	3945	3945
q22	480	370	376	370
Total cold run time: 53269 ms
Total hot run time: 48749 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	4614	4640	4616	4616
q2	340	236	261	236
q3	3978	3969	3966	3966
q4	2673	2672	2668	2668
q5	9642	9510	9385	9385
q6	242	127	123	123
q7	3004	2480	2480	2480
q8	4502	4445	4492	4445
q9	12854	12850	12780	12780
q10	4098	4149	4191	4149
q11	792	637	659	637
q12	982	808	805	805
q13	4297	3554	3564	3554
q14	384	345	356	345
q15	588	528	530	528
q16	728	672	671	671
q17	3929	3888	3980	3888
q18	9590	9076	8998	8998
q19	1830	1789	1786	1786
q20	2405	2086	2043	2043
q21	8769	8556	8441	8441
q22	877	836	783	783
Total cold run time: 81118 ms
Total hot run time: 77327 ms

@doris-robot
Copy link

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.69 seconds
stream load tsv: 565 seconds loaded 74807831229 Bytes, about 126 MB/s
stream load json: 27 seconds loaded 2358488459 Bytes, about 83 MB/s
stream load orc: 71 seconds loaded 1101869774 Bytes, about 14 MB/s
stream load parquet: 33 seconds loaded 861443392 Bytes, about 24 MB/s
insert into select: 28.8 seconds inserted 10000000 Rows, about 347K ops/s
storage size: 17099430555 Bytes

@Yukang-Lian
Copy link
Collaborator Author

run buildall

@Yukang-Lian
Copy link
Collaborator Author

run pipelinex_p0

@doris-robot
Copy link

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G' 
Tpch sf100 test result on commit c52983ff246da42608ed8dad04e09e791c1a5a91, data reload: false

run tpch-sf100 query with default conf and session variables
q1	4902	4687	4635	4635
q2	353	151	172	151
q3	2067	1988	1915	1915
q4	1389	1266	1277	1266
q5	3950	3936	4014	3936
q6	250	135	137	135
q7	1446	881	875	875
q8	2761	2781	2799	2781
q9	9627	9501	9622	9501
q10	3459	3520	3534	3520
q11	388	247	239	239
q12	435	293	287	287
q13	4552	3837	3803	3803
q14	323	296	290	290
q15	591	544	525	525
q16	665	582	582	582
q17	1156	947	929	929
q18	7757	7258	7327	7258
q19	1675	1698	1683	1683
q20	555	312	294	294
q21	4382	4008	3974	3974
q22	477	381	364	364
Total cold run time: 53160 ms
Total hot run time: 48943 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	4596	4602	4554	4554
q2	340	228	265	228
q3	4014	3995	3978	3978
q4	2701	2662	2665	2662
q5	9592	9611	9520	9520
q6	242	125	124	124
q7	3002	2447	2467	2447
q8	4496	4484	4471	4471
q9	12922	12884	12905	12884
q10	4112	4169	4185	4169
q11	760	658	659	658
q12	974	836	801	801
q13	4276	3634	3605	3605
q14	383	344	352	344
q15	580	526	523	523
q16	727	653	658	653
q17	3950	3950	3909	3909
q18	9510	8823	8923	8823
q19	1787	1804	1798	1798
q20	2388	2071	2050	2050
q21	8831	8514	8536	8514
q22	885	795	806	795
Total cold run time: 81068 ms
Total hot run time: 77510 ms

@doris-robot
Copy link

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.53 seconds
stream load tsv: 566 seconds loaded 74807831229 Bytes, about 126 MB/s
stream load json: 25 seconds loaded 2358488459 Bytes, about 89 MB/s
stream load orc: 71 seconds loaded 1101869774 Bytes, about 14 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 29.0 seconds inserted 10000000 Rows, about 344K ops/s
storage size: 17099111032 Bytes

@Yukang-Lian Yukang-Lian force-pushed the Fix_Group_Commit_Core_2 branch from c52983f to fd2c967 Compare November 26, 2023 11:47
@Yukang-Lian
Copy link
Collaborator Author

run buildall

dataroaring
dataroaring approved these changes Nov 26, 2023
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Nov 26, 2023
@doris-robot
Copy link

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G' 
Tpch sf100 test result on commit fd2c967300228552d33e9ebf67da77522a61be09, data reload: false

run tpch-sf100 query with default conf and session variables
q1	4926	4685	4660	4660
q2	363	159	156	156
q3	2047	1877	1862	1862
q4	1386	1262	1259	1259
q5	3933	3939	4060	3939
q6	252	134	136	134
q7	1460	883	876	876
q8	2782	2813	2796	2796
q9	9867	9490	9465	9465
q10	3473	3537	3523	3523
q11	374	248	235	235
q12	447	292	287	287
q13	4561	3758	3767	3758
q14	318	286	282	282
q15	587	531	529	529
q16	672	585	581	581
q17	1157	1000	932	932
q18	7855	7356	7344	7344
q19	1680	1697	1696	1696
q20	552	330	288	288
q21	4346	3966	3969	3966
q22	473	369	377	369
Total cold run time: 53511 ms
Total hot run time: 48937 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	4614	4606	4590	4590
q2	333	242	270	242
q3	4003	3985	3974	3974
q4	2770	2673	2673	2673
q5	9667	9579	9536	9536
q6	240	125	125	125
q7	3003	2464	2456	2456
q8	4474	4443	4466	4443
q9	12988	12794	12790	12790
q10	4073	4153	4194	4153
q11	809	736	704	704
q12	970	813	799	799
q13	4285	3559	3542	3542
q14	383	345	349	345
q15	587	520	517	517
q16	728	676	666	666
q17	3984	3913	3855	3855
q18	9655	8971	9082	8971
q19	1842	1780	1798	1780
q20	2409	2057	2073	2057
q21	8742	8481	8646	8481
q22	879	819	774	774
Total cold run time: 81438 ms
Total hot run time: 77473 ms

@doris-robot
Copy link

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.18 seconds
stream load tsv: 567 seconds loaded 74807831229 Bytes, about 125 MB/s
stream load json: 26 seconds loaded 2358488459 Bytes, about 86 MB/s
stream load orc: 70 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.5 seconds inserted 10000000 Rows, about 350K ops/s
storage size: 17098833145 Bytes

@dataroaring dataroaring merged commit 13b26ee into apache:master Nov 27, 2023
16 checks passed
seawinde pushed a commit to seawinde/doris that referenced this pull request Nov 28, 2023
@Yukang-Lian @seawinde
[Fix](core) Fix wal space back pressure core and add regression test (a…
9352cae 
…pache#27311)
XuJianxu pushed a commit to XuJianxu/doris that referenced this pull request Dec 14, 2023
@Yukang-Lian
[Fix](core) Fix wal space back pressure core and add regression test (a…
0749159 
…pache#27311)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@github-actions github-actions[bot] github-actions[bot] left review comments

@dataroaring dataroaring dataroaring approved these changes

@hust-hhb hust-hhb hust-hhb approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@Yukang-Lian @doris-robot @hust-hhb @dataroaring