Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[regression](compaction) Add case to test single replica compaction #27199

Merged
merged 3 commits into from
Nov 30, 2023
Merged

[regression](compaction) Add case to test single replica compaction #27199

merged 3 commits into from
Nov 30, 2023

Conversation

Xiaoccer
Copy link
Contributor

No description provided.

@Xiaoccer
Copy link
Contributor Author

run buildall

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 36.74% (8413/22898)
Line Coverage: 29.25% (68422/233883)
Region Coverage: 27.84% (35373/127052)
Branch Coverage: 24.59% (18065/73462)
Coverage Report: http://coverage.selectdb-in.cc/coverage/f59ffc84ac4e977734b3c4e045df7a766b0dfc0f_f59ffc84ac4e977734b3c4e045df7a766b0dfc0f/report/index.html

@doris-robot
Copy link

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G' 
Tpch sf100 test result on commit f59ffc84ac4e977734b3c4e045df7a766b0dfc0f, data reload: false

run tpch-sf100 query with default conf and session variables
q1	4930	4654	4652	4652
q2	352	163	158	158
q3	2044	1891	1918	1891
q4	1409	1262	1226	1226
q5	3937	3932	3999	3932
q6	242	128	131	128
q7	1374	871	889	871
q8	2781	2779	2785	2779
q9	9721	9644	9583	9583
q10	3457	3524	3526	3524
q11	373	248	253	248
q12	434	286	285	285
q13	4552	3803	3838	3803
q14	317	285	295	285
q15	594	530	528	528
q16	655	581	592	581
q17	1127	978	944	944
q18	7807	7381	7358	7358
q19	1658	1683	1678	1678
q20	572	291	305	291
q21	4416	3952	3980	3952
q22	466	384	370	370
Total cold run time: 53218 ms
Total hot run time: 49067 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	4564	4572	4557	4557
q2	347	237	284	237
q3	4028	4000	3993	3993
q4	2701	2686	2689	2686
q5	9641	9686	9589	9589
q6	241	120	124	120
q7	2617	2305	2315	2305
q8	4497	4497	4445	4445
q9	13187	13106	13176	13106
q10	4096	4175	4186	4175
q11	746	666	643	643
q12	976	824	804	804
q13	4266	3622	3586	3586
q14	390	364	346	346
q15	576	516	514	514
q16	745	664	683	664
q17	3888	3877	3843	3843
q18	9575	9036	9023	9023
q19	1806	1785	1767	1767
q20	2402	2079	2049	2049
q21	8794	8849	8646	8646
q22	904	780	820	780
Total cold run time: 80987 ms
Total hot run time: 77878 ms

@doris-robot
Copy link

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.28 seconds
stream load tsv: 567 seconds loaded 74807831229 Bytes, about 125 MB/s
stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 29.3 seconds inserted 10000000 Rows, about 341K ops/s
storage size: 17099304272 Bytes

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

dataroaring
dataroaring previously approved these changes Nov 17, 2023
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Nov 17, 2023
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@dataroaring
Copy link
Contributor

run buildall

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 36.77% (8431/22927)
Line Coverage: 29.28% (68603/234278)
Region Coverage: 27.87% (35460/127251)
Branch Coverage: 24.60% (18098/73570)
Coverage Report: http://coverage.selectdb-in.cc/coverage/2c2425f2208ac42f9db55c07a7c95d42dc8c5a50_2c2425f2208ac42f9db55c07a7c95d42dc8c5a50/report/index.html

@doris-robot
Copy link

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 44.96 seconds
stream load tsv: 564 seconds loaded 74807831229 Bytes, about 126 MB/s
stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.8 seconds inserted 10000000 Rows, about 347K ops/s
storage size: 17098657392 Bytes

@doris-robot
Copy link

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G' 
Tpch sf100 test result on commit 2c2425f2208ac42f9db55c07a7c95d42dc8c5a50, data reload: false

run tpch-sf100 query with default conf and session variables
q1	8033	5236	4690	4690
q2	500	144	138	138
q3	2816	1903	1882	1882
q4	5401	1138	1265	1138
q5	4978	3980	4244	3980
q6	256	131	134	131
q7	1440	881	892	881
q8	7628	2823	2778	2778
q9	10264	9537	9659	9537
q10	9902	3569	3552	3552
q11	404	250	246	246
q12	446	291	290	290
q13	21319	3833	3768	3768
q14	321	297	282	282
q15	585	530	530	530
q16	683	589	584	584
q17	1164	983	941	941
q18	7892	7451	7451	7451
q19	1731	1697	1692	1692
q20	592	315	304	304
q21	4481	4056	4039	4039
q22	478	387	375	375
Total cold run time: 91314 ms
Total hot run time: 49209 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	4591	4533	4591	4533
q2	350	236	254	236
q3	4028	4031	4020	4020
q4	2741	2726	2725	2725
q5	9707	9629	9683	9629
q6	251	122	121	121
q7	3043	2507	2519	2507
q8	4458	4570	4453	4453
q9	13008	13078	12797	12797
q10	4070	4215	4148	4148
q11	829	646	643	643
q12	974	800	809	800
q13	4299	3594	3647	3594
q14	376	346	346	346
q15	576	511	522	511
q16	743	694	681	681
q17	3910	3856	3907	3856
q18	9686	9098	9007	9007
q19	1810	1804	1791	1791
q20	2413	2099	2083	2083
q21	8882	8766	8538	8538
q22	910	770	757	757
Total cold run time: 81655 ms
Total hot run time: 77776 ms

xiaokang
xiaokang reviewed Nov 19, 2023
View reviewed changes
regression-test/suites/compaction/test_single_replica_compaction.groovy Show resolved Hide resolved
be/src/olap/tablet.cpp Outdated
if (tablet_meta()->tablet_schema()->enable_single_replica_compaction() &&
StorageEngine::instance()->get_peer_replica_info(tablet_id(), &replica_info,
&dummp_token)) {
std::string addr = replica_info.host;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

std::string addr = replica_info.host + ":" + std::to_string(replica_info.brpc_port);

@github-actions github-actions bot removed the approved Indicates a PR has been approved by one committer. label Nov 20, 2023
@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@dataroaring
Copy link
Contributor

run buildall

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 36.51% (8447/23134)
Line Coverage: 28.86% (68667/237900)
Region Coverage: 27.83% (35502/127576)
Branch Coverage: 24.59% (18106/73624)
Coverage Report: http://coverage.selectdb-in.cc/coverage/c3ab985b943c4240ea318d6b2943584521c792f5_c3ab985b943c4240ea318d6b2943584521c792f5/report/index.html

@doris-robot
Copy link

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G' 
Tpch sf100 test result on commit c3ab985b943c4240ea318d6b2943584521c792f5, data reload: false

run tpch-sf100 query with default conf and session variables
q1	4957	4677	4661	4661
q2	377	178	159	159
q3	2055	1948	1989	1948
q4	1407	1241	1243	1241
q5	4025	3992	4109	3992
q6	242	132	127	127
q7	1394	881	907	881
q8	2771	2816	2802	2802
q9	9772	9817	9674	9674
q10	3490	3538	3550	3538
q11	385	247	249	247
q12	442	300	291	291
q13	4600	3844	3809	3809
q14	319	289	278	278
q15	594	542	526	526
q16	657	584	585	584
q17	1132	965	962	962
q18	7793	7316	7399	7316
q19	1688	1696	1704	1696
q20	583	321	307	307
q21	4406	3949	4004	3949
q22	484	375	365	365
Total cold run time: 53573 ms
Total hot run time: 49353 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	4601	4607	4603	4603
q2	341	232	243	232
q3	4043	4005	4006	4005
q4	2711	2710	2687	2687
q5	9701	9866	9761	9761
q6	245	120	127	120
q7	3007	2490	2488	2488
q8	4469	4471	4460	4460
q9	13292	13187	13222	13187
q10	4102	4205	4233	4205
q11	797	647	719	647
q12	976	828	809	809
q13	4314	3545	3561	3545
q14	394	356	352	352
q15	572	529	525	525
q16	739	662	676	662
q17	3840	3859	3918	3859
q18	9512	8980	8978	8978
q19	1815	1800	1785	1785
q20	2402	2044	2073	2044
q21	8931	8619	8695	8619
q22	874	764	784	764
Total cold run time: 81678 ms
Total hot run time: 78337 ms

@doris-robot
Copy link

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G' 
Tpch sf100 test result on commit c3ab985b943c4240ea318d6b2943584521c792f5, data reload: false

run tpch-sf100 query with default conf and session variables
q1	4944	4689	4650	4650
q2	357	174	159	159
q3	2030	1967	1880	1880
q4	1386	1289	1249	1249
q5	4007	3966	4028	3966
q6	255	130	137	130
q7	1413	895	886	886
q8	2740	2770	2762	2762
q9	9725	9767	9616	9616
q10	3484	3544	3509	3509
q11	375	244	246	244
q12	447	300	306	300
q13	4555	3828	3787	3787
q14	319	307	286	286
q15	581	539	525	525
q16	666	593	580	580
q17	1118	973	918	918
q18	7831	7312	7493	7312
q19	1667	1688	1672	1672
q20	563	305	302	302
q21	4423	3950	3944	3944
q22	478	371	377	371
Total cold run time: 53364 ms
Total hot run time: 49048 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	4603	4558	4597	4558
q2	343	273	263	263
q3	4039	3989	3999	3989
q4	2706	2689	2699	2689
q5	9673	9599	9554	9554
q6	239	122	128	122
q7	3042	2517	2519	2517
q8	4473	4438	4459	4438
q9	13254	13094	13099	13094
q10	4103	4235	4207	4207
q11	835	652	653	652
q12	979	806	804	804
q13	4317	3576	3591	3576
q14	375	339	360	339
q15	578	520	526	520
q16	741	681	689	681
q17	3825	3899	3871	3871
q18	9600	9030	9197	9030
q19	1803	1771	1781	1771
q20	2386	2069	2063	2063
q21	8913	8787	8792	8787
q22	937	824	786	786
Total cold run time: 81764 ms
Total hot run time: 78311 ms

@doris-robot
Copy link

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 44.57 seconds
stream load tsv: 564 seconds loaded 74807831229 Bytes, about 126 MB/s
stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.9 seconds inserted 10000000 Rows, about 346K ops/s
storage size: 17097972544 Bytes

dataroaring
dataroaring approved these changes Nov 24, 2023
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Nov 24, 2023
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@dataroaring dataroaring self-assigned this Nov 25, 2023
@dataroaring dataroaring merged commit 97105e9 into apache:master Nov 30, 2023
18 of 19 checks passed
csun5285
csun5285 reviewed Dec 1, 2023
View reviewed changes
regression-test/suites/compaction/test_single_replica_compaction.groovy
def triggerCompaction = { be_host, be_http_port, compact_type, tablet_id ->
StringBuilder sb = new StringBuilder();
sb.append("curl -X POST http://${be_host}:${be_http_port}")
sb.append("/api/compaction/run?tablet_id=")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It does not employ single replica compaction. use table_id =

XuJianxu pushed a commit to XuJianxu/doris that referenced this pull request Dec 14, 2023
@Xiaoccer
[regression](compaction) Add case to test single replica compaction (a…
e4651c1 
…pache#27199)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@csun5285 csun5285 csun5285 left review comments

@xiaokang xiaokang xiaokang left review comments

@dataroaring dataroaring dataroaring approved these changes

@bobhan1 bobhan1 bobhan1 approved these changes

Assignees

@dataroaring dataroaring

Labels
approved Indicates a PR has been approved by one committer. dev/2.0.4 reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
6 participants
@Xiaoccer @doris-robot @dataroaring @xiaokang @bobhan1 @csun5285