Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[opt](resource) Implement ResourceContext to manage task resources #47274

Open
wants to merge 5 commits into
base: master
Choose a base branch
from
Open

[opt](resource) Implement ResourceContext to manage task resources #47274

wants to merge 5 commits into from

Conversation

xinyiZzz
Copy link
Contributor

@xinyiZzz xinyiZzz commented Jan 21, 2025 •
edited
Loading

What problem does this PR solve?

All tasks in Doris, query/load/compaction/etc., must have a unique ResourceContext to manage the resource usage of this Task, including CpuContext, MemoryContext, IOContext, WorkloadGroupContext and TaskController.

What can ResourceContext do:

  1. Save task resource stat, such as profile counter of CPU/Memory/IO.

  2. Hold resource objects such as WorkloadGroupPtr, MemTrackerPtr.

  3. Provide methods for operating task resources, such as revokable_bytes, cancel task.

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@hello-stephen
Copy link
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@xinyiZzz xinyiZzz force-pushed the 20250120_fix_resource_ctx branch from ba42938 to 3d24170 Compare January 21, 2025 13:09
@xinyiZzz
Copy link
Contributor Author

run buildall

@xinyiZzz xinyiZzz force-pushed the 20250120_fix_resource_ctx branch from 3d24170 to daf382f Compare January 22, 2025 11:45
@xinyiZzz
Copy link
Contributor Author

run buildall

@xinyiZzz xinyiZzz force-pushed the 20250120_fix_resource_ctx branch from daf382f to 414079f Compare January 22, 2025 12:16
@xinyiZzz
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 41.57% (10854/26113)
Line Coverage: 31.96% (91632/286667)
Region Coverage: 31.10% (46939/150910)
Branch Coverage: 27.18% (23756/87408)
Coverage Report: http://coverage.selectdb-in.cc/coverage/414079f3ebf3027a1e0271c2dadbf4b847f39245_414079f3ebf3027a1e0271c2dadbf4b847f39245/report/index.html

@xinyiZzz xinyiZzz force-pushed the 20250120_fix_resource_ctx branch from 414079f to 43513d3 Compare January 22, 2025 18:20
@xinyiZzz
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 41.57% (10856/26116)
Line Coverage: 31.98% (91687/286723)
Region Coverage: 31.10% (46947/150935)
Branch Coverage: 27.19% (23765/87410)
Coverage Report: http://coverage.selectdb-in.cc/coverage/43513d3203db125dcc0fa7eb015673edd41c0688_43513d3203db125dcc0fa7eb015673edd41c0688/report/index.html

@doris-robot
Copy link

TPC-H: Total hot run time: 32805 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 43513d3203db125dcc0fa7eb015673edd41c0688, data reload: false

------ Round 1 ----------------------------------
q1	17588	5720	5398	5398
q2	2051	334	190	190
q3	10369	1266	782	782
q4	10213	1002	557	557
q5	7523	2412	2216	2216
q6	197	171	135	135
q7	915	776	617	617
q8	9236	1454	1253	1253
q9	5350	4898	4955	4898
q10	6881	2336	1903	1903
q11	475	291	262	262
q12	359	375	230	230
q13	17785	3729	3170	3170
q14	226	238	222	222
q15	525	478	464	464
q16	626	631	595	595
q17	610	888	345	345
q18	7154	6443	6501	6443
q19	2083	978	566	566
q20	324	330	195	195
q21	2929	2256	2050	2050
q22	374	341	314	314
Total cold run time: 103793 ms
Total hot run time: 32805 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5596	5525	5500	5500
q2	243	340	242	242
q3	2265	2701	2371	2371
q4	1493	1814	1402	1402
q5	4393	4805	4788	4788
q6	179	159	129	129
q7	2143	2003	1811	1811
q8	2662	2865	2756	2756
q9	7328	7192	7285	7192
q10	3071	3325	2738	2738
q11	573	506	500	500
q12	652	742	601	601
q13	3595	4026	3281	3281
q14	285	287	288	287
q15	528	478	463	463
q16	661	680	649	649
q17	1286	1756	1282	1282
q18	7776	7579	7390	7390
q19	875	1178	1142	1142
q20	2007	2041	1935	1935
q21	5829	5304	4955	4955
q22	613	623	587	587
Total cold run time: 54053 ms
Total hot run time: 52001 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 193906 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 43513d3203db125dcc0fa7eb015673edd41c0688, data reload: false

query1	1290	952	950	950
query2	6379	2029	1947	1947
query3	10967	4543	4496	4496
query4	61029	33969	23115	23115
query5	5467	614	454	454
query6	366	201	201	201
query7	4999	512	304	304
query8	313	258	220	220
query9	6161	2608	2580	2580
query10	424	321	255	255
query11	15289	15015	15437	15015
query12	169	111	109	109
query13	1069	557	416	416
query14	10793	7222	6944	6944
query15	220	205	197	197
query16	7371	655	447	447
query17	1163	749	639	639
query18	1794	412	294	294
query19	200	181	161	161
query20	123	120	106	106
query21	207	124	113	113
query22	4452	4647	4393	4393
query23	34652	33076	33756	33076
query24	5653	2389	2335	2335
query25	477	457	406	406
query26	645	258	169	169
query27	1750	461	330	330
query28	3933	2481	2448	2448
query29	561	547	432	432
query30	210	197	162	162
query31	957	919	826	826
query32	73	63	58	58
query33	436	369	306	306
query34	761	872	517	517
query35	861	844	797	797
query36	1026	1040	951	951
query37	129	104	81	81
query38	4315	4293	4250	4250
query39	1505	1472	1441	1441
query40	202	128	115	115
query41	55	50	49	49
query42	120	104	105	104
query43	530	522	525	522
query44	1305	836	841	836
query45	187	179	181	179
query46	893	1084	650	650
query47	1862	1905	1826	1826
query48	426	411	330	330
query49	744	507	416	416
query50	674	665	402	402
query51	7108	7147	7010	7010
query52	105	103	95	95
query53	237	263	189	189
query54	492	506	446	446
query55	82	83	82	82
query56	267	307	254	254
query57	1191	1172	1114	1114
query58	232	228	248	228
query59	3065	3099	2885	2885
query60	274	268	261	261
query61	118	119	117	117
query62	720	730	662	662
query63	223	183	184	183
query64	1367	1138	719	719
query65	3266	3249	3181	3181
query66	761	423	304	304
query67	16029	15872	15459	15459
query68	5018	826	535	535
query69	510	322	258	258
query70	1184	1168	1115	1115
query71	410	305	250	250
query72	6043	3924	3800	3800
query73	788	773	381	381
query74	10073	9193	8864	8864
query75	3230	3143	2683	2683
query76	3769	1167	757	757
query77	519	370	271	271
query78	10077	9963	9383	9383
query79	3746	823	589	589
query80	1634	539	458	458
query81	560	275	244	244
query82	653	155	127	127
query83	288	167	164	164
query84	299	100	79	79
query85	814	358	301	301
query86	474	305	296	296
query87	4550	4552	4320	4320
query88	4387	2216	2207	2207
query89	407	333	334	333
query90	1540	188	190	188
query91	139	142	110	110
query92	70	56	53	53
query93	2631	902	530	530
query94	812	391	272	272
query95	328	272	267	267
query96	483	611	283	283
query97	2838	2846	2762	2762
query98	230	203	197	197
query99	1287	1362	1270	1270
Total cold run time: 310515 ms
Total hot run time: 193906 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.96 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 43513d3203db125dcc0fa7eb015673edd41c0688, data reload: false

query1	0.03	0.03	0.04
query2	0.07	0.04	0.03
query3	0.23	0.07	0.06
query4	1.62	0.10	0.10
query5	0.42	0.41	0.41
query6	1.13	0.65	0.64
query7	0.03	0.02	0.02
query8	0.04	0.04	0.04
query9	0.60	0.50	0.51
query10	0.56	0.57	0.54
query11	0.14	0.10	0.10
query12	0.14	0.11	0.11
query13	0.61	0.60	0.60
query14	2.72	2.73	2.87
query15	0.90	0.83	0.83
query16	0.37	0.36	0.39
query17	1.06	1.00	1.02
query18	0.23	0.21	0.21
query19	1.86	1.76	1.98
query20	0.02	0.01	0.01
query21	15.38	0.92	0.57
query22	0.75	0.81	0.90
query23	15.06	1.42	0.53
query24	2.94	1.12	2.28
query25	0.15	0.12	0.16
query26	0.34	0.15	0.13
query27	0.06	0.05	0.04
query28	14.55	0.93	0.45
query29	12.56	4.13	3.41
query30	0.25	0.09	0.08
query31	2.81	0.60	0.38
query32	3.23	0.54	0.47
query33	2.98	2.99	3.03
query34	16.30	5.15	4.51
query35	4.49	4.49	4.51
query36	0.65	0.49	0.48
query37	0.09	0.06	0.06
query38	0.05	0.04	0.03
query39	0.04	0.02	0.02
query40	0.17	0.13	0.12
query41	0.08	0.03	0.02
query42	0.03	0.02	0.03
query43	0.03	0.03	0.03
Total cold run time: 105.77 s
Total hot run time: 30.96 s

@xinyiZzz xinyiZzz force-pushed the 20250120_fix_resource_ctx branch from 43513d3 to 02cd0be Compare January 23, 2025 03:31
@xinyiZzz
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 32534 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 02cd0be5c780f8181caf31ba7dd42e27cf40013a, data reload: false

------ Round 1 ----------------------------------
q1	17578	6247	5424	5424
q2	2054	305	185	185
q3	10691	1222	739	739
q4	10215	988	529	529
q5	7550	2411	2185	2185
q6	195	171	138	138
q7	930	754	608	608
q8	9251	1370	1188	1188
q9	5283	5087	4895	4895
q10	6836	2342	1882	1882
q11	480	271	259	259
q12	353	358	223	223
q13	17774	3764	3141	3141
q14	238	222	212	212
q15	515	479	467	467
q16	637	607	582	582
q17	566	866	323	323
q18	7145	6517	6644	6517
q19	1777	961	541	541
q20	313	318	195	195
q21	2923	2234	1989	1989
q22	364	333	312	312
Total cold run time: 103668 ms
Total hot run time: 32534 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5596	5475	5541	5475
q2	239	335	232	232
q3	2283	2665	2379	2379
q4	1475	1852	1378	1378
q5	4313	4792	4771	4771
q6	177	164	127	127
q7	2059	1933	1890	1890
q8	2661	2804	2728	2728
q9	7348	7234	7292	7234
q10	2994	3311	2756	2756
q11	572	500	477	477
q12	683	717	612	612
q13	3601	3995	3329	3329
q14	278	292	276	276
q15	526	472	461	461
q16	676	690	665	665
q17	1223	1776	1253	1253
q18	7809	7747	7499	7499
q19	814	1167	1091	1091
q20	1974	2104	1958	1958
q21	5772	5247	5019	5019
q22	628	624	597	597
Total cold run time: 53701 ms
Total hot run time: 52207 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 193613 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 02cd0be5c780f8181caf31ba7dd42e27cf40013a, data reload: false

query1	1326	945	931	931
query2	6388	2079	2042	2042
query3	11100	4643	4558	4558
query4	61675	29006	23161	23161
query5	5403	605	455	455
query6	417	193	173	173
query7	5558	506	314	314
query8	320	229	221	221
query9	8652	2616	2630	2616
query10	457	315	280	280
query11	17758	15161	15508	15161
query12	168	122	119	119
query13	1475	573	431	431
query14	10322	7174	6630	6630
query15	208	230	192	192
query16	7237	651	440	440
query17	1105	739	606	606
query18	1940	420	349	349
query19	201	175	155	155
query20	119	108	121	108
query21	220	122	101	101
query22	4337	4434	4591	4434
query23	34653	33618	33347	33347
query24	5520	2363	2371	2363
query25	479	455	416	416
query26	635	288	157	157
query27	1730	449	333	333
query28	4326	2510	2467	2467
query29	535	558	411	411
query30	228	185	151	151
query31	972	914	831	831
query32	72	60	58	58
query33	417	355	313	313
query34	737	856	528	528
query35	829	858	737	737
query36	1032	1040	960	960
query37	123	100	75	75
query38	4328	4261	4168	4168
query39	1509	1463	1519	1463
query40	206	110	99	99
query41	79	49	51	49
query42	122	101	115	101
query43	509	525	491	491
query44	1374	835	834	834
query45	181	180	172	172
query46	879	1078	674	674
query47	1895	1890	1856	1856
query48	386	415	339	339
query49	725	495	389	389
query50	651	704	401	401
query51	7053	7129	6865	6865
query52	104	101	96	96
query53	227	256	183	183
query54	499	504	412	412
query55	86	81	79	79
query56	256	262	291	262
query57	1269	1251	1131	1131
query58	237	230	226	226
query59	2950	3090	2797	2797
query60	290	264	259	259
query61	121	117	122	117
query62	752	711	645	645
query63	224	185	185	185
query64	1323	1011	658	658
query65	3273	3184	3158	3158
query66	709	386	328	328
query67	15879	15488	15388	15388
query68	5021	832	540	540
query69	506	305	261	261
query70	1178	1110	1120	1110
query71	423	288	261	261
query72	5449	3907	3936	3907
query73	793	761	368	368
query74	10219	9268	8667	8667
query75	3193	3139	2647	2647
query76	3656	1151	761	761
query77	480	364	276	276
query78	10177	10059	9320	9320
query79	3432	810	591	591
query80	1417	557	454	454
query81	555	268	239	239
query82	349	149	134	134
query83	254	167	153	153
query84	291	97	76	76
query85	802	343	292	292
query86	454	305	307	305
query87	4401	4434	4547	4434
query88	4763	2205	2228	2205
query89	395	319	286	286
query90	1562	192	190	190
query91	134	147	105	105
query92	60	71	60	60
query93	2788	869	540	540
query94	758	413	270	270
query95	325	269	247	247
query96	493	615	281	281
query97	2812	2909	2710	2710
query98	243	200	197	197
query99	1281	1355	1277	1277
Total cold run time: 315434 ms
Total hot run time: 193613 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.92 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 02cd0be5c780f8181caf31ba7dd42e27cf40013a, data reload: false

query1	0.04	0.05	0.03
query2	0.07	0.04	0.03
query3	0.24	0.07	0.07
query4	1.62	0.10	0.10
query5	0.42	0.43	0.41
query6	1.14	0.65	0.66
query7	0.02	0.02	0.02
query8	0.04	0.03	0.03
query9	0.59	0.49	0.52
query10	0.57	0.56	0.55
query11	0.15	0.10	0.10
query12	0.14	0.10	0.11
query13	0.62	0.61	0.60
query14	2.76	2.81	2.83
query15	0.91	0.84	0.84
query16	0.37	0.39	0.38
query17	1.05	1.00	1.05
query18	0.23	0.20	0.21
query19	1.96	1.75	1.98
query20	0.02	0.02	0.01
query21	15.36	0.93	0.58
query22	0.76	0.81	0.80
query23	15.10	1.43	0.55
query24	3.04	0.90	1.45
query25	0.23	0.12	0.12
query26	0.27	0.17	0.14
query27	0.05	0.05	0.06
query28	13.52	0.99	0.43
query29	12.55	4.05	3.34
query30	0.25	0.09	0.07
query31	2.81	0.59	0.40
query32	3.23	0.55	0.46
query33	3.00	3.00	3.10
query34	16.63	5.22	4.53
query35	4.57	4.60	4.59
query36	0.65	0.50	0.48
query37	0.09	0.07	0.06
query38	0.05	0.04	0.04
query39	0.03	0.03	0.02
query40	0.17	0.13	0.12
query41	0.09	0.03	0.02
query42	0.04	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 105.49 s
Total hot run time: 30.92 s

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 41.57% (10857/26116)
Line Coverage: 31.97% (91671/286711)
Region Coverage: 31.11% (46945/150924)
Branch Coverage: 27.19% (23761/87404)
Coverage Report: http://coverage.selectdb-in.cc/coverage/02cd0be5c780f8181caf31ba7dd42e27cf40013a_02cd0be5c780f8181caf31ba7dd42e27cf40013a/report/index.html

@xinyiZzz xinyiZzz force-pushed the 20250120_fix_resource_ctx branch from 02cd0be to 72ad8eb Compare January 23, 2025 08:31
@xinyiZzz
Copy link
Contributor Author

run buildall

@xinyiZzz xinyiZzz force-pushed the 20250120_fix_resource_ctx branch from 72ad8eb to 0ab9736 Compare January 23, 2025 08:32
@xinyiZzz
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 32335 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 0ab973614d85e7c1dd630aac5317a938518f0b08, data reload: false

------ Round 1 ----------------------------------
q1	17770	5501	5426	5426
q2	2049	309	168	168
q3	10508	1278	733	733
q4	10230	965	547	547
q5	7676	2422	2215	2215
q6	195	162	132	132
q7	927	788	624	624
q8	9227	1385	1163	1163
q9	5173	4844	4887	4844
q10	6820	2330	1885	1885
q11	467	268	260	260
q12	351	352	225	225
q13	17783	3685	3127	3127
q14	220	236	214	214
q15	532	471	459	459
q16	637	623	589	589
q17	570	882	342	342
q18	7356	6479	6342	6342
q19	1976	952	542	542
q20	319	330	182	182
q21	2943	2188	2013	2013
q22	365	345	303	303
Total cold run time: 104094 ms
Total hot run time: 32335 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5766	5474	5470	5470
q2	239	332	233	233
q3	2260	2638	2353	2353
q4	1392	1806	1393	1393
q5	4303	4786	4666	4666
q6	179	165	130	130
q7	2099	1973	1829	1829
q8	2677	2822	2676	2676
q9	7366	7216	7252	7216
q10	2977	3280	2797	2797
q11	598	505	494	494
q12	671	805	614	614
q13	3467	3871	3281	3281
q14	277	287	272	272
q15	515	481	469	469
q16	645	694	628	628
q17	1228	1714	1251	1251
q18	7728	7572	7276	7276
q19	793	1179	1074	1074
q20	1974	2012	1887	1887
q21	5852	5309	4963	4963
q22	580	636	563	563
Total cold run time: 53586 ms
Total hot run time: 51535 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 184449 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 0ab973614d85e7c1dd630aac5317a938518f0b08, data reload: false

query1	981	386	380	380
query2	6525	2008	2010	2008
query3	6785	221	219	219
query4	33191	23525	22854	22854
query5	4330	605	438	438
query6	282	194	170	170
query7	4592	484	313	313
query8	296	244	223	223
query9	9284	2654	2647	2647
query10	491	321	249	249
query11	18304	15154	14926	14926
query12	148	107	107	107
query13	1668	533	422	422
query14	9707	6254	6995	6254
query15	234	193	185	185
query16	8034	633	464	464
query17	1587	727	573	573
query18	2077	391	291	291
query19	225	179	151	151
query20	115	139	108	108
query21	212	125	104	104
query22	4212	4401	4367	4367
query23	33940	33154	33278	33154
query24	6474	2369	2295	2295
query25	476	441	386	386
query26	1127	251	155	155
query27	2013	459	336	336
query28	5416	2473	2429	2429
query29	535	550	406	406
query30	230	185	163	163
query31	945	866	786	786
query32	75	67	57	57
query33	507	358	297	297
query34	749	863	514	514
query35	836	823	730	730
query36	978	1029	950	950
query37	113	100	83	83
query38	4194	4238	4005	4005
query39	1472	1384	1414	1384
query40	212	113	98	98
query41	54	51	50	50
query42	121	100	104	100
query43	512	521	479	479
query44	1397	835	806	806
query45	184	174	163	163
query46	846	1056	639	639
query47	1813	1852	1745	1745
query48	380	405	317	317
query49	795	494	394	394
query50	625	663	411	411
query51	4235	4165	4144	4144
query52	107	104	89	89
query53	225	258	190	190
query54	482	533	427	427
query55	80	78	76	76
query56	266	269	240	240
query57	1144	1177	1072	1072
query58	244	235	236	235
query59	2907	2941	2901	2901
query60	277	270	255	255
query61	118	118	137	118
query62	820	731	674	674
query63	227	199	194	194
query64	4293	1035	667	667
query65	3248	3169	3145	3145
query66	1070	402	316	316
query67	15808	15554	15425	15425
query68	2241	829	548	548
query69	399	304	262	262
query70	1168	1139	1114	1114
query71	305	290	254	254
query72	4969	3951	3900	3900
query73	639	744	359	359
query74	10026	9214	8954	8954
query75	3082	3167	2675	2675
query76	2258	1174	776	776
query77	331	402	275	275
query78	10108	10178	9263	9263
query79	989	869	602	602
query80	948	517	451	451
query81	525	278	237	237
query82	1130	160	129	129
query83	235	171	152	152
query84	231	95	71	71
query85	762	348	306	306
query86	373	289	304	289
query87	4534	4478	4425	4425
query88	3367	2205	2170	2170
query89	389	337	297	297
query90	1717	189	194	189
query91	131	140	109	109
query92	57	58	54	54
query93	926	863	535	535
query94	578	400	308	308
query95	339	277	259	259
query96	484	609	291	291
query97	2828	2841	2765	2765
query98	224	197	197	197
query99	1300	1381	1280	1280
Total cold run time: 273443 ms
Total hot run time: 184449 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.97 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 0ab973614d85e7c1dd630aac5317a938518f0b08, data reload: false

query1	0.03	0.03	0.03
query2	0.09	0.03	0.03
query3	0.23	0.06	0.07
query4	1.61	0.10	0.10
query5	0.42	0.42	0.40
query6	1.18	0.67	0.65
query7	0.03	0.01	0.02
query8	0.04	0.03	0.03
query9	0.58	0.51	0.50
query10	0.56	0.56	0.57
query11	0.14	0.10	0.10
query12	0.13	0.11	0.11
query13	0.61	0.59	0.59
query14	2.74	2.77	2.90
query15	0.89	0.83	0.83
query16	0.40	0.40	0.38
query17	1.02	1.03	1.06
query18	0.22	0.21	0.21
query19	1.86	1.82	1.96
query20	0.01	0.01	0.01
query21	15.46	1.00	0.59
query22	0.77	0.73	0.76
query23	15.26	1.48	0.59
query24	2.59	1.60	1.06
query25	0.16	0.25	0.08
query26	0.33	0.15	0.15
query27	0.06	0.07	0.05
query28	14.00	1.08	0.43
query29	12.58	3.90	3.26
query30	0.24	0.08	0.07
query31	2.84	0.59	0.39
query32	3.22	0.56	0.46
query33	2.98	3.04	3.04
query34	16.29	5.18	4.55
query35	4.59	4.51	4.54
query36	0.66	0.49	0.48
query37	0.10	0.06	0.06
query38	0.05	0.04	0.03
query39	0.04	0.03	0.02
query40	0.16	0.14	0.12
query41	0.07	0.03	0.03
query42	0.04	0.02	0.03
query43	0.04	0.03	0.03
Total cold run time: 105.32 s
Total hot run time: 30.97 s

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 41.95% (10958/26119)
Line Coverage: 32.22% (92391/286732)
Region Coverage: 31.35% (47326/150957)
Branch Coverage: 27.38% (23934/87414)
Coverage Report: http://coverage.selectdb-in.cc/coverage/0ab973614d85e7c1dd630aac5317a938518f0b08_0ab973614d85e7c1dd630aac5317a938518f0b08/report/index.html

@xinyiZzz
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 32696 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 815a434d048e39327555d49d966aa5d508773b4c, data reload: false

------ Round 1 ----------------------------------
q1	17581	5698	5380	5380
q2	2045	296	171	171
q3	10429	1250	758	758
q4	10214	987	533	533
q5	7553	2421	2228	2228
q6	196	168	141	141
q7	927	781	634	634
q8	9272	1431	1191	1191
q9	5248	4893	4965	4893
q10	6828	2350	1899	1899
q11	484	296	258	258
q12	355	354	223	223
q13	17782	3748	3161	3161
q14	233	221	204	204
q15	518	476	465	465
q16	654	617	598	598
q17	584	891	324	324
q18	7260	6757	6553	6553
q19	1659	961	575	575
q20	326	320	191	191
q21	3064	2189	1997	1997
q22	376	338	319	319
Total cold run time: 103588 ms
Total hot run time: 32696 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5588	5529	5515	5515
q2	246	340	236	236
q3	2261	2674	2350	2350
q4	1417	1796	1379	1379
q5	4379	4779	4718	4718
q6	176	162	130	130
q7	2090	2028	1853	1853
q8	2641	2849	2728	2728
q9	7316	7174	7264	7174
q10	3063	3296	2808	2808
q11	583	511	496	496
q12	634	783	629	629
q13	3618	3929	3322	3322
q14	273	289	274	274
q15	505	487	461	461
q16	656	690	649	649
q17	1257	1786	1257	1257
q18	7679	7462	7448	7448
q19	853	1156	1114	1114
q20	2009	2045	1911	1911
q21	5922	5187	5076	5076
q22	668	647	588	588
Total cold run time: 53834 ms
Total hot run time: 52116 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 196632 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 815a434d048e39327555d49d966aa5d508773b4c, data reload: false

query1	1332	935	958	935
query2	6075	2042	2013	2013
query3	10982	4500	4610	4500
query4	60847	28616	25749	25749
query5	5629	650	490	490
query6	433	202	194	194
query7	5518	551	309	309
query8	343	256	240	240
query9	8085	2665	2662	2662
query10	463	336	264	264
query11	17322	15045	15948	15045
query12	166	111	112	111
query13	1457	553	427	427
query14	11557	7642	7672	7642
query15	235	212	195	195
query16	7316	669	513	513
query17	1151	762	616	616
query18	1919	447	334	334
query19	209	195	180	180
query20	127	118	119	118
query21	216	128	112	112
query22	4336	4712	4411	4411
query23	34000	33386	33532	33386
query24	5620	2420	2410	2410
query25	483	494	401	401
query26	684	299	163	163
query27	1749	499	339	339
query28	3863	2510	2483	2483
query29	575	593	437	437
query30	213	191	160	160
query31	954	893	830	830
query32	98	56	60	56
query33	430	363	307	307
query34	781	931	526	526
query35	818	838	772	772
query36	1033	1039	981	981
query37	125	104	76	76
query38	4379	4352	4410	4352
query39	1486	1444	1467	1444
query40	224	123	104	104
query41	58	59	58	58
query42	125	105	106	105
query43	546	534	517	517
query44	1414	860	834	834
query45	185	188	172	172
query46	950	1109	703	703
query47	1904	1893	1837	1837
query48	391	442	335	335
query49	743	531	450	450
query50	709	730	431	431
query51	4344	4392	4229	4229
query52	113	107	100	100
query53	241	279	197	197
query54	501	530	436	436
query55	87	86	81	81
query56	276	282	282	282
query57	1194	1186	1179	1179
query58	242	239	247	239
query59	3321	3426	3103	3103
query60	288	296	271	271
query61	127	120	131	120
query62	783	761	655	655
query63	243	191	191	191
query64	1641	1079	674	674
query65	3269	3145	3174	3145
query66	745	394	302	302
query67	16145	15540	15462	15462
query68	4273	852	541	541
query69	505	301	259	259
query70	1228	1110	1164	1110
query71	413	308	268	268
query72	6102	3978	3975	3975
query73	754	821	367	367
query74	10305	9198	9074	9074
query75	3319	3203	2718	2718
query76	3498	1265	775	775
query77	503	372	304	304
query78	10071	11508	9653	9653
query79	2216	793	620	620
query80	1450	603	448	448
query81	539	279	235	235
query82	244	154	136	136
query83	270	173	226	173
query84	285	94	72	72
query85	738	400	322	322
query86	454	301	303	301
query87	4503	4471	4666	4471
query88	3492	2202	2197	2197
query89	393	316	292	292
query90	1554	195	197	195
query91	135	144	107	107
query92	66	58	53	53
query93	1130	773	547	547
query94	750	408	309	309
query95	340	271	264	264
query96	485	684	294	294
query97	2811	2862	2707	2707
query98	225	206	191	191
query99	1283	1375	1259	1259
Total cold run time: 308152 ms
Total hot run time: 196632 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.13 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 815a434d048e39327555d49d966aa5d508773b4c, data reload: false

query1	0.04	0.04	0.03
query2	0.07	0.03	0.03
query3	0.25	0.06	0.06
query4	1.62	0.10	0.10
query5	0.43	0.41	0.38
query6	1.18	0.66	0.66
query7	0.02	0.02	0.01
query8	0.04	0.03	0.04
query9	0.58	0.50	0.49
query10	0.56	0.56	0.56
query11	0.15	0.11	0.11
query12	0.14	0.10	0.10
query13	0.62	0.60	0.61
query14	2.74	2.77	2.77
query15	0.91	0.84	0.83
query16	0.38	0.38	0.39
query17	1.03	0.99	1.06
query18	0.24	0.21	0.21
query19	1.92	1.81	2.01
query20	0.01	0.02	0.01
query21	15.35	0.94	0.59
query22	0.76	0.77	0.70
query23	15.30	1.42	0.53
query24	3.72	1.05	0.48
query25	0.12	0.09	0.22
query26	0.38	0.15	0.14
query27	0.06	0.05	0.04
query28	12.88	1.01	0.45
query29	12.61	3.91	3.20
query30	0.25	0.10	0.06
query31	2.83	0.59	0.38
query32	3.23	0.56	0.46
query33	2.99	3.02	3.10
query34	16.44	5.14	4.54
query35	4.57	4.50	4.51
query36	0.64	0.49	0.48
query37	0.10	0.06	0.06
query38	0.04	0.04	0.03
query39	0.03	0.02	0.03
query40	0.16	0.14	0.12
query41	0.08	0.03	0.03
query42	0.04	0.02	0.03
query43	0.04	0.03	0.04
Total cold run time: 105.55 s
Total hot run time: 30.13 s

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 41.98% (10968/26125)
Line Coverage: 32.24% (92500/286868)
Region Coverage: 31.38% (47404/151055)
Branch Coverage: 27.45% (24012/87490)
Coverage Report: http://coverage.selectdb-in.cc/coverage/815a434d048e39327555d49d966aa5d508773b4c_815a434d048e39327555d49d966aa5d508773b4c/report/index.html

@yiguolei yiguolei force-pushed the 20250120_fix_resource_ctx branch from 815a434 to dcfaeab Compare January 25, 2025 02:20
@yiguolei
Copy link
Contributor

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 32347 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit dcfaeabc7b7d84dd5f5df1c8f9854cf8f4db63d2, data reload: false

------ Round 1 ----------------------------------
q1	17566	5484	5366	5366
q2	2050	307	187	187
q3	10391	1239	744	744
q4	10227	970	539	539
q5	8127	2405	2181	2181
q6	198	169	130	130
q7	908	774	593	593
q8	9226	1337	1187	1187
q9	5128	4880	4871	4871
q10	6828	2345	1883	1883
q11	453	275	260	260
q12	339	361	215	215
q13	17781	3671	3214	3214
q14	233	230	204	204
q15	513	460	473	460
q16	633	629	581	581
q17	570	851	323	323
q18	6941	6421	6492	6421
q19	1241	964	537	537
q20	323	330	191	191
q21	2864	2180	1944	1944
q22	359	332	316	316
Total cold run time: 102899 ms
Total hot run time: 32347 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5443	5564	5488	5488
q2	234	322	232	232
q3	2263	2668	2391	2391
q4	1392	1822	1423	1423
q5	4312	4749	4687	4687
q6	170	159	127	127
q7	2010	1970	1826	1826
q8	2593	2821	2699	2699
q9	7396	7237	7259	7237
q10	3046	3302	2790	2790
q11	596	503	490	490
q12	667	756	618	618
q13	3533	4012	3304	3304
q14	284	305	271	271
q15	535	480	481	480
q16	667	709	649	649
q17	1212	1754	1259	1259
q18	7753	7625	7375	7375
q19	811	1185	1051	1051
q20	1996	2029	1893	1893
q21	5773	5244	4869	4869
q22	617	580	567	567
Total cold run time: 53303 ms
Total hot run time: 51726 ms

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 42.06% (10990/26127)
Line Coverage: 32.34% (92772/286907)
Region Coverage: 31.49% (47581/151080)
Branch Coverage: 27.53% (24087/87506)
Coverage Report: http://coverage.selectdb-in.cc/coverage/dcfaeabc7b7d84dd5f5df1c8f9854cf8f4db63d2_dcfaeabc7b7d84dd5f5df1c8f9854cf8f4db63d2/report/index.html

@doris-robot
Copy link

TPC-DS: Total hot run time: 191605 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit dcfaeabc7b7d84dd5f5df1c8f9854cf8f4db63d2, data reload: false

query1	1293	957	934	934
query2	6142	2141	2067	2067
query3	10967	4544	4492	4492
query4	61029	28182	23158	23158
query5	5480	602	429	429
query6	430	196	198	196
query7	5497	504	304	304
query8	328	263	229	229
query9	7963	2607	2607	2607
query10	455	339	254	254
query11	17236	15023	15523	15023
query12	154	112	116	112
query13	1417	549	420	420
query14	10866	7424	6713	6713
query15	235	220	198	198
query16	7079	658	481	481
query17	1119	742	603	603
query18	1141	388	319	319
query19	208	186	161	161
query20	124	117	117	117
query21	213	124	102	102
query22	4431	4549	4593	4549
query23	35294	33455	33472	33455
query24	5492	2337	2314	2314
query25	469	453	407	407
query26	709	254	151	151
query27	1977	485	345	345
query28	3974	2486	2471	2471
query29	522	567	445	445
query30	216	194	157	157
query31	967	924	844	844
query32	67	60	57	57
query33	434	364	299	299
query34	739	882	511	511
query35	834	833	764	764
query36	1008	1053	972	972
query37	123	94	80	80
query38	4329	4368	4219	4219
query39	1461	1446	1418	1418
query40	197	122	108	108
query41	50	51	47	47
query42	126	109	102	102
query43	527	535	516	516
query44	1341	834	873	834
query45	200	179	164	164
query46	892	1066	654	654
query47	1930	1970	1880	1880
query48	370	411	337	337
query49	723	518	415	415
query50	666	676	390	390
query51	4341	4317	4215	4215
query52	106	107	97	97
query53	232	259	188	188
query54	507	495	417	417
query55	89	80	79	79
query56	288	286	244	244
query57	1182	1217	1156	1156
query58	238	244	238	238
query59	3269	3411	3094	3094
query60	282	261	247	247
query61	147	113	112	112
query62	709	753	647	647
query63	223	183	178	178
query64	1825	1034	647	647
query65	3230	3131	3136	3131
query66	670	395	289	289
query67	16127	15539	15390	15390
query68	4928	831	532	532
query69	487	284	259	259
query70	1204	1151	1186	1151
query71	406	288	263	263
query72	5999	3851	3865	3851
query73	774	745	383	383
query74	9975	8906	9085	8906
query75	3249	3151	2708	2708
query76	3769	1162	756	756
query77	476	369	267	267
query78	10110	9997	9326	9326
query79	2483	801	621	621
query80	681	525	484	484
query81	505	272	233	233
query82	220	152	129	129
query83	173	171	147	147
query84	288	89	77	77
query85	733	352	297	297
query86	413	326	303	303
query87	4375	4502	4364	4364
query88	3559	2186	2160	2160
query89	395	325	289	289
query90	1646	194	192	192
query91	135	138	104	104
query92	73	60	54	54
query93	2591	845	531	531
query94	705	411	279	279
query95	336	270	257	257
query96	482	622	289	289
query97	2839	2877	2736	2736
query98	222	196	189	189
query99	1271	1328	1275	1275
Total cold run time: 308411 ms
Total hot run time: 191605 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.77 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit dcfaeabc7b7d84dd5f5df1c8f9854cf8f4db63d2, data reload: false

query1	0.04	0.04	0.03
query2	0.08	0.03	0.03
query3	0.24	0.07	0.06
query4	1.61	0.11	0.11
query5	0.44	0.41	0.42
query6	1.17	0.66	0.66
query7	0.02	0.02	0.02
query8	0.05	0.04	0.03
query9	0.58	0.50	0.51
query10	0.56	0.54	0.55
query11	0.15	0.11	0.10
query12	0.15	0.12	0.11
query13	0.62	0.60	0.59
query14	2.86	2.76	2.76
query15	0.89	0.83	0.82
query16	0.38	0.38	0.37
query17	1.05	1.03	1.06
query18	0.22	0.20	0.21
query19	1.97	1.88	2.00
query20	0.01	0.02	0.01
query21	15.35	0.93	0.58
query22	0.76	0.75	0.57
query23	15.42	1.37	0.52
query24	3.62	1.76	1.21
query25	0.12	0.14	0.05
query26	0.28	0.15	0.13
query27	0.06	0.06	0.04
query28	14.23	0.95	0.44
query29	12.56	3.90	3.25
query30	0.26	0.09	0.06
query31	2.83	0.59	0.39
query32	3.23	0.55	0.47
query33	3.10	2.99	3.04
query34	16.54	5.22	4.54
query35	4.55	4.47	4.48
query36	0.65	0.51	0.49
query37	0.09	0.06	0.06
query38	0.05	0.04	0.03
query39	0.04	0.03	0.02
query40	0.17	0.13	0.13
query41	0.08	0.03	0.02
query42	0.04	0.02	0.03
query43	0.04	0.03	0.03
Total cold run time: 107.16 s
Total hot run time: 30.77 s

yiguolei
yiguolei approved these changes Jan 25, 2025
View reviewed changes
Copy link
Contributor

@yiguolei yiguolei left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jan 25, 2025
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

dataroaring
dataroaring approved these changes Jan 26, 2025
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@dataroaring dataroaring dataroaring approved these changes

@yiguolei yiguolei yiguolei approved these changes

@gavinchou gavinchou Awaiting requested review from gavinchou gavinchou is a code owner

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@xinyiZzz @hello-stephen @doris-robot @yiguolei @dataroaring