Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[cherry-pick](branch-2.0) Pick "[Fix](Rowset Id) Use a randomly generated rowset ID to handle memory write failures (#42949) #46102

Merged
merged 3 commits into from
Jan 2, 2025

Conversation

Yukang-Lian
Copy link
Collaborator

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Pick #42949 #46074

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@hello-stephen
Copy link
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@Yukang-Lian
Copy link
Collaborator Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 49097 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 47586ac61aad984bd5157c18f4fe6c35efcf35f2, data reload: false

------ Round 1 ----------------------------------
q1	17625	4404	4383	4383
q2	2065	157	148	148
q3	10265	1870	1940	1870
q4	10308	1254	1313	1254
q5	8511	3878	3882	3878
q6	236	125	123	123
q7	2065	1569	1593	1569
q8	9311	2747	2727	2727
q9	10319	10025	9842	9842
q10	8618	3491	3565	3491
q11	425	254	247	247
q12	471	296	305	296
q13	18356	3978	4034	3978
q14	354	324	337	324
q15	520	457	447	447
q16	551	456	463	456
q17	1141	974	952	952
q18	7349	6861	6818	6818
q19	1717	1544	1510	1510
q20	507	330	314	314
q21	4395	4198	4083	4083
q22	509	387	398	387
Total cold run time: 115618 ms
Total hot run time: 49097 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4341	4302	4325	4302
q2	327	223	218	218
q3	4192	4150	4172	4150
q4	2768	2738	2764	2738
q5	7205	7138	7110	7110
q6	242	122	118	118
q7	3272	2818	2873	2818
q8	4345	4490	4515	4490
q9	13618	13604	13566	13566
q10	4272	4265	4257	4257
q11	770	700	688	688
q12	1041	880	876	876
q13	6951	3734	3774	3734
q14	453	442	443	442
q15	500	450	460	450
q16	640	608	597	597
q17	3809	3815	3861	3815
q18	8759	8738	8839	8738
q19	1738	1661	1710	1661
q20	2388	2089	2135	2089
q21	8450	8411	8424	8411
q22	1028	960	937	937
Total cold run time: 81109 ms
Total hot run time: 76205 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 212534 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 47586ac61aad984bd5157c18f4fe6c35efcf35f2, data reload: false

query1	957	422	384	384
query2	6527	2137	1976	1976
query3	6925	207	202	202
query4	23360	21639	21630	21630
query5	19743	6509	6516	6509
query6	292	221	238	221
query7	4336	303	315	303
query8	281	280	243	243
query9	3136	2718	2628	2628
query10	468	315	300	300
query11	16063	15320	14957	14957
query12	125	82	76	76
query13	1025	453	444	444
query14	17250	13190	13451	13190
query15	356	218	233	218
query16	6490	288	273	273
query17	1699	947	933	933
query18	899	309	309	309
query19	220	154	160	154
query20	108	96	102	96
query21	191	100	95	95
query22	5246	5068	5060	5060
query23	34140	33293	33560	33293
query24	7875	6394	6310	6310
query25	516	432	446	432
query26	1272	168	166	166
query27	2419	302	301	301
query28	6126	2308	2265	2265
query29	2793	2709	2869	2709
query30	241	168	169	168
query31	964	755	736	736
query32	74	75	63	63
query33	447	267	271	267
query34	870	508	466	466
query35	1133	923	906	906
query36	1187	1411	1152	1152
query37	181	62	61	61
query38	3095	2941	2881	2881
query39	1364	1330	1319	1319
query40	311	94	97	94
query41	45	39	49	39
query42	87	87	93	87
query43	634	525	606	525
query44	1156	736	741	736
query45	251	231	228	228
query46	1228	958	966	958
query47	1886	1740	1763	1740
query48	518	408	412	408
query49	664	376	379	376
query50	855	612	577	577
query51	4779	4670	4665	4665
query52	83	78	88	78
query53	229	182	189	182
query54	2671	2487	2449	2449
query55	92	83	86	83
query56	238	205	214	205
query57	1286	1169	1179	1169
query58	222	211	207	207
query59	3370	3164	3193	3164
query60	226	200	207	200
query61	97	93	106	93
query62	807	449	476	449
query63	207	180	174	174
query64	3484	1633	1496	1496
query65	3659	3537	3562	3537
query66	803	390	396	390
query67	15732	16644	15248	15248
query68	9406	679	647	647
query69	498	272	269	269
query70	1703	1371	1335	1335
query71	410	322	317	317
query72	6749	4884	4816	4816
query73	767	314	329	314
query74	6317	5903	5816	5816
query75	5197	3685	3683	3683
query76	5097	1146	1168	1146
query77	858	267	269	267
query78	12616	12731	11920	11920
query79	11378	663	651	651
query80	847	396	405	396
query81	490	236	238	236
query82	1102	100	98	98
query83	170	133	137	133
query84	264	69	73	69
query85	867	312	311	311
query86	339	303	297	297
query87	3188	3030	3048	3030
query88	5271	2350	2328	2328
query89	405	313	285	285
query90	1900	218	217	217
query91	156	134	127	127
query92	66	53	51	51
query93	5846	639	604	604
query94	750	209	207	207
query95	2029	1940	1978	1940
query96	669	328	340	328
query97	6500	6402	6405	6402
query98	230	225	208	208
query99	2756	886	833	833
Total cold run time: 320855 ms
Total hot run time: 212534 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 47586ac61aad984bd5157c18f4fe6c35efcf35f2, data reload: false

query1	0.02	0.03	0.02
query2	0.07	0.03	0.03
query3	0.25	0.05	0.05
query4	1.76	0.06	0.05
query5	0.53	0.52	0.52
query6	1.23	0.62	0.62
query7	0.01	0.01	0.01
query8	0.04	0.02	0.02
query9	0.53	0.48	0.47
query10	0.55	0.53	0.53
query11	0.13	0.09	0.09
query12	0.12	0.09	0.09
query13	0.63	0.62	0.60
query14	0.79	0.80	0.77
query15	0.79	0.76	0.74
query16	0.38	0.37	0.36
query17	1.01	1.03	1.02
query18	0.23	0.24	0.26
query19	1.93	1.86	1.85
query20	0.02	0.01	0.01
query21	15.44	0.57	0.55
query22	2.55	2.27	1.88
query23	17.26	1.02	1.00
query24	4.69	0.95	1.51
query25	0.34	0.07	0.06
query26	0.57	0.15	0.16
query27	0.06	0.03	0.04
query28	8.08	0.78	0.78
query29	12.68	2.34	2.20
query30	0.63	0.63	0.53
query31	2.82	0.39	0.37
query32	3.35	0.50	0.49
query33	3.05	3.07	3.07
query34	15.23	4.77	4.80
query35	4.88	4.82	4.84
query36	1.06	1.03	1.01
query37	0.05	0.05	0.05
query38	0.04	0.02	0.02
query39	0.03	0.01	0.02
query40	0.16	0.14	0.14
query41	0.07	0.02	0.01
query42	0.02	0.01	0.02
query43	0.02	0.01	0.02
Total cold run time: 104.1 s
Total hot run time: 31 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 47586ac61aad984bd5157c18f4fe6c35efcf35f2 with default session variables
Stream load json:         21 seconds loaded 2358488459 Bytes, about 107 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select:       21.3 seconds inserted 10000000 Rows, about 469K ops/s

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jan 1, 2025
Copy link
Contributor

github-actions bot commented Jan 1, 2025

PR approved by at least one committer and no changes requested.

Copy link
Contributor

github-actions bot commented Jan 1, 2025

PR approved by anyone and no changes requested.

@Yukang-Lian
Copy link
Collaborator Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 49509 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ce05c47313f7434d66eb312bb51d5cb030a1b822, data reload: false

------ Round 1 ----------------------------------
q1	18345	4476	4359	4359
q2	2030	152	148	148
q3	10411	1940	1919	1919
q4	10277	1270	1351	1270
q5	8507	3892	3922	3892
q6	231	146	125	125
q7	2029	1609	1594	1594
q8	9321	2742	2735	2735
q9	10609	10139	10040	10040
q10	8675	3532	3506	3506
q11	419	259	258	258
q12	480	306	307	306
q13	18384	3960	4024	3960
q14	347	328	332	328
q15	505	470	449	449
q16	534	458	473	458
q17	1131	955	921	921
q18	7237	6924	6993	6924
q19	1749	1592	1515	1515
q20	527	293	326	293
q21	4340	4146	4109	4109
q22	493	400	406	400
Total cold run time: 116581 ms
Total hot run time: 49509 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4352	4325	4318	4318
q2	329	235	225	225
q3	4172	4192	4150	4150
q4	2751	2760	2770	2760
q5	7206	7148	7117	7117
q6	240	123	124	123
q7	3206	2917	2838	2838
q8	4371	4476	4482	4476
q9	14411	13944	13916	13916
q10	4231	4275	4248	4248
q11	785	673	666	666
q12	1024	871	845	845
q13	6986	3741	3753	3741
q14	461	435	417	417
q15	500	452	465	452
q16	639	593	590	590
q17	3915	3845	3842	3842
q18	8830	8750	8743	8743
q19	1716	1653	1640	1640
q20	2404	2140	2094	2094
q21	8559	8429	8417	8417
q22	1015	919	972	919
Total cold run time: 82103 ms
Total hot run time: 76537 ms

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 38.55% (8310/21555)
Line Coverage: 30.24% (68713/227215)
Region Coverage: 29.68% (35398/119278)
Branch Coverage: 25.44% (18196/71530)
Coverage Report: http://coverage.selectdb-in.cc/coverage/ce05c47313f7434d66eb312bb51d5cb030a1b822_ce05c47313f7434d66eb312bb51d5cb030a1b822/report/index.html

@doris-robot
Copy link

TPC-DS: Total hot run time: 214193 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit ce05c47313f7434d66eb312bb51d5cb030a1b822, data reload: false

query1	991	394	375	375
query2	6544	2380	2108	2108
query3	6932	203	203	203
query4	23301	21769	21555	21555
query5	19759	6501	6567	6501
query6	300	221	231	221
query7	4337	310	307	307
query8	273	254	243	243
query9	3121	2710	2626	2626
query10	466	321	299	299
query11	15464	15183	14924	14924
query12	133	79	78	78
query13	1048	463	455	455
query14	17718	13687	13660	13660
query15	371	228	229	228
query16	6478	293	285	285
query17	1704	938	937	937
query18	887	323	328	323
query19	223	149	151	149
query20	100	92	92	92
query21	195	95	94	94
query22	5158	4764	4863	4764
query23	34263	33606	33599	33599
query24	7906	6318	6306	6306
query25	530	442	429	429
query26	1267	164	165	164
query27	2450	297	294	294
query28	6183	2329	2303	2303
query29	2871	2883	2715	2715
query30	248	170	166	166
query31	940	779	735	735
query32	73	60	59	59
query33	461	271	273	271
query34	873	477	476	476
query35	1124	915	955	915
query36	1363	1357	1190	1190
query37	177	62	62	62
query38	3101	2969	2920	2920
query39	1380	1346	1325	1325
query40	327	98	95	95
query41	40	40	39	39
query42	87	91	88	88
query43	669	630	522	522
query44	1175	721	726	721
query45	250	234	234	234
query46	1249	984	964	964
query47	1817	1771	1731	1731
query48	515	427	413	413
query49	654	394	422	394
query50	853	676	639	639
query51	4719	4685	4697	4685
query52	90	83	80	80
query53	231	186	197	186
query54	2681	2467	2513	2467
query55	88	75	81	75
query56	228	219	214	214
query57	1247	1199	1297	1199
query58	253	214	205	205
query59	3437	3373	3362	3362
query60	223	216	214	214
query61	100	104	100	100
query62	740	496	435	435
query63	206	178	180	178
query64	3554	1628	1485	1485
query65	3610	3545	3571	3545
query66	769	452	389	389
query67	16473	16296	15665	15665
query68	10323	665	679	665
query69	506	273	282	273
query70	1770	1330	1435	1330
query71	417	306	310	306
query72	6861	4945	4901	4901
query73	778	313	322	313
query74	6270	5848	5824	5824
query75	5358	3714	3735	3714
query76	6080	1185	1199	1185
query77	1035	257	259	257
query78	12952	12515	11795	11795
query79	10630	649	658	649
query80	804	395	397	395
query81	497	245	244	244
query82	703	97	96	96
query83	178	137	136	136
query84	258	71	70	70
query85	911	328	331	328
query86	342	305	306	305
query87	3285	3079	3047	3047
query88	4930	2339	2340	2339
query89	364	292	288	288
query90	2673	209	210	209
query91	160	143	131	131
query92	60	51	50	50
query93	4932	606	596	596
query94	1305	204	209	204
query95	2092	2006	2047	2006
query96	667	327	331	327
query97	6563	6471	6501	6471
query98	217	205	213	205
query99	3125	952	854	854
Total cold run time: 324044 ms
Total hot run time: 214193 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.59 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit ce05c47313f7434d66eb312bb51d5cb030a1b822, data reload: false

query1	0.02	0.02	0.02
query2	0.07	0.03	0.03
query3	0.26	0.04	0.04
query4	1.78	0.07	0.06
query5	0.53	0.51	0.52
query6	1.24	0.63	0.62
query7	0.01	0.01	0.01
query8	0.03	0.02	0.02
query9	0.52	0.46	0.47
query10	0.53	0.54	0.54
query11	0.13	0.09	0.08
query12	0.12	0.09	0.09
query13	0.63	0.62	0.62
query14	0.79	0.78	0.78
query15	0.78	0.76	0.78
query16	0.36	0.37	0.39
query17	1.03	1.01	1.02
query18	0.22	0.25	0.26
query19	1.93	1.87	1.83
query20	0.02	0.01	0.01
query21	15.49	0.56	0.55
query22	2.10	1.88	2.03
query23	17.45	0.97	0.92
query24	8.46	0.60	0.75
query25	0.38	0.09	0.04
query26	0.84	0.17	0.16
query27	0.05	0.04	0.04
query28	5.36	0.80	0.72
query29	12.77	2.26	2.37
query30	0.62	0.53	0.53
query31	2.82	0.39	0.38
query32	3.35	0.50	0.48
query33	3.07	3.05	3.03
query34	15.27	4.78	4.80
query35	4.87	4.86	4.87
query36	1.07	1.00	1.02
query37	0.06	0.05	0.05
query38	0.04	0.02	0.02
query39	0.02	0.01	0.02
query40	0.16	0.14	0.15
query41	0.07	0.01	0.02
query42	0.02	0.01	0.02
query43	0.03	0.01	0.01
Total cold run time: 105.37 s
Total hot run time: 30.59 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit ce05c47313f7434d66eb312bb51d5cb030a1b822 with default session variables
Stream load json:         20 seconds loaded 2358488459 Bytes, about 112 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       21.4 seconds inserted 10000000 Rows, about 467K ops/s

@gavinchou gavinchou merged commit 6debbc3 into apache:branch-2.0 Jan 2, 2025
17 of 20 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by one committer. reviewed
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants