Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

branch-3.0: [fix](restore) Make the DirMoveTask idempotent. #47313 #47426

Open
wants to merge 1 commit into
base: branch-3.0
Choose a base branch
from

Conversation

github-actions[bot]
Copy link
Contributor

Cherry-picked from #47313

Since the DirMoveTask is executed asynchronously, the FE might send the
task again to ensure its completion eventually. But the rowsets
committed during two DirMoveTasks (if any) will be dropped, which causes
the data loss.

This PR adds a LOADED tag file to indicate that the snapshot has been
loaded into a tablet and should not be reloaded again.
@github-actions github-actions bot requested a review from dataroaring as a code owner January 24, 2025 09:54
@hello-stephen
Copy link
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@dataroaring dataroaring reopened this Jan 24, 2025
@hello-stephen
Copy link
Contributor

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 41439 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ef6a4ee3677d9d9f5fa380bcbb317f886f4eda3f, data reload: false

------ Round 1 ----------------------------------
q1	17582	7645	7393	7393
q2	2057	167	177	167
q3	10611	1081	1165	1081
q4	10565	749	755	749
q5	7757	2887	2854	2854
q6	244	152	149	149
q7	1017	623	611	611
q8	9366	2041	2114	2041
q9	6681	6413	6448	6413
q10	6988	2302	2308	2302
q11	458	267	269	267
q12	438	217	217	217
q13	17779	2997	2990	2990
q14	240	214	220	214
q15	574	517	524	517
q16	637	597	576	576
q17	994	549	578	549
q18	7227	6612	6744	6612
q19	1413	1179	1208	1179
q20	486	208	200	200
q21	4213	3496	3375	3375
q22	1117	983	1001	983
Total cold run time: 108444 ms
Total hot run time: 41439 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7358	7388	7332	7332
q2	332	234	232	232
q3	3043	2956	2964	2956
q4	2123	1971	1825	1825
q5	5738	5782	5733	5733
q6	242	150	148	148
q7	2287	1830	1815	1815
q8	3350	3547	3448	3448
q9	8994	8978	8993	8978
q10	3630	3602	3574	3574
q11	591	487	495	487
q12	832	632	655	632
q13	8924	3194	3119	3119
q14	305	273	284	273
q15	581	521	541	521
q16	706	658	632	632
q17	1866	1658	1647	1647
q18	8340	7909	7794	7794
q19	1777	1628	1716	1628
q20	2174	1902	1899	1899
q21	5629	5336	5512	5336
q22	1164	1057	1058	1057
Total cold run time: 69986 ms
Total hot run time: 61066 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 197669 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit ef6a4ee3677d9d9f5fa380bcbb317f886f4eda3f, data reload: false

query1	1326	913	909	909
query2	6251	2102	2073	2073
query3	10981	4397	4465	4397
query4	66735	27073	23487	23487
query5	5063	445	434	434
query6	411	177	171	171
query7	5557	300	303	300
query8	311	217	230	217
query9	8728	2697	2703	2697
query10	463	265	265	265
query11	17394	15237	15919	15237
query12	154	102	102	102
query13	1501	424	439	424
query14	9997	7359	7013	7013
query15	197	189	184	184
query16	7108	518	519	518
query17	1097	595	632	595
query18	1860	349	326	326
query19	212	162	163	162
query20	118	111	117	111
query21	207	110	107	107
query22	4943	4718	4526	4526
query23	34897	34118	34391	34118
query24	6105	3008	2940	2940
query25	537	433	424	424
query26	660	178	179	178
query27	1796	363	357	357
query28	4269	2515	2436	2436
query29	713	473	447	447
query30	239	161	164	161
query31	985	814	856	814
query32	72	61	59	59
query33	435	302	292	292
query34	910	516	550	516
query35	852	759	723	723
query36	1071	967	980	967
query37	127	83	77	77
query38	4138	4084	3985	3985
query39	1544	1496	1481	1481
query40	205	101	96	96
query41	48	47	49	47
query42	117	101	100	100
query43	535	507	507	507
query44	1160	813	812	812
query45	181	180	172	172
query46	1164	744	742	742
query47	2009	1925	1919	1919
query48	496	388	383	383
query49	703	380	394	380
query50	885	441	449	441
query51	7412	7238	7008	7008
query52	101	89	87	87
query53	261	182	180	180
query54	568	452	434	434
query55	76	76	77	76
query56	247	233	250	233
query57	1227	1131	1105	1105
query58	213	202	220	202
query59	3213	2884	2955	2884
query60	273	250	258	250
query61	111	109	107	107
query62	753	645	669	645
query63	224	187	189	187
query64	1364	651	673	651
query65	3235	3188	3224	3188
query66	690	322	306	306
query67	15950	15709	15775	15709
query68	3917	603	558	558
query69	424	264	269	264
query70	1194	1137	1143	1137
query71	349	260	256	256
query72	6443	4021	4219	4021
query73	768	348	347	347
query74	10231	8898	9003	8898
query75	3378	2684	2681	2681
query76	1888	1016	1106	1016
query77	520	274	277	274
query78	10714	9710	9624	9624
query79	1442	597	588	588
query80	842	431	423	423
query81	500	243	240	240
query82	1239	120	122	120
query83	199	144	197	144
query84	282	80	80	80
query85	869	315	297	297
query86	367	299	313	299
query87	4433	4468	4259	4259
query88	3971	2364	2324	2324
query89	417	285	295	285
query90	1999	190	190	190
query91	197	151	152	151
query92	68	51	51	51
query93	1850	541	548	541
query94	772	309	307	307
query95	357	254	252	252
query96	607	275	281	275
query97	3319	3195	3189	3189
query98	211	210	196	196
query99	1676	1302	1301	1301
Total cold run time: 318828 ms
Total hot run time: 197669 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 32.6 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit ef6a4ee3677d9d9f5fa380bcbb317f886f4eda3f, data reload: false

query1	0.04	0.03	0.03
query2	0.07	0.03	0.04
query3	0.23	0.07	0.07
query4	1.62	0.10	0.11
query5	0.51	0.50	0.50
query6	1.13	0.73	0.72
query7	0.02	0.01	0.01
query8	0.04	0.03	0.05
query9	0.57	0.50	0.49
query10	0.56	0.55	0.55
query11	0.14	0.10	0.11
query12	0.14	0.12	0.11
query13	0.60	0.60	0.60
query14	2.76	2.75	2.74
query15	0.90	0.83	0.81
query16	0.38	0.41	0.37
query17	1.06	1.02	1.01
query18	0.24	0.21	0.22
query19	1.89	1.77	1.98
query20	0.01	0.02	0.01
query21	15.38	0.62	0.57
query22	2.49	1.63	1.64
query23	17.05	0.90	0.74
query24	3.20	1.32	1.00
query25	0.18	0.04	0.14
query26	0.53	0.14	0.14
query27	0.04	0.03	0.05
query28	10.29	1.11	1.07
query29	12.62	3.34	3.35
query30	0.25	0.06	0.05
query31	2.85	0.40	0.39
query32	3.23	0.45	0.47
query33	3.04	3.04	3.06
query34	17.27	4.58	4.53
query35	4.65	4.52	4.61
query36	0.70	0.48	0.48
query37	0.09	0.06	0.07
query38	0.05	0.04	0.04
query39	0.03	0.02	0.03
query40	0.17	0.13	0.13
query41	0.08	0.02	0.02
query42	0.04	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 107.17 s
Total hot run time: 32.6 s

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants