Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[improvement](statistics)External table getRowCount return -1 when row count is not available or row count is 0. (#43009) #43087

Merged
merged 1 commit into from
Nov 2, 2024

Conversation

Jibing-Li
Copy link
Contributor

backport: #43009

@Jibing-Li Jibing-Li marked this pull request as ready for review November 1, 2024 11:06
@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@Jibing-Li
Copy link
Contributor Author

run buildall

…w count is not available or row count is 0. (apache#43009)

External table getRowCount return -1 when row count is not available or
row count is 0. So the behavior of external table could match with
internal olap table.
@Jibing-Li Jibing-Li force-pushed the externalrowcount2.0 branch from a3bd58b to 27f72a4 Compare November 1, 2024 11:12
@Jibing-Li
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 49512 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 27f72a49ade7ba09ed2abea13049e3b2c4bdf412, data reload: false

------ Round 1 ----------------------------------
q1	18151	4356	4303	4303
q2	2065	156	147	147
q3	10291	1925	1947	1925
q4	10694	1252	1346	1252
q5	8426	3928	3972	3928
q6	234	124	125	124
q7	2063	1672	1596	1596
q8	9374	2756	2703	2703
q9	10216	9831	10089	9831
q10	9717	3573	3523	3523
q11	422	254	247	247
q12	475	303	293	293
q13	18546	4164	4144	4144
q14	377	339	352	339
q15	532	456	450	450
q16	560	498	491	491
q17	1138	958	938	938
q18	7622	7309	6922	6922
q19	1710	1555	1507	1507
q20	575	331	310	310
q21	4442	4155	4138	4138
q22	504	401	429	401
Total cold run time: 118134 ms
Total hot run time: 49512 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4371	4313	4284	4284
q2	321	233	224	224
q3	4163	4114	4119	4114
q4	2729	2732	2728	2728
q5	7110	7084	7076	7076
q6	237	119	124	119
q7	3240	2845	2870	2845
q8	4344	4489	4448	4448
q9	13679	13777	13597	13597
q10	4190	4261	4254	4254
q11	747	659	693	659
q12	1026	859	846	846
q13	6576	3753	3757	3753
q14	449	417	413	413
q15	493	454	450	450
q16	635	595	593	593
q17	3865	3893	3889	3889
q18	8784	8755	8885	8755
q19	1704	1672	1672	1672
q20	2371	2133	2098	2098
q21	8578	8535	8432	8432
q22	1056	944	953	944
Total cold run time: 80668 ms
Total hot run time: 76193 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 211203 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 27f72a49ade7ba09ed2abea13049e3b2c4bdf412, data reload: false

query1	929	424	375	375
query2	6531	2151	2114	2114
query3	6916	199	205	199
query4	23319	21562	21427	21427
query5	19722	6505	6521	6505
query6	287	216	238	216
query7	4341	290	299	290
query8	256	251	285	251
query9	3069	2612	2542	2542
query10	462	313	301	301
query11	15782	15086	14710	14710
query12	122	72	75	72
query13	1042	428	427	427
query14	17265	13263	13163	13163
query15	379	220	234	220
query16	5681	278	263	263
query17	1718	949	935	935
query18	883	317	310	310
query19	188	161	159	159
query20	107	95	102	95
query21	189	97	98	97
query22	5229	5112	5012	5012
query23	34099	33430	33645	33430
query24	7879	6273	6297	6273
query25	523	430	440	430
query26	1385	162	158	158
query27	2551	296	292	292
query28	6086	2214	2191	2191
query29	2894	2734	2840	2734
query30	248	168	168	168
query31	967	762	770	762
query32	68	62	61	61
query33	453	254	259	254
query34	864	455	465	455
query35	1125	873	899	873
query36	1221	1083	1111	1083
query37	174	58	60	58
query38	3100	2919	2969	2919
query39	1402	1342	1359	1342
query40	314	94	98	94
query41	42	39	38	38
query42	94	105	95	95
query43	657	630	647	630
query44	1179	735	715	715
query45	243	228	227	227
query46	1218	954	947	947
query47	2101	1645	1785	1645
query48	511	419	418	418
query49	661	381	382	381
query50	855	646	599	599
query51	4771	4712	4649	4649
query52	86	78	72	72
query53	220	195	187	187
query54	2666	2459	2435	2435
query55	95	84	83	83
query56	231	208	206	206
query57	1239	1207	1136	1136
query58	212	195	212	195
query59	3332	3144	3472	3144
query60	222	209	188	188
query61	97	95	97	95
query62	808	458	479	458
query63	194	169	180	169
query64	3629	1615	1507	1507
query65	3629	3590	3569	3569
query66	793	395	441	395
query67	15596	14966	15109	14966
query68	10289	651	657	651
query69	501	265	264	264
query70	1746	1412	1396	1396
query71	401	313	315	313
query72	6797	4818	4790	4790
query73	760	321	314	314
query74	6331	5823	5818	5818
query75	5404	3740	3757	3740
query76	6009	1102	1192	1102
query77	1022	266	260	260
query78	12512	11759	11641	11641
query79	7793	627	623	623
query80	1490	379	380	379
query81	488	241	237	237
query82	1692	98	98	98
query83	169	130	132	130
query84	259	69	71	69
query85	886	315	315	315
query86	320	292	287	287
query87	3320	3030	2978	2978
query88	4609	2286	2297	2286
query89	483	294	290	290
query90	1956	213	211	211
query91	154	132	125	125
query92	59	52	52	52
query93	6241	570	539	539
query94	703	208	211	208
query95	1981	1924	1966	1924
query96	643	325	319	319
query97	6571	6326	6426	6326
query98	217	203	205	203
query99	2804	1031	906	906
Total cold run time: 319761 ms
Total hot run time: 211203 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.37 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 27f72a49ade7ba09ed2abea13049e3b2c4bdf412, data reload: false

query1	0.02	0.03	0.03
query2	0.07	0.03	0.03
query3	0.25	0.05	0.04
query4	1.78	0.10	0.07
query5	0.53	0.52	0.52
query6	1.27	0.61	0.60
query7	0.02	0.00	0.01
query8	0.04	0.02	0.02
query9	0.53	0.50	0.48
query10	0.54	0.53	0.54
query11	0.12	0.08	0.10
query12	0.12	0.09	0.09
query13	0.62	0.61	0.61
query14	0.78	0.78	0.79
query15	0.80	0.76	0.77
query16	0.39	0.37	0.35
query17	0.98	1.02	0.98
query18	0.21	0.28	0.24
query19	1.96	1.84	1.84
query20	0.01	0.02	0.01
query21	15.45	0.55	0.56
query22	2.10	2.56	1.49
query23	17.13	0.92	0.97
query24	5.78	3.37	0.92
query25	0.37	0.11	0.07
query26	0.82	0.16	0.15
query27	0.04	0.03	0.04
query28	5.34	0.76	0.72
query29	12.75	2.29	2.20
query30	0.62	0.53	0.55
query31	2.80	0.38	0.37
query32	3.38	0.49	0.49
query33	3.09	3.05	3.08
query34	15.29	4.78	4.78
query35	4.86	4.80	4.81
query36	1.06	1.02	1.01
query37	0.06	0.04	0.05
query38	0.04	0.02	0.02
query39	0.02	0.01	0.01
query40	0.16	0.14	0.13
query41	0.07	0.02	0.01
query42	0.02	0.02	0.01
query43	0.02	0.02	0.01
Total cold run time: 102.31 s
Total hot run time: 30.37 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 27f72a49ade7ba09ed2abea13049e3b2c4bdf412 with default session variables
Stream load json:         20 seconds loaded 2358488459 Bytes, about 112 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       21.5 seconds inserted 10000000 Rows, about 465K ops/s

@Jibing-Li
Copy link
Contributor Author

run p0

3 similar comments
@Jibing-Li
Copy link
Contributor Author

run p0

@Jibing-Li
Copy link
Contributor Author

run p0

@Jibing-Li
Copy link
Contributor Author

run p0

@Jibing-Li Jibing-Li merged commit 3b9a240 into apache:branch-2.0 Nov 2, 2024
21 of 23 checks passed
@Jibing-Li Jibing-Li deleted the externalrowcount2.0 branch November 4, 2024 01:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants