Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[pick](branch-3.0) pick #44177 #44286 #44992 #46217 #46842

Open
wants to merge 4 commits into
base: branch-3.0
Choose a base branch
from
Open

[pick](branch-3.0) pick #44177 #44286 #44992 #46217 #46842

wants to merge 4 commits into from

Conversation

xinyiZzz
Copy link
Contributor

@xinyiZzz xinyiZzz commented Jan 12, 2025 •
edited
Loading

What problem does this PR solve?

[fix](arrow-flight-sql) Fix conf public_host and arrow_flight_sql_proxy_port #44177
[fix](arrow-flight-sql) Arrow Flight support multiple endpoints #44286
[fix](arrow-flight-sql) Revert arrow_flight_sql group from regression test #44992
[opt](arrow-flight-sql) Support arrow-flight-sql protocol getStreamCatalogs, getStreamSchemas, getStreamTables #46217

@xinyiZzz
[fix](arrow-flight-sql) Fix conf public_host and `arrow_flight_sql_…
729504a 
…proxy_port` (apache#44177)

### What problem does this PR solve?

Problem Summary:

Rename `public_access_ip` to `public_host` and `public_access_port` to
`arrow_flight_sql_proxy_port`, they do not have to be used together.
@xinyiZzz
[fix](arrow-flight-sql) Arrow Flight support multiple endpoints (apac…
65853bd 
…he#44286)

Problem Summary:

By default, the query results of all BE nodes will be aggregated to one
BE node. ADBC Client will only receive one endpoint and pull data from
the BE node corresponding to this endpoint.

`set global enable_parallel_result_sink=true;` to allow each BE to
return query results separately. ADBC Client will receive multiple
endpoints and pull data from each endpoint.
@xinyiZzz
[fix](arrow-flight-sql) Revert arrow_flight_sql group from regressi…
d162dcd 
…on test (apache#44992)

After pipeline `P0 Regression (Doris Regression)` executes
`./run-regression-test.sh --run -g p0`, it will execute
`./run-regression-test.sh -g arrow_flight_sql` to run regression-test
again, can use `jdbc:arrow-flight-sql` to run all Suites whose group
contains arrow_flight_sql.
@xinyiZzz
[opt](arrow-flight-sql) Support arrow-flight-sql protocol `getStrea…
10fe27e 
…mCatalogs`, `getStreamSchemas`, `getStreamTables` (apache#46217)

### What problem does this PR solve?

Implement the `getStreamCatalogs`, `getStreamSchemas`, `getStreamTables`
methods in the arrow-flight-sql protocol, which can support BI tools to
correctly display the metadata tree when using the `arrow-flight-sql`
Driver to connect to Doris.

DBeaver uses the `arrow-flight-sql` Driver connecting to Doris:

1. list all catalogs and show properties

![image](https://github.com/user-attachments/assets/f1de6e87-ba5d-4d67-a7bb-06fd91b6cbb1)
2. list dbSchemas and show properties

![image](https://github.com/user-attachments/assets/e706a065-420a-4137-a6a1-6e4f807fad8a)
3. list tables and list table columns.

![image](https://github.com/user-attachments/assets/f9929da9-8cc3-4d74-9f73-b3837854c349)
4. external catalog

![image](https://github.com/user-attachments/assets/ef9ebee1-36d8-4f7f-b97a-e4720ed45d1c)

![image](https://github.com/user-attachments/assets/58f1e0d9-17ed-48a9-be89-dfb818672525)

How to connect to Doris: (will be organized into documents later)

https://www.dremio.com/blog/jdbc-driver-for-arrow-flight-sql/#h-how-to-use-jdbc-driver-with-dbeaver-client

https://docs.dremio.com/current/sonar/client-applications/clients/dbeaver/?_gl=1*1epgwh0*_gcl_au*MjUyNjE1ODM0LjE3MzQwMDExNDg.
@hello-stephen
Copy link
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@xinyiZzz
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 41368 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 10fe27e8c3615a7fe059639992a3396eb868fa32, data reload: false

------ Round 1 ----------------------------------
q1	17653	7728	7335	7335
q2	2063	172	158	158
q3	10780	1095	1159	1095
q4	10442	819	815	815
q5	7749	2892	2888	2888
q6	240	152	157	152
q7	1001	620	614	614
q8	9357	2028	2043	2028
q9	6673	6455	6503	6455
q10	7044	2283	2322	2283
q11	465	263	258	258
q12	411	217	229	217
q13	17809	3045	2989	2989
q14	245	206	204	204
q15	566	532	534	532
q16	676	627	608	608
q17	987	521	590	521
q18	7435	6746	6817	6746
q19	1392	1125	1098	1098
q20	487	210	199	199
q21	4065	3314	3193	3193
q22	1098	980	1036	980
Total cold run time: 108638 ms
Total hot run time: 41368 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7357	7294	7291	7291
q2	327	234	241	234
q3	3088	2982	2972	2972
q4	2037	1841	1828	1828
q5	5762	5740	5761	5740
q6	225	144	149	144
q7	2250	1807	1773	1773
q8	3457	3576	3438	3438
q9	8971	8881	8824	8824
q10	3613	3561	3570	3561
q11	593	522	510	510
q12	854	614	661	614
q13	9905	3178	3216	3178
q14	325	282	286	282
q15	576	533	531	531
q16	736	677	662	662
q17	1860	1623	1602	1602
q18	8353	7916	7600	7600
q19	1697	1669	1511	1511
q20	2127	1871	1861	1861
q21	5466	5338	5346	5338
q22	1171	1045	1060	1045
Total cold run time: 70750 ms
Total hot run time: 60539 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 196888 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 10fe27e8c3615a7fe059639992a3396eb868fa32, data reload: false

query1	1342	922	954	922
query2	6239	2140	2088	2088
query3	10951	4326	4340	4326
query4	66654	29134	23400	23400
query5	5007	461	449	449
query6	409	176	164	164
query7	5581	306	317	306
query8	315	230	235	230
query9	8841	2674	2654	2654
query10	454	266	265	265
query11	17460	15183	15743	15183
query12	152	104	106	104
query13	1492	447	415	415
query14	10284	6647	7162	6647
query15	204	180	184	180
query16	6734	485	485	485
query17	1086	605	618	605
query18	1732	343	344	343
query19	210	177	164	164
query20	121	110	113	110
query21	215	104	107	104
query22	5049	4601	4839	4601
query23	34778	34220	34181	34181
query24	6187	2930	2956	2930
query25	539	413	425	413
query26	652	172	177	172
query27	1819	365	357	357
query28	4230	2536	2528	2528
query29	731	485	475	475
query30	242	162	173	162
query31	1030	822	833	822
query32	71	61	60	60
query33	469	288	278	278
query34	932	511	527	511
query35	868	732	742	732
query36	1081	975	971	971
query37	121	71	71	71
query38	4147	4075	4021	4021
query39	1528	1458	1474	1458
query40	196	98	98	98
query41	50	49	54	49
query42	118	114	98	98
query43	529	478	501	478
query44	1191	846	852	846
query45	187	164	177	164
query46	1184	740	724	724
query47	2026	1948	1957	1948
query48	482	380	383	380
query49	741	391	387	387
query50	837	433	438	433
query51	7316	7188	7346	7188
query52	100	84	87	84
query53	260	180	185	180
query54	552	461	449	449
query55	76	80	74	74
query56	255	235	250	235
query57	1259	1111	1124	1111
query58	211	207	207	207
query59	3399	2964	2880	2880
query60	286	264	254	254
query61	103	106	111	106
query62	766	651	686	651
query63	224	186	185	185
query64	1380	648	633	633
query65	3267	3205	3184	3184
query66	709	309	301	301
query67	16145	15796	15514	15514
query68	4190	596	562	562
query69	434	262	257	257
query70	1186	1125	1085	1085
query71	344	258	264	258
query72	6331	4033	3736	3736
query73	766	347	351	347
query74	10477	9012	8966	8966
query75	3331	2625	2640	2625
query76	1791	1083	1027	1027
query77	498	274	264	264
query78	10596	9604	9614	9604
query79	1249	591	603	591
query80	839	421	418	418
query81	509	240	240	240
query82	1265	114	116	114
query83	241	148	137	137
query84	281	83	78	78
query85	879	314	292	292
query86	337	301	307	301
query87	4423	4432	4268	4268
query88	3425	2381	2335	2335
query89	413	283	293	283
query90	1999	187	185	185
query91	177	144	146	144
query92	60	46	48	46
query93	1392	542	544	542
query94	775	314	261	261
query95	368	258	260	258
query96	611	286	285	285
query97	3431	3170	3237	3170
query98	210	218	205	205
query99	1643	1307	1276	1276
Total cold run time: 318277 ms
Total hot run time: 196888 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 33.5 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 10fe27e8c3615a7fe059639992a3396eb868fa32, data reload: false

query1	0.03	0.04	0.03
query2	0.07	0.03	0.03
query3	0.23	0.06	0.06
query4	1.63	0.11	0.11
query5	0.50	0.51	0.50
query6	1.15	0.72	0.73
query7	0.02	0.02	0.01
query8	0.04	0.03	0.04
query9	0.57	0.50	0.50
query10	0.56	0.55	0.56
query11	0.14	0.11	0.11
query12	0.15	0.12	0.10
query13	0.61	0.59	0.60
query14	2.98	3.07	2.99
query15	0.91	0.83	0.82
query16	0.37	0.38	0.37
query17	1.06	1.05	1.06
query18	0.24	0.22	0.21
query19	1.89	1.73	1.96
query20	0.01	0.01	0.01
query21	15.35	0.56	0.56
query22	3.17	2.71	1.87
query23	16.92	0.96	0.77
query24	2.74	1.51	1.77
query25	0.22	0.31	0.08
query26	0.37	0.14	0.14
query27	0.05	0.03	0.04
query28	10.02	1.11	1.06
query29	12.63	3.22	3.28
query30	0.25	0.06	0.06
query31	2.87	0.39	0.38
query32	3.26	0.46	0.47
query33	3.02	3.00	3.05
query34	16.96	4.53	4.54
query35	4.62	4.52	4.63
query36	0.68	0.50	0.48
query37	0.09	0.06	0.06
query38	0.05	0.04	0.04
query39	0.03	0.02	0.03
query40	0.16	0.13	0.12
query41	0.08	0.03	0.02
query42	0.04	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 106.77 s
Total hot run time: 33.5 s

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@xinyiZzz @hello-stephen @doris-robot