Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Improvement](inverted index) enable bkd index reader cache and refactor inverted index searcher cache #27367

Closed
wants to merge 3 commits into from

Conversation

airborne12
Copy link
Member

Proposed changes

Issue Number: close #xxx

Further comments

If this is a relatively large or complex change, kick off the discussion at [email protected] by explaining why you chose the solution you did and what alternatives you considered, etc...

Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

@airborne12
Copy link
Member Author

run buildall

Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

Comment on lines +59 to +60
namespace util {
namespace bkd {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

warning: nested namespaces can be concatenated [modernize-concat-nested-namespaces]

Suggested change
namespace util {
namespace bkd {
namespace util::bkd {

be/src/olap/rowset/segment_v2/inverted_index_cache.h:61:

- }
- } // namespace util
+ } // namespace util

@doris-robot
Copy link

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
Tpch sf100 test result on commit cca061c27b632987726159f33b6f95a18f98192f, data reload: false

run tpch-sf100 query with default conf and session variables
q1	4870	4665	4626	4626
q2	352	148	168	148
q3	2046	1855	1870	1855
q4	1374	1233	1263	1233
q5	3963	3920	4013	3920
q6	244	127	128	127
q7	1414	891	905	891
q8	2734	2768	2756	2756
q9	9713	9658	9507	9507
q10	3459	3507	3537	3507
q11	381	258	252	252
q12	443	295	298	295
q13	4542	3822	3767	3767
q14	332	300	281	281
q15	580	535	532	532
q16	658	575	583	575
q17	1124	982	922	922
q18	7817	7411	7410	7410
q19	1672	1665	1664	1664
q20	564	299	295	295
q21	4323	3964	3948	3948
q22	485	373	386	373
Total cold run time: 53090 ms
Total hot run time: 48884 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	4612	4542	4573	4542
q2	338	214	296	214
q3	3991	3974	3994	3974
q4	2689	2701	2690	2690
q5	9731	9710	9734	9710
q6	237	121	123	121
q7	3011	2480	2472	2472
q8	4477	4461	4498	4461
q9	13244	13096	13171	13096
q10	4098	4197	4170	4170
q11	825	709	664	664
q12	976	805	813	805
q13	4258	3589	3547	3547
q14	395	335	349	335
q15	575	531	532	531
q16	739	667	685	667
q17	3873	3852	3875	3852
q18	9591	8883	8923	8883
q19	1811	1766	1765	1765
q20	2394	2093	2040	2040
q21	8819	8648	8506	8506
q22	911	815	792	792
Total cold run time: 81595 ms
Total hot run time: 77837 ms

@doris-robot
Copy link

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 47.35 seconds
stream load tsv: 589 seconds loaded 74807831229 Bytes, about 121 MB/s
stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 30.4 seconds inserted 10000000 Rows, about 328K ops/s
storage size: 17100033800 Bytes

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants