Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[CARBONDATA-4346] Remove list files while query and invalid cache #4287

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open

[CARBONDATA-4346] Remove list files while query and invalid cache #4287

wants to merge 1 commit into from

Conversation

ShreelekhyaG
Copy link
Contributor

@ShreelekhyaG ShreelekhyaG commented Jul 4, 2022 •
edited
Loading

Why is this PR needed?

  1. Performance degradation for Incremental updates is observed in the partition table.
  • During the update, in the prune step we are listing files from segment path to get the carbondata files and create fileNameToMetaInfoMapping map. On incremental update for partition table, the number of invalid files keep on increasing each time which is causing the degradation in createCarbonDataFileBlockMetaInfoMapping method.
  • Example: Assume a single partition with 1000 carbondata files.
    Perform 1st update: adds 900 new carbondata files.
    Perform 2nd update (same update query): adds another 900 carbondata files. Now the files added by 1st update are invalid.
    Perform query: It does list files. Here, considers invalid files also and adds to fileNameToMetaInfoMapping map.
    The number of invalid files keeps on increasing with each update which is causing the degradation in creating fileNameToMetaInfoMapping map.
  1. Invalid segments cache is not removed after delete/update.

What changes were proposed in this PR?

  1. Instead of listing files, made a change to get the carbon file from the file name and create BlockMetaInfo directly in createBlockMetaInfo.
    Impact when tested on a single partition with 100 segments:
    - There is significant improvement observed in the Incremental update operation.
    - 95% improvement seen in 1st time select count(*) operation. Because in select count(*) flow it was listing files for each segment and the map was not reused.
    Impact when tested on a non-partition table with 100 segments:
    - Almost the same or no improvement for the non-partition table

  2. Clearing invalid/deleted segments from cache after delete and update.

Does this PR introduce any user interface change?

  • No

Is any new testcase added?

  • Yes

@CarbonDataQA2
Copy link

Build Failed with Spark 3.1, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_3.1/772/

@CarbonDataQA2
Copy link

Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/6386/

@CarbonDataQA2
Copy link

Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/4641/

@CarbonDataQA2
Copy link

Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/6387/

@CarbonDataQA2
Copy link

Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/4642/

@CarbonDataQA2
Copy link

Build Success with Spark 3.1, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_3.1/773/

@ShreelekhyaG ShreelekhyaG changed the title [WIP] Remove list files while query and invalid cache [CARBONDATA-4346] Remove list files while query and invalid cache Jul 6, 2022
@CarbonDataQA2
Copy link

Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/6390/

@CarbonDataQA2
Copy link

Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/4645/

@CarbonDataQA2
Copy link

Build Success with Spark 3.1, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_3.1/776/

@Indhumathi27
Copy link
Contributor

LGTM

@CarbonDataQA2
Copy link

Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/6393/

@CarbonDataQA2
Copy link

Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/4650/

@CarbonDataQA2
Copy link

Build Success with Spark 3.1, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_3.1/779/

@ShreelekhyaG
Copy link
Contributor Author

retest this please

@CarbonDataQA2
Copy link

Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/6395/

@CarbonDataQA2
Copy link

Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/4652/

@CarbonDataQA2
Copy link

Build Failed with Spark 3.1, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_3.1/781/

@ShreelekhyaG
Copy link
Contributor Author

retest this please

@CarbonDataQA2
Copy link

Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/4653/

@CarbonDataQA2
Copy link

Build Failed with Spark 3.1, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_3.1/782/

@CarbonDataQA2
Copy link

Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/6396/

@akashrn5
Copy link
Contributor

LGTM

@ShreelekhyaG
Copy link
Contributor Author

retest this please

@CarbonDataQA2
Copy link

Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/6397/

@CarbonDataQA2
Copy link

Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/4654/

@CarbonDataQA2
Copy link

Build Failed with Spark 3.1, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_3.1/783/

@shenjiayu17
Copy link
Contributor

Build Failed with Spark 2.4.5, Please check CI http://159.138.8.58:12602/job/ApacheCarbon_PR_Builder_2.4.5/4659/

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Indhumathi27 Indhumathi27 Indhumathi27 left review comments

@akashrn5 akashrn5 akashrn5 left review comments

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@ShreelekhyaG @CarbonDataQA2 @Indhumathi27 @akashrn5 @shenjiayu17