[CARBONDATA-4270]Delete segment expect remain_number #4203

W1thOut · 2021-08-14T16:02:24Z

Why is this PR needed?

In some scenarios, need to delete old segments in batches and keep only latest few segments.

What changes were proposed in this PR?

Add a DML.

Does this PR introduce any user interface change?

No

Is any new testcase added?

Yes

CarbonDataQA2 · 2021-08-14T16:04:37Z

Can one of the admins verify this patch?

W1thOut · 2021-08-14T16:10:05Z

start build

W1thOut · 2021-08-14T16:21:14Z

retest this please

integration/spark/src/main/scala/org/apache/spark/sql/parser/CarbonSpark2SqlParser.scala

MarvinLitt · 2021-08-16T01:50:13Z

...rg/apache/carbondata/spark/testsuite/deletesegment/DeleteSegmentByRemainNumberTestCase.scala

+    val ex = intercept[Exception] {
+      sql("delete from table deleteSegmentTable expect segment.remain_number = -1")
+    }
+    assert(ex.getMessage.contains("not found in database"))


"not found in database" has nothing to do with -1, why choose this msg.

...rg/apache/carbondata/spark/testsuite/deletesegment/DeleteSegmentByRemainNumberTestCase.scala

MarvinLitt · 2021-08-16T02:00:49Z

...rg/apache/carbondata/spark/testsuite/deletesegment/DeleteSegmentByRemainNumberTestCase.scala

+    sql(
+      s"""LOAD DATA local inpath '$resourcesPath/dataretention3.csv'
+         | INTO TABLE deleteSegmentTable OPTIONS('DELIMITER'= ',', 'QUOTECHAR'= '"')""".stripMargin)
+  }


need some after all function to make sure the table has been removed.

i have define function "beforeEach" to drop table

MarvinLitt · 2021-08-16T02:09:56Z

...rg/apache/spark/sql/execution/command/management/CarbonDeleteLoadByRemainNumberCommand.scala

+    // if insert overwrite in progress, do not allow delete segment
+    if (SegmentStatusManager.isOverwriteInProgressInTable(carbonTable)) {
+      throw new ConcurrentOperationException(carbonTable, "insert overwrite", "delete segment")
+    }


why this pr do not support insert overwrite

The table cannot be updated when deleted

MarvinLitt · 2021-08-16T02:21:39Z

...rg/apache/spark/sql/execution/command/management/CarbonDeleteLoadByRemainNumberCommand.scala

+    }
+
+    val segments = CarbonStore.readSegments(carbonTable.getTablePath, showHistory = false, None)
+    if (segments.length == remaining.toInt) {


remaining.toInt means the segment count shoule be int range, does this restriction necessary

if there has some segment with status SegmentStatus.MARKED_FOR_DELETE, does this condition is right or not?

interger value from -2147483648 to 2147483647. i think the remaining number of segment will not exceed this range.

SegmentStatus.MARKED_FOR_DELETE won't be in remaining range.

remaining.toInt can throw java.lang.NumberFormatException , handle appropriately.

ok. i have modify sql parse code. transfer 'remain_number' to Integer value directly.

W1thOut · 2021-08-16T13:16:56Z

retest this please

MarvinLitt · 2021-08-16T15:00:11Z

jenkins,add to whitelist

MarvinLitt · 2021-08-16T15:09:30Z

add to whitelist

W1thOut · 2021-08-17T03:26:20Z

retest this please

MarvinLitt · 2021-08-17T03:32:47Z

@CarbonDataQA2 why this pr can not trigger the CI, please help.

W1thOut · 2021-08-17T03:48:55Z

@CarbonDataQA2 why this pr can not trigger the CI, please help.

ajantha-bhat · 2021-08-17T03:53:40Z

add to whitelist

vikramahuja1001 · 2021-08-17T05:25:03Z

...rg/apache/spark/sql/execution/command/management/CarbonDeleteLoadByRemainNumberCommand.scala

+    }
+
+    // Through the remaining number, get the delete id
+    val deleteSegmentIds = segments.filter(segment => segment.getSegmentStatus != SegmentStatus.MARKED_FOR_DELETE)


Instead of filtering out MFD segments, can only take success and compacted segments and avoid In-progress segments, because clean files will anyways take care of those segments

i have modify this problem

CarbonDataQA2 · 2021-08-17T05:31:46Z

Build Failed with Spark 3.1, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_3.1/263/

CarbonDataQA2 · 2021-08-17T05:34:50Z

Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/5860/

vikramahuja1001 · 2021-08-17T05:35:50Z

Hi @W1thOut , can you raise the discussion in the community first? As you want to expose a new DDL, please check the community guidelines here: https://www.mail-archive.com/[email protected]/msg01835.html

CarbonDataQA2 · 2021-08-17T05:35:59Z

Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/4117/

vikramahuja1001 · 2021-08-17T05:48:37Z

...rg/apache/carbondata/spark/testsuite/deletesegment/DeleteSegmentByRemainNumberTestCase.scala

+  override def beforeEach(): Unit = {
+    initTestTable
+  }
+


add test cases with SI as well

ok. i have add test cases with SI

MarvinLitt · 2021-08-17T11:21:22Z

retest this please

W1thOut · 2021-08-17T11:55:52Z

retest this please

CarbonDataQA2 · 2021-08-17T13:14:34Z

Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/5865/

CarbonDataQA2 · 2021-08-17T13:19:11Z

Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/4123/

CarbonDataQA2 · 2021-08-17T13:37:09Z

Build Failed with Spark 3.1, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_3.1/268/

CarbonDataQA2 · 2021-08-17T18:06:57Z

Build Failed with Spark 3.1, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_3.1/269/

CarbonDataQA2 · 2021-08-17T18:11:00Z

Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/5866/

CarbonDataQA2 · 2021-08-17T18:15:21Z

Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/4124/

W1thOut · 2021-08-18T14:28:54Z

Hi @W1thOut , can you raise the discussion in the community first? As you want to expose a new DDL, please check the community guidelines here: https://www.mail-archive.com/[email protected]/msg01835.html

okay. but i can not access community: http://apache-carbondata-mailing-list-archive.1130556.n5.nabble.com/
Is it because I don't have permission?

nihal0107 · 2021-08-18T14:32:33Z

http://apache-carbondata-dev-mailing-list-archive.168.s1.nabble.com/
Please use this link.

W1thOut · 2021-08-18T14:58:57Z

retest this please

CarbonDataQA2 · 2021-08-18T15:10:39Z

Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/4127/

CarbonDataQA2 · 2021-08-18T15:12:44Z

Build Failed with Spark 3.1, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_3.1/272/

W1thOut · 2021-08-18T16:03:04Z

retest this please

CarbonDataQA2 · 2021-08-18T16:16:22Z

Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/5871/

CarbonDataQA2 · 2021-08-18T16:16:51Z

Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/4129/

CarbonDataQA2 · 2021-08-18T17:45:46Z

Build Failed with Spark 3.1, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_3.1/274/

Revise modify code add SI test add no partition table teset code checkstyle

W1thOut · 2021-08-19T16:07:12Z

retest this please

CarbonDataQA2 · 2021-08-19T17:46:07Z

Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/4133/

CarbonDataQA2 · 2021-08-19T17:55:57Z

Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/5876/

CarbonDataQA2 · 2021-08-19T18:12:06Z

Build Success with Spark 3.1, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_3.1/278/

W1thOut · 2021-08-21T09:24:19Z

Hi @W1thOut , can you raise the discussion in the community first? As you want to expose a new DDL, please check the community guidelines here: https://www.mail-archive.com/[email protected]/msg01835.html

okay. but i can not access community: http://apache-carbondata-mailing-list-archive.1130556.n5.nabble.com/
Is it because I don't have permission?

Could you give me your valuable comments/inputs/suggestions.
http://apache-carbondata-dev-mailing-list-archive.168.s1.nabble.com/DISCUSSION-CARBONDATA-4270-New-DDL-delete-table-tableName-expect-segment-remain-number-n-td109013.html

MarvinLitt · 2021-09-02T02:31:24Z

does this pr can be a new kind of segment management.
there may be some scenario to aging segments just left few ones.

CarbonDataQA2 · 2021-11-21T11:27:58Z

Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/4383/

CarbonDataQA2 · 2021-11-21T11:48:33Z

Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/6126/

CarbonDataQA2 · 2021-11-21T11:50:54Z

Build Failed with Spark 3.1, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_3.1/516/

W1thOut changed the title ~~Delete segment expect remain_number~~ [CARBONDATA-4270]Delete segment expect remain_number Aug 14, 2021

MarvinLitt reviewed Aug 16, 2021

View reviewed changes

W1thOut force-pushed the master branch from b2d9615 to 7fa6471 Compare August 16, 2021 13:14

W1thOut force-pushed the master branch from 5eb06ba to e967350 Compare August 16, 2021 15:59

vikramahuja1001 reviewed Aug 17, 2021

View reviewed changes

W1thOut force-pushed the master branch from e967350 to 21a8c72 Compare August 17, 2021 16:26

W1thOut force-pushed the master branch from 21a8c72 to f8fdb75 Compare August 18, 2021 14:08

delete segment expect remaining numbe

93b9c7a

Revise modify code add SI test add no partition table teset code checkstyle

W1thOut force-pushed the master branch from f8fdb75 to 93b9c7a Compare August 19, 2021 16:06

W1thOut added 3 commits November 21, 2021 17:28

partition table

b62286c

partition table

8cd250b

partition table

38b2855

[CARBONDATA-4270]Delete segment expect remain_number #4203

Are you sure you want to change the base?

[CARBONDATA-4270]Delete segment expect remain_number #4203

Conversation

W1thOut commented Aug 14, 2021

Why is this PR needed?

What changes were proposed in this PR?

Does this PR introduce any user interface change?

Is any new testcase added?

CarbonDataQA2 commented Aug 14, 2021

W1thOut commented Aug 14, 2021

W1thOut commented Aug 14, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

W1thOut commented Aug 16, 2021

MarvinLitt commented Aug 16, 2021

MarvinLitt commented Aug 16, 2021

W1thOut commented Aug 17, 2021

MarvinLitt commented Aug 17, 2021

W1thOut commented Aug 17, 2021

ajantha-bhat commented Aug 17, 2021

vikramahuja1001 Aug 17, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

CarbonDataQA2 commented Aug 17, 2021

CarbonDataQA2 commented Aug 17, 2021

vikramahuja1001 commented Aug 17, 2021

CarbonDataQA2 commented Aug 17, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

MarvinLitt commented Aug 17, 2021

W1thOut commented Aug 17, 2021

CarbonDataQA2 commented Aug 17, 2021

CarbonDataQA2 commented Aug 17, 2021

CarbonDataQA2 commented Aug 17, 2021

CarbonDataQA2 commented Aug 17, 2021

CarbonDataQA2 commented Aug 17, 2021

CarbonDataQA2 commented Aug 17, 2021

W1thOut commented Aug 18, 2021

nihal0107 commented Aug 18, 2021 • edited Loading

W1thOut commented Aug 18, 2021

CarbonDataQA2 commented Aug 18, 2021

CarbonDataQA2 commented Aug 18, 2021

W1thOut commented Aug 18, 2021

CarbonDataQA2 commented Aug 18, 2021

CarbonDataQA2 commented Aug 18, 2021

CarbonDataQA2 commented Aug 18, 2021

W1thOut commented Aug 19, 2021

CarbonDataQA2 commented Aug 19, 2021

CarbonDataQA2 commented Aug 19, 2021

CarbonDataQA2 commented Aug 19, 2021

W1thOut commented Aug 21, 2021

MarvinLitt commented Sep 2, 2021

CarbonDataQA2 commented Nov 21, 2021

CarbonDataQA2 commented Nov 21, 2021

CarbonDataQA2 commented Nov 21, 2021

vikramahuja1001 Aug 17, 2021 •

edited

Loading

nihal0107 commented Aug 18, 2021 •

edited

Loading