[Feature] [File Connector] Supports writing column names when the output type is file (CSV) #5459

zck573693104 · 2023-09-11T01:51:03Z

Purpose of this pull request

Check list

Code changed are covered with tests, or it does not need tests for reason:
If any new Jar binary package adding in your PR, please add License Notice according
New License Guide
If necessary, please update the documentation to describe the new feature. https://github.com/apache/seatunnel/tree/dev/docs
If you are contributing the connector code, please check that the following files are updated:
1. Update change log that in connector document. For more details you can refer to connector-v2
2. Update plugin-mapping.properties and add new connector information in it
3. Update the pom file of seatunnel-dist
Update the release-note.

…put type is file (CSV) apache#5443

EricJoy2048 · 2023-09-12T09:25:35Z

.../main/java/org/apache/seatunnel/connectors/seatunnel/file/sink/writer/TextWriteStrategy.java

@@ -133,15 +133,21 @@ private FSDataOutputStream getOrCreateOutputStream(@NonNull String filePath) {
                        OutputStream out =
                                lzo.createOutputStream(fileSystemUtils.getOutputStream(filePath));
                        fsDataOutputStream = new FSDataOutputStream(out, null);
+                        fsDataOutputStream.write(String.join(",", seaTunnelRowType.getFieldNames()).getBytes());
+                        fsDataOutputStream.write(System.lineSeparator().getBytes());


Use fsDataOutputStream.write(rowDelimiter.getBytes()); ?

yeah,i will fix it

EricJoy2048 · 2023-09-12T09:25:52Z

.../main/java/org/apache/seatunnel/connectors/seatunnel/file/sink/writer/TextWriteStrategy.java

                        break;
                    case NONE:
                        fsDataOutputStream = fileSystemUtils.getOutputStream(filePath);
+                        fsDataOutputStream.write(String.join(",", seaTunnelRowType.getFieldNames()).getBytes());
+                        fsDataOutputStream.write(System.lineSeparator().getBytes());


Same as above.

EricJoy2048 · 2023-09-12T09:30:37Z

Please fix code style by run sh tools/spotless_check/pre-commit.sh

zck573693104 · 2023-09-13T02:24:03Z

Please fix code style by run sh tools/spotless_check/pre-commit.sh

i will fix it

…eader,true:write header. apache#5443

ruanwenjun · 2023-09-14T08:43:52Z

docs/en/connector-v2/sink/LocalFile.md

@@ -166,6 +167,10 @@ When File Format is Excel,The maximum number of data items that can be cached in

 Writer the sheet of the workbook

+### enable_header_write [boolean]
+
+false:dont write header,true:write header.


It seems you only implement this in text format, if so it's needed to mention this only support in text format.

yeah，i will fix it

support text and csv

EricJoy2048 · 2023-09-14T11:36:57Z

Please fix CI error.

zck573693104 · 2023-09-15T01:26:06Z

Please fix CI error.

yeah. fix it ok.

docs/en/connector-v2/sink/LocalFile.md

…put type is file (CSV) apache#5443

…eader,true:write header. apache#5443

…put type is file (CSV) apache#5443

…eader,true:write header. apache#5443

FlechazoW

Nice job!

…put type is file (CSV) (apache#5459) * [Feature] [File Connector] Supports writing column names when the output type is file (CSV) apache#5443 * [Feature] [File Connector] fix code style and lineSeparator apache#5443 * [Feature] [File Connector] add enable_header_write,false:dont write header,true:write header. apache#5443 * [Feature] [File Connector] fix code style apache#5443 * [Feature] [File Connector] add enable_header_write explain apache#5443 * [Feature] [File Connector]fix code style apache#5443 * Update docs/en/connector-v2/sink/LocalFile.md * Update docs/en/connector-v2/sink/LocalFile.md * [Feature] [File Connector]fix code style apache#5443 * [Feature] [File Connector]add junit test apache#5443 * [Feature] [File Connector]add license header: apache#5443 * [Feature] [File Connector] Supports writing column names when the output type is file (CSV) apache#5443 * [Feature] [File Connector] fix code style and lineSeparator apache#5443 * [Feature] [File Connector] add enable_header_write,false:dont write header,true:write header. apache#5443 * [Feature] [File Connector] fix code style apache#5443 * [Feature] [File Connector] add enable_header_write explain apache#5443 * [Feature] [File Connector]fix code style apache#5443 * Update docs/en/connector-v2/sink/LocalFile.md * Update docs/en/connector-v2/sink/LocalFile.md * [Feature] [File Connector]fix code style apache#5443 * [Feature] [File Connector]add junit test apache#5443 * [Feature] [File Connector]add license header: apache#5443 * [Feature] [File Connector]add junit: apache#5443 * [Feature] [File Connector]add junit: apache#5443 * [Feature] [File Connector]remove scala: apache#5443 * [Feature] [File Connector]modify md style: apache#5443 * [Feature] [File Connector] Supports writing column names when the output type is file (CSV) apache#5443 * [Feature] [File Connector] fix code style and lineSeparator apache#5443 * [Feature] [File Connector] add enable_header_write,false:dont write header,true:write header. apache#5443 * [Feature] [File Connector] fix code style apache#5443 * [Feature] [File Connector] add enable_header_write explain apache#5443 * [Feature] [File Connector]fix code style apache#5443 * Update docs/en/connector-v2/sink/LocalFile.md * Update docs/en/connector-v2/sink/LocalFile.md * [Feature] [File Connector]fix code style apache#5443 * [Feature] [File Connector]add junit test apache#5443 * [Feature] [File Connector]add license header: apache#5443 * [Feature] [File Connector]add junit: apache#5443 * [Feature] [File Connector]add junit: apache#5443 * [Feature] [File Connector]remove scala: apache#5443 * [Feature] [File Connector]modify md style: apache#5443 * [Feature] [File Connector]junit modify: apache#5443 --------- Co-authored-by: zck <[email protected]> Co-authored-by: Eric <[email protected]>

[Feature] [File Connector] Supports writing column names when the out…

7a44e95

…put type is file (CSV) apache#5443

EricJoy2048 reviewed Sep 12, 2023

View reviewed changes

[Feature] [File Connector] fix code style and lineSeparator apache#5443

de2859b

zck added 3 commits September 13, 2023 16:08

[Feature] [File Connector] add enable_header_write,false:dont write h…

4acdda8

…eader,true:write header. apache#5443

[Feature] [File Connector] fix code style apache#5443

b7c90dc

Merge branch 'dev' of https://github.com/apache/seatunnel into dev

ee306b7

ruanwenjun changed the title ~~[Feature] [File Connector] Supports writing column names when the out…~~ [Feature] [File Connector] Supports writing column names when the output type is file (CSV) Sep 14, 2023

ruanwenjun assigned zck573693104 Sep 14, 2023

ruanwenjun reviewed Sep 14, 2023

View reviewed changes

ruanwenjun added the feature New feature label Sep 14, 2023

zck added 2 commits September 14, 2023 17:36

Merge branch 'dev' of https://github.com/apache/seatunnel into dev

388771b

[Feature] [File Connector] add enable_header_write explain apache#5443

d1da88a

[Feature] [File Connector]fix code style apache#5443

dc26b73

EricJoy2048 reviewed Sep 15, 2023

View reviewed changes

docs/en/connector-v2/sink/LocalFile.md Outdated Show resolved Hide resolved

docs/en/connector-v2/sink/LocalFile.md Outdated Show resolved Hide resolved

EricJoy2048 and others added 12 commits September 15, 2023 11:52

Update docs/en/connector-v2/sink/LocalFile.md

6da518a

Update docs/en/connector-v2/sink/LocalFile.md

98d2571

Merge branch 'dev' of https://github.com/apache/seatunnel into dev

5bad203

Merge branch 'dev' of https://github.com/zck573693104/seatunnel into dev

84c0ff5

[Feature] [File Connector]fix code style apache#5443

17e6710

Merge branch 'dev' of https://github.com/apache/seatunnel into dev

c30321a

[Feature] [File Connector]add junit test apache#5443

6846e37

Merge branch 'dev' of https://github.com/apache/seatunnel into dev

d305e6f

[Feature] [File Connector]add license header: apache#5443

03a24a9

[Feature] [File Connector] Supports writing column names when the out…

1ffa54f

…put type is file (CSV) apache#5443

[Feature] [File Connector] fix code style and lineSeparator apache#5443

cbc07f6

[Feature] [File Connector] add enable_header_write,false:dont write h…

2b1e15c

…eader,true:write header. apache#5443

zck and others added 18 commits September 22, 2023 14:37

[Feature] [File Connector]modify md style: apache#5443

03c369f

[Feature] [File Connector] Supports writing column names when the out…

bf7df66

…put type is file (CSV) apache#5443

[Feature] [File Connector] fix code style and lineSeparator apache#5443

301058e

[Feature] [File Connector] add enable_header_write,false:dont write h…

147302a

…eader,true:write header. apache#5443

[Feature] [File Connector] fix code style apache#5443

9f7ea83

[Feature] [File Connector] add enable_header_write explain apache#5443

afb7ce5

[Feature] [File Connector]fix code style apache#5443

c9b7b38

Update docs/en/connector-v2/sink/LocalFile.md

e0885bb

Update docs/en/connector-v2/sink/LocalFile.md

0f9987f

[Feature] [File Connector]fix code style apache#5443

1f37a64

[Feature] [File Connector]add junit test apache#5443

5761244

[Feature] [File Connector]add license header: apache#5443

af3b5f2

[Feature] [File Connector]add junit: apache#5443

37f3a15

[Feature] [File Connector]add junit: apache#5443

3b809e7

[Feature] [File Connector]remove scala: apache#5443

738e36a

[Feature] [File Connector]modify md style: apache#5443

bc5a7e0

Merge branch 'dev' of https://github.com/zck573693104/seatunnel into dev

f6c115a

[Feature] [File Connector]junit modify: apache#5443

2bc2fa5

zck573693104 requested review from ruanwenjun, zhilinli123 and EricJoy2048 September 25, 2023 06:43

EricJoy2048 approved these changes Sep 25, 2023

View reviewed changes

github-actions bot added approved reviewed labels Sep 25, 2023

FlechazoW approved these changes Sep 25, 2023

View reviewed changes

lightzhao approved these changes Sep 25, 2023

View reviewed changes

liugddx approved these changes Sep 25, 2023

View reviewed changes

liugddx merged commit f73b372 into apache:dev Sep 25, 2023
8 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Feature] [File Connector] Supports writing column names when the output type is file (CSV) #5459

[Feature] [File Connector] Supports writing column names when the output type is file (CSV) #5459

zck573693104 commented Sep 11, 2023 •

edited by ruanwenjun

Loading

EricJoy2048 Sep 12, 2023

zck573693104 Sep 13, 2023

EricJoy2048 Sep 12, 2023

EricJoy2048 commented Sep 12, 2023

zck573693104 commented Sep 13, 2023

ruanwenjun Sep 14, 2023

zck573693104 Sep 14, 2023

zck573693104 Sep 14, 2023

EricJoy2048 commented Sep 14, 2023

zck573693104 commented Sep 15, 2023

FlechazoW left a comment

[Feature] [File Connector] Supports writing column names when the output type is file (CSV) #5459

[Feature] [File Connector] Supports writing column names when the output type is file (CSV) #5459

Conversation

zck573693104 commented Sep 11, 2023 • edited by ruanwenjun Loading

Purpose of this pull request

Check list

EricJoy2048 Sep 12, 2023

Choose a reason for hiding this comment

zck573693104 Sep 13, 2023

Choose a reason for hiding this comment

EricJoy2048 Sep 12, 2023

Choose a reason for hiding this comment

EricJoy2048 commented Sep 12, 2023

zck573693104 commented Sep 13, 2023

ruanwenjun Sep 14, 2023

Choose a reason for hiding this comment

zck573693104 Sep 14, 2023

Choose a reason for hiding this comment

zck573693104 Sep 14, 2023

Choose a reason for hiding this comment

EricJoy2048 commented Sep 14, 2023

zck573693104 commented Sep 15, 2023

FlechazoW left a comment

Choose a reason for hiding this comment

zck573693104 commented Sep 11, 2023 •

edited by ruanwenjun

Loading