Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Support partitioned reads for DateTime column type in JDBC to BigQuery template #2084

Draft
wants to merge 4 commits into
base: main
Choose a base branch
from
Draft

Support partitioned reads for DateTime column type in JDBC to BigQuery template #2084

wants to merge 4 commits into from

Conversation

sharan-malyala
Copy link
Contributor

No description provided.

stankiewicz
stankiewicz suggested changes Dec 30, 2024
View reviewed changes
v2/jdbc-to-googlecloud/src/main/java/com/google/cloud/teleport/v2/templates/JdbcToBigQuery.java Outdated Show resolved Hide resolved
v2/jdbc-to-googlecloud/src/main/java/com/google/cloud/teleport/v2/templates/JdbcToBigQuery.java Outdated Show resolved Hide resolved
...c/test/java/com/google/cloud/teleport/v2/templates/JdbcToBigQueryReadWithPartitionsTest.java Outdated

public void testE2EWithDateTimePartitionColumnTypeWithIncorrectBounds() throws IOException, InterruptedException {
options.setPartitionColumn("SELL_TIME");
options.setPartitionColumnType("datetime");
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please also add spomewhere test for missing column type to validate backward compatibility and test for invalid type.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

razvanculea
razvanculea reviewed Dec 30, 2024
View reviewed changes
...c/test/java/com/google/cloud/teleport/v2/templates/JdbcToBigQueryReadWithPartitionsTest.java Outdated
try (Connection conn = DriverManager.getConnection("jdbc:derby:memory:jdc2bq;create=true");
Statement statement = conn.createStatement()) {
statement.execute("CREATE TABLE book (BOOK_ID bigint primary key, TITLE varchar(128), SELL_TIME timestamp)");
statement.execute("INSERT INTO book VALUES (1, 'ABC', '2024-12-24 06:00:00')");
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

add timezone

Copy link
Contributor Author

@sharan-malyala sharan-malyala Jan 10, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We use derby in Unit tests and Derby database does not support timezone.
ref : https://db.apache.org/derby/docs/10.13/ref/rrefsqlj27620.html

ij> INSERT INTO myTable (id, name, created_at) VALUES (2,'Tej','2025-01-09 10:00:00.000+0:00');
ERROR 22007: The syntax of the string representation of a date/time value is incorrect.

ij> INSERT INTO myTable (id, name, created_at) VALUES (2,'Tej','2025-01-09 10:00:00.000Z');
ERROR 22007: The syntax of the string representation of a date/time value is incorrect.

Will document this observation somewhere.

razvanculea
razvanculea reviewed Dec 30, 2024
View reviewed changes
...c/test/java/com/google/cloud/teleport/v2/templates/JdbcToBigQueryReadWithPartitionsTest.java Outdated
.waitUntilFinish();

assertThat(fakeDatasetService.getAllRows(PROJECT, DATASET, TABLE))
.isEqualTo(ImmutableList.of(new TableRow().set("BOOK_ID", 1).set("TITLE", "ABC").set("SELL_TIME", "2024-12-24 06:00:00")));
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

add timezone

razvanculea
razvanculea reviewed Dec 30, 2024
View reviewed changes
...c/test/java/com/google/cloud/teleport/v2/templates/JdbcToBigQueryReadWithPartitionsTest.java Outdated
.waitUntilFinish();

assertThat(fakeDatasetService.getAllRows(PROJECT, DATASET, TABLE))
.isEqualTo(ImmutableList.of(new TableRow().set("BOOK_ID", 1).set("TITLE", "ABC").set("SELL_TIME", "2024-12-24 06:00:00")));
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

add timezone

razvanculea
razvanculea reviewed Dec 30, 2024
View reviewed changes
...c/test/java/com/google/cloud/teleport/v2/templates/JdbcToBigQueryReadWithPartitionsTest.java Outdated
.waitUntilFinish();

assertThat(fakeDatasetService.getAllRows(PROJECT, DATASET, TABLE))
.isEqualTo(ImmutableList.of(new TableRow().set("BOOK_ID", 1).set("TITLE", "ABC").set("SELL_TIME", "2024-12-24 06:00:00")));
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

add timezone

razvanculea
razvanculea reviewed Dec 30, 2024
View reviewed changes
...c/test/java/com/google/cloud/teleport/v2/templates/JdbcToBigQueryReadWithPartitionsTest.java Outdated
.waitUntilFinish();

assertThat(fakeDatasetService.getAllRows(PROJECT, DATASET, TABLE))
.isEqualTo(ImmutableList.of(new TableRow().set("BOOK_ID", 1).set("TITLE", "ABC").set("SELL_TIME", "2024-12-24 06:00:00")));
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

add timezone

razvanculea
razvanculea reviewed Dec 30, 2024
View reviewed changes
...c/test/java/com/google/cloud/teleport/v2/templates/JdbcToBigQueryReadWithPartitionsTest.java Outdated
public void testE2EWithDateTimePartitionColumnTypeWithBounds() throws IOException, InterruptedException {
options.setPartitionColumn("SELL_TIME");
options.setPartitionColumnType("datetime");
options.setLowerBound("2024-12-24 05:00:00");
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

add timezone for both bounds

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@razvanculea razvanculea razvanculea left review comments

@stankiewicz stankiewicz stankiewicz requested changes

At least 1 approving review is required to merge this pull request.

Assignees
No one assigned
Labels
size/L
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@sharan-malyala @stankiewicz @razvanculea