CASSGO-1 CASSGO-30 Native Protocol 5 Support #1822

worryg0d · 2024-09-16T08:01:11Z

Overview

C* 4.0 introduced Native Protocol 5, so we should make gocql to support it.
This PR consists of other my PRs, which also provides support for the v5 version of the protocol.

This PR provides support of Native Protocol 5 features as:

New frames format CASSANDRA-15299
result_metadata_id field for EXECUTE and PREPARED messages CASSANDRA-10786
now_in_seconds field for BATCH and QUERY messages CASSANDRA-14664
keyspace field for BATCH and QUERY messages

worryg0d · 2024-10-01T15:01:29Z

This PR is merged from #1778, #1773, #1782, and #1781, so we could close them in favor of this one

joao-r-reis · 2024-10-08T15:17:34Z

What about #1783

lukasz-antoniak

Copied unsolved comments fromCopied unresolved comments from previous PRs.

frame_test.go

lukasz-antoniak · 2024-10-08T15:43:12Z

What about #1783

@worryg0d, IMHO changes in #1783 are still needed and improve code maintainability.

worryg0d · 2024-10-09T04:17:51Z

Thanks a lot for reaching out!

This PR is lacking changes from #1783. I will ask @testisnullus if we could merge it into this PR.

testisnullus · 2024-10-09T09:29:35Z

@worryg0d @lukasz-antoniak I think we can keep the current implementation and merge the #1783 PR into this one.

worryg0d · 2024-10-10T12:36:12Z

I merged changes from #1783 into this one.

joao-r-reis

First pass review. I'll need some extra time to properly review the envelope/message format changes.

batch_test.go

cassandra_test.go

compressor.go

conn.go

session.go

frame.go

conn.go

worryg0d · 2024-10-15T18:01:13Z

Hey, @joao-r-reis. Thanks a lot for your feedback! I'm pleased this PR is moving. I will take a look at your comments and suggestions tomorrow. I hope everything will be fine here.

…to ver < 5. Using ptr based logic for now_in_seconds field instead of additional flag

worryg0d · 2024-10-16T12:47:09Z

Hello, @joao-r-reis. I added some changes following your comments and suggestions. Could you please take a look at this one again? Also, I have a question regarding to cache of prepared statements. Could you please look at this one too?

Thanks a lot.

…orthIt integration tests

…n METADATA_CHANGED is presented in ROWS/RESULT only if the prepared stmt is presented in the cache

conn.go

joao-r-reis

Almost there! I think the functionality is now done and everything looks to be working correctly 👍 These last few comments are small things.

Still need another committer to take a look at this PR though.

conn.go

joao-r-reis · 2024-10-29T12:18:52Z

conn.go

+	// 2.3.1 Initial Handshake
+	// 	In order to support both v5 and earlier formats, the v5 framing format is not
+	//  applied to message exchanges before an initial handshake is completed.
+	startupCompleted := &atomic.Bool{}


I'm going to keep this one open until we have a second reviewer to look at this.

conn.go

joao-r-reis · 2024-10-29T15:44:43Z

Added semver-major label due to the Compressor interface changes

2. The driver should relly on No_metadata flag instead of Skip_metadata when handling RESULT/ROWS responses from C*

…_changed handling logic by forcing the driver to execute queries with old metadata 2. Added checking of integrity of the result to the test

…rent code base uses initial keyspace provided by the user to construct the keys. Since proto v5 we also should account for keyspace bounding for a specific query, so the driver should use the bounded keyspace instead of the initial to construct the key. 2. Changed the way how routing key cache keys are constructed to account the keyspace overriding as well.

joao-r-reis

Good stuff 👍 , I left 2 conversations open so that we get input from the other committer that looks at this:

bumping go.mod version to 1.19 so we can use atomic.Bool (current minimum version is 1.17 but go.mod is not up to date
Compressor interface changes

I'm +1 on both of these though.

joao-r-reis · 2024-10-31T15:48:19Z

oh I just noticed the integration tests on GH are running with snappy compression, we should change it to lz4 so we get coverage on compression+v5

worryg0d · 2024-10-31T17:32:11Z

Good stuff 👍 , I left 2 conversations open so that we get input from the other committer that looks at this:

bumping go.mod version to 1.19 so we can use atomic.Bool (current minimum version is 1.17 but go.mod is not up to date

Compressor interface changes

I'm +1 on both of these though.

Thanks a lot! I'm really happy to hear that. I'm waiting for a review from another committer and hoping everything will be fine here.

oh I just noticed the integration tests on GH are running with snappy compression, we should change it to lz4 so we get coverage on compression+v5

Oh, yeah. I'll try to change CI to run tests here.

worryg0d · 2024-10-31T18:03:01Z

I changed CI here to run tests over proto 5. One thing I had to replace lz4 dependency in go.mod to use the updated version of the pkg...

…CacheUsesOverriddenKeyspace integration tests when running on proto < 5 because they requires overridding keyspace feature that is available for proto 5 and higher

joao-r-reis · 2024-11-06T15:11:34Z

Hmm it does look weird to have gocql's go.mod list gocql/lz4 as a dependency when gocql/lz4's go.mod lists gocql as a dependency... A possible solution is to make lz4 a package instead of its own module, I brought this up in slack and here.

If lz4 stays a separate module then I'm not sure what is the best way to use lz4 in the integration tests... Maybe your approach is the best one, if I'm not mistaken the lz4 dependency will not actually be downloaded by users since it's only used in tests.

go.mod

ribaraka

lgtm

worryg0d · 2024-11-12T16:03:39Z

@joao-r-reis I adjusted the CI in this PR to run integration tests over proto 5 and the lz4 compressor. Could you please approve running the workflow to see if nothing breaks?

worryg0d mentioned this pull request Sep 16, 2024

Support for result_metadata_id in PREPARED and EXECUTE messages #1782

Closed

worryg0d force-pushed the native-proto-5-support-all-merged branch 4 times, most recently from d67fcb1 to 93933c6 Compare September 19, 2024 12:49

worryg0d mentioned this pull request Sep 19, 2024

Proto v5 support was added #1773

Closed

worryg0d force-pushed the native-proto-5-support-all-merged branch 3 times, most recently from 47a3ae9 to b617638 Compare October 1, 2024 07:03

joao-r-reis added the protocol_v5 Tickets related to supporting protocol version 5 label Oct 1, 2024

This was referenced Oct 1, 2024

Support for now_in_seconds option field for QUERY, BATCH and EXECUTE messages #1781

Closed

Support of keyspace field for BATCH message #1778

Closed

lukasz-antoniak reviewed Oct 8, 2024

View reviewed changes

frame_test.go Show resolved Hide resolved

worryg0d force-pushed the native-proto-5-support-all-merged branch from b617638 to 3f50a90 Compare October 10, 2024 12:06

Support for Native Protocol 5 release version

fbdaa2b

worryg0d force-pushed the native-proto-5-support-all-merged branch from 3f50a90 to fbdaa2b Compare October 10, 2024 12:14

tengu-alt mentioned this pull request Oct 15, 2024

CASSGO-11 Support vector type #1828

Open

joao-r-reis reviewed Oct 15, 2024

View reviewed changes

worryg0d added 3 commits October 16, 2024 13:45

Added 'IF NOT EXISTS' to table creation queries in integration tests

93fb4ad

Clarified comment on startup completed

d926108

Returned panics on setting keyspace and now_in_seconds field when pro…

a1baea2

…to ver < 5. Using ptr based logic for now_in_seconds field instead of additional flag

Added description for TestLargeSizeQuery and TestQueryCompressionNotW…

b77aa18

…orthIt integration tests

worryg0d added 2 commits October 25, 2024 18:52

Fix reading <new_metadata_id> in parseResultMetadata

1343003

Updating prepared stmt metadata result set and result_metadata_id whe…

98e37c1

…n METADATA_CHANGED is presented in ROWS/RESULT only if the prepared stmt is presented in the cache

joao-r-reis reviewed Oct 25, 2024

View reviewed changes

conn.go Show resolved Hide resolved

worryg0d added 2 commits October 25, 2024 20:41

Avoiding deadlock after updating the lru cache entry

31d10f9

Integration test for Metadata_changed mechanism

2a5056d

worryg0d force-pushed the native-proto-5-support-all-merged branch from 2b7d196 to 2a5056d Compare October 28, 2024 09:52

joao-r-reis reviewed Oct 29, 2024

View reviewed changes

conn.go Show resolved Hide resolved

joao-r-reis added the semver-major label Oct 29, 2024

worryg0d added 2 commits October 30, 2024 09:57

1. Updating info to ensure the code looking at the updated prepared stmt

bf0d7fa

2. The driver should relly on No_metadata flag instead of Skip_metadata when handling RESULT/ROWS responses from C*

1. Updated TestPrepareExecuteMetadataChangedFlag to validate Metadata…

0592a90

…_changed handling logic by forcing the driver to execute queries with old metadata 2. Added checking of integrity of the result to the test

worryg0d force-pushed the native-proto-5-support-all-merged branch 2 times, most recently from 6370ae0 to b5d76bd Compare October 30, 2024 16:58

worryg0d force-pushed the native-proto-5-support-all-merged branch from b5d76bd to 0298a00 Compare October 31, 2024 14:56

joao-r-reis approved these changes Oct 31, 2024

View reviewed changes

Added skip for TestStmtCacheUsesOverriddenKeyspace and TestRoutingKey…

a1a613c

…CacheUsesOverriddenKeyspace integration tests when running on proto < 5 because they requires overridding keyspace feature that is available for proto 5 and higher

joao-r-reis changed the title ~~CASSGO-1 Native Protocol 5 Support~~ CASSGO-1 CASSGO-30 Native Protocol 5 Support Nov 6, 2024

joao-r-reis mentioned this pull request Nov 6, 2024

CASSGO-30 DSE Search Results May Not Be Queryable #1689

Open

joao-r-reis reviewed Nov 6, 2024

View reviewed changes

go.mod Show resolved Hide resolved

Go bump to 1.19

e779e73

ribaraka approved these changes Nov 7, 2024

View reviewed changes

ribaraka mentioned this pull request Nov 11, 2024

CASSGO-21 Remove HostPoolHostPolicy from gocql package #1770

Open

Changed CI to run integration tests over proto 5 + lz4 compressor

5ca5382

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CASSGO-1 CASSGO-30 Native Protocol 5 Support #1822

CASSGO-1 CASSGO-30 Native Protocol 5 Support #1822

worryg0d commented Sep 16, 2024 •

edited

Loading

worryg0d commented Oct 1, 2024 •

edited

Loading

joao-r-reis commented Oct 8, 2024

lukasz-antoniak left a comment

lukasz-antoniak commented Oct 8, 2024

worryg0d commented Oct 9, 2024

testisnullus commented Oct 9, 2024

worryg0d commented Oct 10, 2024

joao-r-reis left a comment

worryg0d commented Oct 15, 2024

worryg0d commented Oct 16, 2024

joao-r-reis left a comment

joao-r-reis Oct 29, 2024

joao-r-reis commented Oct 29, 2024

joao-r-reis left a comment

joao-r-reis commented Oct 31, 2024

worryg0d commented Oct 31, 2024

worryg0d commented Oct 31, 2024 •

edited

Loading

joao-r-reis commented Nov 6, 2024

ribaraka left a comment

worryg0d commented Nov 12, 2024

CASSGO-1 CASSGO-30 Native Protocol 5 Support #1822

Are you sure you want to change the base?

CASSGO-1 CASSGO-30 Native Protocol 5 Support #1822

Conversation

worryg0d commented Sep 16, 2024 • edited Loading

Overview

worryg0d commented Oct 1, 2024 • edited Loading

joao-r-reis commented Oct 8, 2024

lukasz-antoniak left a comment

Choose a reason for hiding this comment

lukasz-antoniak commented Oct 8, 2024

worryg0d commented Oct 9, 2024

testisnullus commented Oct 9, 2024

worryg0d commented Oct 10, 2024

joao-r-reis left a comment

Choose a reason for hiding this comment

worryg0d commented Oct 15, 2024

worryg0d commented Oct 16, 2024

joao-r-reis left a comment

Choose a reason for hiding this comment

joao-r-reis Oct 29, 2024

Choose a reason for hiding this comment

joao-r-reis commented Oct 29, 2024

joao-r-reis left a comment

Choose a reason for hiding this comment

joao-r-reis commented Oct 31, 2024

worryg0d commented Oct 31, 2024

worryg0d commented Oct 31, 2024 • edited Loading

joao-r-reis commented Nov 6, 2024

ribaraka left a comment

Choose a reason for hiding this comment

worryg0d commented Nov 12, 2024

worryg0d commented Sep 16, 2024 •

edited

Loading

worryg0d commented Oct 1, 2024 •

edited

Loading

worryg0d commented Oct 31, 2024 •

edited

Loading