EX-115: add kafka publisher with protobuf #105
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
lehainam-dev
force-pushed
the
EX-115/kafka-publisher
branch
from
27e56f9 to
6223770
Compare
lehainam-dev
changed the title
lehainam-dev
commented
Jan 16, 2024
| listener.WithEventLogs(nil, nil)) | ||
|
|
||
| return listener.New(l, wsEVMClient, httpEVMClient, handler, sanityEVMClient, sanityCheckInterval, | ||
| listener.WithEventLogs(nil, nil)), nil | ||
| } | ||
|
|
||
| func getPublisher(c *cli.Context, redisClient *redis.Client, topic string) (publisherpkg.Publisher, error) { |
There was a problem hiding this comment.
Review note: At first I put the logic below into NewListener function, however, it causes cyclomatic complexity error when linting. So I separate this logic into its own function.
The parameters passed into this function seem quite counterintuitive to me, though.
lehainam-dev
force-pushed
the
EX-115/kafka-publisher
branch
from
f993069 to
c495538
Compare
lehainam-dev
force-pushed
the
EX-115/kafka-publisher
branch
from
c495538 to
e3dfc52
Compare
lehainam-dev
changed the title
hiepnv90
reviewed
Jan 18, 2024
lehainam-dev
changed the title
lehainam-dev
force-pushed
the
EX-115/kafka-publisher
branch
2 times, most recently
from
710bd29 to
8dba19c
Compare
lehainam-dev
force-pushed
the
EX-115/kafka-publisher
branch
from
8dba19c to
6cb2463
Compare
lehainam-dev
changed the title
lehainam-dev
commented
Jan 22, 2024
Comment on lines
+12
to
+14
| bytes hash = 2; | ||
| bytes parent_hash = 3; | ||
| bytes reorged_hash = 4; |
There was a problem hiding this comment.
Review note: Previously in Go struct, these fields are defined as string. I updated them to bytes to save data size.
0xh9n
approved these changes
Jan 22, 2024
hiepnv90
reviewed
Jan 23, 2024
lehainam-dev
force-pushed
the
EX-115/kafka-publisher
branch
from
0bd891e to
b9b06f3
Compare
lehainam-dev
added a commit
that referenced
this pull request
Mar 14, 2024
* feat: add kafka publisher * feat: handler config publisher type * chore: remove cleanup producer * feat: add producer wait acks * chore: refactor function name * refactor: reduce cyclomatic complexity to pass linter * chore: add kafka topic name validation * fix: check topic name match regex * test: add validate topic name test * test: add kafka publisher test * feat: change interface of publisher to publish byte * feat: add protobuf support * feat: add encoder * feat: add protobuf encoder * chore: separate encoder type
lehainam-dev
added a commit
that referenced
this pull request
May 13, 2024
* feat: add kafka publisher * feat: handler config publisher type * chore: remove cleanup producer * feat: add producer wait acks * chore: refactor function name * refactor: reduce cyclomatic complexity to pass linter * chore: add kafka topic name validation * fix: check topic name match regex * test: add validate topic name test * test: add kafka publisher test * feat: change interface of publisher to publish byte * feat: add protobuf support * feat: add encoder * feat: add protobuf encoder * chore: separate encoder type
lehainam-dev
added a commit
that referenced
this pull request
Nov 12, 2024
* feat: add kafka publisher * feat: handler config publisher type * chore: remove cleanup producer * feat: add producer wait acks * chore: refactor function name * refactor: reduce cyclomatic complexity to pass linter * chore: add kafka topic name validation * fix: check topic name match regex * test: add validate topic name test * test: add kafka publisher test * feat: change interface of publisher to publish byte * feat: add protobuf support * feat: add encoder * feat: add protobuf encoder * chore: separate encoder type
lehainam-dev
added a commit
that referenced
this pull request
Nov 13, 2024
* feat: add kafka publisher * feat: handler config publisher type * chore: remove cleanup producer * feat: add producer wait acks * chore: refactor function name * refactor: reduce cyclomatic complexity to pass linter * chore: add kafka topic name validation * fix: check topic name match regex * test: add validate topic name test * test: add kafka publisher test * feat: change interface of publisher to publish byte * feat: add protobuf support * feat: add encoder * feat: add protobuf encoder * chore: separate encoder type
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
TDD: https://www.notion.so/kybernetwork/TDD-Longer-retention-message-broker-for-EVM-Listener-2da22223cd5d41ac897d73ea244ad65f
This PR is based on the idea of running EVM Listener + Kafka in a separate instance vs EVM Listener + Redis stream.
Although this PR is backward-compatible, we still need to discuss if we want to merge this PR into
mainbranch.