Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

PostgreSQL CDC Plugin #2917

Open
wants to merge 127 commits into
base: main
Choose a base branch
from
Open

PostgreSQL CDC Plugin #2917

wants to merge 127 commits into from

Conversation

le-vlad
Copy link

@le-vlad le-vlad commented Oct 7, 2024

Added:

  • Support for PostgreSQL versions (17,....10)

    • Implemented a different approach for taking snapshots in earlier versions (<15)
    • Using pg_create_logical_replication_slot instead of CREATE_REPLICATION_SLOT
    • Note: Snapshot name not provided by default with this method
    • Investigating discrepancy with documentation (https://www.postgresql.org/docs/12/protocol-replication.html)
    • Aim to use newer API instead of old one once resolved
  • Replication monitoring

    • Added progress tracking for snapshot tables (%)
    • Added WAL lag monitoring (bytes) when streaming is started
  • Stream uncommitted changes

    • New stream_uncommitted flag for use with pgoutput plugin
    • Allows receiving updates mid-transaction
    • Contrasts with wal2json, which sends updates with LSN of committed transactions
  • Configurable decoding plugin

    • Added decoding_plugin option to specify plugin for streaming data
    • Supports wal2json or pgoutput
  • Temporary replication slots

    • New temporary_slot setting to create temporary slots
    • Instructs PostgreSQL to drop replication slot on connection close
    • Useful for testing or when persistent replication slots are not needed
  • Comprehensive integration tests for various scenarios

@CLAassistant
Copy link

CLAassistant commented Oct 7, 2024

CLA assistant check
All committers have signed the CLA.

@le-vlad le-vlad force-pushed the pg_cdc branch 2 times, most recently from 2b958eb to 0418bcf Compare October 16, 2024 12:01

// Subsequent characters must be letters, numbers, underscores, or dots
for i, char := range name {
if !unicode.IsLetter(char) && !unicode.IsDigit(char) && char != '_' && char != '.' {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Isn't a dot the delimiter for schema.table? Any concerns there?

@rockwotj
Copy link
Collaborator

Wonderful work here @le-vlad - I ran the integration tests and they are really good so thanks for those 👍

I think this is good to go pending any other feedback from the rest of the team. I will followup this change with some modifications to the message format and removing the "wait for ack" step in the unbuffered handler.

Copy link
Collaborator

@Jeffail Jeffail left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, can one of you stick an entry in the CHANGELOG.md pls?

@rockwotj
Copy link
Collaborator

Done! I also snuck in another changelog entry

@rockwotj rockwotj dismissed mihaitodor’s stale review November 15, 2024 18:36

PR has changed substantially and all feedback has been addressed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants