Skip to content

Release v0.10.0-redis-connect
Compare
Choose a tag to compare
@viragtripathi viragtripathi released this 23 Mar 01:55
· 53 commits to main since this release
v0.10.0-redis-connect-5
145bf85

🚀 Changelog

🧰 Enhancements

  • Added GEMFIRE connector to support both initial load and stream jobs.
  • Added FILES connector to support initial load jobs from CVS, TSV, PSV, JSON, and XML files types.
  • Added SPLUNK connector to support HEC Collector stream jobs as a source and a HEC_FORWARDER_SINK to forward logs to Splunk Enterprise. Initial load jobs are not supported.
  • Added REDIS_STREAMS_MESSAGE_BROKER source type to support Redis Streams brokered deployments in which the source and sink connectors can scale independently and communicate via a structured schema.
  • Added REDIS_STREAMS_MESSAGE_BROKER sink type to support Redis Streams brokered deployments in which the source and sink connectors can scale independently and communicate via a structured schema.
  • Added REDIS_STRING_SINK to support String blobs (often JSON) from sources such as Gemfire, Splunk, etc.
  • Enhanced delivery guarantees by adding checkpoint rollback for failed Redis transactions.
  • Enhanced delivery guarantees by adding checkpoint rollback for failed replication-to-backup Redis shards using the WAIT command.
  • Added checkpoint key as an attribute to the checkpoint hash value. In the event of an exception, the checkpoint value will be logged and its key will make it clear which hashslot should be updated to move past the corrupted source offset.
  • Enhanced save checkpoint process to no longer assume that the latest checkpoint key should be used during the administrative save checkpoint process.
  • Added Time-Sequence Strategies including INITIAL_LOAD, LRU, NO, PASS_THROUGH and SEQUENTIAL and standardized all usage of sourceTxTime and sourceTxSequence around them.
  • Added Domain Model Strategies including DICTIONARY, KEY_ONLY, STRING, and MESSAGE_BROKER.
  • Added Redis Streams Eviction Strategies including THRESHOLD and SCHEDULER. They are used in conjunction with a MESSAGE_BROKER deployment to clean up Redis Streams partitions in coordination with committed checkpoints so data is not lost.
  • Added thread-safe in-memory ChangeEventQueue which transitions changed data events between the source and the JobProducer's polling loop for non-debezium connectors with write-through persistence to Redis Streams.
  • Added Redis Streams distributed ChangeEventQueue which transitions changed data events between the source and the JobProducer's polling loop for non-debezium connectors.
  • Added new LOGGER called "redis-connect-pipeline" which only captures DEBUG level logs solely dedicated to the critical path across every layer of the replication pipeline at the change event level. This will be helpful for debugging in production since it avoids all other noise.
  • Added JobPipelineStageDTO configuration called "checkpointDatabase" to be used for offset management when the target database is not Redis. This would be typical in a MESSAGE_BROKER deployment.
  • Added JobSourceTableColumnDTO configuration called "passThroughEnabled" which does not allow column-level values to be passed beyond the transformation layer when disabled. This is useful if they are only used for enrichment.
  • Enhanced JobReaper service with distributed two-phase commit to avoid split-brain / zombie jobs (running but not claimed) in the event of network split failure type scenarios and un-synchronized job manager interval threshold configurations.
  • Added job-level configuration and validations for max partitions per cluster member. Each JobClaimer service will now check to see if its cluster member has reached a threshold for max claimed job partitions before attempting to claim another. This is supported for both the stream and initial load jobs.
  • Removed validation to allow a Custom Stage to access custom configs without requiring a database credential validation.
  • Removed optional secrets encryption logic since we no longer store credentials within Redis Connect.
  • Enhanced error messages across all REST, CLI, and SWAGGER endpoints by consistently including the first line of the exception stack trace. This specifically helps identify which field is missing/incorrect within the job config file in the event of save failure.
  • Enhanced validation of source connections by handling stream and initial load jobs separately.
  • Added validation for event handler (pipeline stages) configuration which will be enforced as part of the start job process.
  • Added validation for source configuration which will be enforced as part of the start job process.
  • Added validation to the start and migrate job processes which calculates available capacity with respect to the job's max partitions per cluster member configuration.
  • Added various minor validations to avoid corner case misconfigurations.
  • Added granularity for trustStore and keyStore configuration across source, target, and checkpoint databases.
  • Disabled debezium's tombstone.on.delete configuration by default since it emits events meant for kafka compaction which are not relevant to Redis Connect. Existing workaround code was left in place for backward compatibility.
  • Enhanced metrics to report at the jobId level instead of jobName for granularity at the job partition level.
  • Refactored Producer, Transformation, Connection, Event Handler (pipeline stages), and Utils layers for easier future extension and standardization.
  • (Deprecated - Not backward compatible) JobSourceDTO.Table configuration called "schemaAndTableName". It was redundant with the table name. Extra care should be taken to remove this configuration during upgrades as it will cause the save job config process to fail.
  • (Not backward compatible for custom stages) Refactored ChangeEventDTO to account for dependent events. An example of a dependent event is a primary key change within a relational database which creates both a delete and insert within the target.
  • (Not backward compatible for custom stages) Changed package for BaseCustomStageHandler from com.redis.connect.pipeline.event.handler to com.redis.connect.pipeline.event.handler.impl.
  • (Not backward compatible for custom stages) Broke up ConnectConstants into 4 separate Constants classes and moved them to a dedicated package.
  • Added feature to receive Email alert notifications for STOPPED jobs due to exceptions.
  • Added feature to build Redis Connect docker image based on users choice of base images.
  • Added Multi-Architecture build with Docker.

🐛 Bug Fixes

  • Handled edge case in which Redis client could be successfully created and cached however the connection would fail and the client wasn't removed from the cache.

Tested Versions

Java 11+
Redis Connect 0.10.x
DB2 (Initial Loader) Database: 11.5.x
JDBC Driver: 11.5.6.0
Files (Initial Loader) CSV, TSV, PSV, JSON, XML
Gemfire (CDC and Initial Loader) Database: 1.12.9
Driver: 1.12.9
MongoDB (CDC and Initial Loader) Database: 4.4+
Driver: 4.7.1
MySQL (CDC and Initial Loader) Database: 8.0.x
JDBC Driver: 8.0.29
Oracle (CDC and Initial Loader) Database: 11g, 12c, 19c, 21c
JDBC Driver: 12.2.0.1, 19.8.0.0, 21.1.0.0, 21.3.0.0, 21.4.0.0, 21.5.0.0, 21.6.0.0
Adapter: logminer
PostgreSQL (CDC and Initial Loader) Database: 10, 11, 12, 13, 14, 15
JDBC Driver: 42.5.1
Plug-ins: pgoutput
Splunk (CDC) 8.1.2
SQL Server (CDC and Initial Loader) Database: 2017, 2019
JDBC Driver: 10.2.1.jre8
Vertica (Initial Loader) Database: 11.1.0-0
JDBC Driver: 11.1.0-0
Assets 5
Loading