Queue Extract and Transformation library

A helper library to help to implement ETL process or process that requires of consuming data from a queue and execute transformations.

The library uses the standard template package to define the transformation between the source Json file and the generation of the payload (that can be a Json, a SQL sentence or whatever text file format)

A processor orchestrates the consumption of data from a queue and launches parallel payload generators to execute the transformations. The result of the transformations is input for the user-defined executor.

Multiple source queues can be implemented. The project currently supports:

• RabbitMQ, via the streadway/amqp Go client
• Kafka, via the Confluent's Apache Kafka Golan client

Error management is included, support retry mechanism (different implementation per queue) and reconnect mechanism to deal with service interruptions in the source queue.

The errors managed by the retry mechanism are:

• error in payload generation (no transformation available or error to execute the template)
• error in queue connectivity
• error in payload execution (implemented by the user and error propagated to the library), like error calling an API or an error in a database insertion

Working flow:

+--------------------------------+
|             Queue              |
|                                |
| json-1, json-2, json-3, json-n |
+---------------+----------------+
                |
               json
                |
     +----------v---------+
     | Payload Generator  |
     |                    |
     |     template 1     |
     |     template 2     |
     |     template n     |
     +----------+---------+
                |
             payload
                |
      +---------v--------+
      |     Executor     |
      +------------------+

Use library dependencies

To obtain the library:

go get github.com/rormartin/qet

Libraries to include:

• base transformation suite and processor: github.com/rormartin/qet/pkg/transform
• RabbitMQ data source: github.com/rormartin/qet/pkg/rabbitmq
• Kafka data source: github.com/rormartin/qet/pkg/kafka

Library dependencies

The dependencies are managed using Go Modules

• Mergo as a helper for the transformation library to inject data for the template transformations
• Logrus as a structured logger system for Go
• Go RabbitMQ Client Library
• Confluent's Apache Kafka Golan client
• Testify as helper for testing

Project structure

The project contains the main libraries under the pkg folder and the examples folder with a implementation reference examples.