0.6.3

Core

• Improved master selection algorithm to produce more balanced assignments
• fixed partition transitioning from master to standby bug caused by missing boot before tail
• Patterns utility for filtering textual,literal,nominal,numeral,decimal and alphanum strings
• Added OutputDataStream.delete(key) method to use generalized form of tombstone append rather than kafka-specific null

Http

• fixed vulnerability to header injection in GatewayHttp.errorResponse - when handling RequestException, the serverMessage instead of clientReason was returned to the client
• not disclosing Akka Http server header by default
• added CLIENTIP extractor, that works in the order: X-Forwarded-For, Remote-Address, X-Real-Ip

Avro

• removing default avro serde class and instead making it required - it could lead to serious problems if a production application is misconfigured and unknowingly falls back to default MemorySchemaRegistry
• fixed java serialization transient fields in AvroRecord - this is useful in cases where AvroRecords are serialized as part of objects that for some reason have to use java serialization
• fixed wrong default constructor and subject serialization in LocalSchemaRegistry

Kafka

• added print.type property to AvroMessageFormatter

Build

• 100% incremental gradle build and cleanup of all scala warnings

0.6.2

Core

• improved supervision strategy in the controller - high level exceptions resume, low-level restart and throwables escalate
• fixed gateway shutdown sequence during failure

Kafka

• configuring AdminClient in KafkaLogStorage with security settings inherited from kafka consumer

Spark

• LogRDD.json and .xml support for raw data
• LogRDD supports topics with null keys (and throws exception if null key is found in a compacted topic)

CLI

• added timelog utility for visualising time-compaction status of a kafka topic and time/offset ranges

Testing

• fixed flaky failover spec

0.6.1

Akka

• affinity.conf now holds all affinity-specific akka settings instead of layering on top of reference.conf

Avro

• complete support for conversions scala-avro-json and json-avro-scala
• fixed bug which caused avro formatter to expected runtime class to be avilable however formatter the avro serde needs to produce generic record if not runtime class is present for the fqn of the schema
• optimizing serialization and deserialization by removing all runtime type expressions and caching readers and writers in thread-local context

Http

• renamed default handler fulfillAndHandleErrors to handleWith to align with naming convention other default handlers

Examples

• cleaned up graph example from the original prototype
• started example for authentication methods

Testing

• fixed flaky spec for failing kafka writes

0.6.0

Core

• State range lookup support with avro-based compound keys
• added MemStoreSortedMap which supports prefix range queries without persistence
• updated MemStoreRocksDb to support prefix range queries
• deprecated State.transform method in favour of several devolved methods: get, updateAndGet, getAndUpdate
• Gateway graceful shutdown closes all input streams, state stores and keyspaces with the final commit on the input stream before confirming

Http

• RequestException(status: StatusCode) - used by the default handler as the first match, logging status.reason on the server and sending status.defaultMessage with status.intValue as http response
• WebSocket default error handler for handling downstream message - a) it can be extended with a application specific partial function b) it sends upstream a Map object which will be serialized automatically by the provided avro and json web sockets - the format is Map("type": "error", "code": Int , "message": String )

Avro

• Support for avro-encoded compound keys made up of a sequence of FIXED fields - used in range lookups in memstore implementations
• AvroRecord FIXED type can be applied to String fields as @Fixed(len) field: String
• AvroRecord FIXED type can be applied to Int fields as @Fixed field: Int
• AvroRecord FIXED type can be applied to Long fields as @Fixed field: Int

Kafka

• Asynchronous commit() implementation in KafkaLogStorage now provides at-least-once guarantee in input stream processors
• KafkaAvroFormatter options --property print.offset --property print.partition --property no.values

Spark

• simpler abstraction for spark logs:
• LogRDD(s,r) represents a view on the binary stream within the time range r uncompacted
• LogRDD.compact - gives a fully compacted view on the binary log rdd
• LogRDD.present[K,V](keySerde, valSerde): RDD[(K,V)] renders the binary LogRDD to runtime types
• LogRDD.timelog(s,r) gives a 2-dimensional plot of event/processing time of a LogRDD
• LogRDD.join[K,V,X](keySerde, valSerde, other:RDD[(K,X)]): RDD[K, (V,X)] - serialization optimized join on the binary log

Testing

• fixed some of the flaky tests
• improved temporary files cleanup in all tests
• switched all modules to use logback instead of log4j in tests

0.5.4

Core

• unified BinaryStream and Storage into a single concept LogStorage whose API is minimal to provide what is expected of a distributed, replayable, append-only log
• refactored State management around the new LogStorage: Log FSM is wrapped around a concrete LogStorage instance and ensures correct switching between BOOT, TAIL and WRITE states.
• added tests for bootstrapping, checkpointing and tailing mechanisms
• fixed bug which caused partition actors to become online and available before all state stores were consistent with the storage from previous master writes

Spark

• fixed wrong package in CompactRDD
• Spark version upgrade: 2.2.1 as master, 2.0.2 as variant
• LogTimeRDD - for analysing time segmentation of data streams

Testing

• added circleci integration
• added example for external state which also serve as integration tests
• added example for stateful stream processing guarantees which also serve as integration tests

0.5.3

Core

• #140 gateway output stream

Avro

• #133 simplified schema registry abstraction after on-demand initialisation was introduced

Kafka

• #137 --print-key and --print-timestamp option for KafkaAvroMessageFormatter

Spark

• #139 Support for binary stream range scan based on time interval and its application to CompactRDD

0.5.2

Avro

• hotfix for avro enums, using the correct scala reflection mirror

0.5.1

Core

• readonly config option for state which protects the underlying storage from writes

Avro

• @Alias annotation for avro schema aliases mapping

Kafka

• registering schema topic subjects automatically for state store topics

Build and Test Tools

• MemorySchemaRegistry can be used in isolation or shared via schema.registry.id

0.5.0

Core

• introducing Stream Gateway - with suspend / resume behaviour informed by the Coordinator
• switch from object hashcode to binary partitioner - only using serialized key for partitioning
• Storage can now be used for reprocessing/repartitioning purpose
• got rid of the lightweight transactions pending further design
• preventing and handling state checkpoint files corruption
• unifying StreamClient and ManagedConsumer into a single BinaryStream
• EventTime - time conversion utils
• better shutdown hook and logging around it
• enabling a simple gateway with no keyspaces to operate
• fix: propagating the same exception as caught in the ack after retries instead of wrapping it in RuntimeException
• updated documentation

Http

• Refactor of the WebSocketSupport layer: 1. avro 2. json 3. custom
• handleAsText which deals with futures and options automatically
• minor refactor of web socket client

Kafka

• Upgrade to Kafka 1.0
• KafkaAvroSerde and InternalKafkaAvroSerde for KafkaStreams applications
• Kafka 0.10 and Kafka 0.11 cross-builds
• fixed a bad bug in the kafka bootstrap sequence

Spark

• simplified CompactRDD
• no need for StreamSplit anymore
• binding BinaryStream from the storage module

Avro

• refactored names,configs and packages for avro tools
• Array[Byte] handled as Avro Bytes
• improved schema validation
• various fixes in interoperability of different registries
• and added README.md for avro module

Build and Test Tools

• new cross-building structure for different versions of Kafka and Spark
• zzz script for testing and releasing cross-builds
• replace onejar with shadowjar with createscripts in the examples

0.4.0

Core

• config validation
• memstore checkpointing
• full state consistency with improved performance
• state per-key ttl support
• scatter-gather operator in Keyspace via AckSupport
• various bug fixes

Kafka

• kafka module upgrade to Kafka 0.11.x
• storage topic auto-configuration

Avro

• avro module performance comparable to GenericRecord serdes

Build and Test Tools

• docker images for local tests