v0.99.0

It's been a while since we've officially tagged a release. Sorry about that.

What's Changed

• client: AppendService must release buffers of error'd AsyncAppends by @jgraettinger in #298
• This refactors the gazctl command to allow importing the cli parsing to another project by @snowzach in #296
• client: AppendService owns its buffer sync.Pool by @jgraettinger in #299
• brokertest: fix graceful stop test flake by @jgraettinger in #302
• consumer: flexible transaction boundaries by @jgraettinger in #303
• performance instrumentation and configurable read channel sizes by @jgraettinger in #306
• build(deps): bump babel from 2.7.0 to 2.9.1 in /docs by @dependabot in #308
• protocols: explicitly mark messages for Equal generation by @jgraettinger in #310
• multiple improvements to facilitate local testing contexts by @jgraettinger in #311
• Add shard unassign cmd by @saterus in #312
• consumer: shard transitions account for delete-then-create race by @jgraettinger in #315
• Improve Unassign Shard Workflow by @saterus in #316
• client: add new metric instrumentation for reads & appends by @jgraettinger in #317
• Register Protobuf Enums by @saterus in #320
• Add DeferPublishUncommitted to message.Publisher by @psFried in #319
• work around for platform dependent test setup by @michaelschiff in #321
• parse SSEKMSKeyId from store url and pass to store config by @davidgmonical in #323
• address WithBalancerName deleted in grpc v1.46.0 by @apesternikov in #324
• gazctl: add shards recoverylog-play command by @mdibaiee in #325
• shards unassign command no longer stalls by @saterus in #322
• Allow customizing server Listener by @psFried in #327
• consumer: synchronously fail if StartCommit returns a pre-resolved error by @jgraettinger in #329
• error when s3 client is missing region config by @psFried in #330
• Fix keyspace missing watch events by @psFried in #336
• keyspace: update MemberId/RaftTerm on watch events by @psFried in #338
• consumer: add Shard.PrimaryLoop() client.OpFuture by @jgraettinger in #339
• broker: allow further banking of unused flow-control credit by @jgraettinger in #340
• feature: Implement azure-ad:// as a fragment store URI scheme by @jshearer in #341
• chore: Bump github.com/Azure/azure-storage-blob-go to v0.15.0 by @jshearer in #342
• shard unassign: handle non-PRIMARY assignments which are FAILED by @jgraettinger in #343
• Fix azure pre-signed urls by @psFried in #344
• message: Sequencer incrementally tracks pending partial sequences by @jgraettinger in #345
• broker: add optional ReadRequest.BeginModTime constraint by @jgraettinger in #346
• Phil/recoverylog pruning by @psFried in #347
• gazctl: re-use shard and journal clients by @psFried in #348
• Azure AD: fix my dumb mistake when refreshing user delegation credentials by @jshearer in #349
• gazctl: shard pruning should consider all hints by @jgraettinger in #350
• allocator: make assignment much much faster by @jgraettinger in #351
• align with grpc base/balancer to trigger reconnect in Idle state (when we move to gRPC 1.41+) by @ddowker in #335
• go mod: update grpc module to latest 1.59.0 by @mdibaiee in #352
• fix: Azure fragment store with multiple tenants by @jshearer in #354
• recoverylog: relax FSM to allow updating the content of a property by @jgraettinger in #356
• broker/client: discard async appends of deleted journals by @jgraettinger in #357
• take two: relax ErrJournalNotFound only when writing recovered ACKs by @jgraettinger in #358
• AppendService should retry on JOURNAL_NOT_FOUND by @jgraettinger in #359
• SegmentSet.Add allow missing LastOffset by @psFried in #360
• broker/consumer: add CreateRevision to ListResponse by @jgraettinger in #361
• Phil/pruning is hard by @psFried in #362
• Phil/pruning paranoia by @psFried in #363
• go.mod: update pinned gorocksdb and x/net by @jgraettinger in #364
• broker/fragment: region support for s3 stores by @williamhbaker in #365
• consumer: bound the size of an error stored as Etcd shard status by @jgraettinger in #366
• generalize first-write persistence and relax byte-zero read skipping by @jgraettinger in #367
• journal client: force using http1.1 when fetching fragments from clou… by @psFried in #368
• Phil/http client errors by @psFried in #369
• journal client: use http1.1 for fetching fragments, for real this time by @psFried in #370
• mainboilerplate: use a maximum-size connection flow control window by @jgraettinger in #371
• build(deps): bump golang.org/x/net from 0.17.0 to 0.23.0 by @dependabot in #372

New Contributors

• @snowzach made their first contribution in #296
• @saterus made their first contribution in #312
• @davidgmonical made their first contribution in #323
• @apesternikov made their first contribution in #324
• @mdibaiee made their first contribution in #325
• @jshearer made their first contribution in #341
• @ddowker made their first contribution in #335
• @williamhbaker made their first contribution in #365

Full Changelog: v0.89.0...v0.99.0

v0.89.0

• Optional consumer.MessageProducer interface allows applications
  to provide an alternate driver of message reading & parsing,
  outside of the built-in mechanisms for reading journals named
  in the ShardSpec.
• Binary-compatible enhancement of recovery logs allows
  for cheap "forks", where historical content of one log is used
  from within another.
  This introduced a breaking change to the recoverylog.Recorder,
  which now has a dedicated constructor and non-exported fields.
• Allow journal names in Read and Append RPCs to be extended with a
  'name;suffix', where the suffix is captured into consumer checkpoint offsets.
  This allows a single consumer to have multiple, concurrent reads of a journal.
• New instrumentation for testing and building custom consumer executables.
• Fix a memory leak in append flow control.
• Fix an allocator bug which could cause delayed convergence under pretty uncommon conditions.
• Fix some corner cases on timely reporting of shard progress in the Stat RPC.
• Update the versions of all dependencies.

v0.88.0

• Behavior change: journal registers are a small set of labels which are used
  for cooperative, transactional locking. Where before a set of checked registers
  must always match current journal registers for an append to proceed, now an
  exception exists in the case where a journal has a fully empty set
  of registers (as is the case on recovery if journal consistency is lost).
• gazctl journals reset-head now issues concurrent resets to enumerated journals,
  and no longer fails if an attempt to reset a consistent journal loses an append race.
• Makefiles now build local Docker images gazette/broker:latest and
  gazette/examples:latest, instead of gazette-broker:latest and gazette-examples:latest.
• Additional sanity checks to ensure badly-behaved clients or peers cannot
  block an exiting broker from draining its server.
• Gazette no longer logs "wrong fragment format" warnings when encountering "directory" objects
  (those where the object key ends with a '/') in cloud storage. These were just adding noise and
  were not actually anything to worry about.
• Fix various test flakes.

v0.87.3

• BUGFIX: Add a catch-all timeout on server GracefulStop, to ensure poorly
  behaved clients cannot keep a terminating broker around indefinitely (GH-266).

v0.87.2

• % is now an allowed character in label values and journal names. This makes their grammar compatible with URL-encoding.
• Added grpc-prometheus instrumentation to client & server gRPC's.
• Several new metrics added:
  • gazette_shard_*: operational metrics of consumer shards
  • gazette_spool_*: metrics of journal fragment spools
  • gazette_journal_server_*: metrics of gRPC JournalServer invocations
• Some metrics are marked as DEPRECATED, and will be removed in the future:
  • gazette_commits_total
  • gazette_committed_bytes_total
  • gazette_consumer_*
• Some already-broken or unused metrics were removed.
• Improvements to benchmarks and fixed test flakes.

v0.87.1

• BUGFIX: Fix potential UUID sequencing race in concurrent calls to PublishCommitted of the same message.Publisher, and add new regression testing (GH-259).
• Instrument message.Sequencer with Prometheus metrics.
• Add curl to release images, making it much simpler for node-zone.sh mappings to directly query cloud metadata APIs.
• Fix omitted registration of broker WriteHead metric.
• Various improvements to tests, addressing or instrumenting flakes.

v0.86.1

• Update RocksDB to 6.7.3, and Go to 1.14.2, along with many other package dependencies.
• The etcdtest package no longer embeds an Etcd server. Instead, an etcd
  binary must be available on the PATH and is invoked as a sub-process,
  using Unix domain sockets. Users who use etcdtest themselves, or are running
  Gazette tests outside of the hermetic Docker build environment, must provide a
  reasonably recent version of etcd.
• Relatedly, the gazette client package no longer depends on Etcd (including the Etcd client).
• JSON-framed messages are now able to use custom marshal/demarshal routines.
• gazctl journals/shards apply now interprets revision -1 (previously
  disallowed) to mean "don't care", allowing specs to be applied which will always
  overwrite what's in Etcd.
  This makes sense for specs managed in another source-of-truth (eg, git).
• Mitigations and improved logging for issue GH-248.
• Remove explicit TCP keep-alive management, as this is now Go default.
• Various doc improvements & cleanups.

v0.85.2

• Fix: as-ci target downloads go modules before invoking inner make, to allow external repos to include Makefiles of the gazette/core repo which are dynamically determined using the go mod tool.
• Update to Go 1.13.4

v0.85.1

• Added MaxAppendRate JournalSpec field and global broker flag. Append RPCs now use a token-bucket flow control strategy, where RPC chunks are evaluated and potentially throttled or policed against maximum and minimum allowed flow-rates.
• Added PathPostfixTemplate JournalSpec field. Path postfixes are evaluated and applied to individual Fragments as they're persisted. A primary use case is to support Hive-compatible partitioning of Fragments based on their creation time. Journal names and labels may now include the '=' rune, to facilitate the layout of multiple journals as a Hive-partitioned table.
• Reworked almost all documentation into reStructuredText / Sphinx / ReadTheDocs format.
• Make-based build system is refactored to make it easier to integrate and reuse in external repositories and consumer application projects.
• Add DisableWaitForAck ShardSpec field, which toggles the consumer transaction behavior of waiting for ACKs of read pending messages. Most applications won't want to set this, but it can be helpful to avoid stalls in applications with cyclic message flows.

v0.84.2

Add ca-certificates to release images (Issue #227)