Release v2.10.25

Changelog

Refer to the 2.10 Upgrade Guide for backwards compatibility notes with 2.9.x.

Go Version

• 1.23.5 (#6379)

Dependencies

• golang.org/x/sys v0.29.0 (#6323)
• golang.org/x/time v0.9.0 (#6324)
• golang.org/x/crypto v0.32.0 (#6367)

Improved

JetStream

• Raft groups will no longer snapshot too often in some situations, improving performance (#6277)
• Optimistically perform stream and consumer snapshots on a normal shutdown (#6279)
• The stream snapshot interval has been removed, now relying on the compaction minimum, which improves performance (#6289)
• Raft groups will no longer report current while they are paused with pending commits (#6317)
• Unnecessary client info fields have been removed from stream and consumer assignment proposals, API advisories and stream snapshot/restore advisories (#6326, #6338)
• Reduced lock contention between the JetStream lock and Raft group locks (#6335)
• Advisories will only be encoded and sent when there is interest, reducing CPU usage (#6341)
• Consumers with inactivity thresholds will now start less clean-up goroutines, which can reduce load on the goroutine scheduler (#6344)
• Consumer cleanup goroutines will now stop faster when the server shuts down (#6351)

Fixed

JetStream

• Subject state consistency with some message removal patterns (#6226)
• A performance issue has been fixed when updating the per-subject state (#6235)
• Fixed consistency issues with detecting partial writes in the filestore (#6283)
• A race condition between removing peers and updating replica counts has been fixed (#6316)
• Pre-acks for a sequence are now removed when the message is removed, correcting a potential memory leak (#6325)
• Metalayer snapshot errors are now surfaced correctly (#6361)
• Healthchecks no longer re-evaluate stream and consumer assignments, avoiding some streams and consumers being unexpectedly recreated shortly after a deletion (#6362)
• Clients should no longer timeout on a retried ack with the AckAll policy after a server restart (#6392)
• Replicated consumers should no longer get stuck after leader changes due to incorrect accounting (#6387)
• Consumers will now correctly handle the case where messages queued for delivery have been removed, fixing a delivery slowdown (#6387, #6399)
• The API in-flight metric has been fixed so that it does not drift after the queue has been dropped (#6373)
• Handles for temporary files are now closed correctly if compression errors occur (#6390) — Thanks to @deem0n for the contribution!
• JetStream will now shut down correctly when detecting that the store directory underlying filesystem has become read-only (#6292) — Thanks to @souravagrawal for the contribution!

Leafnodes

• Fixed an interest propagation issue that could occur when the hub has a user with subscribe permissions on a literal subject (#6291)
• Fixed a bug where all queue interest across leafnodes could be dropped over gateways in a supercluster deployment after a leafnode connection drops (#6377)

Tests

• A number of unit tests have been improved (#6150, #6278, #6297, #6300, #6343, #6329, #6330, #6331, #6331, #6334, #6364)

Complete Changes

v2.10.24...v2.10.25

Release v2.10.25-RC.3

Changelog

Refer to the 2.10 Upgrade Guide for backwards compatibility notes with 2.9.x.

Go Version

• 1.23.5 (#6379)

Fixed

JetStream

• Clients should no longer timeout on a retried ack with the AckAll policy after a server restart (#6392)
• Replicated consumers should no longer get stuck after leader changes due to incorrect accounting (#6387)
• Consumers will now correctly handle the case where messages queued for delivery have been removed, fixing a delivery slowdown (#6387)
• The API in-flight metric has been fixed so that it does not drift after the queue has been dropped (#6373)
• Handles for temporary files are now closed correctly if compression errors occur (#6390) — Thanks to @deem0n for the contribution!
• JetStream will now shut down correctly when detecting that the store directory underlying filesystem has become read-only (#6292) — Thanks to @souravagrawal for the contribution!

Leafnodes

• Fixed a bug where all queue interest across leafnodes could be dropped over gateways in a supercluster deployment after a leafnode connection drops (#6377)

Complete Changes

v2.10.25-RC.2...v2.10.25-RC.3

Release v2.10.25-RC.2

Changelog

Refer to the 2.10 Upgrade Guide for backwards compatibility notes with 2.9.x.

Go Version

• 1.23.4

Dependencies

• golang.org/x/crypto v0.32.0 (#6367)

Fixed

JetStream

• Metalayer snapshot errors are now surfaced correctly (#6361)
• Healthchecks no longer re-evaluate stream and consumer assignments, avoiding some streams and consumers being unexpectedly recreated shortly after a deletion (#6362)

Tests

• A number of unit tests have been improved (#6364)

Complete Changes

v2.10.25-RC.1...v2.10.25-RC.2

Release v2.10.25-RC.1

Changelog

Refer to the 2.10 Upgrade Guide for backwards compatibility notes with 2.9.x.

Go Version

• 1.23.4

Dependencies

• golang.org/x/sys v0.29.0 (#6323)
• golang.org/x/time v0.9.0 (#6324)

Improved

JetStream

• Raft groups will no longer snapshot too often in some situations, improving performance (#6277)
• Optimistically perform stream and consumer snapshots on a normal shutdown (#6279)
• The stream snapshot interval has been removed, now relying on the compaction minimum, which improves performance (#6289)
• Raft groups will no longer report current while they are paused with pending commits (#6317)
• Unnecessary client info fields have been removed from stream and consumer assignment proposals, API advisories and stream snapshot/restore advisories (#6326, #6338)
• Reduced lock contention between the JetStream lock and Raft group locks (#6335)
• Advisories will only be encoded and sent when there is interest, reducing CPU usage (#6341)
• Consumers with inactivity thresholds will now start less clean-up goroutines, which can reduce load on the goroutine scheduler (#6344)
• Consumer cleanup goroutines will now stop faster when the server shuts down (#3651)

Fixed

JetStream

• Subject state consistency with some message removal patterns (#6226)
• A performance issue has been fixed when updating the per-subject state (#6235)
• Fixed consistency issues with detecting partial writes in the filestore (#6283)
• A race condition between removing peers and updating replica counts has been fixed (#6316)
• Pre-acks for a sequence are now removed when the message is removed, correcting a potential memory leak (#6325)

Leafnodes

• Fixed an interest propagation issue that could occur when the hub has a user with subscribe permissions on a literal subject (#6291)

Tests

• A number of unit tests have been improved (#6150, #6278, #6297, #6300, #6343, #6329, #6330, #6331, #6331, #6334)

Complete Changes

v2.10.24...v2.10.25-RC.1

Release v2.10.24

Changelog

Refer to the 2.10 Upgrade Guide for backwards compatibility notes with 2.9.x.

CVEs

• Vulnerability check warnings for CVE-2024-45337 are addressed by the dependency update to x/crypto, although the NATS Server does not use the affected functionality and is therefore not vulnerable

Go Version

• 1.23.4

Dependencies

• golang.org/x/crypto v0.31.0 (#6246)
• github.com/nats-io/jwt/v2 v2.7.3 (#6256)
• github.com/nats-io/nkeys v0.4.9 (#6255)

Fixed

General

• Request/reply tracking with allow_responses permission is now pruned more regularly, fixing performance issues that can get worse over time (#6064)

JetStream

• Revert a change introduced in 2.10.23 that could potentially cause a consumer info call to fail if it takes place immediately after the consumer was created in some large or heavily-loaded clustered setups (#6250)
• Minor fixes to subject state tracking (#6244)
• Minor fixes to healthz and healthchecks (#6247, #6248, #6232)
• A calculation used to determine if exceeding limits has been corrected (#6264)
• Raft groups will no longer spin when truncating the log fails, i.e. during shutdown (#6271)

WebSockets

• A WebSocket close frame will no longer incorrectly include a status code when not needed (#6260)

Complete Changes

v2.10.23...v2.10.24

Release v2.10.24-RC.3

Changelog

Refer to the 2.10 Upgrade Guide for backwards compatibility notes with 2.9.x.

Go Version

• 1.23.4

Fixed

JetStream

• A calculation used to determine if exceeding limits has been corrected (#6264)

WebSockets

• A WebSocket close frame will no longer incorrectly include a status code when not needed (#6260)

Complete Changes

v2.10.24-RC.2...v2.10.24-RC.3

Release v2.10.24-RC.2

Changelog

Refer to the 2.10 Upgrade Guide for backwards compatibility notes with 2.9.x.

Go Version

• 1.23.4

Dependencies

• github.com/nats-io/jwt/v2 v2.7.3 (#6256)
• github.com/nats-io/nkeys v0.4.9 (#6255)

Complete Changes

v2.10.24-RC.1...v2.10.24-RC.2

Release v2.10.24-RC.1

Changelog

Refer to the 2.10 Upgrade Guide for backwards compatibility notes with 2.9.x.

Go Version

• 1.23.4

Dependencies

• golang.org/x/crypto v0.31.0 (#6246)

Fixed

General

• Request/reply tracking with allow_responses permission is now pruned more regularly, fixing performance issues that can get worse over time (#6064)

JetStream

• Revert a change introduced in 2.10.23 that could potentially cause a consumer info call to fail if it takes place immediately after the consumer was created in some large or heavily-loaded clustered setups (#6250)
• Minor fixes to subject state tracking (#6226, #6235, #6244)
• Minor fixes to healthz and healthchecks (#6247, #6248, #6232)

Complete Changes

v2.10.23...v2.10.24-RC.1

Release v2.10.23

Changelog

Refer to the 2.10 Upgrade Guide for backwards compatibility notes with 2.9.x.

Go Version

• 1.23.4 (#6228)

Dependencies

• golang.org/x/crypto v0.30.0 (#6230)
• golang.org/x/sys v0.27.0 (#6229)
• golang.org/x/time v0.8.0 (#6105)
• github.com/nats-io/nkeys v0.4.8 (#6192)

Added

JetStream

• Support for responding to forwarded proposals (for future use, #6157)

Windows

• New ca_certs_match option has been added in the tls block for searching the certificate store for only certificates matching the specified CAs (#5115)
• New cert_match_skip_invalid option has been added in the tls block for ignoring certificates that have expired or are not valid yet (#6042)
• The cert_match_by option can now be set to thumbprint, allowing an SHA1 thumbprint to be specified in cert_match (#6042, #6047)

Improved

JetStream

• Reduced the number of allocations in consumers from get-next requests and when returning some error codes (#6039)
• Metalayer recovery at startup will now more reliably group assets for creation/update/deletion and handle pending consumers more reliably, reducing the chance of ghost consumers and misconfigured streams happening after restarts (#6066, #6069, #6088, #6092)
• Creation of filtered consumers is now considerably faster with the addition of a new multi-subject num-pending calculation (#6089, #6112)
• Consumer backoff should now be correctly respected with multiple in-flight deliveries to clients (#6104)
• Add node10 node size to stree, providing better memory utilisation for some subject spaces, particularly those that are primarily numeric or with numeric tokens (#6106)
• Some JetStream log lines have been made more consistent (#6065)
• File-backed Raft groups will now use the same sync intervals as the filestore, including when sync always is in use (#6041)
• Metalayer snapshots will now always be attempted on shutdown (#6067)
• Consumers will now detect if an ack is received past the stream last sequence and will no longer register pre-acks from a snapshot if this happens, reducing memory usage (#6109)
• Reduced copies and number of allocations when generating headers for republished messages (#6127)
• Adjusted the spread of filestore sync timers (#6128)
• Reduced the number of allocations in Raft group send queues, improving performance (#6132)
• Improvements to Raft append entry handling and log consistency (#5661, #5689, #5714, #5957, #6027, #6073)
• Improvements to Raft stepdown behaviour (#5666, #5344, #5717)
• Improvements to Raft elections and vote handling (#5671, #6056)
• Improvements to Raft term handling (#5684, #5792, #5975, #5848, #6060)
• Improvements to Raft catchups (#5987, #6038, #6072)
• Improvements to Raft snapshot handling (#6053, #6055)
• Reduced the overall metalayer snapshot frequency by increasing the minimum interval and no longer pre-empting consumer deletes (#6165)
• Consumer info requests for non-existent consumers will no longer be relayed, reducing overall load on the metaleader (#6176)
• The metaleader will now log if it takes a long time to perform a metalayer snapshot (#6178)
• Unnecessary client and subject information will no longer be included in the meta snapshots, reducing the size and encoding time (#6185)
• Sourcing consumers for R1 streams will now be set up inline when the stream is recovered (#6219)
• Introduced additional jitter to the timer for writing stream state, to smooth out sudden spikes in I/O (#6220)

Fixed

General

• Load balancing queue groups from leaf nodes in a cluster (#6043)

JetStream

• Invalidate the stream state when a drift between the tracking states has been detected (#6034)
• Fixed a panic in the subject tree when checking for full wildcards (#6049)
• Snapshot processing should no longer spin when there is no leader (#6050)
• Replicated stream message framing can no longer overflow with extremely long subjects, headers or reply subjects (#6052)
• Don’t replace the leader’s snapshot when shutting down, potentially causing a desync (#6053)
• Consumer start sequence when specifying an optional start time has been fixed (#6082)
• Raft snapshots will no longer be incorrectly removed when truncating the log back to applied (#6055)
• Raft state will no longer be deleted if creating a stream/consumer failed because the server was shutting down (#6061)
• Fixed a panic when shutting down whilst trying to set up the metagroup (#6075)
• Raft entries that we cannot be sure were applied during a shutdown will no longer be reported as applied (#6087)
• Corrected an off-by-one error in the run-length encoding of interior deletes, which could incorrectly remove an extra message (#6111)
• Don’t process duplicate stream assignment responses when the stream is being reassigned due to placement issues (#6121)
• Corrected use of the stream mutex when checking interest (#6122)
• Raft entries for consumers that we cannot be sure were applied during a shutdown will no longer be reported as applied (#6133)
• Consistent state update behavior between file store and memory store, including a fixed integer underflow (#6147)
• No longer send a state snapshot when becoming a consumer leader as it may not include all applied state (#6151)
• Do not install snapshots on shutdown from outside the monitor goroutines as it may race with upper layer state (#6153)
• The consumer Backoff configuration option now correctly checks the MaxDelivery constraint (#6154)
• Consumer check floor will no longer surpass the store ack floor (#6146)
• Replicated consumers will no longer update their delivered state until quorum is reached, fixing some drifts that can occur on a leader change (#6139)
• Resolved a deadlock when removing the leader from the peer set (#5912)
• Don’t delete disk state if a stream or consumer creation fails during shutdown (#6061)
• The metalayer will no longer generate and send snapshots when switching leaders, reducing the chance that snapshots can be sent with stale assignments (#5700)
• When restarting a Raft group, wait for previous goroutines to shut down, fixing a race condition (#5832)
• Correctly empty the Raft snapshots directory for in-memory assets (#6169)
• A race condition when accessing the stream assignments has been fixed (#6173)
• Subject state will now be correctly cleared when compacting in-memory streams, fixing some potential replica drift issues (#6187)
• Stream-level catchups no longer return more than they should (#6213)

Leafnodes

• Fixed queue distribution where a leafnode expressed interest on behalf of a gateway in complex setups (#6126)
• A number of leafnode interest propagation issues have been fixed, making it possible to distinguish leaf subscriptions from local routed subscriptions (#6161)
• Credential files containing CRLF line endings will no longer error (#6175)

WebSockets

• Ensure full writes are made when compression is in use (#6091)

Windows

• Using the LocalMachine certificate store is now possible from a non-administrator user (#6019)

Tests

• A number of unit tests have been improved (#6086, #6096, #6098, #6097, #5691, #5707, #5991, #6107, #6183, #6218)

Complete Changes

v2.10.22...v2.10.23

Release v2.10.23-RC.12

Changelog

Refer to the 2.10 Upgrade Guide for backwards compatibility notes with 2.9.x.

Go Version

• 1.23.4

Dependencies

• golang.org/x/crypto v0.30.0 (#6230)
• golang.org/x/sys v0.27.0 (#6229)

Fixed

JetStream

• Fixed datarace in filestore calling tombs() (#6233)

Complete Changes

v2.10.23-RC.11...v2.10.23-RC.12