v1.3.3

Major Changes

• Expose containerPort to user defined container (#2135)
• Fixed the race condition of terminating source vertex pods and draining the buffers during pipeline pausing (#2131)
• Create buffers and buckets before updating Vertices (#2112)

Installation

kubectl create namespace numaflow-system

# Install numaflow controller and UI
kubectl apply -n numaflow-system -f https://github.com/numaproj/numaflow/releases/download/v1.3.3/install.yaml

# Install validating webhook (Optional)
kubectl apply -n numaflow-system -f https://github.com/numaproj/numaflow/releases/download/v1.3.3/validating-webhook-install.yaml

Full Changelog: v1.3.2...v1.3.3

v1.3.2

Major Changes

• Rolling update support for pipeline changes (#2040)
• Allow customization on readyz and livez config (#2068)
• Fixed the issue of keys not being kept in built-in transformers (#2047)
• Bugfix for resources not being orchestrated well for new pipelines with desiredPhase: Paused (#2028)

Installation

kubectl create namespace numaflow-system

# Install numaflow controller and UI
kubectl apply -n numaflow-system -f https://github.com/numaproj/numaflow/releases/download/v1.3.2/install.yaml

# Install validating webhook (Optional)
kubectl apply -n numaflow-system -f https://github.com/numaproj/numaflow/releases/download/v1.3.2/validating-webhook-install.yaml

Full Changelog: v1.3.1...v1.3.2

v1.3.1

Major Changes

• Add retry mechanism for sinks [#2010]
• Deprecate replicasPerScale, and introduce replicasPerScaleUp and replicasPerScaleDown for more flexible autoscaling strategy [#2003]
• Enable resourceClaims for Pipeline Vertex [#2009]
• Introduce readyReplicas for Vertex status [#2014]

Installation

kubectl create namespace numaflow-system

# Install numaflow controller and UI
kubectl apply -n numaflow-system -f https://github.com/numaproj/numaflow/releases/download/v1.3.1/install.yaml

# Install validating webhook (Optional)
kubectl apply -n numaflow-system -f https://github.com/numaproj/numaflow/releases/download/v1.3.1/validating-webhook-install.yaml

Full Changelog: v1.3.0...v1.3.1

v1.3.0

Major Features 🔥

• Introducing Batch Map Capability
• Built-in Jetstream source

🚨 Breaking Change 🚨

• SDKs have to be updated to the latest version (internal message format optimizations)

New Features 🔥

• feat: read only view for UI by @veds-g in #1628
• feat: expose controller leader election duration and renew opts by @whynowy in #1657
• feat: controller change for fallback sink by @whynowy in #1664
• feat: Built-in Jetstream source implementation (Closes #1695) by @BulkBeing in #1723
• feat: adding numaflow version to the UI by @veds-g in #1744
• feat(config): standardize boolean value with YAML tags. Fixes #1742 by @charans29 in #1749
• feat: use protobuf to store header and messages in ISB by @yhl25 in #1771
• feat: use protobuf to store wmb in KV by @yhl25 in #1782
• feat: crate for retry with backoff strategy by @vigith in #1785
• feat: implement map batch by @kohlisid in #1778
• feat: enable restful daemon client option for UX server by @whynowy in #1826
• feat: add ObservedGeneration to Pipeline and ISBService Status by @xdevxy in #1799
• feat: use same server-info file for all map modes by @kohlisid in #1828
• feat: add controller and pipeline info metrics by @whynowy in #1855
• feat: pipeline and isbsvc resource health status metrics and detailed vertex type stats by @whynowy in #1856
• feat: new edge path by @veds-g in #1864
• feat: expose replica metrics for ISB Service and Vertex by @whynowy in #1859
• feat: Sync the health status of ISBService, pipeline and vertex by @chandankumar4 in #1860
• feat: Add ObservedGeneration field in vertex status and use it for calculating status by @chandankumar4 in #1892
• feat: Rust k8s model for Numaflow by @vigith in #1898

Major Fixes 🐞

• fix: dedup in user defined source by @yhl25 in #1613
• fix: allow pipeline to start with redis isbsvc (Fixes: #1513) by @QuentinFAIDIDE in #1567
• fix: watermark progression during pods creation/deletion by @yhl25 in #1619
• fix: numaflow package style fixes by @veds-g in #1622
• fix: bug in late message handling for sliding window by @yhl25 in #1471
• fix: race condition while publishing wm inside reduce by @yhl25 in #1599
• fix: readonly view by @veds-g in #1640
• fix: Dedup not working for multi-partitioned edge by @yhl25 in #1639
• fix: flaky TestDropOnFull by @yhl25 in #1647
• fix: avoid publishing watermarks for duplicate messages. by @yhl25 in #1649
• fix: pass headers to transfomer by @yhl25 in #1651
• fix: add headers to custom sinkrequest by @nagarajatantry in #1653
• fix: routing fixes by @veds-g in #1671
• fix: version downgrade for monaco-editor by @veds-g in #1673
• fix: summary bar overlay fix for plugin by @veds-g in #1710
• fix: height fixes to render pipeline view by @veds-g in #1720
• fix: Read from Oldest Offset for Idle Source Kafka e2e by @yhl25 in #1731
• fix: update SDKs to stable image by @samhith-kakarla in #1746
• fix: add retries when writing to redis and resp headers const by @vigith in #1766
• fix: save trait should accept Self as mutable by @vigith in #1795
• fix: value can be null by @vigith in #1831
• fix: should never scale down to < min by @whynowy in #1832
• fix(#1832): scale down to >=min, but not 0 when there's direct back pressure by @whynowy in #1834
• fix: api docs for jetstream service by @KeranYang in #1851
• fix: configure discard policy for WorkQueue/Interest by @kohlisid in #1884

New Contributors

• @QuentinFAIDIDE made their first contribution in #1567
• @nagarajatantry made their first contribution in #1653
• @magelisk made their first contribution in #1733
• @tolmanam made their first contribution in #1729
• @samhith-kakarla made their first contribution in #1746
• @charans29 made their first contribution in #1750

Installation

kubectl create namespace numaflow-system

# Install numaflow controller and UI
kubectl apply -n numaflow-system -f https://github.com/numaproj/numaflow/releases/download/v1.3.0/install.yaml

# Install validating webhook (Optional)
kubectl apply -n numaflow-system -f https://github.com/numaproj/numaflow/releases/download/v1.3.0/validating-webhook-install.yaml

Full Changelog: v1.2.0...v1.3.0

v1.3.0-rc1

Installation

kubectl create namespace numaflow-system

# Install numaflow controller and UI
kubectl apply -n numaflow-system -f https://github.com/numaproj/numaflow/releases/download/v1.3.0-rc1/install.yaml

# Install validating webhook (Optional)
kubectl apply -n numaflow-system -f https://github.com/numaproj/numaflow/releases/download/v1.3.0-rc1/validating-webhook-install.yaml

Full Changelog: v1.2.1...v1.3.0-rc1

v1.2.1

Major Fixes

• fix: pipeline page rendering issues by @veds-g in #1720

Installation

kubectl create namespace numaflow-system

# Install numaflow controller and UI
kubectl apply -n numaflow-system -f https://github.com/numaproj/numaflow/releases/download/v1.2.1/install.yaml

# Install validating webhook (Optional)
kubectl apply -n numaflow-system -f https://github.com/numaproj/numaflow/releases/download/v1.2.1/validating-webhook-install.yaml

Full Changelog: v1.2.0...v1.2.1

v1.2.0

Major Features 🔥

• Introducing Session Window and Streaming Reduce Capability
• Ability to pass headers from Source to Sink
• Fallback Sink

🚨 Breaking Change 🚨

• SDKs have to be updated to the latest version
• Python SDK has interface changes.

New Features 🔥

• feat: health status implementation by @kohlisid in #1406
• feat: Session Window and Reduce Streaming by @yhl25 in #1384
• feat: update tcp client connections by @kohlisid in #1429
• feat: enhance autoscaling peeking logic by @akashjkhamkar in #1432
• feat: pipeline health status for UI by @veds-g in #1460
• feat: configure standardResources via controller configmap by @inishchith in #1490
• feat: add support for SASL SCRAM 256 and 512 for Kafka by @Dnomyar in #1518
• feat: noop persistence store for reduce by @yhl25 in #1532
• feat: unaligned wal by @yhl25 in #1511
• feat: numaflow package by @veds-g in #1579
• feat: support headers for message by @yhl25 in #1578
• feat: Fallback Sink by @yhl25 in #1669

Major Fixes 🐞

• fix: terminate reduce vertex pods when pausing pipeline by @dpadhiar in #1481
• fix: configmap const name by @whynowy in #1423
• fix: server-secrets-init container restart by @veds-g in #1433
• fix: Kafka source reads duplicated messages by @jy4096 in #1438
• fix: UI Filter by status for pipelines doesn't work as expected by @inishchith in #1444
• fix: GetDownstreamEdges is not cycle safe by @inishchith in #1447
• fix: memory leak inside session windower by @yhl25 in #1445
• fix: bug where dashed line is treated as permalink + spelling/formatting by @ayildirim21 in #1467
• fix(controller): incorrect cpu/mem resources calculation by @whynowy in #1477
• fix: add idle handler offset nil check by @jy4096 in #1489
• fix: unknown for ISB details in pipeline card by @veds-g in #1497
• fix: avoid panic when ctx is canceled by @yhl25 in #1515
• fix: idle manager refactor for multi partitions by @jy4096 in #1512
• fix: podSpec incorrectly configured in case template exist by @inishchith in #1516
• fix: incorrect json schema for tls config by @whynowy in #1520
• fix(controller): vertex template metadata nil check by @inishchith in #1527
• fix: break from retry loop when key is not found by @yhl25 in #1535
• fix: initialize inflightAcks channel to not nil channel by @afugazzotto in #1548
• fix(codegen): protobuf upgrade by @whynowy in #1558
• fix: error message that cause Buttons overflow in the UI by @a49ibrahim in #1591
• fix: flaky e2e tests by @vigith in #1590
• fix: dedup in user-defined source #1613
• fix: allow pipeline to start with redis isbsvc (Fixes: #1513) #1567
• fix: watermark progression during pods creation/deletion #1619
• fix: numaflow package style fixes in #1622
• fix: bug in late message handling for sliding window #1471
• fix: race condition while publishing wm inside reduce #1599
• fix: read-only view for UI #1628

New Contributors ❤️

• @akashjkhamkar made their first contribution in #1432
• @inishchith made their first contribution in #1444
• @ayildirim21 made their first contribution in #1467
• @syayi made their first contribution in #1510
• @Dnomyar made their first contribution in #1518
• @afugazzotto made their first contribution in #1548
• @a49ibrahim made their first contribution in #1591
• @QuentinFAIDIDE made their first contribution in #1567

Installation

kubectl create namespace numaflow-system

# Install numaflow controller and UI
kubectl apply -n numaflow-system -f https://github.com/numaproj/numaflow/releases/download/v1.2.0/install.yaml

# Install validating webhook (Optional)
kubectl apply -n numaflow-system -f https://github.com/numaproj/numaflow/releases/download/v1.2.0/validating-webhook-install.yaml

Full Changelog: v1.1.0...v1.2.0

v1.2.0-rc5

Major Changes

• Skip sdk compatibility check if server info does not contain sdk version (#1703)

Pull Requests

v1.2.0-rc4...v1.2.0-rc5

Installation

kubectl create namespace numaflow-system

# Install numaflow controller and UI
kubectl apply -n numaflow-system -f https://github.com/numaproj/numaflow/releases/download/v1.2.0-rc5/install.yaml

# Install validating webhook (Optional)
kubectl apply -n numaflow-system -f https://github.com/numaproj/numaflow/releases/download/v1.2.0-rc5/validating-webhook-install.yaml

v1.2.0-rc4

Major Features 🔥

• Fallback Sink #1660

🚨 Breaking Change 🚨

• SDKs have to be updated to the latest version, user-defined sink has a breaking change.

Other changes

• feat: expose controller leader election duration and renew opts #1657
• chore: remove pods/exec from UX server RBAC #1659
• chore: do not create init-secrets when auth is disabled #1661
• fix: routing fixes #1671
• fix: version downgrade for monaco-editor #1673
• chore: add version compatibility check #1683

Full Changelog: v1.2.0-rc3...v1.2.0-rc4

v1.2.0-rc3

Major Changes

• Fix: pass headers to custom UDSink (#1653)
• Fix: pass headers to Source Transformer (#1651)
• Fix: avoid publishing watermarks for duplicate messages (#1649)
• Fix: Dedup not working for multi-partitioned edge (#1639)
• Fix: readonly view (#1640)

Pull Requests

v1.2.0-rc2...v1.2.0-rc3

Installation

kubectl create namespace numaflow-system

# Install numaflow controller and UI
kubectl apply -n numaflow-system -f https://github.com/numaproj/numaflow/releases/download/v1.2.0-rc3/install.yaml

# Install validating webhook (Optional)
kubectl apply -n numaflow-system -f https://github.com/numaproj/numaflow/releases/download/v1.2.0-rc3/validating-webhook-install.yaml