Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactor all the things #3

Merged
merged 14 commits into from
May 31, 2024
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 7 additions & 7 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -62,15 +62,15 @@ All notable changes to this project will be documented in this file.

### Fixed

- Fixed a regression in v4.25.0 where [template based components](https://www.benthos.dev/docs/configuration/templating) were not parsing correctly from configs.
- Fixed a regression in v4.25.0 where [template based components](https://warpstreamlabs.github.io/bento/docs/configuration/templating) were not parsing correctly from configs.

## 4.25.0 - 2024-03-01

### Added

- Field `address_cache` added to the `socket_server` input.
- Field `read_header` added to the `amqp_1` input.
- All inputs with a `codec` field now support a new field `scanner` to replace it. Scanners are more powerful as they are configured in a structured way similar to other component types rather than via a single string field, for more information [check out the scanners page](https://www.benthos.dev/docs/components/scanners/about).
- All inputs with a `codec` field now support a new field `scanner` to replace it. Scanners are more powerful as they are configured in a structured way similar to other component types rather than via a single string field, for more information [check out the scanners page](https://warpstreamlabs.github.io/bento/docs/components/scanners/about).
- New `diff` and `patch` Bloblang methods.
- New `processors` processor.
- Field `read_header` added to the `amqp_1` input.
Expand Down Expand Up @@ -680,7 +680,7 @@ to the [`database/sql` docs](https://pkg.go.dev/database/sql#DB.SetMaxIdleConns)

## 4.0.0 - 2022-04-20

This is a major version release, for more information and guidance on how to migrate please refer to [https://benthos.dev/docs/guides/migration/v4](https://www.benthos.dev/docs/guides/migration/v4).
This is a major version release, for more information and guidance on how to migrate please refer to [https://benthos.dev/docs/guides/migration/v4](https://warpstreamlabs.github.io/bento/docs/guides/migration/v4).

### Added

Expand Down Expand Up @@ -720,7 +720,7 @@ This is a major version release, for more information and guidance on how to mig
- The `switch` output field `retry_until_success` now defaults to `false`.
- All AWS components now have a default `region` field that is empty, allowing environment variables or profile values to be used by default.
- Serverless distributions of Bento (AWS lambda, etc) have had the default output config changed to reject messages when the processing fails, this should make it easier to handle errors from invocation.
- The standard metrics emitted by Bento have been largely simplified and improved, for more information [check out the metrics page](https://www.benthos.dev/docs/components/metrics/about).
- The standard metrics emitted by Bento have been largely simplified and improved, for more information [check out the metrics page](https://warpstreamlabs.github.io/bento/docs/components/metrics/about).
- The default metrics type is now `prometheus`.
- The `http_server` metrics type has been renamed to `json_api`.
- The `stdout` metrics type has been renamed to `logger`.
Expand Down Expand Up @@ -1993,7 +1993,7 @@ This is a major version release, for more information and guidance on how to mig

## 3.0.0 - 2019-09-17

This is a major version release, for more information and guidance on how to migrate please refer to [https://benthos.dev/docs/guides/migration/v3](https://www.benthos.dev/docs/guides/migration/v3).
This is a major version release, for more information and guidance on how to migrate please refer to [https://benthos.dev/docs/guides/migration/v3](https://warpstreamlabs.github.io/bento/docs/guides/migration/v3).

### Added

Expand Down Expand Up @@ -2347,11 +2347,11 @@ This is a major version release, for more information and guidance on how to mig

### Changed

This is a major version released due to a series of minor breaking changes, you can read the [full migration guide here](https://www.benthos.dev/docs/guides/migration/v2).
This is a major version released due to a series of minor breaking changes, you can read the [full migration guide here](https://warpstreamlabs.github.io/bento/docs/guides/migration/v2).

#### Configuration

- Bento now attempts to infer the `type` of config sections whenever the field is omitted, for more information please read this overview: [Concise Configuration](https://www.benthos.dev/docs/configuration/about#concise-configuration).
- Bento now attempts to infer the `type` of config sections whenever the field is omitted, for more information please read this overview: [Concise Configuration](https://warpstreamlabs.github.io/bento/docs/configuration/about#concise-configuration).
- Field `unsubscribe_on_close` of the `nats_stream` input is now `false` by default.

#### Service
Expand Down
30 changes: 15 additions & 15 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -171,23 +171,23 @@ docker run --rm \

Contributions are welcome, please [read the guidelines](CONTRIBUTING.md), come and chat (links are on the [community page][community]), and watch your back.

[inputs]: https://www.benthos.dev/docs/components/inputs/about
[about-categories]: https://www.benthos.dev/docs/about#components
[processors]: https://www.benthos.dev/docs/components/processors/about
[outputs]: https://www.benthos.dev/docs/components/outputs/about
[metrics]: https://www.benthos.dev/docs/components/metrics/about
[tracers]: https://www.benthos.dev/docs/components/tracers/about
[config-interp]: https://www.benthos.dev/docs/configuration/interpolation
[streams-api]: https://www.benthos.dev/docs/guides/streams_mode/streams_api
[streams-mode]: https://www.benthos.dev/docs/guides/streams_mode/about
[general-docs]: https://www.benthos.dev/docs/about
[bloblang-about]: https://www.benthos.dev/docs/guides/bloblang/about
[config-doc]: https://www.benthos.dev/docs/configuration/about
[serverless]: https://www.benthos.dev/docs/guides/serverless/about
[cookbooks]: https://www.benthos.dev/cookbooks
[inputs]: https://warpstreamlabs.github.io/bento/docs/components/inputs/about
[about-categories]: https://warpstreamlabs.github.io/bento/docs/about#components
[processors]: https://warpstreamlabs.github.io/bento/docs/components/processors/about
[outputs]: https://warpstreamlabs.github.io/bento/docs/components/outputs/about
[metrics]: https://warpstreamlabs.github.io/bento/docs/components/metrics/about
[tracers]: https://warpstreamlabs.github.io/bento/docs/components/tracers/about
[config-interp]: https://warpstreamlabs.github.io/bento/docs/configuration/interpolation
[streams-api]: https://warpstreamlabs.github.io/bento/docs/guides/streams_mode/streams_api
[streams-mode]: https://warpstreamlabs.github.io/bento/docs/guides/streams_mode/about
[general-docs]: https://warpstreamlabs.github.io/bento/docs/about
[bloblang-about]: https://warpstreamlabs.github.io/bento/docs/guides/bloblang/about
[config-doc]: https://warpstreamlabs.github.io/bento/docs/configuration/about
[serverless]: https://warpstreamlabs.github.io/bento/docs/guides/serverless/about
[cookbooks]: https://warpstreamlabs.github.io/bento/cookbooks
[releases]: https://github.com/warpstreamlabs/bento/releases
[plugin-repo]: https://github.com/warpstreamlabs/bento-plugin-example
[getting-started]: https://www.benthos.dev/docs/guides/getting_started
[getting-started]: https://warpstreamlabs.github.io/bento/docs/guides/getting_started

[godoc-badge]: https://pkg.go.dev/badge/github.com/warpstreamlabs/bento/v4/public
[godoc-url]: https://pkg.go.dev/github.com/warpstreamlabs/bento/v4/public
Expand Down
2 changes: 1 addition & 1 deletion config/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,4 +9,4 @@ If you're looking for specific config examples for a use case you have then try
bento create kafka/schema_registry_decode/nats_jetstream > example.yaml
```

[unit-tests]: https://www.benthos.dev/docs/configuration/unit_testing
[unit-tests]: https://warpstreamlabs.github.io/bento/docs/configuration/unit_testing
2 changes: 1 addition & 1 deletion config/test/cookbooks/filtering.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -4,5 +4,5 @@ pipeline:
root = match {
meta("topic").or("") == "foo" ||
doc.type.or("") == "bar" ||
doc.urls.contains("https://www.benthos.dev/").catch(false) => deleted()
doc.urls.contains("https://warpstreamlabs.github.io/bento/").catch(false) => deleted()
}
2 changes: 1 addition & 1 deletion config/test/cookbooks/filtering_benthos_test.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,6 @@ tests:
metadata:
topic: foo
- content: '{"doc":{"should":"not remain","type":"bar"},"id":"3"}'
- content: '{"doc":{"should":"not remain","urls":["https://www.benthos.dev/"]},"id":"4"}'
- content: '{"doc":{"should":"not remain","urls":["https://warpstreamlabs.github.io/bento/"]},"id":"4"}'
output_batches:
- - content_equals: '{"doc":{"should":"remain"},"id":"1"}'
2 changes: 1 addition & 1 deletion internal/bloblang/query/methods_strings.go
Original file line number Diff line number Diff line change
Expand Up @@ -1247,7 +1247,7 @@ var _ = registerSimpleMethod(
MethodCategoryParsing, "",
NewExampleSpec("",
`root.foo_url = this.foo_url.parse_url()`,
`{"foo_url":"https://www.benthos.dev/docs/guides/bloblang/about"}`,
`{"foo_url":"https://warpstreamlabs.github.io/bento/docs/guides/bloblang/about"}`,
`{"foo_url":{"fragment":"","host":"www.benthos.dev","opaque":"","path":"/docs/guides/bloblang/about","raw_fragment":"","raw_path":"","raw_query":"","scheme":"https"}}`,
),
NewExampleSpec("",
Expand Down
2 changes: 1 addition & 1 deletion internal/cli/run.go
Original file line number Diff line number Diff line change
Expand Up @@ -115,7 +115,7 @@ func App(opts *common.CLIOpts) *cli.App {

app := &cli.App{
Name: "Bento",
Usage: "A stream processor for mundane tasks - https://www.benthos.dev",
Usage: "A stream processor for mundane tasks - https://warpstreamlabs.github.io/bento",
Description: `
Either run Bento as a stream processor or choose a command:

Expand Down
2 changes: 1 addition & 1 deletion internal/impl/cockroachdb/input_changefeed.go
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,7 @@ func crdbChangefeedInputConfig() *service.ConfigSpec {
Description("CSV of tables to be included in the changefeed").
Example([]string{"table1", "table2"}),
service.NewStringField("cursor_cache").
Description("A [cache resource](https://www.benthos.dev/docs/components/caches/about) to use for storing the current latest cursor that has been successfully delivered, this allows Bento to continue from that cursor upon restart, rather than consume the entire state of the table.").
Description("A [cache resource](https://warpstreamlabs.github.io/bento/docs/components/caches/about) to use for storing the current latest cursor that has been successfully delivered, this allows Bento to continue from that cursor upon restart, rather than consume the entire state of the table.").
Optional(),
service.NewStringListField("options").
Description("A list of options to be included in the changefeed (WITH X, Y...).\n**NOTE: Both the CURSOR option and UPDATED will be ignored from these options when a `cursor_cache` is specified, as they are set explicitly by Bento in this case.**").
Expand Down
2 changes: 1 addition & 1 deletion internal/impl/pure/processor_catch.go
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@ When messages leave the catch block their fail flags are cleared. This processor

More information about error handling can be found [here](/docs/configuration/error_handling).`).
LintRule(`if this.or([]).any(pconf -> pconf.type.or("") == "try" || pconf.try.type() == "array" ) {
"'catch' block contains a 'try' block which will never execute due to errors only being cleared at the end of the 'catch', for more information about nesting 'try' within 'catch' read: https://www.benthos.dev/docs/components/processors/try#nesting-within-a-catch-block"
"'catch' block contains a 'try' block which will never execute due to errors only being cleared at the end of the 'catch', for more information about nesting 'try' within 'catch' read: https://warpstreamlabs.github.io/bento/docs/components/processors/try#nesting-within-a-catch-block"
}`).
Field(service.NewProcessorListField("").Default([]any{})),
func(conf *service.ParsedConfig, res *service.Resources) (service.BatchProcessor, error) {
Expand Down
4 changes: 2 additions & 2 deletions public/service/message.go
Original file line number Diff line number Diff line change
Expand Up @@ -265,7 +265,7 @@ func (m *Message) SetStructuredMut(i any) {

// SetError marks the message as having failed a processing step and adds the
// error to it as context. Messages marked with errors can be handled using a
// range of methods outlined in https://www.benthos.dev/docs/configuration/error_handling.
// range of methods outlined in https://warpstreamlabs.github.io/bento/docs/configuration/error_handling.
func (m *Message) SetError(err error) {
if m.onErr != nil {
m.onErr(err)
Expand All @@ -275,7 +275,7 @@ func (m *Message) SetError(err error) {

// GetError returns an error associated with a message, or nil if there isn't
// one. Messages marked with errors can be handled using a range of methods
// outlined in https://www.benthos.dev/docs/configuration/error_handling.
// outlined in https://warpstreamlabs.github.io/bento/docs/configuration/error_handling.
func (m *Message) GetError() error {
return m.part.ErrorGet()
}
Expand Down
2 changes: 1 addition & 1 deletion public/service/processor.go
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ type Processor interface {
// When an error is returned the input message will continue down the
// pipeline but will be marked with the error with *message.SetError, and
// metrics and logs will be emitted. The failed message can then be handled
// with the patterns outlined in https://www.benthos.dev/docs/configuration/error_handling.
// with the patterns outlined in https://warpstreamlabs.github.io/bento/docs/configuration/error_handling.
//
// The Message types returned MUST be derived from the provided message, and
// CANNOT be custom implementations of Message. In order to copy the
Expand Down
4 changes: 2 additions & 2 deletions public/wasm/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,6 @@ In this directory are libraries and examples tailored to help developers create

Most of these are adapted from the fantastic range of examples provided by [the Wazero library][wazero_examples]. Our goal is to eventually provide libraries and examples for all popular languages and we'll be tackling them one at a time based on demand. Please be patient but also make [yourself heard][community].

[processor.wasm]: https://www.benthos.dev/docs/components/processors/wasm
[processor.wasm]: https://warpstreamlabs.github.io/bento/docs/components/processors/wasm
[wazero_examples]: https://github.com/tetratelabs/wazero/tree/main/examples
[community]: https://www.benthos.dev/community
[community]: https://warpstreamlabs.github.io/bento/community
2 changes: 1 addition & 1 deletion public/wasm/examples/tinygo/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,4 +17,4 @@ pipeline:
```

[TinyGo]: https://tinygo.org/
[processor.wasm]: https://www.benthos.dev/docs/components/processors/wasm
[processor.wasm]: https://warpstreamlabs.github.io/bento/docs/components/processors/wasm
2 changes: 1 addition & 1 deletion resources/scripts/release_notes.sh
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
#!/bin/sh
echo "For installation instructions check out the [getting started guide](https://www.benthos.dev/docs/guides/getting_started)."
echo "For installation instructions check out the [getting started guide](https://warpstreamlabs.github.io/bento/docs/guides/getting_started)."
cat CHANGELOG.md | awk '
/^## [0-9]/ {
release++;
Expand Down
2 changes: 1 addition & 1 deletion website/blog/2019-05-27-compiling-benthos-to-wasm.md
Original file line number Diff line number Diff line change
Expand Up @@ -169,7 +169,7 @@ to completion without rushing the conclusion. Yes, I'm still bitter about Game
of Thrones.

[meltwater]: https://underthehood.meltwater.com/blog/2019/06/17/benthos-lab-a-case-study-of-hackathon-innovation/
[Bento]: https://www.benthos.dev/
[Bento]: https://warpstreamlabs.github.io/bento/
[wasm-go-wiki]: https://github.com/golang/go/wiki/WebAssembly
[syscall-js-func]: https://godoc.org/syscall/js#Func
[go-build-constraint]: https://golang.org/pkg/go/build/#hdr-Build_Constraints
Expand Down
2 changes: 1 addition & 1 deletion website/blog/2019-06-17-introducing-benthos-lab.md
Original file line number Diff line number Diff line change
Expand Up @@ -102,6 +102,6 @@ missing (or broken, obviously) then please [open an issue][lab-issues].
[wasm-blog]: /blog/2019/05/27/compiling-benthos-to-wasm/
[lab-repo]: https://github.com/warpstreamlabs/bento-lab
[lab-issues]: https://github.com/warpstreamlabs/bento-lab/issues
[benthos]: https://www.benthos.dev
[benthos]: https://warpstreamlabs.github.io/bento
[under-the-hood]: https://underthehood.meltwater.com/
[benthos-procs]: https://benthos.dev/docs/components/processors/about
2 changes: 1 addition & 1 deletion website/blog/2019-08-20-write-a-benthos-plugin.md
Original file line number Diff line number Diff line change
Expand Up @@ -288,7 +288,7 @@ metrics:

I'm sure you'll make great use of Bento plugins with your extremely important work /s.

[benthos]: https://www.benthos.dev
[benthos]: https://warpstreamlabs.github.io/bento
[benthos-repo]: https://github.com/warpstreamlabs/bento
[plugin-repo]: https://github.com/warpstreamlabs/bento-plugin-example
[processors]: https://benthos.dev/docs/components/processors/about
Expand Down
10 changes: 5 additions & 5 deletions website/blog/2021-03-09-redpanda.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,11 +17,11 @@ Combining the power of Redpanda and Bento for your streaming needs is so simple

<!--truncate-->

[Bento](https://www.benthos.dev/) is an open source stream processor that provides data mapping, filtering, hydration and enrichment capabilities across a wide range of connectors. It is driven by a minimal, declarative configuration spec, and with a transaction based architecture it eliminates the development effort of building resilient stream processing pipelines.
[Bento](https://warpstreamlabs.github.io/bento/) is an open source stream processor that provides data mapping, filtering, hydration and enrichment capabilities across a wide range of connectors. It is driven by a minimal, declarative configuration spec, and with a transaction based architecture it eliminates the development effort of building resilient stream processing pipelines.

Likewise, with its simplicity and high performance, Redpanda eliminates the operational effort of data persistence and availability by providing a Kafka-compatible streaming platform without the moving parts.

With so much taken care of you're well in for a boring, uneventful time when you combine the two. Make sure you've grabbed a copy of both services, full instructions can be found in the [getting started guide for Bento](https://www.benthos.dev/docs/guides/getting_started) and [the Redpanda docs](https://vectorized.io/docs). In this post we'll be running them with Docker so we'll start by pulling both images:
With so much taken care of you're well in for a boring, uneventful time when you combine the two. Make sure you've grabbed a copy of both services, full instructions can be found in the [getting started guide for Bento](https://warpstreamlabs.github.io/bento/docs/guides/getting_started) and [the Redpanda docs](https://vectorized.io/docs). In this post we'll be running them with Docker so we'll start by pulling both images:

```
docker pull vectorized/redpanda:latest
Expand Down Expand Up @@ -82,7 +82,7 @@ but this here ain't structured at all!

When you're finished hit CTRL+C and it'll exit.

Next, let's try reading that data back out from Redpanda, this time let's also add a [processor](https://www.benthos.dev/docs/components/processors/about) in order to mutate our data, copy the following into a file `consumer.yaml`:
Next, let's try reading that data back out from Redpanda, this time let's also add a [processor](https://warpstreamlabs.github.io/bento/docs/components/processors/about) in order to mutate our data, copy the following into a file `consumer.yaml`:

```yaml
input:
Expand Down Expand Up @@ -119,6 +119,6 @@ Now you should see it print mutated versions of the messages you sent to Stdout:
{"doc":[{"id":"2"},"also structured in a different (but totally valid) way"],"length":69,"topic":"topic_A"}
```

The [Bloblang processor](https://www.benthos.dev/docs/components/processors/bloblang) in our consumer config has remapped the original message to a new field `doc`, first attempting to extract it as a structured document, but falling back to a stringified version of it when it's unstructured. We've also added a field `length` which contains the length of the original message, and `topic` which contains the Kafka topic the message was consumed from.
The [Bloblang processor](https://warpstreamlabs.github.io/bento/docs/components/processors/bloblang) in our consumer config has remapped the original message to a new field `doc`, first attempting to extract it as a structured document, but falling back to a stringified version of it when it's unstructured. We've also added a field `length` which contains the length of the original message, and `topic` which contains the Kafka topic the message was consumed from.

That's it for now, if you're still hungry for more then check out the Bento website at [https://www.benthos.dev](https://www.benthos.dev/), and you can learn more about the Bento mapping language Bloblang [in this guide](https://www.benthos.dev/docs/guides/bloblang/about).
That's it for now, if you're still hungry for more then check out the Bento website at [https://warpstreamlabs.github.io/bento](https://warpstreamlabs.github.io/bento/), and you can learn more about the Bento mapping language Bloblang [in this guide](https://warpstreamlabs.github.io/bento/docs/guides/bloblang/about).
Loading
Loading