-
Notifications
You must be signed in to change notification settings - Fork 839
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Merge pull request #2939 from redpanda-data/telemetry
Add telemetry to our builds
- Loading branch information
Showing
15 changed files
with
392 additions
and
18 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,44 @@ | ||
| Telemetry | ||
| ========= | ||
|
|
||
| ## What is this for? | ||
|
|
||
| Our main goal is to find out the frequency with which each plugin is used in production environments, as this helps us prioritise enhancements and bug fixes for various plugin families on our roadmap. | ||
|
|
||
| Ideally, we'd also like to identify common patterns in plugin usage that may help us plan new work or identify gaps in our functionality. For example, if we were to see that almost all `aws_s3` outputs were paired with a `mutation` processor then we might conclude that embedding a mutation field into the plugin itself could be a useful feature. | ||
|
|
||
| ## What is being sent? | ||
|
|
||
| When a Redpanda Connect instance exports telemetry data to our collection server it sends a JSON payload that contains a high-level and anonymous summary of the contents of the config file being executed. Specific field values are never transmitted, nor are decorations of the config such as label names. For example, with an instance running the following config: | ||
|
|
||
| ```yaml | ||
| input: | ||
| label: fooer | ||
| generate: | ||
| interval: 1s | ||
| mapping: 'root.foo = "bar"' | ||
|
|
||
| output: | ||
| label: bazer | ||
| aws_s3: | ||
| bucket: baz | ||
| path: meow.txt | ||
| ``` | ||
| We would extract the following information: | ||
| - A unique identifier for the Redpanda Connect instance. | ||
| - The duration for which the config has been running thus far. | ||
| - That the config contains a `generate` input and an `aws_s3` output. | ||
| - The IP address of the running Redpanda Connect instance (as a byproduct of the data delivery mechanism). | ||
|
|
||
| The code responsible for extracting this data is simple enough to dig into, and we encourage curious users to do so. A good place to start is the data format, which can be found at [`./payload.go`](./payload.go). | ||
|
|
||
| ## When is it sent? | ||
|
|
||
| Telemetry data is sent from an instance of Redpanda Connect that has been running for at least 5 minutes, this is in order to avoid sending data from instances used for testing or experimentation. Once telemetry data starts being emitted it is sent once every 24 hours. | ||
|
|
||
| ## How do I avoid it? | ||
|
|
||
| Any custom build of Redpanda Connect will not send this data, as it is only included in the build artifacts published by us either through Github releases or our official Docker images. You can also prevent telemetry by blocking the internet traffic, Redpanda Connect will continue operating as normal if it is unable to deliver telemetry data. | ||
|
|
Empty file.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,33 @@ | ||
| // Copyright 2024 Redpanda Data, Inc. | ||
| // | ||
| // Licensed under the Apache License, Version 2.0 (the "License"); | ||
| // you may not use this file except in compliance with the License. | ||
| // You may obtain a copy of the License at | ||
| // | ||
| // http://www.apache.org/licenses/LICENSE-2.0 | ||
| // | ||
| // Unless required by applicable law or agreed to in writing, software | ||
| // distributed under the License is distributed on an "AS IS" BASIS, | ||
| // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| // See the License for the specific language governing permissions and | ||
| // limitations under the License. | ||
|
|
||
| package telemetry | ||
|
|
||
| import "github.com/redpanda-data/benthos/v4/public/service" | ||
|
|
||
| type logWrapper struct { | ||
| l *service.Logger | ||
| } | ||
|
|
||
| func (l *logWrapper) Errorf(format string, v ...interface{}) { | ||
| l.l.With("component", "resty").Debugf(format, v...) | ||
| } | ||
|
|
||
| func (l *logWrapper) Warnf(format string, v ...interface{}) { | ||
| // Ignore | ||
| } | ||
|
|
||
| func (l *logWrapper) Debugf(format string, v ...interface{}) { | ||
| // Ignore | ||
| } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,84 @@ | ||
| // Copyright 2024 Redpanda Data, Inc. | ||
| // | ||
| // Licensed under the Apache License, Version 2.0 (the "License"); | ||
| // you may not use this file except in compliance with the License. | ||
| // You may obtain a copy of the License at | ||
| // | ||
| // http://www.apache.org/licenses/LICENSE-2.0 | ||
| // | ||
| // Unless required by applicable law or agreed to in writing, software | ||
| // distributed under the License is distributed on an "AS IS" BASIS, | ||
| // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| // See the License for the specific language governing permissions and | ||
| // limitations under the License. | ||
|
|
||
| package telemetry | ||
|
|
||
| import ( | ||
| "fmt" | ||
| "time" | ||
|
|
||
| "github.com/redpanda-data/benthos/v4/public/service" | ||
| ) | ||
|
|
||
| // Information gathered from each component present in the running config. | ||
| type componentInfo struct { | ||
| // The type (input, output, etc) of the plugin. | ||
| Type string `json:"type"` | ||
|
|
||
| // The name (aws_s3, generate, etc) of the plugin. | ||
| Name string `json:"name"` | ||
| } | ||
|
|
||
| // Contains all of the information which is delivered during a telemetry | ||
| // export, serialisable in JSON format. | ||
| type payload struct { | ||
| // A unique identifier for the Redpanda Connect instance. | ||
| ID string `json:"id"` | ||
|
|
||
| // Uptime of the Redpanda Connect instance. | ||
| Uptime int64 `json:"uptime"` | ||
|
|
||
| // A slice representing each component within a config. | ||
| Components []componentInfo `json:"components"` | ||
| } | ||
|
|
||
| // All information sent during a telemetry export is extracted within this | ||
| // function and stored within the payload. | ||
| func extractPayload(identifier string, logger *service.Logger, schema *service.ConfigSchema, conf *service.ParsedConfig) (*payload, error) { | ||
| p := payload{ID: identifier, Uptime: 0} | ||
|
|
||
| rootValue, err := conf.FieldAny() | ||
| if err != nil { | ||
| return nil, fmt.Errorf("failed to obtain root of config: %w", err) | ||
| } | ||
|
|
||
| if err := schema.NewStreamConfigWalker().WalkComponentsAny(rootValue, func(w *service.WalkedComponent) error { | ||
| p.Components = append(p.Components, componentInfo{ | ||
| Type: w.ComponentType, | ||
| Name: w.Name, | ||
| }) | ||
| return nil | ||
| }); err != nil { | ||
| logger.With("error", err).Debug("Failed to walk config") | ||
| } | ||
|
|
||
| return &p, nil | ||
| } | ||
|
|
||
| // This function runs asynchronously and is solely where telemetry data is | ||
| // exported. | ||
| func exporterLoop(p *payload, exportDelay, exportPeriod time.Duration, exporter *telemetryExporter) { | ||
| started := time.Now() | ||
|
|
||
| // First, wait until after the export delay has passed. | ||
| time.Sleep(exportDelay) | ||
|
|
||
| for { | ||
| p.Uptime = int64(time.Since(started) / time.Second) | ||
| exporter.export(p) | ||
|
|
||
| // Now wait for the next export. | ||
| time.Sleep(exportPeriod) | ||
| } | ||
| } |
Oops, something went wrong.