-
-## Adapter
-An adapter connects to any external data source and forwards received events to the internal StreamPipes system. Within StreamPipes, the output of adapters are available in form of the two primary building blocks **Data Set** and **Data Stream**.
-Adapters can be either created by using StreamPipes Connect, a module to easily connect to new data sources directly from the user interface, or by defining an adapter using the provided Software Development Kit (SDK).
-
-## Data Set / Data Stream
-**Data Streams** and **Data Sets** represent the primary source for working with events in StreamPipes.
-A stream is an ordered sequence of events, where an event typically consists of one or more observation values and additional metadata. The "structure" (or schema) of an event provided by a data stream or set is stored in the internal semantic schema registry of StreamPipes.
-While data streams are typically unbounded, data sets have a fixed end and are internally "replayed" by the system from beginning to end once they are used as part of a pipeline.
-As follows, although when referring to data streams, most concepts also apply for data sets.
-
-## Data Processor
-**Data Processors** in StreamPipes transform one or more input data streams into an output data stream.
-Such transformations can be rather simple, e.g. filtering based on a predefined rule or more complex, e.g. applying rule-based or learning-based algorithms on the data.
-Data Processors can be applied on any data stream that matches the input requirements of a processor. In addition, most processors can be configured by providing user-defined parameters directly in the user interface.
-Processing elements define stream requirements that are a set of minimum properties an incoming event stream must provide. Data processors can keep state or perform stateless operations.
-At runtime, data streams are processed by using one of the underlying runtime wrappers (see the developer guide for more details).
-
-## Data Sink
-**Data Sinks** consume event streams similar to Data Processors, but do not provide an output data stream. As such, data sinks typically perform some action or trigger a visualization as a result of a stream transformation.
-Similar to data processors, sinks also require for the presence of specific input requirements of any bound data stream and can be customized.
-StreamPipes provides several internal data sinks, e.g., to create notifications, visualize live data or persist historical data of incoming streams. In addition, various data sinks are provided to forward data streams to external systems such as databases.
-
-## Pipeline
-A pipeline in Apache StreamPipes describes the transformation process from a data stream to a data sink. Typically, a pipeline consists of at least one data stream (or data set), zero or more data processors and at least one data sink.
-Pipelines are built by users in a graphical way using the **Pipeline Editor** and can be started and stopped at any time.
diff --git a/website-v2/versioned_docs/version-0.70.0/05_deploy-docker.md b/website-v2/versioned_docs/version-0.70.0/05_deploy-docker.md
deleted file mode 100644
index e7926c372..000000000
--- a/website-v2/versioned_docs/version-0.70.0/05_deploy-docker.md
+++ /dev/null
@@ -1,75 +0,0 @@
----
-id: deploy-docker
-title: Docker Deployment
-sidebar_label: Docker Deployment
-original_id: deploy-docker
----
-
-StreamPipes Compose is a simple collection of user-friendly `docker-compose` files that easily lets gain first-hand experience with Apache StreamPipes.
-
-> **NOTE**: We recommend StreamPipes Compose to only use for initial try-out and testing. If you are a developer and
-> want to develop new pipeline elements or core feature, use the [StreamPipes CLI](06_extend-cli.md).
-
-#### TL;DR: A one-liner to rule them all :-)
-
-```bash
-docker-compose up -d
-```
-Go to http://localhost to finish the installation in the browser. Once finished, switch to the pipeline editor and start the interactive tour or check the [online tour](https://streampipes.apache.org/docs/docs/user-guide-tour/) to learn how to create your first pipeline!
-
-## Prerequisites
-* Docker >= 17.06.0
-* Docker-Compose >= 1.17.0 (Compose file format: 3.4)
-* Google Chrome (recommended), Mozilla Firefox, Microsoft Edge
-
-Tested on: **macOS, Linux, Windows 10** (CMD, PowerShell, GitBash)
-
-**macOS** and **Windows 10** (Pro, Enterprise, Education) users can easily get Docker and Docker-Compose on their systems by installing **Docker for Mac/Windows** (recommended).
-
-> **NOTE**: On purpose, we disabled all port mappings except of http port **80** to access the StreamPipes UI to provide minimal surface for conflicting ports.
-
-## Usage
-We provide two options to get you going:
-
-- **default**: a light-weight option with few pipeline elements, needs less memory
-- **full**: contains more pipeline elements, requires **>16 GB RAM** (recommended)
-
-**Starting** the **default** option is as easy as simply running:
-> **NOTE**: Starting might take a while since `docker-compose up` also initially pulls all Docker images from Dockerhub.
-
-```bash
-docker-compose up -d
-# go to after all services are started http://localhost
-```
-After all containers are successfully started just got to your browser and visit http://localhost to finish the installation. Once finished, switch to the pipeline editor and start the interactive tour or check the [online tour](https://streampipes.apache.org/docs/docs/user-guide-tour/) to learn how to create your first pipeline!
-
-**Stopping** the **default** option is similarly easy:
-```bash
-docker-compose down
-# if you want to remove mapped data volumes, run:
-# docker-compose down -v
-```
-
-Starting the **full** option is almost the same, just specify the `docker-compose.full.yml` file:
-```bash
-docker-compose -f docker-compose.full.yml up -d
-# go to after all services are started http://localhost
-```
-Stopping the **full** option:
-```bash
-docker-compose -f docker-compose.full.yml down
-#docker-compose -f docker-compose.full.yml down -v
-```
-
-## Update services
-To actively pull the latest available Docker images use:
-```bash
-docker-compose pull
-# docker-compose -f docker-compose.full.yml pull
-```
-
-## Upgrade
-To upgrade to another StreamPipes version, simply edit the `SP_VERSION` in the `.env` file.
-```
-SP_VERSION=
-
-
-
-## Project setup
-
-Instead of creating a new project from scratch, we recommend to use the Maven archetype to create a new project skeleton (streampipes-archetype-extensions-jvm).
-Enter the following command in a command line of your choice (Apache Maven needs to be installed):
-
-```
-mvn archetype:generate \
--DarchetypeGroupId=org.apache.streampipes -DarchetypeArtifactId=streampipes-archetype-extensions-jvm \
--DarchetypeVersion=0.70.0 -DgroupId=my.groupId \
--DartifactId=my-example -DclassNamePrefix=MyExample -DpackageName=mypackagename
-```
-
-You will see a project structure similar to the structure shown in the [archetypes](06_extend-archetypes.md) section.
-
-Note
-The implementation in this tutorial is pretty simple - our processor will fire an event every time the GPS location is inside the geofence.
- In a real-world application, you would probably want to define a pattern that recognizes the _first_ event a vehicle enters the geofence.
- This can be easily done using a CEP library.
-
-
-Now you're ready to create your first data processor for StreamPipes!
-
-## Adding data processor requirements
-
-First, we will add a new stream requirement.
-Create a new class `GeofencingProcessor` which should look as follows:
-
-```java
-package org.apache.streampipes.pe.example;
-
-import org.apache.streampipes.commons.exceptions.SpRuntimeException;
-import org.apache.streampipes.model.DataProcessorType;
-import org.apache.streampipes.model.graph.DataProcessorDescription;
-import org.apache.streampipes.model.runtime.Event;
-import org.apache.streampipes.sdk.builder.ProcessingElementBuilder;
-import org.apache.streampipes.sdk.builder.StreamRequirementsBuilder;
-import org.apache.streampipes.sdk.helpers.EpRequirements;
-import org.apache.streampipes.sdk.helpers.Labels;
-import org.apache.streampipes.sdk.helpers.Locales;
-import org.apache.streampipes.sdk.helpers.OutputStrategies;
-import org.apache.streampipes.sdk.utils.Assets;
-import org.apache.streampipes.wrapper.context.EventProcessorRuntimeContext;
-import org.apache.streampipes.wrapper.routing.SpOutputCollector;
-import org.apache.streampipes.wrapper.standalone.ProcessorParams;
-import org.apache.streampipes.wrapper.standalone.StreamPipesDataProcessor;
-
-public class GeofencingProcessor extends StreamPipesDataProcessor {
-
- private static final String LATITUDE_CENTER = "latitude-center";
- private static final String LONGITUDE_CENTER = "longitude-center";
-
- @Override
- public DataProcessorDescription declareModel() {
- return ProcessingElementBuilder.create("org.apache.streampipes.tutorial-geofencing")
- .category(DataProcessorType.ENRICH)
- .withAssets(Assets.DOCUMENTATION, Assets.ICON)
- .build();
- }
-
- @Override
- public void onInvocation(ProcessorParams parameters, SpOutputCollector spOutputCollector, EventProcessorRuntimeContext runtimeContext) throws SpRuntimeException {
-
- }
-
- @Override
- public void onEvent(Event event, SpOutputCollector collector) throws SpRuntimeException {
-
- }
-
- @Override
- public void onDetach() throws SpRuntimeException {
-
- }
-}
-
-
-```
-
-In this class, we need to implement three methods: The `declareModel` method is used to define abstract stream requirements such as event properties that must be present in any input stream that is later connected to the element using the StreamPipes UI.
-The second method, `onInvocation` is triggered once a pipeline is started. Finally, the `onEvent` method
-
-Similar to data sources, the SDK provides a builder class to generate the description for data processors.
-Delete the content within the ``declareModel`` method and add the following lines to the `declareModel` method:
-
-```java
-return ProcessingElementBuilder.create("org.apache.streampipes.tutorial.geofencing", "Geofencing", "A simple geofencing data processor")
-```
-
-This creates a new data processor with the ID, title and description assigned to the element builder.
-Next, we add some _stream requirements_ to the description. As we'd like to develop a generic pipeline element that works with any event that provides a lat/lng pair, we define two stream requirements as stated below:
-
-```java
-.requiredStream(StreamRequirementsBuilder
- .create()
- .requiredPropertyWithUnaryMapping(EpRequirements.domainPropertyReq(Geo.lat),
- Labels.from("latitude-field", "Latitude", "The event " +
- "property containing the latitude value"), PropertyScope.MEASUREMENT_PROPERTY)
- .requiredPropertyWithUnaryMapping(EpRequirements.domainPropertyReq(Geo.lng),
- Labels.from("longitude-field", "Longitude", "The event " +
- "property containing the longitude value"), PropertyScope.MEASUREMENT_PROPERTY)
- .build())
-```
-
-The first line, `.requiredStream()` defines that we want a data processor with exactly one input stream. Adding more stream requirements would create elements with multiple input connectors in StreamPipes.
-Stream requirements can be assigned by using the `StreamRequirementsBuilder` class.
-In our example, we define two requirements, so-called _domain property requirements_. In contrast to _data type requirements_ where we'd expect an event property with a field of a specific data type (e.g., float), domain property requirements expect a specific domain property, e.g., from a vocabulary such as the WGS84 Geo vocab.
-
-Once a pipeline is deployed, we are interested in the actual field (and its field name) that contains the latitude and longitude values.
-In some cases, there might me more than one field that satisfies a property requirement and we would like users to select the property the geofencing component should operate on.
-Therefore, our example uses the method `requiredPropertyWithUnaryMapping`, which will map a requirement to a real event property of an input stream and let the user choose the appropriate field in the StreamPipes UI when pipelines are defined.
-
-Finally, the `PropertyScope` indicates that the required property is a measurement value (in contrast to a dimension value). This allows us later to provide improved user guidance in the pipeline editor.
-
-Besides requirements, users should be able to define the center coordinate of the Geofence and the size of the fence defined as a radius around the center in meters.
-The radius can be defined by adding a simple required text field to the description:
-
-```java
-.requiredIntegerParameter("radius", "Geofence Size", "The size of the circular geofence in meters.", 0, 1000, 1)
-```
-
-Similar to mapping properties, text parameters have an internalId (radius), a label and a description.
-In addition, we can assign a _value specification_ to the parameter indicating the value range we support.
-Our example supports a radius value between 0 and 1000 with a granularity of 1.
-In the StreamPipes UI, a required text parameter is rendered as a text input field, in case we provide an optional value specification, a slider input is automatically generated.
-
-Such user-defined parameters are called _static properties_. There are many different types of static properties (see
- the [Processor SDK](06_extend-sdk-static-properties.md) for an overview).
-
-In this example, we'll further add two very simple input fields to let users provide latitude and longitude of the geofence center.
-
-Add the following line to the `declareModel` method:
-
-```java
- .requiredFloatParameter(Labels.from(LATITUDE_KEY, "Latitude", "The latitude value"))
- .requiredFloatParameter(Labels.from(LONGITUDE_KEY, "Longitude", "The longitude value"))
-
-```
-
-Now we need to define the output of our Geofencing pipeline element.
-As explained in the first section, the element should fire every time some geo-located entity arrives within the defined geofence.
-Therefore, the processor outputs the same schema as it receives as an input.
-Although we don't know the exact input right now as it depends on the stream users connect in StreamPipes when creating pipelines, we can define an _output strategy_ as follows:
-
-```java
-.outputStrategy(OutputStrategies.keep())
-```
-
-This defines a _KeepOutputStrategy_, i.e., the input event schema is not modified by the processor.
-There are many more output strategies you can define depending on the functionality you desire, e.g., _AppendOutput_ for defining a processor that enriches events or _CustomOutput_ in case you would like users to select the output by themselves.
-
-That's it! We've now defined input requirements, required user input and an output strategy.
-In the next section, you will learn how to extract these parameters once the pipeline element is invoked after a pipeline was created.
-
-## Pipeline element invocation
-
-Once users start a pipeline that uses our geofencing component, the _onInvocation_ method in our class is called. The class `ProcessorParams` includes convenient access to user-configured parameters a users has selected in the pipeline editor and information on the acutal streams that are connected to the pipeline element.
-
-Next, we are interested in the fields of the input event stream that contains the latitude and longitude value we would like to compute against the geofence center location as follows:
-
-```java
-String latitudeFieldName = parameters.extractor().mappingPropertyValue("latitude-field");
-String longitudeFieldName = parameters.extractor().mappingPropertyValue("longitude-field");
-```
-
-We use the same `internalId` we've used to define the mapping property requirements in the `declareModel` method.
-
-Next, for extracting the geofence center coordinates, add to class variables centerLatitude and centerLongitude and assign the selected values using the following statements:
-
-```java
-this.centerLatitude = parameters.extractor().singleValueParameter(LATITUDE_CENTER, Float.class);
-this.centerLongitude = parameters.extractor().singleValueParameter(LONGITUDE_CENTER, Float.class);
-```
-
-The radius value can be extracted as follows:
-
-```java
-int radius = parameters.extractor().singleValueParameter("radius", Float.class);
-```
-
-Great! That's all we need to describe a data processor for usage in StreamPipes. Your processor class should look as follows:
-
-```java
-package org.apache.streampipes.pe.example;
-
-import org.apache.streampipes.commons.exceptions.SpRuntimeException;
-import org.apache.streampipes.model.DataProcessorType;
-import org.apache.streampipes.model.graph.DataProcessorDescription;
-import org.apache.streampipes.model.runtime.Event;
-import org.apache.streampipes.model.schema.PropertyScope;
-import org.apache.streampipes.sdk.builder.ProcessingElementBuilder;
-import org.apache.streampipes.sdk.builder.StreamRequirementsBuilder;
-import org.apache.streampipes.sdk.helpers.EpRequirements;
-import org.apache.streampipes.sdk.helpers.Labels;
-import org.apache.streampipes.sdk.helpers.Locales;
-import org.apache.streampipes.sdk.helpers.OutputStrategies;
-import org.apache.streampipes.sdk.utils.Assets;
-import org.apache.streampipes.vocabulary.Geo;
-import org.apache.streampipes.wrapper.context.EventProcessorRuntimeContext;
-import org.apache.streampipes.wrapper.routing.SpOutputCollector;
-import org.apache.streampipes.wrapper.standalone.ProcessorParams;
-import org.apache.streampipes.wrapper.standalone.StreamPipesDataProcessor;
-
-public class GeofencingProcessor extends StreamPipesDataProcessor {
-
- private static final String LATITUDE_CENTER = "latitude-center";
- private static final String LONGITUDE_CENTER = "longitude-center";
-
- private float centerLatitude;
- private float centerLongitude;
- private String latitudeFieldName;
- private String longitudeFieldName;
-
- private int radius;
-
- @Override
- public DataProcessorDescription declareModel() {
- return ProcessingElementBuilder.create("org.streampipes.tutorial-geofencing")
- .category(DataProcessorType.ENRICH)
- .withAssets(Assets.DOCUMENTATION, Assets.ICON)
- .withLocales(Locales.EN)
- .requiredStream(StreamRequirementsBuilder
- .create()
- .requiredPropertyWithUnaryMapping(EpRequirements.domainPropertyReq(Geo.lat),
- Labels.from("latitude-field", "Latitude", "The event " +
- "property containing the latitude value"), PropertyScope.MEASUREMENT_PROPERTY)
- .requiredPropertyWithUnaryMapping(EpRequirements.domainPropertyReq(Geo.lng),
- Labels.from("longitude-field", "Longitude", "The event " +
- "property containing the longitude value"), PropertyScope.MEASUREMENT_PROPERTY)
- .build())
- .outputStrategy(OutputStrategies.keep())
- .requiredIntegerParameter("radius", "Geofence Size", "The size of the circular geofence in meters.", 0, 1000, 1)
- .requiredFloatParameter(Labels.from(LATITUDE_CENTER, "Latitude", "The latitude value"))
- .requiredFloatParameter(Labels.from(LONGITUDE_CENTER, "Longitude", "The longitude value"))
- .build();
- }
-
- @Override
- public void onInvocation(ProcessorParams parameters,
- SpOutputCollector spOutputCollector,
- EventProcessorRuntimeContext runtimeContext) throws SpRuntimeException {
- this.centerLatitude = parameters.extractor().singleValueParameter(LATITUDE_CENTER, Float.class);
- this.centerLongitude = parameters.extractor().singleValueParameter(LONGITUDE_CENTER, Float.class);
- this.latitudeFieldName = parameters.extractor().mappingPropertyValue("latitude-field");
- this.longitudeFieldName = parameters.extractor().mappingPropertyValue("longitude-field");
- this.radius = parameters.extractor().singleValueParameter("radius", Integer.class);
- }
-
- @Override
- public void onEvent(Event event, SpOutputCollector collector) throws SpRuntimeException {
-
- }
-
- @Override
- public void onDetach() throws SpRuntimeException {
-
- }
-}
-
-
-```
-
-## Adding an implementation
-
-Everything we need to do now is to add an implementation.
-
-Add the following piece of code to the onEvent method, which realizes the Geofencing functionality:
-
-```java
-
- @Override
- public void onEvent(Event event, SpOutputCollector collector) throws SpRuntimeException {
- float latitude = event.getFieldBySelector(latitudeFieldName).getAsPrimitive().getAsFloat();
- float longitude = event.getFieldBySelector(longitudeFieldName).getAsPrimitive().getAsFloat();
-
- float distance = distFrom(latitude, longitude, centerLatitude, centerLongitude);
-
- if (distance <= radius) {
- collector.collect(event);
- }
- }
-
- public static float distFrom(float lat1, float lng1, float lat2, float lng2) {
- double earthRadius = 6371000;
- double dLat = Math.toRadians(lat2-lat1);
- double dLng = Math.toRadians(lng2-lng1);
- double a = Math.sin(dLat/2) * Math.sin(dLat/2) +
- Math.cos(Math.toRadians(lat1)) * Math.cos(Math.toRadians(lat2)) *
- Math.sin(dLng/2) * Math.sin(dLng/2);
- double c = 2 * Math.atan2(Math.sqrt(a), Math.sqrt(1-a));
- return (float) (earthRadius * c);
- }
-
-}
-```
-
-We won't go into details here as this isn't StreamPipes-related code, but in general the class extracts latitude and longitude fields from the input event (which is provided as a map data type) and calculates the distance between the geofence center and these coordinates.
-If the distance is below the given radius, the event is forwarded to the next operator.
-
-See the [event model](06_extend-sdk-event-model.md) guide to learn how to extract parameters from events.
-
-## Registering the pipeline element
-The final step is to register the data processor in the `Init` method. Add the following line to the `SpServiceDefinitionBuilder`:
-
-```java
- .registerPipelineElement(new GeofencingProcessor())
-```
-
-## Starting the service
-Tip
-Besides the basic project skeleton, the sample project also includes an example Dockerfile you can use to package your application into a Docker container.
-
-
-
-
-Tip
-Once you start the service, it will register in StreamPipes with the hostname. The hostname will be auto-discovered and should work out-of-the-box. -In some cases, the detected hostname is not resolvable from within a container (where the core is running). In this case, provide a SP_HOST environment variable to override the auto-discovery. -
-
-
-
-Now we are ready to start our service!
-
-Configure your IDE to provide an environment variable called ``SP_DEBUG`` with value ``true`` when starting the project.
-
-Execute the main method in the class `Init` we've just created.
-
-The service automatically registers itself in StreamPipes.
-To install the just created element, open the StreamPipes UI and follow the manual provided in the [user guide](03_use-install-pipeline-elements.md).
-
-## Read more
-
-Congratulations! You've just created your first data processor for StreamPipes.
-There are many more things to explore and data processors can be defined in much more detail using multiple wrappers.
-Follow our [SDK guide](06_extend-sdk-static-properties.md) to see what's possible!
diff --git a/website-v2/versioned_docs/version-0.70.0/06_extend-tutorial-data-sinks.md b/website-v2/versioned_docs/version-0.70.0/06_extend-tutorial-data-sinks.md
deleted file mode 100644
index a2bf82c6f..000000000
--- a/website-v2/versioned_docs/version-0.70.0/06_extend-tutorial-data-sinks.md
+++ /dev/null
@@ -1,231 +0,0 @@
----
-id: extend-tutorial-data-sinks
-title: "Tutorial: Data Sinks"
-sidebar_label: "Tutorial: Data Sinks"
-original_id: extend-tutorial-data-sinks
----
-
-In this tutorial, we will add a new data sink using the standalone wrapper.
-
-From an architectural point of view, we will create a self-contained service that includes the description of the data sink and a corresponding implementation.
-
-## Objective
-
-We are going to create a new data sink that calls an external HTTP endpoint to forward data to an external service.
-
-For each incoming event, an external service is invoked using an HTTP POST request. In this example, we'll call an endpoint provided by [RequestBin](https://requestbin.com/).
-To setup your own endpoint, go to [https://requestbin.com/](https://requestbin.com/) and click "Create a request bin". Copy the URL of the newly created endpoint.
-
-
-## Project setup
-
-Instead of creating a new project from scratch, we recommend to use the Maven archetype to create a new project skeleton (streampipes-archetype-extensions-jvm).
-Enter the following command in a command line of your choice (Apache Maven needs to be installed):
-
-```
-mvn archetype:generate -DarchetypeGroupId=org.apache.streampipes \
--DarchetypeArtifactId=streampipes-archetype-extensions-jvm -DarchetypeVersion=0.70.0 \
--DgroupId=org.streampipes.tutorial -DartifactId=sink-tutorial -DclassNamePrefix=Rest -DpackageName=mypackage
-```
-
-You will see a project structure similar to the structure shown in the [archetypes](06_extend-archetypes.md) section.
-
-Tip
-The default port of all pipeline element services as defined in the `create` method is port 8090. - If you'd like to run multiple services at the same time on your development machine, change the port here. As an alternative, you can also provide an env variable `SP_PORT` which overrides the port settings. This is useful to use different configs for dev and prod environments. -
-
-
-
-Now you're ready to create your first data sink for StreamPipes!
-
-## Adding data sink requirements
-
-First, we will add a new stream requirement.
-Create a class `RestSink` which should look as follows:
-
-```java
-package org.apache.streampipes.pe.example;
-
-import org.apache.streampipes.commons.exceptions.SpRuntimeException;
-import org.apache.streampipes.model.DataSinkType;
-import org.apache.streampipes.model.graph.DataSinkDescription;
-import org.apache.streampipes.model.runtime.Event;
-import org.apache.streampipes.model.schema.PropertyScope;
-import org.apache.streampipes.sdk.builder.DataSinkBuilder;
-import org.apache.streampipes.sdk.builder.StreamRequirementsBuilder;
-import org.apache.streampipes.sdk.helpers.EpRequirements;
-import org.apache.streampipes.sdk.helpers.Labels;
-import org.apache.streampipes.sdk.helpers.Locales;
-import org.apache.streampipes.sdk.utils.Assets;
-import org.apache.streampipes.wrapper.context.EventSinkRuntimeContext;
-import org.apache.streampipes.wrapper.standalone.SinkParams;
-import org.apache.streampipes.wrapper.standalone.StreamPipesDataSink;
-
-public class RestSink extends StreamPipesDataSink {
-
- @Override
- public DataSinkDescription declareModel() {
- return DataSinkBuilder.create("org.apache.streampipes.tutorial.pe.sink.rest")
- .category(DataSinkType.NOTIFICATION)
- .withAssets(Assets.DOCUMENTATION, Assets.ICON)
- .withLocales(Locales.EN)
- .requiredStream(StreamRequirementsBuilder
- .create()
- .requiredPropertyWithNaryMapping(EpRequirements.anyProperty(), Labels.withId(
- "fields-to-send"), PropertyScope.NONE)
- .build())
- .build();
- }
-
- @Override
- public void onInvocation(SinkParams parameters, EventSinkRuntimeContext runtimeContext) throws SpRuntimeException {
-
- }
-
- @Override
- public void onEvent(Event event) throws SpRuntimeException {
-
- }
-
- @Override
- public void onDetach() throws SpRuntimeException {
-
- }
-}
-
-
-```
-
-In this class, we need to implement three methods: The `declareModel` method is used to define abstract stream requirements such as event properties that must be present in any input stream that is later connected to the element using the StreamPipes UI.
-The second method, `onInvocation` is called once a pipeline using this sink is started. The third method, `onEvent`, is called for every incoming event.
-
-The ``declareModel`` method describes the properties of our data sink:
-* ``category`` defines a category for this sink.
-* ``withAssets`` denotes that we will provide an external documentation file and an icon, which can be found in the ``resources`` folder
-* ``withLocales`` defines that we will provide an external language file, also available in the ``resources`` folder
-* ``requiredStream`` defines requirements any input stream connected to this sink must provide. In this case, we do not have any specific requirements, we just forward all incoming events to the REST sink. However, we want to let the user display a list of available fields from the connected input event, where users can select a subset. This is defined by defining a Mapping from the empty requirement. This will later on render a selection dialog in the pipeline editor.
-
-The ``onInvocation`` method is called when a pipeline containing the sink is started. Once a pipeline is started, we would like to extract user-defined parameters.
-In this example, we simply extract the fields selected by users that should be forwarded to the REST sink. Finally, we return a new configured event sink containing the parameters.
-
-## Pipeline element invocation
-
-Once users start a pipeline that uses our geofencing component, the _onInvocation_ method in our class is called. The class `SinkParams` includes a graph that contains information on the configuration parameters a users has selected in the pipeline editor and information on the acutal streams that are connected to the pipeline element.
-
-
-## Adding an implementation
-
-Now we'll add a proper implementation (i.e., the Rest call executed for every incoming event) to the following methods:
-
-Our final class should look as follows:
-
-```java
-package org.apache.streampipes.pe.example;
-
-import com.google.common.base.Charsets;
-import org.apache.http.client.fluent.Request;
-import org.apache.http.entity.StringEntity;
-import org.apache.streampipes.commons.exceptions.SpRuntimeException;
-import org.apache.streampipes.dataformat.SpDataFormatDefinition;
-import org.apache.streampipes.dataformat.json.JsonDataFormatDefinition;
-import org.apache.streampipes.model.DataSinkType;
-import org.apache.streampipes.model.graph.DataSinkDescription;
-import org.apache.streampipes.model.runtime.Event;
-import org.apache.streampipes.model.schema.PropertyScope;
-import org.apache.streampipes.sdk.builder.DataSinkBuilder;
-import org.apache.streampipes.sdk.builder.StreamRequirementsBuilder;
-import org.apache.streampipes.sdk.helpers.EpRequirements;
-import org.apache.streampipes.sdk.helpers.Labels;
-import org.apache.streampipes.sdk.helpers.Locales;
-import org.apache.streampipes.sdk.utils.Assets;
-import org.apache.streampipes.wrapper.context.EventSinkRuntimeContext;
-import org.apache.streampipes.wrapper.standalone.SinkParams;
-import org.apache.streampipes.wrapper.standalone.StreamPipesDataSink;
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
-
-import java.io.IOException;
-import java.util.List;
-import java.util.Map;
-
-public class RestSink extends StreamPipesDataSink {
-
- private static final Logger LOG = LoggerFactory.getLogger(RestSink.class);
-
- private static final String REST_ENDPOINT_URI = YOUR_REQUEST_BIN_URL;
- private ListTip
-Besides the basic project skeleton, the sample project also includes an example Dockerfile you can use to package your application into a Docker container.
-
-
-
-
-Tip
-Once you start the service, it will register in StreamPipes with the hostname. The hostname will be auto-discovered and should work out-of-the-box. -In some cases, the detected hostname is not resolvable from within a container (where the core is running). In this case, provide a SP_HOST environment variable to override the auto-discovery. -
-
-
-
-Now we are ready to start our service!
-
-Configure your IDE to provide an environment variable called ``SP_DEBUG`` with value ``true`` when starting the project.
-
-Execute the main method in the class `Init` we've just created. The service automatically registers itself in StreamPipes.
-
-To install the created element, open the StreamPipes UI and follow the manual provided in the [user guide](03_use-install-pipeline-elements.md).
-
-## Read more
-
-Congratulations! You've just created your first data sink for StreamPipes.
-There are many more things to explore and data sinks can be defined in much more detail using multiple wrappers.
-Follow our [SDK guide](../dev-guide-sdk-guide-sinks) to see what's possible!
diff --git a/website-v2/versioned_docs/version-0.70.0/06_extend-tutorial-data-sources.md b/website-v2/versioned_docs/version-0.70.0/06_extend-tutorial-data-sources.md
deleted file mode 100644
index a2d95f094..000000000
--- a/website-v2/versioned_docs/version-0.70.0/06_extend-tutorial-data-sources.md
+++ /dev/null
@@ -1,214 +0,0 @@
----
-id: extend-tutorial-data-sources
-title: "Tutorial: Data Sources"
-sidebar_label: "Tutorial: Data Sources"
-original_id: extend-tutorial-data-sources
----
-
-In this tutorial, we will add a new data source consisting of a single data stream. The source will be provided as a standalone component (i.e., the description will be accessible through an integrated web server).
-
-## Objective
-
-We are going to create a new data stream that is produced by a GPS sensor installed in a delivery vehicle.
-The sensor produces a continuous stream of events that contain the current timestamp, the current lat/lng position of the vehicle and the plate number of the vehicle.
-Events are published in a JSON format as follows:
-```json
-{
- "timestamp" : 145838399,
- "latitude" : 37.04,
- "longitude" : 17.04,
- "plateNumber" : "KA-AB 123"
-}
-```
-
-These events are published to a Kafka broker using the topic `org.streampipes.tutorial.vehicle`.
-
-In the following section, we show how to describe this stream in a form that allows you to import and use it in StreamPipes.
-
-## Project setup
-
-Instead of creating a new project from scratch, we recommend to use the Maven archetype to create a new project skeleton (streampipes-archetype-extensions-jvm).
-Enter the following command in a command line of your choice (Apache Maven needs to be installed):
-
-```
-mvn archetype:generate \
--DarchetypeGroupId=org.apache.streampipes -DarchetypeArtifactId=streampipes-archetype-extensions-jvm \
--DarchetypeVersion=0.69.0 -DgroupId=my.groupId \
--DartifactId=my-source -DclassNamePrefix=MySource -DpackageName=mypackagename
-```
-
-You will see a project structure similar to the structure shown in the [archetypes](06_extend-archetypes.md) section.
-
-Tip
-The default port of all pipeline element services as defined in the `create` method is port 8090. - If you'd like to run multiple services at the same time on your development machine, change the port here. As an alternative, you can also provide an env variable `SP_PORT` which overrides the port settings. This is useful to use different configs for dev and prod environments. -
-
-
-
-## Adding a data stream description
-
-Now we will add a new data stream definition.
-First, create a new class `MyVehicleStream` which should look as follows:
-
-```java
-
-package org.apache.streampipes.pe.example;
-
-import org.apache.streampipes.model.SpDataStream;
-import org.apache.streampipes.sources.AbstractAdapterIncludedStream;
-
-public class MyVehicleStream extends AbstractAdapterIncludedStream {
-
- @Override
- public SpDataStream declareModel() {
- return null;
- }
-
- @Override
- public void executeStream() {
-
- }
-}
-```
-
-This class extends the class ``AbstractAdapterIncludedStream``, which indicates that this source continuously produces data (configured in the ``executeStream()`` method.
-In contrast, the class `AbstractAlreadyExistingStream` indicates that we only want to describe an already existing stream (e.g., a stream that already sends data to an existing Kafka broker).
-
-Next, we will add the definition of the data stream. Add the following code inside of the `declareModel` method:
-```java
-return DataStreamBuilder.create("org.streampipes.tutorial.vehicle.position", "Vehicle Position", "An event stream " +
- "that produces current vehicle positions")
-```
-
-This line creates a new instance of the SDK's `DataStreamBuilder` by providing three basic parameters:
-The first parameter must be a unique identifier of your data stream.
-The second and third parameters indicate a label and a description of your stream.
-These values will later be used in the StreamPipes UI to display stream details in a human-readable manner.
-
-Next, we will add the properties as stated above to the stream definition by adding the following lines:
-```java
-.property(EpProperties.timestampProperty("timestamp"))
-.property(EpProperties.stringEp(Labels.from("plate-number", "Plate Number", "Denotes the plate number of the vehicle"), "plateNumber", "http://my.company/plateNumber"))
-.property(EpProperties.doubleEp(Labels.from("latitude", "Latitude", "Denotes the latitude value of the vehicle's position"), "latitude", Geo.lat))
-.property(EpProperties.doubleEp(Labels.from("longitude", "Longitude", "Denotes the longitude value of the vehicle's position"), "longitude", Geo.lng))
-```
-These four _event properties_ compose our _event schema_. An event property must, at least, provide the following attributes:
-
-* **Runtime Name**. The runtime name indicates the key of the property at runtime, e.g., if our JSON message contains a structure such as `{"plateNumber" : "KA-F 123"}`, the runtime name must be `plateNumber`.
-* **Runtime Type**. An event property must have a primitive type (we will later see how to model more complex properties such as lists and nested properties).
- The type must be an instance of `XMLSchema` primitives, however, the SDK provides convenience methods to provide the property type.
-* **Domain Property**. The domain property indicates the semantics of the event property. For instance, the `latitude` property is linked to the `http://www.w3.org/2003/01/geo/wgs84_pos#lat` property of the WGS84 vocabulary.
- The domain property should be an URI as part of an existing or domain-specific vocabulary. The SDK provides convenience methods for popuplar vocabularies (e.g., Schema.org, Dolce or WGS84).
-
-In order to complete the minimum required specification of an event stream, we need to provide information on the transport format and protocol of the data stream at runtime.
-
-This can be achieved by extending the builder with the respective properties:
-```java
-.format(Formats.jsonFormat())
-.protocol(Protocols.kafka("localhost", 9094, "TOPIC_SHOULD_BE_CHANGED"))
-.build();
-```
-
-Set ``org.streampipes.tutorial.vehicle`` as your new topic by replacing the term ``TOPIC_SHOULD_BE_CHANGED`.
-
-In this example, we defined that the data stream consists of events in a JSON format and that Kafka is used as a message broker to transmit events.
-The last build() method call triggers the construction of the data stream definition.
-
-That's it! In the next section, we will connect the data stream to a source and inspect the generated RDF description.
-
-## Creating some dummy data
-
-Let's assume our stream should produce some random values that are sent to StreamPipes. We'll add a very simple data simulator to the ``executeStream`` method as follows:
-
-```java
-@Override
- public void executeStream() {
-
- SpKafkaProducer producer = new SpKafkaProducer("localhost:9094", "my-topic", Collections.emptyList());
- Random random = new Random();
- Runnable runnable = () -> {
- for (;;) {
- JsonObject jsonObject = new JsonObject();
- jsonObject.addProperty("timestamp", System.currentTimeMillis());
- jsonObject.addProperty("plateNumber", "KA-FZ 1");
- jsonObject.addProperty("latitude", random.nextDouble());
- jsonObject.addProperty("longitude", random.nextDouble());
-
- producer.publish(jsonObject.toString());
-
- try {
- TimeUnit.SECONDS.sleep(1);
- } catch (InterruptedException e) {
- e.printStackTrace();
- }
-
- }
- };
-
- new Thread(runnable).start();
- }
-```
-
-Change the topic and the URL of your Kafka broker (as stated in the controller).
-
-## Registering the data stream
-
-You need to register the stream in the service definition. Open the ``Init`` class and register the ``MyVehicleStream``:
-
-```java
-
- @Override
- public SpServiceDefinition provideServiceDefinition() {
- return SpServiceDefinitionBuilder.create("org.apache.streampipes",
- "human-readable service name",
- "human-readable service description", 8090)
- .registerPipelineElement(new ExampleDataProcessor())
- .registerPipelineElement(new ExampleDataSink())
- .registerPipelineElement(new MyVehicleStream())
- .registerMessagingFormats(
- new JsonDataFormatFactory(),
- new CborDataFormatFactory(),
- new SmileDataFormatFactory(),
- new FstDataFormatFactory())
- .registerMessagingProtocols(
- new SpKafkaProtocolFactory(),
- new SpJmsProtocolFactory(),
- new SpMqttProtocolFactory())
- .build();
- }
-
-```
-
-You can remove the other two example classes if you want.
-
-## Starting the service
-
-Tip
-Besides the basic project skeleton, the sample project also includes an example Dockerfile you can use to package your application into a Docker container. -
-
-
-
-Now we are ready to start our first container!
-
-Execute the main method in the class `Init`, open a web browser and navigate to http://localhost:8090, or change the port according to the value of the ``SP_PORT`` variable in the env file.
-
-Configure your IDE to provide an environment variable called ``SP_DEBUG`` with value ``true`` when starting the project.
-
-You should see something as follows:
-
-Tip
-Once you start the service, it will register in StreamPipes with the hostname. The hostname will be auto-discovered and should work out-of-the-box. -In some cases, the detected hostname is not resolvable from within a container (where the core is running). In this case, provide a SP_HOST environment variable to override the auto-discovery. -
-
-
-Click on the link of the data source to see the generated description of the pipeline element.
-
-
-
-The container automatically registers itself in StreamPipes.
-
-To install the just created element, open the StreamPipes UI and install the source over the ``Install Pipeline Elements`` section.
-
-## Read more
-
-Congratulations! You've just created your first pipeline element for StreamPipes.
-There are many more things to explore and data sources can be defined in much more detail.
diff --git a/website-v2/versioned_docs/version-0.70.0/07_technicals-architecture.md b/website-v2/versioned_docs/version-0.70.0/07_technicals-architecture.md
deleted file mode 100644
index 4ef1a54f4..000000000
--- a/website-v2/versioned_docs/version-0.70.0/07_technicals-architecture.md
+++ /dev/null
@@ -1,63 +0,0 @@
----
-id: technicals-architecture
-title: Architecture
-sidebar_label: Architecture
-original_id: technicals-architecture
----
-
-
-The following picture illustrates the high-level architecture of StreamPipes:
-
-
-
-Users mainly interact (besides other UI components) with the _Pipeline Editor_ to create stream processing pipelines based on data streams, data processors and data sinks.
-These reusable pipeline elements are provided by self-contained _pipeline element containers_, each of them having a semantic description that specifies their characteristics (e.g., input, output and required user input for data processors).
-Each pipeline element container has a REST endpoint that provides these characteristics as a JSON-LD document.
-
-Pipeline element containers are built using one of several provided _wrappers_.
-Wrappers abstract from the underlying runtime stream processing framework.
-Currently, the StreamPipes framework provides wrappers for Apache Flink, Esper and algorithms running directly on the JVM.
-
-The _pipeline manager_ manages the definition and execution of pipelines.
-When creating pipelines, the manager continuously matches the pipeline against its semantic description and provides user guidance in form of recommendations.
-Once a pipeline is started, the pipeline manager invokes the corresponding pipeline element containers.
-The container prepares the actual execution logic and submits the program to the underlying execution engine, e.g., the program is deployed in the Apache Flink cluster.
-
-Pipeline elements exchange data using one or more message brokers and protocols (e.g., Kafka or MQTT).
-StreamPipes does not rely on a specific broker or message format, but negotiates suitable brokers based on the capabilities of connected pipeline elements.
-
-Thus, StreamPipes provides a higher-level abstraction of existing stream processing technology by leveraging domain experts to create streaming analytics pipelines in a self-service manner.
-
-## Semantic description
-Pipeline elements in StreamPipes are meant to be reusable:
-
-* Data processors and data sink are generic (or domain-specific) elements that express their requirements and are able to operate on any stream that satisfies these requirements.
-* Data processors and data sinks can be manually configured by offering possible configuration parameters which users can individually define when creating pipelines.
-* Data streams can be connected to any data processor or data sink that matches the capabilities of the stream.
-
-When users create pipelines by connecting a data stream with a data processor (or further processors), the pipeline manager _matches_ the input stream of a data processor against its requirements.
-This matching is performed based on the _semantic description of each element.
-The semantic description (technically an RDF graph serialized as JSON-LD) can be best understood by seeing it as an envelope around a pipeline element.
-It only provides metadata information, while we don't rely on any RDF at runtime for exchanging events between pipeline elements.
-While RDF-based metadata ensures good understanding of stream capabilities, lightweight event formats at runtime (such as JSON or Thrift) ensure fast processing of events.
-
-Let's look at an example stream that produces a continuous stream of vehicle positions as illustrated below:
-
-
-
-While the runtime layer produces plain JSON by submitting actual values of the position and the vehicle's plate number, the description layer describes various characteristics of the stream:
-For instance, it defines the event schema (including, besides the data type and the runtime name of each property also a more fine-grained meaning of the property), quality aspects (e.g., the measurement unit of a property or the frequency) and the grounding (e.g., the format used at runtime and the communication protocol used for transmitting events).
-
-The same accounts for data processors and data sinks:
-
-
-
-Data processors (and, with some differences, data sinks) are annotated by providing metadata information on their required input and output.
-For instance, we can define minimum schema requirements (such as geospatial coordinates that need to be provided by any stream that is connected to a processor), but also required (minimum or maximum) quality levels and supported transport protocols and formats.
-In addition, required configuration parameters users can define during the pipeline definition process are provided by the semantic description.
-
-Once new pipeline elements are imported into StreamPipes, we store all information provided by the description layer in a central repository and use this information to guide useres through the pipeline definition process.
-
-Don't worry - you will never be required to model RDF by yourself.
-Our SDK provides convenience methods that help creating the description automatically.
-
diff --git a/website-v2/versioned_docs/version-0.70.0/07_technicals-configuration.md b/website-v2/versioned_docs/version-0.70.0/07_technicals-configuration.md
deleted file mode 100644
index 459909ac6..000000000
--- a/website-v2/versioned_docs/version-0.70.0/07_technicals-configuration.md
+++ /dev/null
@@ -1,59 +0,0 @@
----
-id: technicals-configuration
-title: Configuration
-sidebar_label: Configuration
-original_id: technicals-configuration
----
-
-On this page we explain how the StreamPipes configuration works.
-StreamPipes allows the individual services (pipeline element containers and third-party services) to store configuration parameters in a distributed key-value store.
-This has the advantage that individual services do not need to store any configurations on the local file system, enabling us to run containers anywhere.
-As a key-value store we use [Consul](https://www.consul.io/), which is an essential service for all our services.
-
-
-
-
-## Edit Configurations
-All services in StreamPipes can have configuration parameters.
-You can either change them in the consul user interface (which is by default running on port 8500) or directly in the StreamPipes Configurations Page.
-Once a new pipeline element container is started, it is registered in Consul and the parameters can be edited in the configuration page, as shown below.
-To store changes in Consul, the update button must be clicked.
-
-
- 
-
-
-## Configuration for Developers
-We provide a Configurations API for the use of configuration parameters in your services.
-Each processing element project has a “config” package [[Example]](https://github.com/apache/streampipes-extensions/tree/dev/streampipes-sinks-internal-jvm/src/main/java/org/streampipes/sinks/internal/jvm/config).
-This package usually contains two classes.
-One containing unique keys for the configuration values and one containing the getter and setter methods to access these values.
-For the naming of configuration keys, we recommend to use “SP” as a prefix.
-As we explain later, it is possible to set default configurations as environment variables, this prefix makes them unique on your server.
-A configuration entry needs a unique config key. For this key, a value can be specified containing the configuration, like for example the port number of the service.
-For each configuration, a description explaining the parameter can be provided, further the data type must be specified and whether it is a password or not.
-Below, the schema of a configuration item is shown on the left and an example of a port configuration on the right.
-
-
-
-
-As a developer, you can add as many new configurations to services as you wish, but there are some that are required for all processing element containers.
-Those are **the host**, **the port**, and **the name** of the service.
-
-## Default Values
-You can provide default values for the configurations, which are used when a configuration is read for the first time.
-The first option is to register a configuration parameter in the Config class.
-This is a fallback value, which is used if nothing else is defined.
-Since this value is static, we offer a second option.
-It is possible to provide a default value by setting an environment variable.
-In this case, the convention is that the key of a configuration parameter must be used as the environment variable.
-Now, this value is used instead of the value defined in the Config class.
-During development, the configuration values often need to be changed for debugging purposes, therefore we provide an .env file in all processing element projects and archetypes.
-This file can be used by your IDE to set the environment variables. (e.g., [Intellij Plugin](https://plugins.jetbrains.com/plugin/7861-envfile))
-When you need to change the variable at runtime, you can do this in the StreamPipes configurations as explained before.
-Those changes take effect immediately without the need of a container restart.
-
-
-
diff --git a/website-v2/versioned_docs/version-0.70.0/07_technicals-messaging.md b/website-v2/versioned_docs/version-0.70.0/07_technicals-messaging.md
deleted file mode 100644
index 64d9a2ef8..000000000
--- a/website-v2/versioned_docs/version-0.70.0/07_technicals-messaging.md
+++ /dev/null
@@ -1,8 +0,0 @@
----
-id: technicals-messaging
-title: Messaging
-sidebar_label: Messaging
-original_id: technicals-messaging
----
-
-tbd
\ No newline at end of file
diff --git a/website-v2/versioned_docs/version-0.70.0/07_technicals-runtime-wrappers.md b/website-v2/versioned_docs/version-0.70.0/07_technicals-runtime-wrappers.md
deleted file mode 100644
index dedc3ee18..000000000
--- a/website-v2/versioned_docs/version-0.70.0/07_technicals-runtime-wrappers.md
+++ /dev/null
@@ -1,8 +0,0 @@
----
-id: technicals-runtime-wrappers
-title: Runtime Wrappers
-sidebar_label: Runtime Wrappers
-original_id: technicals-runtime-wrappers
----
-
-tbd
\ No newline at end of file
diff --git a/website-v2/versioned_docs/version-0.70.0/dev-guide-archetype.md b/website-v2/versioned_docs/version-0.70.0/dev-guide-archetype.md
deleted file mode 100644
index 6b2486911..000000000
--- a/website-v2/versioned_docs/version-0.70.0/dev-guide-archetype.md
+++ /dev/null
@@ -1,7 +0,0 @@
----
-id: dev-guide-archetype
-title: Start Developing
-sidebar_label: Start Developing
-original_id: dev-guide-archetype
----
-
diff --git a/website-v2/versioned_docs/version-0.70.0/dev-guide-processor-sdk.md b/website-v2/versioned_docs/version-0.70.0/dev-guide-processor-sdk.md
deleted file mode 100644
index 5ceca4bd7..000000000
--- a/website-v2/versioned_docs/version-0.70.0/dev-guide-processor-sdk.md
+++ /dev/null
@@ -1,12 +0,0 @@
----
-id: dev-guide-sdk-guide-processors
-title: "SDK Guide: Data Processors"
-sidebar_label: "SDK Guide: Data Processors"
-original_id: dev-guide-sdk-guide-processors
----
-
-## Project Setup
-(coming soon, please check the [tutorial](../dev-guide-tutorial-processors) to learn how to define data processors)
-
-## SDK reference
-The complete SDK reference for defining data processors will follow soon. Please check the SDK's Javadoc for now!
diff --git a/website-v2/versioned_docs/version-0.70.0/dev-guide-sink-sdk.md b/website-v2/versioned_docs/version-0.70.0/dev-guide-sink-sdk.md
deleted file mode 100644
index d2e253441..000000000
--- a/website-v2/versioned_docs/version-0.70.0/dev-guide-sink-sdk.md
+++ /dev/null
@@ -1,12 +0,0 @@
----
-id: dev-guide-sdk-guide-sinks
-title: "SDK Guide: Data Sinks"
-sidebar_label: "SDK Guide: Data Sinks"
-original_id: dev-guide-sdk-guide-sinks
----
-
-## Project Setup
-(coming soon, please check the [tutorial](../dev-guide-tutorial-processors) to learn how to define sinks)
-
-## SDK reference
-The complete SDK reference for defining data sinks will follow soon. Please check the SDK's Javadoc for now!
\ No newline at end of file
diff --git a/website-v2/versioned_docs/version-0.70.0/pe/org.apache.streampipes.connect.adapters.coindesk.md b/website-v2/versioned_docs/version-0.70.0/pe/org.apache.streampipes.connect.adapters.coindesk.md
deleted file mode 100644
index 7d5e9b8e8..000000000
--- a/website-v2/versioned_docs/version-0.70.0/pe/org.apache.streampipes.connect.adapters.coindesk.md
+++ /dev/null
@@ -1,45 +0,0 @@
----
-id: org.apache.streampipes.connect.adapters.coindesk
-title: Coindesk Bitcoin Price
-sidebar_label: Coindesk Bitcoin Price
-original_id: org.apache.streampipes.connect.adapters.coindesk
----
-
-
-
-
-
-Installed pipeline elements
-Be cautious, when the configuration is used in the semantic description of a processing element which is already installed in StreamPipes, you have to reload this element in StreamPipes (my elements -> reload). - In addition, changes might affect already running pipelines.
-
- 
-
- 
-
- 
-
- 
-
- 
-
- 
-
- 
-
- 
-
- 
-
- 
-
- 
-
- 
-
- 
-
- 
-
- 
-
- 
-
- 
-
- 
-
- 
-
- 
-
- 
-
- 
-
- 
-
- 
-
- 
-
- 
-
- 
-
- 
-
- 
-
- 
-
- 
-
-
-
-Install Docker
-Go to https://docs.docker.com/installation/ and follow the instructions to install Docker for your OS. Make sure docker can be started as a non-root user (described in the installation manual, don’t forget to log out and in again) and check that Docker is installed correctly by executing docker-run hello-world
-
-
+:::info Install Docker
+Go to https://docs.docker.com/installation/ and follow the instructions to install Docker for your OS. Make sure
+docker can be started as a non-root user (described in the installation manual, don’t forget to log out and in
+again) and check that Docker is installed correctly by executing docker-run hello-world
+:::
### Supported operating systems
-The Docker-based installation supports the operating systems **Linux**, **Mac OS X** and **Windows 10**. Older windows
+The Docker-based installation supports the operating systems **Linux**, **Mac OS X** and **Windows 10 upwards**. Older windows
versions are not fully compatible with Docker. Linux VMs running under Windows might cause network problems with Docker,
therefore some manual work might be needed to make StreamPipes run properly.
@@ -41,7 +33,7 @@ best experience), Firefox or Edge.
## Install StreamPipes
-Configure Docker
-By default, Docker uses only a limited number of CPU cores and memory. - If you run StreamPipes on Windows or on a Mac you need to adjust the default settings. - To do that, click on the Docker icon in your tab bar and open the preferences. - Go to the advanced preferences and set the **number of CPUs to 6** (recommended) and the **Memory to 4GB**. - After changing the settings, Docker needs to be restarted.
- 
-
-
+:::danger Errors during the installation process
+In most cases, errors during the installation are due to an under-powered system.
+If there is a problem with any of the components, please restart the whole system (`docker-compose
+down` and eventually also delete the volumes).
+Please also make sure that you've assigned enough memory available to Docker.
+:::
## Next Steps
-That's it! To ease your first steps with StreamPipes, we've created an [interactive tutorial](try-tutorial).
+That's it! Have a look at the usage guide to learn how to use Apache StreamPipes.
diff --git a/website-v2/versioned_docs/version-0.70.0/01_try-overview.md b/website-v2/versioned_docs/version-0.95.1/01_try-overview.md
similarity index 89%
rename from website-v2/versioned_docs/version-0.70.0/01_try-overview.md
rename to website-v2/versioned_docs/version-0.95.1/01_try-overview.md
index 48be14900..29a059d4a 100644
--- a/website-v2/versioned_docs/version-0.70.0/01_try-overview.md
+++ b/website-v2/versioned_docs/version-0.95.1/01_try-overview.md
@@ -2,12 +2,13 @@
id: user-guide-introduction
title: Apache StreamPipes Documentation
sidebar_label: Overview
-original_id: user-guide-introduction
---
This is the documentation of Apache StreamPipes.
-Errors during the installation process
-In most cases, errors during the installation are due to an under-powered system.
-If there is a problem with any of the components, please restart the whole system (docker-compose down and eventually also delete the volumes).
- Please also make sure that your system meets the hardware requirements as mentioned in the first section of the installation guide.
+
+
@@ -85,7 +86,7 @@ This is the documentation of Apache StreamPipes.
Tutorial Data Sources 🔗,
Tutorial Data Processors 🔗,
Tutorial Data Sinks 🔗,
- Event Model 🔗,
+ Event Model 🔗,
Stream Requirements 🔗,
Static Properties 🔗,
Output Strategies 🔗
@@ -119,4 +120,15 @@ This is the documentation of Apache StreamPipes.
+
diff --git a/website-v2/versioned_docs/version-0.70.0/01_try-tutorial.md b/website-v2/versioned_docs/version-0.95.1/01_try-tutorial.md
similarity index 97%
rename from website-v2/versioned_docs/version-0.70.0/01_try-tutorial.md
rename to website-v2/versioned_docs/version-0.95.1/01_try-tutorial.md
index fb7f86174..c13d6f1dc 100644
--- a/website-v2/versioned_docs/version-0.70.0/01_try-tutorial.md
+++ b/website-v2/versioned_docs/version-0.95.1/01_try-tutorial.md
@@ -2,7 +2,6 @@
id: try-tutorial
title: Interactive Tutorial
sidebar_label: Interactive Tutorial
-original_id: try-tutorial
---
Once you've installed StreamPipes and see the home screen, you'll see a number of modules that are part of the StreamPipes toolbox.
diff --git a/website-v2/versioned_docs/version-0.70.0/02_concepts-adapter.md b/website-v2/versioned_docs/version-0.95.1/02_concepts-adapter.md
similarity index 73%
rename from website-v2/versioned_docs/version-0.70.0/02_concepts-adapter.md
rename to website-v2/versioned_docs/version-0.95.1/02_concepts-adapter.md
index 584f5a543..a94c38268 100644
--- a/website-v2/versioned_docs/version-0.70.0/02_concepts-adapter.md
+++ b/website-v2/versioned_docs/version-0.95.1/02_concepts-adapter.md
@@ -2,7 +2,6 @@
id: concepts-adapter
title: Data Adapters
sidebar_label: Data Adapters
-original_id: concepts-adapter
---
tbd
diff --git a/website-v2/versioned_docs/version-0.70.0/02_concepts-data-streams.md b/website-v2/versioned_docs/version-0.95.1/02_concepts-data-streams.md
similarity index 71%
rename from website-v2/versioned_docs/version-0.70.0/02_concepts-data-streams.md
rename to website-v2/versioned_docs/version-0.95.1/02_concepts-data-streams.md
index a8f25015d..329f9b908 100644
--- a/website-v2/versioned_docs/version-0.70.0/02_concepts-data-streams.md
+++ b/website-v2/versioned_docs/version-0.95.1/02_concepts-data-streams.md
@@ -2,7 +2,6 @@
id: concepts-data-streams
title: Data Streams
sidebar_label: Data Streams
-original_id: concepts-data-streams
---
tbd
\ No newline at end of file
diff --git a/website-v2/versioned_docs/version-0.70.0/02_concepts-glossary.md b/website-v2/versioned_docs/version-0.95.1/02_concepts-glossary.md
similarity index 70%
rename from website-v2/versioned_docs/version-0.70.0/02_concepts-glossary.md
rename to website-v2/versioned_docs/version-0.95.1/02_concepts-glossary.md
index 68a33967c..b401d1829 100644
--- a/website-v2/versioned_docs/version-0.70.0/02_concepts-glossary.md
+++ b/website-v2/versioned_docs/version-0.95.1/02_concepts-glossary.md
@@ -2,7 +2,6 @@
id: concepts-glossary
title: Glossary
sidebar_label: Glossary
-original_id: concepts-glossary
---
tbd
\ No newline at end of file
diff --git a/website-v2/versioned_docs/version-0.95.1/02_concepts-overview.md b/website-v2/versioned_docs/version-0.95.1/02_concepts-overview.md
new file mode 100644
index 000000000..f26f3cc07
--- /dev/null
+++ b/website-v2/versioned_docs/version-0.95.1/02_concepts-overview.md
@@ -0,0 +1,46 @@
+---
+id: concepts-overview
+title: StreamPipes Concepts
+sidebar_label: Overview
+---
+
+To understand how StreamPipes works, it is helpful to understand a few core concepts, which are illustrated below.
+These encompass the entire data journey within StreamPipes: Starting with data collection ([adapters](#adapter)),
+through data exchange ([data streams](#data-stream)) and data processing ([data processors](#data-processor) and [pipelines](#pipeline)),
+to data persistence and distribution ([data sinks](#data-sink)).
+
+
+
+
+ 🐍 StreamPipes Python
+
+
+
+ Discover what we offer for the Python world:
+ Python Documentation 🔗
+ 
+
+## Adapter
+An adapter connects to any external data source (e.g., OPC-UA, MQTT, S7 PLC, Modbus) and forwards the events it receives to the internal StreamPipes system.
+Adapters can either be created by using a predefined adapter for a data source available in our marketplace [StreamPipes Connect](./03_use-connect.md).
+An overview of all available adapters can be found under the menu bar **📚 Pipeline Elements**.
+When you select one of these adapters, you can easily connect to the data source using an intuitive and convenient UI dialog (see the Connect section for more details).
+Alternatively, you can define your own adapter by [using the provided Software Development Kit (SDK)](./06_extend-tutorial-adapters.md).
+Creating an adapter is always the first step when you want to get data into StreamPipes and process it further.
+
+## Data Stream
+**Data streams** are the primary source for working with events in StreamPipes.
+A stream is an ordered sequence of events, where an event typically consists of one or more observation values and additional metadata.
+The `structure` (or `schema` as we call it) of an event provided by a data stream is stored in StreamPipes' internal semantic schema registry.
+Data streams are primarily created by adapters, but can also be created by a [StreamPipes Function](./06_extend-sdk-functions.md).
+
+## Data Processor
+**Data processors** in StreamPipes transform one or more input streams into an output stream.
+Such transformations can be simple, such as filtering based on a predefined rule, or more complex, such as applying rule-based or learning-based algorithms to the data.
+Data processors can be applied to any data stream that meets the input requirements of a processor.
+In addition, most processors can be configured by providing custom parameters directly in the user interface.
+Processing elements define stream requirements, which are a set of minimum characteristics that an incoming event stream must provide.
+Data processors can maintain state or perform stateless operations.
+
+## Data Sink
+**Data sinks** consume event streams similar to data processors, but do not provide an output data stream.
+As such, data sinks typically perform some action or trigger a visualization as a result of a stream transformation.
+Similar to data processors, sinks also require the presence of specific input requirements from each bound data stream and can be customized.
+StreamPipes provides several internal data sinks, for example, to generate notifications, visualize live data, or persist historical data from incoming streams.
+In addition, StreamPipes provides several data sinks to forward data streams to external systems such as databases.
+
+## Pipeline
+A pipeline in Apache StreamPipes describes the transformation process from a data stream to a data sink.
+Typically, a pipeline consists of at least one data stream, zero or more data processors, and at least one data sink.
+Pipelines are created graphically by users using the [Pipeline Editor](./03_use-pipeline-editor.md) and can be started and stopped at any time.
diff --git a/website-v2/versioned_docs/version-0.70.0/02_concepts-pipeline.md b/website-v2/versioned_docs/version-0.95.1/02_concepts-pipeline.md
similarity index 70%
rename from website-v2/versioned_docs/version-0.70.0/02_concepts-pipeline.md
rename to website-v2/versioned_docs/version-0.95.1/02_concepts-pipeline.md
index 282642fc4..3d2c5369b 100644
--- a/website-v2/versioned_docs/version-0.70.0/02_concepts-pipeline.md
+++ b/website-v2/versioned_docs/version-0.95.1/02_concepts-pipeline.md
@@ -2,7 +2,6 @@
id: concepts-pipelines
title: Pipelines
sidebar_label: Pipelines
-original_id: concepts-pipelines
---
tbd
\ No newline at end of file
diff --git a/website-v2/versioned_docs/version-0.95.1/02_introduction.md b/website-v2/versioned_docs/version-0.95.1/02_introduction.md
new file mode 100644
index 000000000..6dc71b384
--- /dev/null
+++ b/website-v2/versioned_docs/version-0.95.1/02_introduction.md
@@ -0,0 +1,85 @@
+---
+id: introduction
+title: Introduction
+sidebar_label: Introduction
+---
+
+## What is StreamPipes?
+
+Apache StreamPipes is a self-service Industrial IoT toolbox to enable non-technical users to connect, analyze and
+explore IoT data streams. The main goal of StreamPipes is to help users bridging the gap between operational
+technology (OT) and information technology (IT). This is achieved by providing a set of tools which help to make
+industrial data accessible for downstream tasks such as data analytics and condition monitoring.
+When working with industrial data and especially when building upon an open source stack for such tasks, users are often
+faced with the management and integration of a variety of different tools for data connectivity, messaging &
+integration, data enrichment, data storage, visualization and analytics. This results in an increasing operational
+complexity and hardly manageable software stacks.
+
+Apache StreamPipes addresses this problem: It provides a complete toolbox with a variety of different tools to easily
+gather data from OT systems such as Programmatic Logic Controllers (PLCs), industrial protocols (e.g., OPC-UA or
+Modbus), IT protocols (e.g., MQTT) and others. Data is integrated in the form of live data streams. Based on connected
+data, StreamPipes provides another module called the pipeline editor, which can be used to apply real-time analytics
+algorithms on connected data stream. To this end, a library of pre-defined algorithms can be used. Out of the box,
+StreamPipes provides more than 100 pipeline elements tailored at manufacturing data analytics. This includes simple
+rule-based algorithms (e.g., flank detection, peak detection, boolean timers), as well as the possibility to integrate
+more sophisticated ML-based algorithms. Finally, the pipeline editor allows to integrate with third-party systems by
+using a variety of data sinks (e.g., to forward data to messaging brokers such as Apache Kafka, MQTT or RocketMQ, to
+store data in databases such as PostgreSQL or Redis or to trigger notifications). Besides pipelines, an included data
+explorer allows to visually analyze industrial IoT data. For this purpose, a number of visualizations are integrated
+that allow non-technical users to quickly get first insights. Examples are correlations between several sensor values,
+value heatmaps, distributions or time-series visualizations. Further tools include a dashboard used for real-time
+monitoring, e.g., for visualizing live KPIs at shopfloor level.
+
+But StreamPipes is much more than just the user interface and an orchestration system for pipelines: It can be used as a
+whole developer platform for Industrial IoT application. Apache StreamPipes is made for extensibility - it provides
+several extension points, which allow the definition of custom algorithms, additional interfaces to third-party tools
+and proprietary data sources.
+
+StreamPipes includes developer support for Java and Python, making it easy to integrate custom-trained machine learning
+models into the data processing environment. With the built-in Python support, it is also possible to run online machine
+learning methods directly on data streams gathered by StreamPipes.
+
+## Where does StreamPipes help?
+
+Being positioned in the industrial IoT domain, the overall goal of StreamPipes is to help manufacturing companies to
+quickly build up an industrial IoT infrastructure and to analyse IIoT data without the need for manual programming.
+Oftentimes, StreamPipes is compared to other tools in this area such as Node-RED for visually wiring of pipelines, which
+is often used together with Grafana for data visualization and InfluxDB for time-series storage. The disadvantage of
+such architectures is the system complexity beyond the first prototype, especially when it comes to production
+deployments. Maintaining and securing multiple software instances is often a hard task requiring for substantial
+development effort. In addition, implementing single-sign-on and providing a unified user experience is another hurdle.
+This is where StreamPipes, as a single integrated tool with production-critical features such as access and role
+management, provides many advantages.
+StreamPipes has already a wide user range from the manufacturing domain. It helps users to quickly do the first steps
+related to industrial analytics but can also be used for monitoring whole production facilities, analysing data streams
+from multiple plants and sensors in real time using the integrated algorithm toolbox. Customization to individual use
+cases is easy due to several extension points:
+
+* Software development kit for adapters, data processors and sinks: The functionality of StreamPipes can be extending by
+ using the integrated SDK. For instance, it is possible to integrate custom-tailored algorithms for proprietary sensors
+ or models into the toolbox. Additional algorithms and data sinks can be installed at runtime.
+* Additional user interface plugins: StreamPipes allows to extend the default installation with additional UI views,
+ making use of a micro frontend approach. For instance, users can extend the system with custom-tailored views for a
+ specific machine or plant. Developers can use a platform API to communicate with the core StreamPipes instance.
+* UI customization: To ensure a consistent look and feel, StreamPipes can be customized to the company’s corporate
+ identity.
+
+## How does StreamPipes technically work in a nutshell?
+
+
+
+
+
+To foster extensibility, Apache StreamPipes is based on a microservice architecture as illustrated above. The main
+services provided or used by StreamPipes are the a) user interface, b) the core, c) a time-series storage, d) a
+publish/subscribe messaging layer and e) extensions services. Adapters are created over the user interface using an
+intuitive configuration wizard and connect to the underlying source systems. Raw events coming from adapters can be
+pre-processed (e.g., measurement unit conversions or datatype conversions). Afterwards, events are sent to the message
+broker, which is the central backbone to provide IIoT data to internal and external applications.
+
+Besides adapters, extensions microservices can also integrate additional business logic in form of data processors and
+data sinks. StreamPipes comes with over 100 built-in processors and sinks, covering basic use cases out-of-the-box. The StreamPipes core cares about orchestration of these pipeline elements and communicates with the user
+interface. In addition, a time-series storage ensures persistence and can be used by any extensions service to write
+data into the internal storage. The StreamPipes core provides a query interface to access historical data, which is, for
+instance, used by the data explorer UI component. The user interface itself provides several built-in modules but can
+also be extended with additional micro frontends.
diff --git a/website-v2/versioned_docs/version-0.70.0/03_use-configurations.md b/website-v2/versioned_docs/version-0.95.1/03_use-configurations.md
similarity index 98%
rename from website-v2/versioned_docs/version-0.70.0/03_use-configurations.md
rename to website-v2/versioned_docs/version-0.95.1/03_use-configurations.md
index fda2f1169..4aa953fbb 100644
--- a/website-v2/versioned_docs/version-0.70.0/03_use-configurations.md
+++ b/website-v2/versioned_docs/version-0.95.1/03_use-configurations.md
@@ -2,7 +2,6 @@
id: use-configurations
title: Configurations
sidebar_label: Configurations
-original_id: use-configurations
---
The configuration section is an admin-only interface for system-wide settings.
diff --git a/website-v2/versioned_docs/version-0.70.0/03_use-connect.md b/website-v2/versioned_docs/version-0.95.1/03_use-connect.md
similarity index 99%
rename from website-v2/versioned_docs/version-0.70.0/03_use-connect.md
rename to website-v2/versioned_docs/version-0.95.1/03_use-connect.md
index 34495cb6e..ba8146481 100644
--- a/website-v2/versioned_docs/version-0.70.0/03_use-connect.md
+++ b/website-v2/versioned_docs/version-0.95.1/03_use-connect.md
@@ -2,7 +2,6 @@
id: use-connect
title: StreamPipes Connect
sidebar_label: StreamPipes Connect
-original_id: use-connect
---
StreamPipes Connect is the module to connect external data sources with Apache StreamPipes directly from the user interface.
diff --git a/website-v2/versioned_docs/version-0.70.0/03_use-dashboard.md b/website-v2/versioned_docs/version-0.95.1/03_use-dashboard.md
similarity index 99%
rename from website-v2/versioned_docs/version-0.70.0/03_use-dashboard.md
rename to website-v2/versioned_docs/version-0.95.1/03_use-dashboard.md
index 4fc75c851..339bf6b57 100644
--- a/website-v2/versioned_docs/version-0.70.0/03_use-dashboard.md
+++ b/website-v2/versioned_docs/version-0.95.1/03_use-dashboard.md
@@ -2,7 +2,6 @@
id: use-dashboard
title: Live Dashboard
sidebar_label: Live Dashboard
-original_id: use-dashboard
---
The live dashboard can be used to visualize live data of data streams using a set of visualizations
diff --git a/website-v2/versioned_docs/version-0.70.0/03_use-data-explorer.md b/website-v2/versioned_docs/version-0.95.1/03_use-data-explorer.md
similarity index 99%
rename from website-v2/versioned_docs/version-0.70.0/03_use-data-explorer.md
rename to website-v2/versioned_docs/version-0.95.1/03_use-data-explorer.md
index af31c65ab..f84323bf6 100644
--- a/website-v2/versioned_docs/version-0.70.0/03_use-data-explorer.md
+++ b/website-v2/versioned_docs/version-0.95.1/03_use-data-explorer.md
@@ -2,7 +2,6 @@
id: use-data-explorer
title: Data Explorer
sidebar_label: Data Explorer
-original_id: use-data-explorer
---
The data explorer can be used to visualize and explore data streams that are persisted by using the **Data Lake** sink.
diff --git a/website-v2/versioned_docs/version-0.70.0/03_use-install-pipeline-elements.md b/website-v2/versioned_docs/version-0.95.1/03_use-install-pipeline-elements.md
similarity index 78%
rename from website-v2/versioned_docs/version-0.70.0/03_use-install-pipeline-elements.md
rename to website-v2/versioned_docs/version-0.95.1/03_use-install-pipeline-elements.md
index 10be7e572..852693200 100644
--- a/website-v2/versioned_docs/version-0.70.0/03_use-install-pipeline-elements.md
+++ b/website-v2/versioned_docs/version-0.95.1/03_use-install-pipeline-elements.md
@@ -2,7 +2,6 @@
id: use-install-pipeline-elements
title: Install Pipeline Elements
sidebar_label: Install Pipeline Elements
-original_id: use-install-pipeline-elements
---
## Install Pipeline Elements
diff --git a/website-v2/versioned_docs/version-0.70.0/03_use-managing-pipelines.md b/website-v2/versioned_docs/version-0.95.1/03_use-managing-pipelines.md
similarity index 98%
rename from website-v2/versioned_docs/version-0.70.0/03_use-managing-pipelines.md
rename to website-v2/versioned_docs/version-0.95.1/03_use-managing-pipelines.md
index 1aba73b1c..2c64b53ee 100644
--- a/website-v2/versioned_docs/version-0.70.0/03_use-managing-pipelines.md
+++ b/website-v2/versioned_docs/version-0.95.1/03_use-managing-pipelines.md
@@ -2,7 +2,6 @@
id: use-managing-pipelines
title: Managing Pipelines
sidebar_label: Managing Pipelines
-original_id: use-managing-pipelines
---
The pipeline view lists all created pipelines and provides several views and actions to manage the lifecycle of pipelines.
diff --git a/website-v2/versioned_docs/version-0.70.0/03_use-notifications.md b/website-v2/versioned_docs/version-0.95.1/03_use-notifications.md
similarity index 97%
rename from website-v2/versioned_docs/version-0.70.0/03_use-notifications.md
rename to website-v2/versioned_docs/version-0.95.1/03_use-notifications.md
index 627efca69..b5c64ed98 100644
--- a/website-v2/versioned_docs/version-0.70.0/03_use-notifications.md
+++ b/website-v2/versioned_docs/version-0.95.1/03_use-notifications.md
@@ -2,7 +2,6 @@
id: use-notifications
title: Notifications
sidebar_label: Notifications
-original_id: use-notifications
---
The notification module can be used to create internal notifications.
diff --git a/website-v2/versioned_docs/version-0.70.0/03_use-pipeline-editor.md b/website-v2/versioned_docs/version-0.95.1/03_use-pipeline-editor.md
similarity index 99%
rename from website-v2/versioned_docs/version-0.70.0/03_use-pipeline-editor.md
rename to website-v2/versioned_docs/version-0.95.1/03_use-pipeline-editor.md
index 9762e7819..f09cf8486 100644
--- a/website-v2/versioned_docs/version-0.70.0/03_use-pipeline-editor.md
+++ b/website-v2/versioned_docs/version-0.95.1/03_use-pipeline-editor.md
@@ -2,7 +2,6 @@
id: use-pipeline-editor
title: Pipeline Editor
sidebar_label: Pipeline Editor
-original_id: use-pipeline-editor
---
The pipeline editor module supports building pipelines that transform a data stream using a set of resuable data processors and data sinks.
diff --git a/website-v2/versioned_docs/version-0.95.1/05_deploy-choosing-the-right-flavor.md b/website-v2/versioned_docs/version-0.95.1/05_deploy-choosing-the-right-flavor.md
new file mode 100644
index 000000000..a140bf46e
--- /dev/null
+++ b/website-v2/versioned_docs/version-0.95.1/05_deploy-choosing-the-right-flavor.md
@@ -0,0 +1,47 @@
+---
+id: choosing-the-right-flavor
+title: Choosing the right flavor
+sidebar_label: Service selection options
+---
+
+
+## Introduction
+
+StreamPipes comes with many different options to customize a deployment. This section introduces the various options you can choose from when installing StreamPipes.
+
+You can choose between various **deployment modes**, choose from two different core packages and several extension packages, wich are described below.
+
+## Deployment Mode
+
+For the deployment model, you choose between a standard multi-container `Docker-Compose` installation and the `Kubernetes` installation.
+we provide several `Docker-Compose` files for the various options shown here and a `helm chart`.
+See [Docker Deployment](05_deploy-docker.md) and [Kubernetes Deployment](05_deploy-kubernetes.md) for more details.
+
+### Running StreamPipes in a non-containerized environment
+
+Of course, it is also possible to launch StreamPipes in a non-containerized environment.
+You will need to build your own executable binaries by running `mvn package`.
+In addition, it is required to install the required 3rd party services (see [Architecture](07_technicals-architecture.md)) and configure the environment variables as described in [Environment Variables](05_deploy-environment-variables.md).
+
+## Core Service
+
+We provide two different pre-packaged versions of core services. The default `streampipes-service-core` is a packaged JAR file which includes client libraries for the various messaging systems StreamPipes supports at the cost of a larger file size.
+In case you plan to run StreamPipes on less resource-intensive hardware, we recommend to switch to the `streampipes-service-core-minimal` package, which only includes support for MQTT and NATS, but has a smaller file size and slightly improved startup performance.
+
+## Extension Services
+
+Similar to the core, we provide several pre-packaged extension services which differ mainly by their file size, number of supported adapters and pipeline elements and messaging systems.
+
+The following packages exist:
+
+* `streampipes-extensions-all-jvm` is the largest package and includes all official StreamPipes adapters and pipeline elements. It also includes support for all messaging systems Streampipes currently supports.
+* `streampipes-extensions-all-iiot` is a subset of the aforementioned package and excludes adapters and pipeline elements which are often not relevant for IIoT use cases. For instance, the package excludes text mining-related pipeline elements.
+* `streampipes-extensions-iiot-minimal` is a subset of the aforementioned package and includes only support for the lightweight messaging systems MQTT and NATS.
+
+Generally said, in cases where you plan to deploy StreamPipes on a resource-limited edge device, we recommend a combination of the `streampipes-service-core-minimal` and `streampipes-extensions-iiot-minimal` package. This could, for instance, be a device with less than 4GB memory.
+In other cases, it depends on the use case and if you need all adapters and pipeline elements or are ok with the IIoT-related extensions.
+
+## Messaging System
+
+StreamPipes can be configured to use different messaging systems for exchanging events between adapters and pipeline elements.
+The section [Messaging](07_technicals-messaging.md) includes detailed information on the configuration of messaging systems.
diff --git a/website-v2/versioned_docs/version-0.95.1/05_deploy-docker.md b/website-v2/versioned_docs/version-0.95.1/05_deploy-docker.md
new file mode 100644
index 000000000..881e16b43
--- /dev/null
+++ b/website-v2/versioned_docs/version-0.95.1/05_deploy-docker.md
@@ -0,0 +1,104 @@
+---
+id: deploy-docker
+title: Docker Deployment
+sidebar_label: Docker Deployment
+---
+
+StreamPipes Compose is a simple collection of user-friendly `docker-compose` files that easily lets gain first-hand experience with Apache StreamPipes.
+
+> **NOTE**: We recommend StreamPipes Compose to only use for initial try-out and testing. If you are a developer and
+> want to develop new pipeline elements or core feature, use the [StreamPipes CLI](06_extend-cli.md).
+
+#### TL;DR: A one-liner to rule them all :-)
+
+```bash
+docker-compose up -d
+```
+Go to http://localhost to finish the installation in the browser. Once finished, switch to the pipeline editor and start the interactive tour or check the [online tour](https://streampipes.apache.org/docs/docs/user-guide-tour/) to learn how to create your first pipeline!
+
+## Prerequisites
+* Docker >= 17.06.0
+* Docker-Compose >= 1.17.0 (Compose file format: 3.4)
+* Google Chrome (recommended), Mozilla Firefox, Microsoft Edge
+
+Tested on: **macOS, Linux, Windows 10 upwards** (CMD, PowerShell, GitBash)
+
+**macOS** and **Windows** users can easily get Docker and Docker-Compose on their systems by installing **Docker for Mac/Windows** (recommended).
+
+> **NOTE**: On purpose, we disabled all port mappings except of http port **80** to access the StreamPipes UI to provide minimal surface for conflicting ports.
+
+## Usage
+We provide several options to get you going:
+
+- **default**: Default docker-compose file, called `docker-compose.yml`.
+- **nats**: The standard installation which uses Nats as message broker,called `docker-compose.nats.yml`.
+- **full**: Contains experimental Flink wrappers, called `docker-compose.full.yml`.
+- **quickstart**: Contains predefined example assets, called `docker-compose.quickstart.yml`. The Quickstart mode is a user-friendly feature which comes with predefined example assets like pipelines, dashboards, and data views. These ready-to-use components allow first-time users to get a feel of StreamPipes in IIoT with ease, serving as a practical demonstration of how StreamPipes can be utilized for efficient monitoring and analysis. We highly recommend first-time users to begin with the Quickstart mode to understand the simplicity and convenience that StreamPipes brings to the IIoT platform. Please follow the [User Guide for Quickstart Mode](user-guide-for-quickstart.md) if you want to explore it.
+
+
+:::info
+
+Other options include configurations for the internally used message broker. The current default is `Kafka`, but you can also start StreamPipes with `Nats`, `MQTT` or `Apache Pulsar`.
+Use one of the other provided docker-compose files.
+
+:::
+
+**Starting** the **default** option is as easy as simply running:
+> **NOTE**: Starting might take a while since `docker-compose up` also initially pulls all Docker images from Dockerhub.
+
+```bash
+docker-compose up -d
+# go to `http://localhost` after all services are started
+```
+After all containers are successfully started just got to your browser and visit http://localhost to finish the installation. Once finished, switch to the pipeline editor and start the interactive tour or check the [online tour](https://streampipes.apache.org/docs/docs/user-guide-tour/) to learn how to create your first pipeline!
+
+**Stopping** the **default** option is similarly easy:
+```bash
+docker-compose down
+# if you want to remove mapped data volumes, run:
+# docker-compose down -v
+```
+
+Starting the **nats** option is almost the same, just specify the `docker-compose.nats.yml` file:
+```bash
+docker-compose -f docker-compose.nats.yml up -d
+# go to `http://localhost` after all services are started
+```
+**Stopping** the **nats** option:
+```bash
+docker-compose -f docker-compose.nats.yml down
+```
+
+
+Starting the **full** option is almost the same, just specify the `docker-compose.full.yml` file:
+```bash
+docker-compose -f docker-compose.full.yml up -d
+#go to `http://localhost` after all services are started
+```
+Stopping the **full** option:
+```bash
+docker-compose -f docker-compose.nats.yml down
+#docker-compose -f docker-compose.nats.yml down -v
+```
+Starting the **quickstart** option:
+```bash
+docker-compose -f docker-compose.quickstart.yml build script-runner
+docker-compose -f docker-compose.quickstart.yml up -d
+#go to `http://localhost` after all services are started
+```
+Stopping the **quickstart** option:
+```bash
+docker-compose -f docker-compose.quickstart.yml down
+```
+
+## Update services
+To actively pull the latest available Docker images use:
+```bash
+docker-compose pull
+```
+
+## Upgrade
+To upgrade to another StreamPipes version, simply edit the `SP_VERSION` in the `.env` file.
+```
+SP_VERSION=
-
## Project structure
@@ -61,5 +44,3 @@ For details, have a look at the other parts of the Developer Guide, where these
## Next steps
Click [here](06_extend-first-processor.md) to learn how to create your first data processor.
-
-
diff --git a/website-v2/versioned_docs/version-0.70.0/06_extend-cli.md b/website-v2/versioned_docs/version-0.95.1/06_extend-cli.md
similarity index 86%
rename from website-v2/versioned_docs/version-0.70.0/06_extend-cli.md
rename to website-v2/versioned_docs/version-0.95.1/06_extend-cli.md
index ee7c7d765..e5f93cfd6 100644
--- a/website-v2/versioned_docs/version-0.70.0/06_extend-cli.md
+++ b/website-v2/versioned_docs/version-0.95.1/06_extend-cli.md
@@ -2,7 +2,6 @@
id: extend-cli
title: StreamPipes CLI
sidebar_label: StreamPipes CLI
-original_id: extend-cli
---
The StreamPipes command-line interface (CLI) is focused on developers in order to provide an easy entrypoint to set up a suitable dev environment, either planning on developing
@@ -24,7 +23,7 @@ pipeline-element
streampipes env --set pipeline-element
streampipes up -d
```
-> **NOTE**: use `./streampipes` if you haven't add it to the PATH and sourced it (see section "Run `streampipes` from anywhere?").
+> **NOTE**: use `./installer/cli/streampipes` if you haven't add it to the PATH and sourced it (see section "Run `streampipes` from anywhere?").
## Prerequisites
The CLI is basically a wrapper around multiple `docker` and `docker-compose` commands plus some additional sugar.
@@ -35,9 +34,9 @@ The CLI is basically a wrapper around multiple `docker` and `docker-compose` com
* For Windows Developer: GitBash only
-Tested on: **macOS**, **Linux**, **Windows***)
+Tested on: (***macOS**, **Linux**, **Windows***)
-> **NOTE**: *) If you're using Windows the CLI only works in combination with GitBash - CMD, PowerShell won't work.
+> **NOTE**: If you're using Windows the CLI only works in combination with GitBash - CMD, PowerShell won't work.
## CLI commands overview
@@ -83,7 +82,7 @@ streampipes env --set pipeline-element
```
**Start** environment ( default: `dev` mode). Here the service definition in the selected environment is used to start the multi-container landscape.
-> **NOTE**: `dev` mode is enabled by default since we rely on open ports to core service such as `consul`, `couchdb`, `kafka` etc. to reach from the IDE when developing. If you don't want to map ports (except the UI port), then use the `--no-ports` flag.
+> **NOTE**: `dev` mode is enabled by default since we rely on open ports to core service such as `couchdb`, `kafka` etc. to reach from the IDE when developing. If you don't want to map ports (except the UI port), then use the `--no-ports` flag.
```bash
streampipes up -d
@@ -92,7 +91,7 @@ streampipes up -d
```
Now you're good to go to write your new pipeline element :tada: :tada: :tada:
-> **HINT for extensions**: Use our [Maven archetypes](https://streampipes.apache.org/docs/docs/dev-guide-archetype/) to setup a project skeleton and use your IDE of choice for development. However, we do recommend using IntelliJ.
+> **HINT for extensions**: Use our [Maven archetypes](https://streampipes.apache.org/docs/docs/extend-archetypes/) to set up a project skeleton and use your IDE of choice for development. However, we do recommend using IntelliJ.
> **HINT for core**: To work on `backend` or `ui` features you need to set the template to `backend` and clone the core repository [streampipes](https://github.com/apache/streampipes) - check the prerequisites there for more information.
@@ -105,12 +104,12 @@ streampipes down
## Additionally, useful commands
-**Start individual services only?** We got you! You chose a template that suits your needs and now you only want to start individual services from it, e.g. only Kafka and Consul.
+**Start individual services only?** We got you! You chose a template that suits your needs and now you only want to start individual services from it, e.g. only Kafka and InfluxDB.
> **NOTE**: the service names need to be present and match your current `.spenv` environment.
```bash
-streampipes up -d kafka consul
+streampipes up -d kafka influxdb
```
**Get current environment** (if previously set using `streampipes env --set Other archetypes
- -## Processors Flink -```bash -mvn archetype:generate \ - -DarchetypeGroupId=org.apache.streampipes \ - -DarchetypeArtifactId=streampipes-archetype-pe-processors-flink \ - -DarchetypeVersion={sp.version} -``` - -## Sinks Flink -```bash -mvn archetype:generate \ - -DarchetypeGroupId=org.apache.streampipes \ - -DarchetypeArtifactId=streampipes-archetype-pe-sinks-flink \ - -DarchetypeVersion={sp.version} -``` -
+
+To communicate with Apache StreamPipes, you need to provide proper credentials. There are two ways to obtain
+credentials:
+
+* An API token, which is bound to a user. The API token can be generate from the UI clicking on the user icon and then
+ navigate to `Profile/API`.
+* A service user, which can be created by users with role `Admin`.
+
+Service users can have their own permissions, while API tokens inherit all permissions from the corresponding user.
+
+## Connecting to StreamPipes
+
+Once you have your API token and configured your dependencies, you can connect to an Apache StreamPipes instance as
+follows:
+
+```java
+
+CredentialsProvider credentials=StreamPipesCredentials
+ .withApiKey("admin@streampipes.apache.org","YOUR_API_KEY");
+
+// Create an instance of the StreamPipes client
+ StreamPipesClient client=StreamPipesClient
+ .create("localhost",8082,credentials,true);
+
+```
+
+The following configurations are required:
+
+* The `withApiKey` method expects the username and the API key. Alternatively, use the `withServiceToken` method to
+ authenticate as a service user.
+* The client instance requires the hostname or IP address of your running StreamPipes instance. In addition, you need to
+ provide the port, the credentials object and a flag which needs to be set in case the StreamPipes instance is not
+ served over HTTPS.
+* There are short-hand convenience options to create a client instance.
+
+## Working with the client
+
+Here are some examples how you can work with the StreamPipes client:
+
+```java
+
+// Get streams
+ListSP_HOST environment variable.
-To check if the service is up and running, open the browser on *'localhost:8090'* (or the port defined in the service definition). The machine-readable description of the processor should be visible as shown below.
-
+To check if the service is up and running, open the browser on *'localhost:8090'* (or the port defined in the service definition). The machine-readable description of the processor should be visible as shown below.
+
-
-
+:::
Now you can go to StreamPipes.
Your new processor *'Example'* should now show up in the installation menu ("Install Pipeline Elements" in the left navigation bar).
diff --git a/website-v2/versioned_docs/version-0.70.0/06_extend-sdk-event-model.md b/website-v2/versioned_docs/version-0.95.1/06_extend-sdk-event-model.md
similarity index 98%
rename from website-v2/versioned_docs/version-0.70.0/06_extend-sdk-event-model.md
rename to website-v2/versioned_docs/version-0.95.1/06_extend-sdk-event-model.md
index f4bb8ed1e..42cc8d472 100644
--- a/website-v2/versioned_docs/version-0.70.0/06_extend-sdk-event-model.md
+++ b/website-v2/versioned_docs/version-0.95.1/06_extend-sdk-event-model.md
@@ -2,7 +2,6 @@
id: extend-sdk-event-model
title: "SDK Guide: Event Model"
sidebar_label: "SDK: Event Model"
-original_id: extend-sdk-event-model
---
## Introduction
@@ -11,7 +10,7 @@ This guide explains the usage of the event model to manipulate runtime events fo
## Prerequisites
-This guide assumes that you are already familiar with the basic setup of [data processors](extend-first-processor).
+This guide assumes that you are already familiar with the basic setup of [data processors](06_extend-first-processor.md).
### Property Selectors
diff --git a/website-v2/versioned_docs/version-0.70.0/06_extend-sdk-functions.md b/website-v2/versioned_docs/version-0.95.1/06_extend-sdk-functions.md
similarity index 94%
rename from website-v2/versioned_docs/version-0.70.0/06_extend-sdk-functions.md
rename to website-v2/versioned_docs/version-0.95.1/06_extend-sdk-functions.md
index 77d2bb966..659690b71 100644
--- a/website-v2/versioned_docs/version-0.70.0/06_extend-sdk-functions.md
+++ b/website-v2/versioned_docs/version-0.95.1/06_extend-sdk-functions.md
@@ -2,7 +2,6 @@
id: extend-sdk-functions
title: "SDK Guide: Functions"
sidebar_label: "SDK: Functions"
-original_id: extend-sdk-functions
---
## Introduction
@@ -25,11 +24,13 @@ and run until the service is stopped.
## Writing a function
-Common Problems
-+:::caution Common Problems If the service description is not shown on 'localhost:8090', you might have to change the port address. This needs to be done in the configuration of your service, further explained in the configurations part of the developer guide. If the service does not show up in the StreamPipes installation menu, click on 'MANAGE ENDPOINTS' and add 'http://YOUR_IP_OR_DNS_NAME:8090'. Use the IP or DNS name you provided as the SP_HOST variable or the IP (if resolvable) found by the auto-discovery service printed in the console. After adding the endpoint, a new processor with the name *Example* should show up. -
-
-
+:::caution Work in Progress
+
+Functions are currently in preview mode and are not yet recommended for production usage.
+APIs are subject to change in a future version.
+
+:::
+
To define a function, create a new extensions service using the [Maven Archetypes](06_extend-archetypes.md) or use an already existing service.
diff --git a/website-v2/versioned_docs/version-0.70.0/06_extend-sdk-migration-sd.md b/website-v2/versioned_docs/version-0.95.1/06_extend-sdk-migration-sd.md
similarity index 84%
rename from website-v2/versioned_docs/version-0.70.0/06_extend-sdk-migration-sd.md
rename to website-v2/versioned_docs/version-0.95.1/06_extend-sdk-migration-sd.md
index 054d11c60..1fd5200ad 100644
--- a/website-v2/versioned_docs/version-0.70.0/06_extend-sdk-migration-sd.md
+++ b/website-v2/versioned_docs/version-0.95.1/06_extend-sdk-migration-sd.md
@@ -2,7 +2,6 @@
id: extend-sdk-migration-service-discovery
title: "Migration Guide: New Service Discovery in 0.69.0"
sidebar_label: "Migration Guide: 0.69.0"
-original_id: extend-sdk-migration-service-discovery
---
@@ -99,15 +98,16 @@ Configs can be easily accessed from the ``EventProcessorRuntimeContext`` (or ``E
### Service Discovery
-An extensions service can be started by executing the Init class. StreamPipes will now automatically select the proper service IP address and register the service in Consul.
+An extensions service can be started by executing the Init class.
+StreamPipes will now automatically select the proper service IP address and register the service at the backend.
You can inspect the selected IP address in the console:
```
-16:41:58.342 SP [main] INFO o.a.s.commons.networking.Networking - Using auto-discovered IP: 172.30.80.1
-16:41:58.364 SP [main] INFO o.a.s.commons.networking.Networking - Using port from provided environment variable SP_PORT: 6025
-16:41:58.367 SP [main] INFO o.a.s.c.init.DeclarersSingleton - Registering 0 configs in key/value store
-16:41:58.400 SP [main] INFO o.a.s.s.consul.ConsulProvider - Checking if consul is available...
-16:41:58.419 SP [main] INFO o.a.s.s.consul.ConsulProvider - Successfully connected to Consul
+2024-05-16T11:03:37.158+02:00 INFO --- [ main] o.a.s.commons.networking.Networking : Using auto-discovered IP: 192.168.178.22
+2024-05-16T11:03:37.158+02:00 INFO --- [ main] o.a.s.commons.networking.Networking : Using port from provided environment variable SP_PORT: 7023
+2024-05-16T11:03:37.372+02:00 INFO --- [ main] a.s.s.e.StreamPipesExtensionsServiceBase : Registering service org.apache.streampipes.extensions.all.jvm with id org.apache.streampipes.extensions.all.jvm-FUt84Y at core
+2024-05-16T11:03:37.814+02:00 INFO --- [ main] o.a.s.s.extensions.CoreRequestSubmitter : Successfully registered service at core.
+2024-05-16T11:03:37.814+02:00 INFO --- [ main] a.s.s.e.StreamPipesExtensionsServiceBase : Registering 1 service configs for service org.apache.streampipes.extensions.all.jvm
```
In some (rare) cases, a non-resolvable IP will be selected. In this case, you can manually override the IP by providing a ``SP_HOST`` environment variable. This falls back to a similar behaviour as in pre-0.69.0-versions and will use the manually provided IP.
diff --git a/website-v2/versioned_docs/version-0.95.1/06_extend-sdk-migrations.md b/website-v2/versioned_docs/version-0.95.1/06_extend-sdk-migrations.md
new file mode 100644
index 000000000..bf822c1b8
--- /dev/null
+++ b/website-v2/versioned_docs/version-0.95.1/06_extend-sdk-migrations.md
@@ -0,0 +1,179 @@
+---
+id: extend-sdk-migration
+title: "SDK Guide: Pipeline Element Migration"
+sidebar_label: "SDK: PE Migration"
+---
+
+Pipeline element migrations allow you to automatically update and migrate both existing and pipeline elements when a new
+version of StreamPipes is installed. This means that whenever you upgrade StreamPipes, all existing and future
+pipeline elements will be directly compatible with the new version without any manual interaction. Pipeline elements
+include adapters, data processors, and data sinks.
+
+:::info
+Migrations will make their debut in StreamPipes version `0.93.0` and will be an integral part of the system going
+forward.
+However, it's important to note that this feature is not available in any of the previous versions of StreamPipes. To
+take full advantage of migrations and their benefits, it is recommended to upgrade to version `0.93.0` or later. This
+will
+ensure that you have access to the latest enhancements and maintain compatibility with the evolving StreamPipes
+platform.
+:::
+
+## Define Migrations
+
+Whenever a pipeline element, be it an adapter, data processor, or data sink, undergoes changes that result in
+modifications to its configuration options, developers must additionally create a migration procedure. This migration
+process should be capable of smoothly transitioning all affected instances from the previous version to the new one.
+The migration itself is automatically managed and executed by StreamPipes. Developers are only responsible for two key
+aspects:
+
+* **Implementing the concrete migration**: Developers need to craft the specific migration logic that facilitates the
+ seamless transition of configuration options.
+* **Registering the migration**: Developers should register their migration procedures at the extensions service,
+ allowing StreamPipes to identify and apply the necessary updates to affected instances.
+
+By adhering to these two essential tasks, developers can ensure a hassle-free evolution of pipeline elements while
+StreamPipes handles the orchestration of the migration process.
+
+The following gives a concrete example of creating a migration for
+the [S7 adapter](./pe/org.apache.streampipes.connect.iiot.adapters.plc4x.s7.md).
+Thereby, we assume this adapter has received a new input element which determines whether the connection should be made
+authenticated or not.
+This is represented by a simple boolean that is visualized as a toggle button in the UI.
+
+### Implementing a Concrete Migration
+
+StreamPipes offers three distinct migration mechanisms tailored to specific types of pipeline
+elements: `IAdapterMigrator`, `IDataProcessorMigrator`, and `IDataSinkMigrator`.
+These migration mechanisms are presented as interfaces and require the implementation of two fundamental methods:
+
+* `config()`: This method defines the configuration for the migration, encompassing all essential metadata related to
+ the migration process.
+* `migrate()`: Within this method, the actual migration logic is to be implemented. It serves as the operational core
+ for facilitating the migration for the respective pipeline element.
+
+In accordance with the example described above, we will implement the `Plc4xS7AdapterMigrationV1` in the following.
+
+:::note
+Before we begin, it's important to familiarize ourselves with two key conventions that guide our approach to migrations:
+
+* To maintain clarity and organization, all migration classes associated with a specific pipeline element are located
+ within a dedicated sub-package named `migration`. This sub-package is nested within the package of the respective
+ pipeline element.
+* Migration classes are named according to a specific schema: `Work in Progress
- Functions are currently in preview mode and are not yet recommended for production usage. -APIs are subject to change in a future version.
-+ +## How Migrations are Handled Internally + +Migrations are handled by an interplay between the Extension Service, which provides the migrations, +and the StreamPipes Core Service, which manages the migrations, as shown in the figure below: +
+
+When an extensions service is initiated and has successfully registered itself with the core, it proceeds to send a
+request to the core. This request includes a comprehensive list of all available migrations that have been registered
+for it. Since this collection of migrations may encompass multiple migrations that affect the same pipeline element,
+the migrations are first de-duplicated and then sorted based on their version range before being transmitted.
+
+Upon receiving these migrations, the core's actions can be categorized into two distinct parts:
+
+* Update descriptions for new elements
+* Update descriptions for existing elements
+
+### Update Descriptions for New Elements
+
+Each migration transmitted from the extensions service to the core triggers the core to update the description of the
+corresponding element stored in CouchDB. This is achieved by requesting the current configuration from the extensions
+service and subsequently overwriting the existing configuration in the storage.
+
+### Update Descriptions for Existing Elements
+
+For each migration sent from the extensions service to the core, the core conducts a thorough check to determine if any
+existing elements are affected by this migration. If such elements are identified, the extensions service is tasked with
+requesting and subsequently executing the migration on behalf of the core.
+
+In scenarios where multiple applicable migrations exist for a single pipeline element, they are sequentially applied.
+Success in this process allows the core to seamlessly update the configuration. However, if any issues arise, the
+corresponding pipeline element is halted. In the case of processors and sinks, the associated pipeline is even marked
+with a `needs attention` label, which comes apparent in the UI.
diff --git a/website-v2/versioned_docs/version-0.70.0/06_extend-sdk-output-strategies.md b/website-v2/versioned_docs/version-0.95.1/06_extend-sdk-output-strategies.md
similarity index 96%
rename from website-v2/versioned_docs/version-0.70.0/06_extend-sdk-output-strategies.md
rename to website-v2/versioned_docs/version-0.95.1/06_extend-sdk-output-strategies.md
index fb0412f8e..feb224856 100644
--- a/website-v2/versioned_docs/version-0.70.0/06_extend-sdk-output-strategies.md
+++ b/website-v2/versioned_docs/version-0.95.1/06_extend-sdk-output-strategies.md
@@ -2,7 +2,6 @@
id: extend-sdk-output-strategies
title: "SDK Guide: Output Strategies"
sidebar_label: "SDK: Output Strategies"
-original_id: extend-sdk-output-strategies
---
## Introduction
@@ -11,12 +10,12 @@ As the exact input schema of a processor is usually not yet known at development
The following reference describes how output strategies can be defined using the SDK.
-
-
+:::tip Code on Github
+
+For all examples, the code can be found on [Github](https://www.github.com/apache/streampipes-examples/tree/dev/streampipes-pipeline-elements-examples-processors-jvm/src/main/java/org/apache/streampipes/pe/examples/jvm/outputstrategy/)
+
+:::
+
## Reference
diff --git a/website-v2/versioned_docs/version-0.70.0/06_extend-sdk-static-properties.md b/website-v2/versioned_docs/version-0.95.1/06_extend-sdk-static-properties.md
similarity index 92%
rename from website-v2/versioned_docs/version-0.70.0/06_extend-sdk-static-properties.md
rename to website-v2/versioned_docs/version-0.95.1/06_extend-sdk-static-properties.md
index 51fec4c3e..39c2d5d6c 100644
--- a/website-v2/versioned_docs/version-0.70.0/06_extend-sdk-static-properties.md
+++ b/website-v2/versioned_docs/version-0.95.1/06_extend-sdk-static-properties.md
@@ -2,7 +2,6 @@
id: extend-sdk-static-properties
title: "SDK Guide: Static Properties"
sidebar_label: "SDK: Static Properties"
-original_id: extend-sdk-static-properties
---
## Introduction
@@ -11,12 +10,11 @@ Processing elements can specify required static properties, which will render di
The following reference describes how static properties can be defined using the SDK.
-Code on Github
-For all examples, the code can be found on Github.
-
-
+:::tip Code on Github
+
+For all examples, the code can be found on [Github](https://github.com/apache/streampipes-examples/tree/dev/streampipes-pipeline-elements-examples-processors-jvm/src/main/java/org/apache/streampipes/pe/examples/jvm/staticproperty).
+
+:::
## Reference
@@ -170,12 +168,12 @@ To extract the selected value, use the following method from the parameter extra
String selectedSingleValue = extractor.selectedSingleValue("id", String.class);
```
-Code on Github
-For all examples, the code can be found on Github.
-
-
+:::tip Declaring options
+
+Sometimes, you may want to use an internal name that differs from the display name of an option.
+For that, you can use the method Options.from(Tuple2{'<'}String, String{'>'}) and the extractor method selectedSingleValueInternalName.
+:::tip
### Multi-Value Selections
@@ -259,9 +257,11 @@ The UI will render a single-value parameter based on the options provided at run
The parameter extraction does not differ from the extraction of static single-value parameters.
-Declaring options
-Sometimes, you may want to use an internal name that differs from the display name of an option. -For that, you can use the method Options.from(Tuple2{'<'}String, String{'>'}) and the extractor method selectedSingleValueInternalName.
-
-
+
+:::info Multi-value selections
+
+Although this example shows the usage of runtime-resolvable selections using single value selections, the same also works for multi-value selections!
+
+:::
diff --git a/website-v2/versioned_docs/version-0.70.0/06_extend-sdk-stream-requirements.md b/website-v2/versioned_docs/version-0.95.1/06_extend-sdk-stream-requirements.md
similarity index 93%
rename from website-v2/versioned_docs/version-0.70.0/06_extend-sdk-stream-requirements.md
rename to website-v2/versioned_docs/version-0.95.1/06_extend-sdk-stream-requirements.md
index 98a9215a7..409c5164d 100644
--- a/website-v2/versioned_docs/version-0.70.0/06_extend-sdk-stream-requirements.md
+++ b/website-v2/versioned_docs/version-0.95.1/06_extend-sdk-stream-requirements.md
@@ -2,7 +2,6 @@
id: extend-sdk-stream-requirements
title: "SDK Guide: Stream Requirements"
sidebar_label: "SDK: Stream Requirements"
-original_id: extend-sdk-stream-requirements
---
## Introduction
@@ -11,12 +10,15 @@ Data processors and data sinks can define ``StreamRequirements``. Stream require
Once users create pipelines in the StreamPipes Pipeline Editor, these requirements are verified against the connected event stream.
By using this feature, StreamPipes ensures that only pipeline elements can be connected that are syntactically and semantically valid.
-This guide covers the creation of stream requirements. Before reading this section, we recommend that you make yourself familiar with the SDK guide on [data processors](dev-guide-processor-sdk.md) and [data sinks](dev-guide-sink-sdk.md).
+This guide covers the creation of stream requirements. Before reading this section, we recommend that you make yourself familiar with the SDK guide on [data processors](extend-first-processor).
+
+
+:::tip Code on Github
+
+For all examples, the code can be found on [Github](https://www.github.com/apache/streampipes-examples/tree/dev/streampipes-pipeline-elements-examples-processors-jvm/src/main/java/org/apache/streampipes/pe/examples/jvm/requirements/).
+
+:::
-Multi-value selections
-Although this example shows the usage of runtime-resolvable selections using single value selections, the same also works for multi-value selections!
-
-
## The StreamRequirementsBuilder
diff --git a/website-v2/versioned_docs/version-0.70.0/06_extend-setup.md b/website-v2/versioned_docs/version-0.95.1/06_extend-setup.md
similarity index 94%
rename from website-v2/versioned_docs/version-0.70.0/06_extend-setup.md
rename to website-v2/versioned_docs/version-0.95.1/06_extend-setup.md
index bea5d2c5f..12a96eb8e 100644
--- a/website-v2/versioned_docs/version-0.70.0/06_extend-setup.md
+++ b/website-v2/versioned_docs/version-0.95.1/06_extend-setup.md
@@ -2,7 +2,6 @@
id: extend-setup
title: Development Setup
sidebar_label: Development Setup
-original_id: extend-setup
---
Pipeline elements in StreamPipes are provided as standalone microservices. New pipeline elements can be easily developed using the provided Maven archetypes and can be installed in StreamPipes at runtime.
@@ -11,7 +10,7 @@ In this section, we describe our recommended minimum setup for locally setting u
## IDE & required dev tools
StreamPipes does not have specific requirements on the IDE - so feel free to choose the IDE of your choice.
-The only requirements in terms of development tools are that you have Java 8 and Maven installed.
+The only requirements in terms of development tools are that you have Java 17 and Maven installed.
## StreamPipes CLI: Docker-based local StreamPipes instance
In order to quickly test developed pipeline elements without needing to install all services required by StreamPipes, we provide a CLI tool that allows you to selectively start StreamPipes components.
@@ -48,4 +47,4 @@ Create the Maven archetype as described in the [Maven Archetypes](06_extend-arch
### Examples
-We provide several examples that explain the usage of some concepts in this [Github repo](https://github.com/apache/incubator-streampipes-examples).
+We provide several examples that explain the usage of some concepts in this [Github repo](https://github.com/apache/streampipes-examples).
diff --git a/website-v2/versioned_docs/version-0.95.1/06_extend-tutorial-adapters.md b/website-v2/versioned_docs/version-0.95.1/06_extend-tutorial-adapters.md
new file mode 100644
index 000000000..4e95cba9a
--- /dev/null
+++ b/website-v2/versioned_docs/version-0.95.1/06_extend-tutorial-adapters.md
@@ -0,0 +1,612 @@
+---
+id: extend-tutorial-adapters
+title: "Tutorial: Build Custom Adapters"
+sidebar_label: "Tutorial: Adapters"
+---
+
+In this tutorial, we will create a new data source consisting of a single data stream.
+By the end of the tutorial, you will be able to implement custom adapters that allow you to connect to data sources
+other than those officially supported by StreamPipes.
+To do this, we will split the tutorial into two parts.
+The [first part](#building-a-basic-adapter) focuses on creating the adapter and defining the event stream.
+At the end, we will have a working adapter that produces an event stream that can be used in StreamPipes.
+This adapter does not provide any way to configure its behavior, so in
+the [second part](#building-a-more-advanced-adapter-by-processing-ui-input) of the tutorial
+to show how we can extend our existing adapter to be configurable via the UI.
+
+:::info
+This tutorial shows how to build your own type of adapter.
+It is intended for people who are interested in extending StreamPipes to meet their own needs.
+If you are here to explore StreamPipes and are interested in using an adapter, you may want to
+continue [here](./03_use-connect.md).
+:::
+
+## Objective
+
+We are going to create an adapter that will simulate a stream of data generated by a control station in a logistics
+center that is used to sort packages.
+This station consists of two sensors: a light barrier that detects when a package passes through, and a weight sensor.
+
+This sensor produces a continuous stream of events containing the current time stamp, an indicator of whether a package
+is present or the conveyor is empty, and the weight of the package in kilograms.
+The events are published in JSON format as follows
+
+```json
+{
+ "timestamp": 1697720916959,
+ "parcelPresent": true,
+ "weight": 3.520
+}
+```
+
+In the following section, we will show you how to develop an adapter that is capable of generating this stream so that
+it is available for further processing in StreamPipes.
+
+## Project Set Up
+
+Instead of creating a new project from scratch, we recommend to use our Maven archetype to create a new project
+skeleton (`streampipes-archetype-extensions-jvm`).
+Enter the following command in a command line of your choice (please ensure
+that [Apache Maven](https://maven.apache.org/install.html) isinstalled):
+
+```bash
+mvn archetype:generate \
+-DarchetypeGroupId=org.apache.streampipes -DarchetypeArtifactId=streampipes-archetype-extensions-jvm \
+-DarchetypeVersion=0.93.0 -DgroupId=org.apache.streampipes \
+-DartifactId=streampipes-archetype-extensions-jvm -DclassNamePrefix=ParcelControlStation -DpackageName=parcelcontrol
+```
+
+This command will ask you for input twice, you can just skip both of them by hitting *enter*.
+The first dialog sets the version to use for our `streampipes-archetype-extensions-jvm` module.
+Feel free to change this if you like.
+
+```bash
+Define value for property 'version' 1.0-SNAPSHOT: :
+
+ Y: :
+```
+
+The `mvn archetype:generate` command generates some required files, the required file structure, and some boilerplate
+code.
+The generated file structure should look like the following:
+
+:::info
+Note that you can customize the parameters of the mvn command to affect the file structure and file naming.
+:::
+
+```bash
+
+|streampipes-archetype-extensions # name is determined by '-DartifactId'
+|-- development
+| |-- env
+|-- src
+| |-- main
+| | |-- java.org.apache.streampipes # name after .java. is determined by '-DgroupId'
+| | | |-- pe.parcelcontrol # name after .pe. is determined by '-DpackageName'
+| | | | |-- ParcelControlStationDataProcessor.java # class name is determined by '-DclassNamePrefix'
+| | | | |-- ParcelControlStationDataSink.java
+| | | | |-- ParcelControlStationGenericAdapter.java
+| | | | |-- ParcelControlStationSpecificAdapter.java
+| | | |-- Init.java
+| | |-- resources
+| | | |-- org.apache.streampipes.pe.parcelcontrol.genericadapter
+| | | | |-- documentation.md
+| | | | |-- icon.png
+| | | | |-- strings.en
+| | | |-- org.apache.streampipes.pe.parcelcontrol.processor
+| | | | |-- documentation.md
+| | | | |-- icon.png
+| | | | |-- strings.en
+| | | |-- org.apache.streampipes.pe.parcelcontrol.sink
+| | | | |-- documentation.md
+| | | | |-- icon.png
+| | | | |-- strings.en
+| | | |-- org.apache.streampipes.pe.parcelcontrol.specificadapter
+| | | | |-- documentation.md
+| | | | |-- icon.png
+| | | | |-- strings.en
+| |-- test.java.org.apache.streampipes # name after .java. is determined by '-DgroupId'
+| | |-- InitTest.java
+|-- Dockerfile
+|-- pom.xml
+
+```
+
+:::tip
+In addition to the basic project skeleton, the sample project also includes a sample `Dockerfile` that you can use to
+package your application into a Docker container.
+:::
+
+## Building a Basic Adapter
+
+In the following, we will demonstrate how to use the boilerplate code generated by the Maven plugin (
+see [Project setup](#project-setup)).
+Within this section, we will focus on creating an event stream that can be used within StreamPipes.
+The following section shows how to configure the created adapter with UI input.
+
+Attentive readers may have noticed that two adapter classes have been generated.
+We will focus on the `ParcelControlStationSpecificAdapter` first, the `ParcelControlStationSimulatorGenericAdapter` will
+be used later for more advanced adapter features.
+First, let us take a look at the `ParcelControlStationSpecificAdapter.java` file as generated by the Maven
+archetype.
+
+```java jsx showLineNumbers
+package org.apache.streampipes.pe.parcelcontrol;
+
+import org.apache.streampipes.commons.exceptions.connect.AdapterException;
+import org.apache.streampipes.extensions.api.connect.IAdapterConfiguration;
+import org.apache.streampipes.extensions.api.connect.IEventCollector;
+import org.apache.streampipes.extensions.api.connect.StreamPipesAdapter;
+import org.apache.streampipes.extensions.api.connect.context.IAdapterGuessSchemaContext;
+import org.apache.streampipes.extensions.api.connect.context.IAdapterRuntimeContext;
+import org.apache.streampipes.extensions.api.extractor.IAdapterParameterExtractor;
+import org.apache.streampipes.model.AdapterType;
+import org.apache.streampipes.model.connect.guess.GuessSchema;
+import org.apache.streampipes.sdk.builder.adapter.AdapterConfigurationBuilder;
+import org.apache.streampipes.sdk.builder.adapter.GuessSchemaBuilder;
+import org.apache.streampipes.sdk.helpers.Labels;
+import org.apache.streampipes.sdk.helpers.Locales;
+
+import java.util.HashMap;
+import java.util.Map;
+
+public class ParcelControlStationSpecificAdapter implements StreamPipesAdapter {
+
+ private boolean running = false;
+
+ @Override
+ public IAdapterConfiguration declareConfig() {
+ return AdapterConfigurationBuilder.create(
+ "org.apache.streampipes.pe.parcelcontrol.specificadapter",
+ ParcelControlStationSpecificAdapter::new
+ )
+ .withAssets(Assets.DOCUMENTATION, Assets.ICON)
+ .withCategory(AdapterType.Manufacturing)
+ .withLocales(Locales.EN)
+ .buildConfiguration();
+ }
+
+ @Override
+ public void onAdapterStarted(IAdapterParameterExtractor extractor,
+ IEventCollector collector,
+ IAdapterRuntimeContext adapterRuntimeContext) throws AdapterException {
+
+ Runnable demo = () -> {
+ while (running) {
+ // make event
+ MapCode on Github
-For all examples, the code can be found on Github.
-
+
+The adapter runs successfully in StreamPipes, you can now play around with the data stream that the
+adapter, or continue with the next section to learn how to make an adapter configurable through the UI.
+
+### Building a more Advanced Adapter by Processing UI Input
+
+In this section, we will extend our previous build apter by adding the ability to configure the minimum and maximum
+package
+in the UI from which the weight value is retrieved.
+The beauty of building adapters for StreamPipes is that you don't have to worry about the UI.
+StreamPipes provides a set of pre-built input elements for adapters that you can simply add to your adapter
+configuration.
+So the first thing we need to customize is `declareConfig()`:
+
+```java jsx {10-11} showLineNumbers
+@Override
+public IAdapterConfiguration declareConfig() {
+ return AdapterConfigurationBuilder.create(
+ "org.apache.streampipes.pe.parcelcontrol.specificadapter",
+ ParcelControlStationSpecificAdapter::new
+ )
+ .withAssets(Assets.DOCUMENTATION, Assets.ICON)
+ .withCategory(AdapterType.Manufacturing)
+ .withLocales(Locales.EN)
+ .requiredFloatParameter(Labels.withId("min-weight"), 0.0f)
+ .requiredFloatParameter(Labels.withId("max-weight"), 10.f)
+ .buildConfiguration();
+}
+
+```
+
+In line `9-10` we have introduced two input parameters that expect float values as input. They have a default value
+of `0` or `10` resp. The defined identifier (`min-weight` and `max-weight`) can be used two provide a caption and
+a description via the `strings.en` file:
+
+```text
+min-weight.title=Minimum Parcel Weight
+min-weight.description=The lower bound from which the weight values are sampled randomly.
+
+max-weight.title=Maximum Parcel Weight
+max-weight.description=The upper bound from which the weight values are sampled randomly.
+```
+
+As a last step, we now need to modify the calculation of the parcel weight, so that the provided parameters are actually
+applied.
+This is done in `onAdapterStarted()`.
+
+```java jsx {6-9,24} showLineNumbers
+@Override
+public void onAdapterStarted(IAdapterParameterExtractor extractor,
+ IEventCollector collector,
+ IAdapterRuntimeContext adapterRuntimeContext) throws AdapterException {
+
+ var ex = extractor.getStaticPropertyExtractor();
+
+ float minWeight = ex.singleValueParameter("min-weight", Float.class);
+ float maxWeight = ex.singleValueParameter("max-weight", Float.class);
+
+ Runnable parcelControl = () -> {
+ while (running) {
+
+ // get the current time in seconds
+ long timestamp = System.currentTimeMillis();
+ long timeInSeconds = (int) timestamp / 1000;
+
+ // make event
+ Map
+
+:::caution
+Please make sure that you uninstall the parcel adapter in `Install Pipeline Elements` before
+you restart the execution of the `Init` class, if you have already done so.
+Otherwise, the changes made in this section will have no effect.
+:::
+
+### Read More
+
+Congratulations! You've just created your first StreamPipes adapter 🎉 + +There are many more things to explore and data sources can be defined in much more detail. +If this is of interest to you, the [advanced section](#advanced) will satisfy your needs. + +If anything within this tutorial did not work for you or you had problems following it, +please feel free to provide some feedback by opening an [issue on GitHub](https://github.com/apache/streampipes/issues/new?assignees=&labels=bug%2Cdocumentation%2Cwebsite&projects=&template=doc_website_issue_report.yml). + + diff --git a/website-v2/versioned_docs/version-0.95.1/06_extend-tutorial-data-processors.md b/website-v2/versioned_docs/version-0.95.1/06_extend-tutorial-data-processors.md new file mode 100644 index 000000000..4c2d5ee25 --- /dev/null +++ b/website-v2/versioned_docs/version-0.95.1/06_extend-tutorial-data-processors.md @@ -0,0 +1,454 @@ +--- +id: extend-tutorial-data-processors +title: "Tutorial: Data Processors" +sidebar_label: "Tutorial: Data Processors" +--- + +In this tutorial, we will add a new data processor. + +From an architectural point of view, we will create a self-contained service that includes the description of the data +processor and an implementation. + +## Objective + +We are going to create a new data processor that realizes a simple geofencing algorithm - we detect vehicles that enter +a specified radius around a user-defined location. +This pipeline element will be a generic element that works with any event stream that provides geospatial coordinates in +form of a latitude/longitude pair. + +The algorithm outputs every location event once the position has entered the geofence. + +:::note + +The implementation in this tutorial is pretty simple - our processor will fire an event every time the GPS location is +inside the geofence. +In a real-world application, you would probably want to define a pattern that recognizes the _first_ event a vehicle +enters the geofence. + +This can be easily done using a CEP library. + +::: + +## Project setup + +Instead of creating a new project from scratch, we recommend to use the Maven archetype to create a new project +skeleton (streampipes-archetype-extensions-jvm). +Enter the following command in a command line of your choice (Apache Maven needs to be installed): + +``` +mvn archetype:generate \ +-DarchetypeGroupId=org.apache.streampipes -DarchetypeArtifactId=streampipes-archetype-extensions-jvm \ +-DarchetypeVersion=0.93.0 -DgroupId=my.groupId \ +-DartifactId=my-example -DclassNamePrefix=MyExample -DpackageName=mypackagename +``` + +You will see a project structure similar to the structure shown in the [archetypes](06_extend-archetypes.md) section. + +:::tip + +Besides the basic project skeleton, the sample project also includes an example Dockerfile you can use to package your +application into a Docker container. + +::: + +Now you're ready to create your first data processor for StreamPipes! + +## Adding data processor requirements + +First, we will add a new stream requirement. +Create a new class `GeofencingProcessor` which should look as follows: + +```java +package org.apache.streampipes.pe.example; + +import org.apache.streampipes.extensions.api.pe.IStreamPipesDataProcessor; +import org.apache.streampipes.extensions.api.pe.config.IDataProcessorConfiguration; +import org.apache.streampipes.extensions.api.pe.context.EventProcessorRuntimeContext; +import org.apache.streampipes.extensions.api.pe.param.IDataProcessorParameters; +import org.apache.streampipes.extensions.api.pe.routing.SpOutputCollector; +import org.apache.streampipes.model.runtime.Event; +import org.apache.streampipes.sdk.builder.ProcessingElementBuilder; +import org.apache.streampipes.sdk.builder.StreamRequirementsBuilder; +import org.apache.streampipes.sdk.builder.processor.DataProcessorConfiguration; +import org.apache.streampipes.sdk.helpers.EpProperties; +import org.apache.streampipes.sdk.helpers.EpRequirements; +import org.apache.streampipes.sdk.helpers.Labels; +import org.apache.streampipes.sdk.helpers.OutputStrategies; +import org.apache.streampipes.sdk.helpers.SupportedFormats; +import org.apache.streampipes.sdk.helpers.SupportedProtocols; +import org.apache.streampipes.vocabulary.SO; + +public class GeofencingProcessor implements IStreamPipesDataProcessor { + + private static final String LATITUDE_CENTER = "latitude-center"; + private static final String LONGITUDE_CENTER = "longitude-center"; + + + public IDataProcessorConfiguration declareConfig() { + return DataProcessorConfiguration.create( + GeofencingProcessor::new, + ProcessingElementBuilder.create( + "org.apache.streampipes.tutorial-geofencing" + ) + .category(DataProcessorType.ENRICH) + .withAssets(Assets.DOCUMENTATION, Assets.ICON) + .build()); + } + + @Override + public void onPipelineStarted(IDataProcessorParameters params, + SpOutputCollector collector, + EventProcessorRuntimeContext runtimeContext) { + + } + + @Override + public void onEvent(Event event, + SpOutputCollector collector) { + + } + + @Override + public void onPipelineStopped() { + + } +} + + +``` + +In this class, we need to implement three methods: The `declareConfig` method is used to define abstract stream +requirements such as event properties that must be present in any input stream that is later connected to the element +using the StreamPipes UI. +The second method, `onPipelineStarted` is triggered once a pipeline is started. +The `onEvent` method is called for every incoming event. +Finally, the `onPipelineStopped` method is called once the pipeline is stopped. + +Similar to data sources, the SDK provides a builder class to generate the description for data processors. + +The current code within the `declareConfig` method creates a new data processor with the ID. +The ID is used as the internal ID of the data processor, but also used to reference additional assets in the `resources` folder, such as a `strings.en` file, used to configure labels and description, and a `documentation.md` file, which will later servce as a markdown documentation in the UI. +But first, we will add some _stream requirements_ to the description. As we'd like to develop a generic pipeline element that +works with any event that provides a lat/lng pair, we define two stream requirements as stated below: + +```java +.requiredStream(StreamRequirementsBuilder + .create() + .requiredPropertyWithUnaryMapping( + EpRequirements.domainPropertyReq(Geo.LAT), + Labels.from("latitude-field","Latitude","The event property containing the latitude value"), + PropertyScope.MEASUREMENT_PROPERTY + ) + .requiredPropertyWithUnaryMapping( + EpRequirements.domainPropertyReq(Geo.LNG), + Labels.from("longitude-field","Longitude","The event property containing the longitude value"), + PropertyScope.MEASUREMENT_PROPERTY + ) + .build()) +``` + +The first line, `.requiredStream()` defines that we want a data processor with exactly one input stream. Adding more +stream requirements would create elements with multiple input connectors in StreamPipes. +Stream requirements can be assigned by using the `StreamRequirementsBuilder` class. +In our example, we define two requirements, so-called _domain property requirements_. In contrast to _data type +requirements_ where we'd expect an event property with a field of a specific data type (e.g., float), domain property +requirements expect a specific semantic type (called domain property), e.g., from a vocabulary such as the WGS84 Geo vocab. + +Once a pipeline is deployed, we are interested in the actual field (and its field name) that contains the latitude and +longitude values. +In some cases, there might be more than one field that satisfies a property requirement, and we would like users to +select the property the geofencing component should operate on. +Therefore, our example uses the method `requiredPropertyWithUnaryMapping`, which will map a requirement to a real event +property of an input stream and let the user choose the appropriate field in the StreamPipes UI when pipelines are +defined. + +Finally, the `PropertyScope` indicates that the required property is a measurement value (in contrast to a dimension +value). This allows us later to provide improved user guidance in the pipeline editor. + +Similar to mapping properties, text parameters have an internalId (radius), a label and a description. +In addition, we can assign a _value specification_ to the parameter indicating the value range we support. +Our example supports a radius value between 0 and 1000 with a granularity of 1. +In the StreamPipes UI, a required text parameter is rendered as a text input field, in case we provide an optional value +specification, a slider input is automatically generated. + +For now, we've assigned parameters with an internal ID, a label and a description. +To decouple human-readable labels and description from the actual data processor description, it is possible to extract the strings to a properties file. +In the `resources` folder, switch to a folder with the same name as the data processor's ID. If you've used the Maven archetype to build our project, there should be a `strings.en` file. +In this file, we can configure labels and descriptions. For instance, instead of writing + +```java + +.requiredPropertyWithUnaryMapping( + EpRequirements.domainPropertyReq(Geo.LAT), + Labels.from("latitude-field","Latitude","The event property containing the latitude value"), + PropertyScope.MEASUREMENT_PROPERTY + ) + +``` + +it is recommended to write + +```java + +.requiredPropertyWithUnaryMapping( + EpRequirements.domainPropertyReq(Geo.LAT), + Labels.withId("latitude-field"), + PropertyScope.MEASUREMENT_PROPERTY + ) + +``` + +and add the following line to the `strings.en` file: + +```properties + +latitude-field.title=Latitude +latitute-field.description=The event property containing the latitude value + +``` + +This feature will also ease future internationalization efforts. + +Besides requirements, users should be able to define the center coordinate of the Geofence and the size of the fence +defined as a radius around the center in meters. +The radius can be defined by adding a simple required text field to the description: + +```java +.requiredIntegerParameter("radius","Geofence Size","The size of the circular geofence in meters.",0,1000,1) +``` + +Such user-defined parameters are called _static properties_. There are many different types of static properties (see +the [Processor SDK](06_extend-sdk-static-properties.md) for an overview). Similar to stream requirements, it is also recommended to type `Labels.withId("radius")` and move labels and descriptions to the resource file. + +In this example, we'll further add two very simple input fields to let users provide latitude and longitude of the +geofence center. + +Add the following line to the `declareConfig` method: + +```java + .requiredFloatParameter(Labels.from(LATITUDE_KEY,"Latitude","The latitude value")) + .requiredFloatParameter(Labels.from(LONGITUDE_KEY,"Longitude","The longitude value")) + +``` + +Now we need to define the output of our Geofencing pipeline element. +As explained in the first section, the element should fire every time some geo-located entity arrives within the defined +geofence. +Therefore, the processor outputs the same schema as it receives as an input. +Although we don't know the exact input right now as it depends on the stream users connect in StreamPipes when creating +pipelines, we can define an _output strategy_ as follows: + +```java +.outputStrategy(OutputStrategies.keep()) +``` + +This defines a _KeepOutputStrategy_, i.e., the input event schema is not modified by the processor. +There are many more output strategies you can define depending on the functionality you desire, e.g., _AppendOutput_ for +defining a processor that enriches events or _CustomOutput_ in case you would like users to select the output by +themselves. + +That's it! We've now defined input requirements, required user input and an output strategy. +In the next section, you will learn how to extract these parameters once the pipeline element is invoked after a +pipeline was created. + +## Pipeline element invocation + +Once users start a pipeline that uses our geofencing component, the _onPipelineStarted_ method in our class is called. The +interface `IDataProcessorParameters` includes convenient access to user-configured parameters a users has selected in the pipeline +editor and information on the actual streams that are connected to the pipeline element. + +Next, we are interested in the fields of the input event stream that contains the latitude and longitude value we would +like to compute against the geofence center location as follows: + +```java + String latitudeFieldName = params.extractor().mappingPropertyValue("latitude-field"); + String longitudeFieldName = params.extractor().mappingPropertyValue("longitude-field"); +``` + +We use the same `internalId` we've used to define the mapping property requirements in the `declareModel` method. + +Next, for extracting the geofence center coordinates, add to class variables centerLatitude and centerLongitude and +assign the selected values using the following statements: + +```java + this.centerLatitude = params.extractor().singleValueParameter(LATITUDE_CENTER,Float.class); + this.centerLongitude = params.extractor().singleValueParameter(LONGITUDE_CENTER,Float.class); +``` + +The radius value can be extracted as follows: + +```java + int radius = params.extractor().singleValueParameter("radius",Float.class); +``` + +Great! That's all we need to describe a data processor for usage in StreamPipes. Your processor class should look as +follows: + +```java + +package org.apache.streampipes.pe.example; + +import org.apache.streampipes.extensions.api.pe.IStreamPipesDataProcessor; +import org.apache.streampipes.extensions.api.pe.config.IDataProcessorConfiguration; +import org.apache.streampipes.extensions.api.pe.context.EventProcessorRuntimeContext; +import org.apache.streampipes.extensions.api.pe.param.IDataProcessorParameters; +import org.apache.streampipes.extensions.api.pe.routing.SpOutputCollector; +import org.apache.streampipes.model.runtime.Event; +import org.apache.streampipes.sdk.builder.ProcessingElementBuilder; +import org.apache.streampipes.sdk.builder.StreamRequirementsBuilder; +import org.apache.streampipes.sdk.builder.processor.DataProcessorConfiguration; +import org.apache.streampipes.sdk.helpers.EpProperties; +import org.apache.streampipes.sdk.helpers.EpRequirements; +import org.apache.streampipes.sdk.helpers.Labels; +import org.apache.streampipes.sdk.helpers.OutputStrategies; +import org.apache.streampipes.sdk.helpers.SupportedFormats; +import org.apache.streampipes.sdk.helpers.SupportedProtocols; +import org.apache.streampipes.vocabulary.SO; + +public class GeofencingProcessor implements IStreamPipesDataProcessor { + + private static final String LATITUDE_CENTER = "latitude-center"; + private static final String LONGITUDE_CENTER = "longitude-center"; + + private float centerLatitude; + private float centerLongitude; + private String latitudeFieldName; + private String longitudeFieldName; + + private int radius; + + public IDataProcessorConfiguration declareConfig() { + return DataProcessorConfiguration.create( + GeofencingProcessor::new, + ProcessingElementBuilder.create("org.streampipes.tutorial-geofencing") + .category(DataProcessorType.ENRICH) + .withAssets(Assets.DOCUMENTATION, Assets.ICON) + .withLocales(Locales.EN) + .requiredStream(StreamRequirementsBuilder + .create() + .requiredPropertyWithUnaryMapping(EpRequirements.domainPropertyReq(Geo.lat), + Labels.from("latitude-field", "Latitude", "The event " + + "property containing the latitude value"), PropertyScope.MEASUREMENT_PROPERTY) + .requiredPropertyWithUnaryMapping(EpRequirements.domainPropertyReq(Geo.lng), + Labels.from("longitude-field", "Longitude", "The event " + + "property containing the longitude value"), PropertyScope.MEASUREMENT_PROPERTY) + .build()) + .outputStrategy(OutputStrategies.keep()) + .requiredIntegerParameter("radius", "Geofence Size", "The size of the circular geofence in meters.", 0, 1000, 1) + .requiredFloatParameter(Labels.from(LATITUDE_CENTER, "Latitude", "The latitude value")) + .requiredFloatParameter(Labels.from(LONGITUDE_CENTER, "Longitude", "The longitude value")) + .build() + ); + } + + @Override + public void onPipelineStarted(IDataProcessorParameters params, + SpOutputCollector collector, + EventProcessorRuntimeContext runtimeContext) { + this.centerLatitude = params.extractor().singleValueParameter(LATITUDE_CENTER, Float.class); + this.centerLongitude = params.extractor().singleValueParameter(LONGITUDE_CENTER, Float.class); + this.latitudeFieldName = params.extractor().mappingPropertyValue("latitude-field"); + this.longitudeFieldName = params.extractor().mappingPropertyValue("longitude-field"); + this.radius = params.extractor().singleValueParameter("radius", Integer.class); + } + + @Override + public void onEvent(Event event, + SpOutputCollector collector) { + + } + + @Override + public void onPipelineStopped() { + + } +} + +``` + +## Adding an implementation + +Everything we need to do now is to add an implementation. + +Add the following piece of code to the onEvent method, which realizes the Geofencing functionality: + +```java + + @Override + public void onEvent(Event event, + SpOutputCollector collector) { + float latitude = event.getFieldBySelector(latitudeFieldName).getAsPrimitive().getAsFloat(); + float longitude = event.getFieldBySelector(longitudeFieldName).getAsPrimitive().getAsFloat(); + + float distance = distFrom(latitude,longitude, centerLatitude, centerLongitude); + + if(distance <= radius){ + collector.collect(event); + } + } + + public static float distFrom(float lat1, float lng1, float lat2, float lng2) { + double earthRadius = 6371000; + double dLat = Math.toRadians(lat2-lat1); + double dLng = Math.toRadians(lng2-lng1); + double a = Math.sin(dLat/2)*Math.sin(dLat/2) + + Math.cos(Math.toRadians(lat1))*Math.cos(Math.toRadians(lat2)) * + Math.sin(dLng/2)*Math.sin(dLng/2); + + double c = 2*Math.atan2(Math.sqrt(a),Math.sqrt(1-a)); + + return(float)(earthRadius*c); + } +``` + +We won't go into details here as this isn't StreamPipes-related code, but in general the class extracts latitude and +longitude fields from the input event (which is provided as a map data type) and calculates the distance between the +geofence center and these coordinates. +If the distance is below the given radius, the event is forwarded to the next operator. + +See the [event model](06_extend-sdk-event-model.md) guide to learn how to extract parameters from events. + +## Registering the pipeline element + +The final step is to register the data processor in the `Init` method. Add the following line to +the `SpServiceDefinitionBuilder`: + +```java + .registerPipelineElement(new GeofencingProcessor()) +``` + +## Starting the service + +:::tip + +Once you start the service, it will register in StreamPipes with the hostname. The hostname will be auto-discovered and +should work out-of-the-box. +In some cases, the detected hostname is not resolvable from within a container (where the core is running). In this +case, provide a SP_HOST environment variable to override the auto-discovery. + +::: + +:::tip + +The default port of all pipeline element services as defined in the `create` method is port 8090. +If you'd like to run multiple services at the same time on your development machine, change the port here. As an +alternative, you can also provide an env variable `SP_PORT` which overrides the port settings. This is useful to use +different configs for dev and prod environments. + +::: + +Now we are ready to start our service! + +Configure your IDE to provide an environment variable called ``SP_DEBUG`` with value ``true`` when starting the project. + +Execute the main method in the class `Init` we've just created. + +The service automatically registers itself in StreamPipes. +To install the just created element, open the StreamPipes UI and follow the manual provided in +the [user guide](03_use-install-pipeline-elements.md). + +## Read more + +Congratulations! You've just created your first data processor for StreamPipes. +There are many more things to explore and data processors can be defined in much more detail using multiple wrappers. +Follow our [SDK guide](06_extend-sdk-static-properties.md) to see what's possible! diff --git a/website-v2/versioned_docs/version-0.95.1/06_extend-tutorial-data-sinks.md b/website-v2/versioned_docs/version-0.95.1/06_extend-tutorial-data-sinks.md new file mode 100644 index 000000000..09baeff71 --- /dev/null +++ b/website-v2/versioned_docs/version-0.95.1/06_extend-tutorial-data-sinks.md @@ -0,0 +1,272 @@ +--- +id: extend-tutorial-data-sinks +title: "Tutorial: Data Sinks" +sidebar_label: "Tutorial: Data Sinks" +--- + +In this tutorial, we will add a new data sink using the standalone wrapper. + +From an architectural point of view, we will create a self-contained service that includes the description of the data +sink and a corresponding implementation. + +## Objective + +We are going to create a new data sink that calls an external HTTP endpoint to forward data to an external service. + +For each incoming event, an external service is invoked using an HTTP POST request. In this example, we'll call an +endpoint provided by [RequestBin](https://requestbin.com/). +To setup your own endpoint, go to [https://requestbin.com/](https://requestbin.com/) and click "Create a request bin". +Copy the URL of the newly created endpoint. + +## Project setup + +Instead of creating a new project from scratch, we recommend to use the Maven archetype to create a new project +skeleton (streampipes-archetype-extensions-jvm). +Enter the following command in a command line of your choice (Apache Maven needs to be installed): + +``` +mvn archetype:generate -DarchetypeGroupId=org.apache.streampipes \ +-DarchetypeArtifactId=streampipes-archetype-extensions-jvm -DarchetypeVersion=0.93.0 \ +-DgroupId=org.streampipes.tutorial -DartifactId=sink-tutorial -DclassNamePrefix=Rest -DpackageName=mypackage +``` + +You will see a project structure similar to the structure shown in the [archetypes](06_extend-archetypes.md) section. + +:::tip + +Besides the basic project skeleton, the sample project also includes an example Dockerfile you can use to package your +application into a Docker container. + +::: + +Now you're ready to create your first data sink for StreamPipes! + +## Adding data sink requirements + +First, we will add a new stream requirement. +Create a class `RestSink` which should look as follows: + +```java +package org.apache.streampipes.pe.example; + +import org.apache.streampipes.extensions.api.pe.IStreamPipesDataSink; +import org.apache.streampipes.extensions.api.pe.config.IDataSinkConfiguration; +import org.apache.streampipes.extensions.api.pe.context.EventSinkRuntimeContext; +import org.apache.streampipes.extensions.api.pe.param.IDataSinkParameters; +import org.apache.streampipes.model.DataSinkType; +import org.apache.streampipes.model.runtime.Event; +import org.apache.streampipes.model.schema.PropertyScope; +import org.apache.streampipes.sdk.builder.DataSinkBuilder; +import org.apache.streampipes.sdk.builder.StreamRequirementsBuilder; +import org.apache.streampipes.sdk.builder.sink.DataSinkConfiguration; +import org.apache.streampipes.sdk.helpers.EpRequirements; +import org.apache.streampipes.sdk.helpers.Labels; +import org.apache.streampipes.sdk.helpers.Locales; +import org.apache.streampipes.sdk.utils.Assets; + +public class RestSink implements IStreamPipesDataSink { + + @Override + public IDataSinkConfiguration declareConfig() { + return DataSinkConfiguration.create( + RestSink::new, + DataSinkBuilder.create("org.apache.streampipes.tutorial.pe.sink.rest") + .category(DataSinkType.NOTIFICATION) + .withAssets(Assets.DOCUMENTATION, Assets.ICON) + .withLocales(Locales.EN) + .requiredStream(StreamRequirementsBuilder + .create() + .requiredPropertyWithNaryMapping(EpRequirements.anyProperty(), Labels.withId( + "fields-to-send"), PropertyScope.NONE) + .build()) + .build() + ); + } + + @Override + public void onPipelineStarted(IDataSinkParameters params, + EventSinkRuntimeContext eventSinkRuntimeContext) { + + } + + @Override + public void onEvent(Event event) { + + } + + @Override + public void onPipelineStopped() { + + } + + +``` + +In this class, we need to implement three methods: The `declareConfig` method is used to define abstract stream +requirements such as event properties that must be present in any input stream that is later connected to the element +using the StreamPipes UI. +The second method, `onPipelineStarted` is called once a pipeline using this sink is started. The third method, `onEvent`, is +called for every incoming event. + +The `DataSinkBuilder` within the ``declareConfig`` method describes the properties of our data sink: + +* ``category`` defines a category for this sink. +* ``withAssets`` denotes that we will provide an external documentation file and an icon, which can be found in + the ``resources`` folder +* ``withLocales`` defines that we will provide an external language file, also available in the ``resources`` folder +* ``requiredStream`` defines requirements any input stream connected to this sink must provide. In this case, we do not + have any specific requirements, we just forward all incoming events to the REST sink. However, we want to let the user + display a list of available fields from the connected input event, where users can select a subset. This is defined by + defining a Mapping from the empty requirement. This will later on render a selection dialog in the pipeline editor. + +The ``onPipelineStarted`` method is called when a pipeline containing the sink is started. Once a pipeline is started, we +would like to extract user-defined parameters. +In this example, we simply extract the fields selected by users that should be forwarded to the REST sink. Finally, we +return a new configured event sink containing the parameters. + +## Pipeline element invocation + +Once users start a pipeline that uses our geofencing component, the _onInvocation_ method in our class is called. The +interface `IDataSinkParameters` includes methods to extract the configuration parameters a user has selected in +the pipeline editor and information on the actual streams that are connected to the pipeline element. + +## Adding an implementation + +Now we'll add a proper implementation (i.e., the Rest call executed for every incoming event) to the following methods: + +Our final class should look as follows: + +```java +package org.apache.streampipes.pe.example; + +import org.apache.streampipes.commons.exceptions.SpRuntimeException; +import org.apache.streampipes.dataformat.SpDataFormatDefinition; +import org.apache.streampipes.dataformat.json.JsonDataFormatDefinition; +import org.apache.streampipes.extensions.api.pe.IStreamPipesDataSink; +import org.apache.streampipes.extensions.api.pe.config.IDataSinkConfiguration; +import org.apache.streampipes.extensions.api.pe.context.EventSinkRuntimeContext; +import org.apache.streampipes.extensions.api.pe.param.IDataSinkParameters; +import org.apache.streampipes.model.DataSinkType; +import org.apache.streampipes.model.runtime.Event; +import org.apache.streampipes.model.schema.PropertyScope; +import org.apache.streampipes.sdk.builder.DataSinkBuilder; +import org.apache.streampipes.sdk.builder.StreamRequirementsBuilder; +import org.apache.streampipes.sdk.builder.sink.DataSinkConfiguration; +import org.apache.streampipes.sdk.helpers.EpRequirements; +import org.apache.streampipes.sdk.helpers.Labels; +import org.apache.streampipes.sdk.helpers.Locales; +import org.apache.streampipes.sdk.utils.Assets; + +import com.google.common.base.Charsets; +import org.apache.http.client.fluent.Request; +import org.apache.http.entity.StringEntity; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.io.IOException; +import java.util.List; +import java.util.Map; + +public class RestSink implements IStreamPipesDataSink { + + private static final Logger LOG = LoggerFactory.getLogger(RestSink.class); + + private static final String REST_ENDPOINT_URI = YOUR_REQUEST_BIN_URL; + private List
+
+Apache StreamPipes implements a microservice architecture as shown in the figure above.
+
+## StreamPipes Core
+
+The StreamPipes Core is the central component to manage all StreamPipes resources.
+It delegates the management of adapters, pipeline elements, pipelines and functions to registered extensions services (see below) and monitors the execution of extensions.
+The Core also provides internal REST interfaces to communicate with the user interface, as well as public REST interfaces that can be used by external applications and StreamPipes clients.
+
+Configuration and user data are stored in an Apache CouchDB database.
+
+## StreamPipes Extensions
+
+An Apache StreamPipes extensions service is a microservice which contains the implementation of specific adapters, data streams, data processors, data sinks and functions.
+Multiple extension services can be part of a single StreamPipes installation.
+Each service might provide its own set of extensions. Extensions services register at the StreamPipes Core at startup. Users are able to install all or a subset of extensions of each service.
+This allows StreamPipes to be extended at runtime by starting a new service with additional extensions.
+
+Extensions can be built using the SDK (see [Extending StreamPipes](06_extend-setup.md)).
+Extensions services can be provided either in Java or in Python.
+
+:::info
+
+As of version 0.93.0, the Python SDK supports functions only. If you would like to develop pipeline elements in Python as well, let us know in a [Github discussions](https://github.com/apache/streampipes/discussions) comment, so that we can better prioritize development.
+
+:::
+
+
+An extensions service interacts with the core by receiving control messages to invoke or detach an extension.
+In addition, the core regularly fetches monitoring and log data from each registered extensions service.
+
+
+## StreamPipes Client
+
+The Apache StreamPipes Client is a lightweight library for Java and Python which can be used to interact with StreamPipes resources programmatically.
+For instance, users use the client to influence the control flow of pipelines, to download raw data from the data lake APIs or to realize custom applications with live data.
+
+
+## Third-party systems
+
+In addition to the core components, an Apache StreamPipes version uses several third-party services, which are part of the standard installation.
+
+* Configurations and user data is stored in an [Apache CouchDB](https://couchdb.apache.org) database.
+* Time-series data is stored in an [InfluxDB](https://github.com/influxdata/influxdb) database.
+* Events are exchanged over a messaging system. Users can choose from various messaging systems that StreamPipes supports. Currently, we support [Apache Kafka](https://kafka.apache.org), [Apache Pulsar](https://pulsar.apache.org), [MQTT](https://mqtt.org/) and [NATS](https://nats.io/). The selection of the right messaging system depends on the use case. See [Messaging](07_technicals-messaging.md) for more information.
+
+:::info
+
+Versions prior to 0.93.0 included Consul for service discovery and registration. Starting from 0.93.0 onwards, we switched to an internal service discovery mechanism.
+
+:::
+
+All mentioned third-party services are part of the default installation and are auto-configured during the installation process.
+
+## Programming Languages
+
+Apache StreamPipes is mainly written in Java.
+Services are based on Spring Boot.
+The included [Python integration](https://streampipes.apache.org/docs/docs/python/latest/) is written in Python.
+
+The user interface is mainly written in TypeScript using the Angular framework.
+
+
+## Data Model
+
+Internally, Apache StreamPipes realizes a stream processing layer where events are continuously exchanged over a messaging system.
+When building a pipeline, data processors consume data from a topic assigned by the core and publish data back to another topic, which is also assigned by the core.
+
+At runtime, events have a flat and easily understandable data structure, consisting of key/value pairs. Events are serialized in JSON, although StreamPipes can be configured to use other (binary) message formats.
+
+This allows for easy integration with other systems which want to consume data from Streampipes, since an event could look as simple as this:
+
+```json
+{
+ "timestamp": 1234556,
+ "deviceId": "ABC",
+ "temperature": 37.5
+}
+```
+
+However, this wouldn't be very expressive, right? To [assist users](07_technicals-user-guidance.md), StreamPipes provides a rich description layer for events. So under the hood, for the `temperature` field shown above StreamPipes can also store the following:
+
+```json
+{
+ "label": "Temperature",
+ "description": "Measures the temperature during leakage tests",
+ "measurementUnit": "https://qudt.org/vocab/unit/DEG_C",
+ "runtimeName": "temperature",
+ "runtimeType": "xsd:float",
+ "semanticType": "https://my-company-vocabulary/leakage-test-temperature"
+}
+```
+
+By dividing the description layer from the runtime representation, we get a good trade-off between expressivity, readability for humans and lightweight runtime message formats.
+The schema is stored in an internal schema registry and available to the client APIs and user interface views to improve validation and user guidance.
+
+StreamPipes also supports arrays and nested structures, although we recommend using flat events where possible to ease integration with downstream systems (such as time-series storage).
+
+
+
+
diff --git a/website-v2/versioned_docs/version-0.95.1/07_technicals-messaging.md b/website-v2/versioned_docs/version-0.95.1/07_technicals-messaging.md
new file mode 100644
index 000000000..d5308a6d8
--- /dev/null
+++ b/website-v2/versioned_docs/version-0.95.1/07_technicals-messaging.md
@@ -0,0 +1,65 @@
+---
+id: technicals-messaging
+title: Messaging
+sidebar_label: Messaging
+---
+
+## Architecture
+
+To exchange messages at runtime between individual [Extensions Services](07_technicals-architecture.md), StreamPipes uses external messaging systems.
+This corresponds to an event-driven architecture with a central message broker and decoupled services which consume and produce events from the messaging system.
+
+There are many different open source messaging systems on the market, which each have individual strengths.
+To provide a flexible system which matches different needs, StreamPipes can be configured to use various messaging systems.
+
+## Supported messaging systems
+
+The following messaging systems are currently supported:
+
+* Apache Kafka
+* Apache Pulsar
+* MQTT
+* NATS
+
+## Configure StreamPipes to use another messaging system
+
+Configuring StreamPipes for one of these messaging system is an installation-time configuration.
+We currently do not recommend to change the configuration at runtime.
+
+The protocol can be configured with the environment variable `SP_PRIORITIZED_PROTOCOL` assigned to the core with one of the following values:
+
+```bash
+SP_PRIORITIZED_PROTOCOL=kafka # Use Kafka as protocol
+SP_PRIORITIZED_PROTOCOL=pulsar # Use Pulsar as protocol
+SP_PRIORITIZED_PROTOCOL=mqtt # Use MQTT as protocol
+SP_PRIORITIZED_PROTOCOL=nats # Use NATS as protocol
+```
+
+Note that each extension service can support an arbitrary number of protocols. For instance, you can have a lightweight extension service which only supports NATS, but have another, cloud-centered service which supports Kafka, both registered at the Core.
+To select a protocol when multiple protocols are supported by two pipeline elements, StreamPipes selects a protocol based on a priority, which can be configured in the [Configuration View](03_use-configurations.md).
+StreamPipes ensures that only pipeline elements which have a commonly supported protocol can be connected.
+
+Note that you might need to change the installation files. For the `Docker-Compose` based installation, we provide various compose file for different messaging setups. For the `Kubernetes` installation, we provide variables which can be set in the helm chart's `values.yaml` file.
+
+### Configure broker addresses
+
+By default, StreamPipes assumes that the messaging system is started from its own environment, e.g., the system configured in the selected `Docker-Compose` file.
+
+Besides that, it is also possible to let StreamPipes connect to an externally provided messaging system. For this purpose, various environment variables exist.
+
+* `SP_PRIORITIZED_PROTOCOL` to set the prioritized protocol to either `kafka`, `mqtt`, `nats` or `pulsar`
+
+* `SP_KAFKA_HOST`, `SP_KAFKA_PORT` to configure Kafka access
+* `SP_MQTT_HOST`, `SP_MQTT_PORT` to configure MQTT access
+* `SP_NATS_HOST`, `SP_NATS_PORT` to configure NATS access
+* `SP_PULSAR_URL` to configure Pulsar access
+
+
+Most settings can also be set in the UI under `Settings->Messaging`.
+
+:::warning Installation-time configurations
+Although it is currently possible to change messaging settings in the user interface, we do not support dynamic modification of messaging systems.
+Choosing a proper system is considered an installation-time setting which should not be changed afterwards.
+Already existing Adapters and pipeline elements are not properly updated after changes of the messaging layer.
+:::
+
diff --git a/website-v2/versioned_docs/version-0.95.1/07_technicals-runtime-wrappers.md b/website-v2/versioned_docs/version-0.95.1/07_technicals-runtime-wrappers.md
new file mode 100644
index 000000000..9cebadfd2
--- /dev/null
+++ b/website-v2/versioned_docs/version-0.95.1/07_technicals-runtime-wrappers.md
@@ -0,0 +1,37 @@
+---
+id: technicals-runtime-wrappers
+title: Runtime Wrappers
+sidebar_label: Runtime Wrappers
+---
+
+## Overview
+
+In general, StreamPipes has an exchangeable runtime layer, e.g., the actual processing of incoming events can be delegated to a third-party stream processing system such as Kafka Streams or Apache Flink.
+
+The default runtime wrapper is the StreamPipes Native Wrapper, called the `StandaloneWrapper`.
+
+Although not recommended for production, we invite interested developers to check out our experimental wrappers:
+
+* Kafka Streams runtime wrapper at [https://github.com/apache/streampipes/tree/dev/streampipes-wrapper-kafka-streams](https://github.com/apache/streampipes/tree/dev/streampipes-wrapper-kafka-streams)
+* Apache Flink runtime wrapper at [https://github.com/apache/streampipes/tree/dev/streampipes-wrapper-flink](https://github.com/apache/streampipes/tree/dev/streampipes-wrapper-flink)
+
+## Assigning a runtime wrapper to an extension service
+
+Runtime wrappers can be assigned in the `Service Definition` of the `Init` class of an extension service:
+
+```java
+
+ @Override
+ public SpServiceDefinition provideServiceDefinition(){
+ return SpServiceDefinitionBuilder.create("org.apache.streampipes.extensions.all.jvm",
+ "StreamPipes Extensions (JVM)",
+ "",8090)
+ ...
+ .registerRuntimeProvider(new StandaloneStreamPipesRuntimeProvider())
+ ...
+ .build();
+ }
+
+```
+
+Please let us know through our communication channels if you are interested in this feature and if you are willing to contribute!
diff --git a/website-v2/versioned_docs/version-0.70.0/07_technicals-user-guidance.md b/website-v2/versioned_docs/version-0.95.1/07_technicals-user-guidance.md
similarity index 70%
rename from website-v2/versioned_docs/version-0.70.0/07_technicals-user-guidance.md
rename to website-v2/versioned_docs/version-0.95.1/07_technicals-user-guidance.md
index 0141dabc1..697411861 100644
--- a/website-v2/versioned_docs/version-0.70.0/07_technicals-user-guidance.md
+++ b/website-v2/versioned_docs/version-0.95.1/07_technicals-user-guidance.md
@@ -2,7 +2,6 @@
id: technicals-user-guidance
title: User Guidance
sidebar_label: User Guidance
-original_id: technicals-user-guidance
---
tbd
\ No newline at end of file
diff --git a/website-v2/versioned_docs/version-0.70.0/08_debugging.md b/website-v2/versioned_docs/version-0.95.1/08_debugging.md
similarity index 70%
rename from website-v2/versioned_docs/version-0.70.0/08_debugging.md
rename to website-v2/versioned_docs/version-0.95.1/08_debugging.md
index 33bedad41..95892c175 100644
--- a/website-v2/versioned_docs/version-0.70.0/08_debugging.md
+++ b/website-v2/versioned_docs/version-0.95.1/08_debugging.md
@@ -2,7 +2,6 @@
id: debugging-debugging
title: Debugging
sidebar_label: Debugging
-original_id: debugging-debugging
---
tbd
\ No newline at end of file
diff --git a/website-v2/versioned_docs/version-0.70.0/08_monitoring.md b/website-v2/versioned_docs/version-0.95.1/08_monitoring.md
similarity index 70%
rename from website-v2/versioned_docs/version-0.70.0/08_monitoring.md
rename to website-v2/versioned_docs/version-0.95.1/08_monitoring.md
index 0712b98e9..6680b5d86 100644
--- a/website-v2/versioned_docs/version-0.70.0/08_monitoring.md
+++ b/website-v2/versioned_docs/version-0.95.1/08_monitoring.md
@@ -2,7 +2,6 @@
id: debugging-monitoring
title: Monitoring
sidebar_label: Monitoring
-original_id: debugging-monitoring
---
tbd
\ No newline at end of file
diff --git a/website-v2/versioned_docs/version-0.70.0/09_contribute.md b/website-v2/versioned_docs/version-0.95.1/09_contribute.md
similarity index 96%
rename from website-v2/versioned_docs/version-0.70.0/09_contribute.md
rename to website-v2/versioned_docs/version-0.95.1/09_contribute.md
index 43ea625f3..119568929 100644
--- a/website-v2/versioned_docs/version-0.70.0/09_contribute.md
+++ b/website-v2/versioned_docs/version-0.95.1/09_contribute.md
@@ -2,7 +2,6 @@
id: community-contribute
title: Contribute
sidebar_label: Contribute
-original_id: community-contribute
---
## Contribute
diff --git a/website-v2/versioned_docs/version-0.70.0/09_get-help.md b/website-v2/versioned_docs/version-0.95.1/09_get-help.md
similarity index 88%
rename from website-v2/versioned_docs/version-0.70.0/09_get-help.md
rename to website-v2/versioned_docs/version-0.95.1/09_get-help.md
index 0e564825f..077f0b62f 100644
--- a/website-v2/versioned_docs/version-0.70.0/09_get-help.md
+++ b/website-v2/versioned_docs/version-0.95.1/09_get-help.md
@@ -2,7 +2,6 @@
id: community-get-help
title: Get Help
sidebar_label: Get Help
-original_id: community-get-help
---
The Apache StreamPipes community is happy to help with any questions or problems you might have.
@@ -12,7 +11,7 @@ Subscribe to our user mailing list to ask a question.
[Mailing Lists](https://streampipes.apache.org/mailinglists.html)
-To subscribe to the user list, send an email to [users-subscribe@streampipes.apache.org](mailto:users-subscribe@streampipes.apache.org)
+To subscribe to the user list, send an email to [users-subscribe@streampipes.apache.org](users-subscribe@streampipes.apache.org)
You can also ask questions on our Github discussions page:
[Github Discussions](https://github.com/apache/streampipes/discussions)
diff --git a/website-v2/versioned_docs/version-0.70.0/faq-common-problems.md b/website-v2/versioned_docs/version-0.95.1/faq-common-problems.md
similarity index 99%
rename from website-v2/versioned_docs/version-0.70.0/faq-common-problems.md
rename to website-v2/versioned_docs/version-0.95.1/faq-common-problems.md
index c7a61f147..14195c0f0 100644
--- a/website-v2/versioned_docs/version-0.70.0/faq-common-problems.md
+++ b/website-v2/versioned_docs/version-0.95.1/faq-common-problems.md
@@ -2,7 +2,6 @@
id: faq-common-problems
title: Common Problems
sidebar_label: Common Problems
-original_id: faq-common-problems
---
* Windows 10: Consul, Kafka, Zookeeper, or Kafka-Rest did not start
diff --git a/website-v2/versioned_docs/version-0.70.0/pe/org.apache.streampipes.connect.adapters.image.stream.md b/website-v2/versioned_docs/version-0.95.1/pe/org.apache.streampipes.connect.adapters.image.stream.md
similarity index 94%
rename from website-v2/versioned_docs/version-0.70.0/pe/org.apache.streampipes.connect.adapters.image.stream.md
rename to website-v2/versioned_docs/version-0.95.1/pe/org.apache.streampipes.connect.adapters.image.stream.md
index 03d8f81cf..a03806d9e 100644
--- a/website-v2/versioned_docs/version-0.70.0/pe/org.apache.streampipes.connect.adapters.image.stream.md
+++ b/website-v2/versioned_docs/version-0.95.1/pe/org.apache.streampipes.connect.adapters.image.stream.md
@@ -2,7 +2,6 @@
id: org.apache.streampipes.connect.adapters.image.stream
title: Image Upload (Stream)
sidebar_label: Image Upload (Stream)
-original_id: org.apache.streampipes.connect.adapters.image.stream
---
+
+
+
+
+ 
+
+ 
+
+ 
+
+ 
+
+ 
+
+ 
+
+ 
+
+
+
+ 
+
- 
+ 
+ 
+
+ 
+
+ 
+
+ 
+
+ 
+
+ 
+
+ 
+
+ 
+
+ 
+
- 
+ 
+ 
+
+ 
+