From a8116a659fc0fe90044579148b53e74fa35a31c8 Mon Sep 17 00:00:00 2001 From: Jiaxiao Zhou Date: Mon, 25 Apr 2022 17:03:40 -0700 Subject: [PATCH 1/6] added cloudevent trigger SIP draft Signed-off-by: Jiaxiao Zhou --- docs/content/sips/00x-cloudevent-trigger.md | 148 ++++++++++++++++++++ 1 file changed, 148 insertions(+) create mode 100644 docs/content/sips/00x-cloudevent-trigger.md diff --git a/docs/content/sips/00x-cloudevent-trigger.md b/docs/content/sips/00x-cloudevent-trigger.md new file mode 100644 index 0000000000..aecbe0b7f1 --- /dev/null +++ b/docs/content/sips/00x-cloudevent-trigger.md @@ -0,0 +1,148 @@ +title = "SIP xxx - CloudEvents trigger" +template = "main" +date = "2022-04-25T14:53:30Z" +--- + +Summary: A CloudEvents trigger for spin. + +Owner: jiazho@microsoft.com + +Created: April 24, 2022 + +Updated: April 24, 2022 + +## Background + +Currently spin supports two triggers, one for Redis messages and one for HTTP requests. [CloudEvents](https://cloudevents.io/) are a new standard for eventing and received huge interests from the major cloud providers. Supporting CloudEvents could make spin a great solution for writing serverless applications. + +## Proposal + +This document proposes adding a new trigger for CloudEvents. The triggers are invoked by a CloudEvent source. The CloudEvent source is a event provider service that sends CloudEvents to spin, such as Kafka topics, HTTP requests, AMQP messages. For example, the [CloudEvents spec](https://github.com/cloudevents/spec/tree/main/cloudevents/bindings) list a few protocol bindings including AMQP, HTTP, Kafka etc. + +This proposal aims at providing a CloudEvent component for the [HTTP Protocol Bindings](https://github.com/cloudevents/spec/blob/main/cloudevents/bindings/http-protocol-binding.md). This example shows the mapping of an event with an HTTP POST request. +``` +POST /someresource HTTP/1.1 +Host: webhook.example.com +ce-specversion: 1.0 +ce-type: com.example.someevent +ce-time: 2018-04-05T03:56:24Z +ce-id: 1234-1234-1234 +ce-source: /mycontext/subcontext + .... further attributes ... +Content-Type: application/json; charset=utf-8 +Content-Length: nnnn +{ + ... application data ... +} +``` + +Creating an CloudEvents trigger is done when [configuring the application](/configuration) +by defining the top-level application trigger: + +```toml +# spin.toml +trigger = { type = "cloudevent" } +``` + +Then, when defining the component (in `spin.toml`), you can set the protocol binding for the component. For example: + +- an HTTP CloudEvents component: + +```toml +[component.trigger] +binding = "http" +``` + +- an Kafka CloudEvents component (optional): + +```toml +[component.trigger] +binding = "kafka" +broker = ["localhost:9092", "localhost:9093"] +topic = "mytopic" +group = "mygroup" +``` + +- an AMQP CloudEvents component (optional WIP): + +```toml +[component.trigger] +binding = "amqp" +broker = "localhost:5672" +exchange = "myexchange" +routing_key = "myroutingkey" +``` + +You can also set the sink address for the component that returns a CloudEvent. For example: + +```toml +[component.trigger] +binding = "http" +sink = "http://localhost:8080/someresource" +``` + +Note that the sink address is only used when the component is invoked. The component will make a outbound HTTP request that includes the CloudEvents to the sink address. + + +## The WebAssembly interface + +The CloudEvent trigger is built on top of the +[WebAssembly component model](https://github.com/WebAssembly/component-model). +The current interface is defined using the +[WebAssembly Interface (WIT)](https://github.com/bytecodealliance/wit-bindgen/blob/main/WIT.md) +format, and is a function that takes the event payload as its only parameter: + +```fsharp +// wit/ephemeral/spin-ce.wit + +// The event payload. +record event { + // The event type. + type: string, + // The event id. + id: string, + // The event source. + source: string, + // The event specversion. + specversion: string, + // The event data content type. + datacontenttype: string, + // The event data schema. + dataschema: string, + // The event subject. + subject: string, + // The event time. + time: option, + // The event data. + data: option +} + +// The entry point for a CloudEvent handler. +handle-cloudevent: function(event: event) -> expected +``` + + +This is the function that all CloudEvents components must implement, and which is +used by the Spin Redis executor when instantiating and invoking the component. + +Notice that the function will return a `expected` value. If the sink address is not set, the function will return `expected` with `event` as the value. If the sink address is set, spin will make an outbound HTTP request to the sink address and return the response as the value. + +Due to the [known issue](https://github.com/bytecodealliance/wit-bindgen/issues/171) of the cannonical ABI representation cannot exceeds the number of parameters (16) in the wasmtime API, the proposed WIT format cannot be compiled. I am proposing an alternative WIT format for CloudEvents: + +```fsharp +// wit/ephemeral/spin-ce.wit + +// The event payload. +type event = string + +// The entry point for a CloudEvent handler. +handle-cloudevent: function(event: event) -> expected +``` + +At the runtime, Spin will use CloudEvent SDK to parse the event payload and invoke the function. For example, take a look at the [CloudEvents SDK Rust](https://github.com/cloudevents/sdk-rust) library. + + +## Future design considerations + +- CloudEvents bindings for different protocols, such as AMQP, Kafka, etc. +- Filter events based on event attributes. \ No newline at end of file From a805b2aaf6adcc3daf3ac3a34fe25eb99519a189 Mon Sep 17 00:00:00 2001 From: Jiaxiao Zhou Date: Mon, 25 Apr 2022 17:12:34 -0700 Subject: [PATCH 2/6] added SDK proposal Signed-off-by: Jiaxiao Zhou --- docs/content/sips/00x-cloudevent-trigger.md | 35 +++++++++++++++++++++ 1 file changed, 35 insertions(+) diff --git a/docs/content/sips/00x-cloudevent-trigger.md b/docs/content/sips/00x-cloudevent-trigger.md index aecbe0b7f1..1313fb22bc 100644 --- a/docs/content/sips/00x-cloudevent-trigger.md +++ b/docs/content/sips/00x-cloudevent-trigger.md @@ -141,6 +141,41 @@ handle-cloudevent: function(event: event) -> expected At the runtime, Spin will use CloudEvent SDK to parse the event payload and invoke the function. For example, take a look at the [CloudEvents SDK Rust](https://github.com/cloudevents/sdk-rust) library. +## The CloudEvents Spin SDK +```rust +// A Spin CloudEvents component written in Rust +use anyhow::Result; +use spin_sdk::{ + event::{Event}, + event_component, +}; + +/// A simple Spin event component. +#[event_component] +fn trigger(event: Event) -> Result { + println!("event is {}", event.id()); + Ok(event) +} +``` + +```go +// A Spin CloudEvents component written in Go +package main + +import ( + "fmt" + "context" + + spin "github.com/fermyon/spin/sdk/go/event" +) + +func main() { + spin.ReceiveEvent(func(ctx context.Context, event spin.Event) { + fmt.Printf("%s", event) + spin.SendEvent(ctx, event) + }) +} +``` ## Future design considerations From 2e6dbb9551e4b09d58b5203e6dc57b4772d6645d Mon Sep 17 00:00:00 2001 From: Jiaxiao Zhou Date: Sat, 30 Apr 2022 21:11:35 -0700 Subject: [PATCH 3/6] change event_component to cloud_event_component Signed-off-by: Jiaxiao Zhou --- docs/content/sips/00x-cloudevent-trigger.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/content/sips/00x-cloudevent-trigger.md b/docs/content/sips/00x-cloudevent-trigger.md index 1313fb22bc..096401033b 100644 --- a/docs/content/sips/00x-cloudevent-trigger.md +++ b/docs/content/sips/00x-cloudevent-trigger.md @@ -151,7 +151,7 @@ use spin_sdk::{ }; /// A simple Spin event component. -#[event_component] +#[cloud_event_component] fn trigger(event: Event) -> Result { println!("event is {}", event.id()); Ok(event) From fb05116e1e62a0aa95b328ea5935c0a02bdb2d83 Mon Sep 17 00:00:00 2001 From: Jiaxiao Zhou Date: Tue, 10 May 2022 02:35:00 -0700 Subject: [PATCH 4/6] updated sip to restrict scope to only ce-http spin-sdk Signed-off-by: Jiaxiao Zhou --- docs/content/sips/00x-cloudevent-trigger.md | 149 ++++++++++++++------ 1 file changed, 108 insertions(+), 41 deletions(-) diff --git a/docs/content/sips/00x-cloudevent-trigger.md b/docs/content/sips/00x-cloudevent-trigger.md index 096401033b..2e6b07b830 100644 --- a/docs/content/sips/00x-cloudevent-trigger.md +++ b/docs/content/sips/00x-cloudevent-trigger.md @@ -15,11 +15,10 @@ Updated: April 24, 2022 Currently spin supports two triggers, one for Redis messages and one for HTTP requests. [CloudEvents](https://cloudevents.io/) are a new standard for eventing and received huge interests from the major cloud providers. Supporting CloudEvents could make spin a great solution for writing serverless applications. -## Proposal -This document proposes adding a new trigger for CloudEvents. The triggers are invoked by a CloudEvent source. The CloudEvent source is a event provider service that sends CloudEvents to spin, such as Kafka topics, HTTP requests, AMQP messages. For example, the [CloudEvents spec](https://github.com/cloudevents/spec/tree/main/cloudevents/bindings) list a few protocol bindings including AMQP, HTTP, Kafka etc. +## Proposal -This proposal aims at providing a CloudEvent component for the [HTTP Protocol Bindings](https://github.com/cloudevents/spec/blob/main/cloudevents/bindings/http-protocol-binding.md). This example shows the mapping of an event with an HTTP POST request. +This document proposes adding features in the spin SDK to support CloudEvents. CloudEvents itself is a envelop for the underlying transport protocol, such as AMQP, Kafka, HTTP, etc. This proposal aims at providing a CloudEvent component for the [HTTP Protocol Bindings](https://github.com/cloudevents/spec/blob/main/cloudevents/bindings/http-protocol-binding.md). Here is an example shows the mapping of an event with an HTTP POST request in CloudEvent's binary format. ``` POST /someresource HTTP/1.1 Host: webhook.example.com @@ -36,53 +35,32 @@ Content-Length: nnnn } ``` -Creating an CloudEvents trigger is done when [configuring the application](/configuration) -by defining the top-level application trigger: - +Creating an HTTP CloudEvents trigger is done by defining the top level application trigger in spin. The following code snippet shows the definition of a HTTP CloudEvents trigger. ```toml # spin.toml -trigger = { type = "cloudevent" } -``` - -Then, when defining the component (in `spin.toml`), you can set the protocol binding for the component. For example: - -- an HTTP CloudEvents component: - -```toml -[component.trigger] -binding = "http" +trigger = { type = "http", base = "/", schema = "cloudevents" } ``` -- an Kafka CloudEvents component (optional): +The added `schema` attribute in trigger will tell spin application that it will expect the HTTP request and responses are CloudEvents. -```toml -[component.trigger] -binding = "kafka" -broker = ["localhost:9092", "localhost:9093"] -topic = "mytopic" -group = "mygroup" -``` +> Note that the `schema` attribute is not the same as the `schema` attribute in the CloudEvents spec. The `schema` attribute in the spec is the schema of the payload. -- an AMQP CloudEvents component (optional WIP): +We also allow users to define individual component to be CloudEvents components. For example, we could define a HTTP CloudEvents component in spin.toml: ```toml -[component.trigger] -binding = "amqp" -broker = "localhost:5672" -exchange = "myexchange" -routing_key = "myroutingkey" -``` - -You can also set the sink address for the component that returns a CloudEvent. For example: +# spin.toml +trigger = { type = "http", base = "/" } -```toml +[[component]] +id = "hello" +source = "target/wasm32-wasi/release/spinhelloworld.wasm" +description = "A simple component that returns hello." +schema = "cloudevents" [component.trigger] -binding = "http" -sink = "http://localhost:8080/someresource" +route = "/hello" ``` -Note that the sink address is only used when the component is invoked. The component will make a outbound HTTP request that includes the CloudEvents to the sink address. - +> Note that if there is no `schema` attribute in the application or individual component, the HTTP request and responses are not CloudEvents. ## The WebAssembly interface @@ -170,7 +148,7 @@ import ( ) func main() { - spin.ReceiveEvent(func(ctx context.Context, event spin.Event) { + spin.ReceiveEvent(func(event spin.Event) { fmt.Printf("%s", event) spin.SendEvent(ctx, event) }) @@ -179,5 +157,94 @@ func main() { ## Future design considerations -- CloudEvents bindings for different protocols, such as AMQP, Kafka, etc. -- Filter events based on event attributes. \ No newline at end of file +#### More transport protocols bindings +- Kafka binding +- AMQP binding +- NATS binding + +#### Filtering based on event attributes +```toml +[[component]] +id = "filter" +source = "target/wasm32-wasi/release/spinhelloworld.wasm" +description = "A simple component that filters events based on event attributes." +schema = "cloudevents" +[component.trigger] +route = "/filter" +[component.filter] +ce.type = "com.example.someevent" +ce.source = "/mycontext/subcontext" +``` + +#### Generic CloudEvent component +A generic CloudEvents component is defined in the following way: +```rust +// A Spin CloudEvents component written in Rust +use anyhow::Result; +use spin_sdk::{ + event::{Event}, + cloud_event_component, +}; + +/// A simple Spin event component. +#[cloud_event_component] +fn trigger(event: Event) -> Result { + println!("event is {}", event.id()); + // do something with the event + Ok(event) +} +``` + +It is trigger-agnostic, at least within the supported CloudEvents protocol bindings. You can see the list of supported protocols in the [CloudEvents documentation](https://github.com/cloudevents/spec/blob/main/cloudevents/bindings). The benefits of doing this are: +1. Rapid prototyping: you can quickly prototype your event components and test them locally using HTTP bindings. Once you are confident that they are working, You can switch the trigger to a different type, without having to modify the code. +2. Reusability: you can reuse the same event components with different protocols bindings, such as AMQP and Kafka. +3. Chaining: you can chain multiple event components together since they share the same component signature. + +#### CloudEvents trigger + +Creating an CloudEvents trigger is done when [configuring the application](/configuration) +by defining the top-level application trigger: + +```toml +# spin.toml +trigger = { type = "cloudevent" } +``` + +Then, when defining the component (in `spin.toml`), you can set the protocol binding for the component. For example: + +- an HTTP CloudEvents component: + +```toml +[component.trigger] +binding = "http" +``` + +- an Kafka CloudEvents component (optional): + +```toml +[component.trigger] +binding = "kafka" +broker = ["localhost:9092", "localhost:9093"] +topic = "mytopic" +group = "mygroup" +``` + +- an AMQP CloudEvents component (optional WIP): + +```toml +[component.trigger] +binding = "amqp" +broker = "localhost:5672" +exchange = "myexchange" +routing_key = "myroutingkey" +``` + +You can also set the sink address for the component that returns a CloudEvent. For example: + +```toml +[component.trigger] +binding = "http" +sink = "http://localhost:8080/someresource" +``` + +Note that the sink address is only used when the component is invoked. The component will make a outbound HTTP request that includes the CloudEvents to the sink address. \ No newline at end of file From 21147d704f6811fe51af2636e6a12818b99ea5dd Mon Sep 17 00:00:00 2001 From: Jiaxiao Zhou Date: Tue, 10 May 2022 02:44:15 -0700 Subject: [PATCH 5/6] added implementation requirements to SIP Signed-off-by: Jiaxiao Zhou --- docs/content/sips/00x-cloudevent-trigger.md | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/docs/content/sips/00x-cloudevent-trigger.md b/docs/content/sips/00x-cloudevent-trigger.md index 2e6b07b830..44aa487670 100644 --- a/docs/content/sips/00x-cloudevent-trigger.md +++ b/docs/content/sips/00x-cloudevent-trigger.md @@ -155,6 +155,19 @@ func main() { } ``` +## Implementation +### Funtional Requirements +1. Enable Rust SDK to serialize/deserialize CloudEvents using CloudEvnets SDK. +2. Enable Go SDK to serialize/deserialize CloudEvents using CloudEvnets SDK. +3. Add a new CloudEvents component macro to Rust SDK. +4. Add CloudEvents webhook to both Rust and Go SDK. +5. Fully test the new CloudEvents component. +6. Write examples for new CloudEvents component. + +### Non-Functional Requirements +1. Scale spin to support thousands of CloudEvents requests per second. +2. Establish reasonable performance metrics for the new CloudEvents component. + ## Future design considerations #### More transport protocols bindings From 6016731d5e74ae8dc107839cb47f0b1cb3af8524 Mon Sep 17 00:00:00 2001 From: Jiaxiao Zhou Date: Tue, 10 May 2022 10:47:13 -0700 Subject: [PATCH 6/6] resolved some comments on the typo and working of the SIP Signed-off-by: Jiaxiao Zhou --- docs/content/sips/00x-cloudevent-trigger.md | 62 +++++++++------------ 1 file changed, 25 insertions(+), 37 deletions(-) diff --git a/docs/content/sips/00x-cloudevent-trigger.md b/docs/content/sips/00x-cloudevent-trigger.md index 44aa487670..de231f1536 100644 --- a/docs/content/sips/00x-cloudevent-trigger.md +++ b/docs/content/sips/00x-cloudevent-trigger.md @@ -1,24 +1,24 @@ -title = "SIP xxx - CloudEvents trigger" +title = "SIP xxx - Spin SDK CloudEvents Support" template = "main" date = "2022-04-25T14:53:30Z" --- -Summary: A CloudEvents trigger for spin. +Summary: Spin SDK support CloudEvents as a alternative message schema. Owner: jiazho@microsoft.com Created: April 24, 2022 -Updated: April 24, 2022 +Updated: May 10, 2022 ## Background -Currently spin supports two triggers, one for Redis messages and one for HTTP requests. [CloudEvents](https://cloudevents.io/) are a new standard for eventing and received huge interests from the major cloud providers. Supporting CloudEvents could make spin a great solution for writing serverless applications. +Currently Spin supports two triggers, one for Redis messages and one for HTTP requests. [CloudEvents](https://cloudevents.io/) is a new specification for eventing and received huge interests from the major cloud providers. Supporting CloudEvents could make Spin a great solution for writing serverless applications. ## Proposal -This document proposes adding features in the spin SDK to support CloudEvents. CloudEvents itself is a envelop for the underlying transport protocol, such as AMQP, Kafka, HTTP, etc. This proposal aims at providing a CloudEvent component for the [HTTP Protocol Bindings](https://github.com/cloudevents/spec/blob/main/cloudevents/bindings/http-protocol-binding.md). Here is an example shows the mapping of an event with an HTTP POST request in CloudEvent's binary format. +This document proposes adding features in the Spin SDK to support CloudEvents. CloudEvents itself is an envelope for the underlying transport protocol, such as AMQP, Kafka, HTTP, etc. This proposal aims at providing a CloudEvents component for the [HTTP Protocol Bindings](https://github.com/cloudevents/spec/blob/main/cloudevents/bindings/http-protocol-binding.md). Here is an example shows the mapping of an event with an HTTP POST request in CloudEvents's binary format. ``` POST /someresource HTTP/1.1 Host: webhook.example.com @@ -35,13 +35,13 @@ Content-Length: nnnn } ``` -Creating an HTTP CloudEvents trigger is done by defining the top level application trigger in spin. The following code snippet shows the definition of a HTTP CloudEvents trigger. +Creating a HTTP CloudEvents trigger is done by defining the top level application trigger in Spin. The following code snippet shows the definition of a HTTP CloudEvents trigger. ```toml # spin.toml trigger = { type = "http", base = "/", schema = "cloudevents" } ``` -The added `schema` attribute in trigger will tell spin application that it will expect the HTTP request and responses are CloudEvents. +The added `schema` attribute in trigger will tell Spin application that it will expect the HTTP request and responses are CloudEvents. > Note that the `schema` attribute is not the same as the `schema` attribute in the CloudEvents spec. The `schema` attribute in the spec is the schema of the payload. @@ -55,23 +55,23 @@ trigger = { type = "http", base = "/" } id = "hello" source = "target/wasm32-wasi/release/spinhelloworld.wasm" description = "A simple component that returns hello." -schema = "cloudevents" [component.trigger] route = "/hello" +schema = "cloudevents" ``` > Note that if there is no `schema` attribute in the application or individual component, the HTTP request and responses are not CloudEvents. ## The WebAssembly interface -The CloudEvent trigger is built on top of the +The CloudEvents HTTP trigger is built on top of the [WebAssembly component model](https://github.com/WebAssembly/component-model). The current interface is defined using the [WebAssembly Interface (WIT)](https://github.com/bytecodealliance/wit-bindgen/blob/main/WIT.md) format, and is a function that takes the event payload as its only parameter: ```fsharp -// wit/ephemeral/spin-ce.wit +// wit/ephemeral/Spin-ce.wit // The event payload. record event { @@ -92,32 +92,18 @@ record event { // The event time. time: option, // The event data. - data: option + data: option> } -// The entry point for a CloudEvent handler. +// The entry point for a CloudEvents handler. handle-cloudevent: function(event: event) -> expected ``` This is the function that all CloudEvents components must implement, and which is -used by the Spin Redis executor when instantiating and invoking the component. - -Notice that the function will return a `expected` value. If the sink address is not set, the function will return `expected` with `event` as the value. If the sink address is set, spin will make an outbound HTTP request to the sink address and return the response as the value. - -Due to the [known issue](https://github.com/bytecodealliance/wit-bindgen/issues/171) of the cannonical ABI representation cannot exceeds the number of parameters (16) in the wasmtime API, the proposed WIT format cannot be compiled. I am proposing an alternative WIT format for CloudEvents: - -```fsharp -// wit/ephemeral/spin-ce.wit +used by the Spin HTTP executor when instantiating and invoking the component. -// The event payload. -type event = string - -// The entry point for a CloudEvent handler. -handle-cloudevent: function(event: event) -> expected -``` - -At the runtime, Spin will use CloudEvent SDK to parse the event payload and invoke the function. For example, take a look at the [CloudEvents SDK Rust](https://github.com/cloudevents/sdk-rust) library. +Notice that the function will return a `expected` value. ## The CloudEvents Spin SDK ```rust @@ -144,15 +130,17 @@ import ( "fmt" "context" - spin "github.com/fermyon/spin/sdk/go/event" + Spin "github.com/fermyon/Spin/sdk/go/event" ) -func main() { - spin.ReceiveEvent(func(event spin.Event) { - fmt.Printf("%s", event) - spin.SendEvent(ctx, event) - }) +func init() { + spincloudevents.Handle(func(in Spin.Event) (out Spin.Event, err error) { + fmt.Printf("%s", event) + return in, nil + }) } + +func main() {} ``` ## Implementation @@ -165,7 +153,7 @@ func main() { 6. Write examples for new CloudEvents component. ### Non-Functional Requirements -1. Scale spin to support thousands of CloudEvents requests per second. +1. Scale Spin to support thousands of CloudEvents requests per second. 2. Establish reasonable performance metrics for the new CloudEvents component. ## Future design considerations @@ -189,7 +177,7 @@ ce.type = "com.example.someevent" ce.source = "/mycontext/subcontext" ``` -#### Generic CloudEvent component +#### Generic CloudEvents component A generic CloudEvents component is defined in the following way: ```rust // A Spin CloudEvents component written in Rust @@ -252,7 +240,7 @@ exchange = "myexchange" routing_key = "myroutingkey" ``` -You can also set the sink address for the component that returns a CloudEvent. For example: +You can also set the sink address for the component that returns a CloudEvents. For example: ```toml [component.trigger]