From 37f5eadc996f5c2957c0d5d8f1facf3423e99ba8 Mon Sep 17 00:00:00 2001 From: Charlie Egan Date: Wed, 19 Jul 2023 15:43:23 +0100 Subject: [PATCH] Update the standalone envoy tutorial The tutorial now uses kind as well as updated versions for envoy. I have made some adjustments to how the bundle is served and the test commands run to exercise the policy too. Signed-off-by: Charlie Egan --- .../envoy-tutorial-standalone-envoy.md | 653 +++++++++++------- 1 file changed, 398 insertions(+), 255 deletions(-) diff --git a/docs/content/envoy-tutorial-standalone-envoy.md b/docs/content/envoy-tutorial-standalone-envoy.md index f03db6d6cf..6b27ca55c5 100644 --- a/docs/content/envoy-tutorial-standalone-envoy.md +++ b/docs/content/envoy-tutorial-standalone-envoy.md @@ -4,33 +4,250 @@ kind: envoy weight: 10 --- -The tutorial shows how Envoy’s External authorization filter can be used with -OPA as an authorization service to enforce security policies over API requests +The tutorial shows how Envoy’s External +[authorization filter](https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overview/security/ext_authz_filter.html) +can be used with OPA as an authorization service to enforce security policies over API requests received by Envoy. The tutorial also covers examples of authoring custom policies over the HTTP request body. -## Prerequisites +## Overview -This tutorial requires Kubernetes 1.20 or later. To run the tutorial locally, we -recommend using [minikube](https://minikube.sigs.k8s.io/docs/start/) in -version `v1.21+` with Kubernetes 1.20+. +In this tutorial we'll see how to use OPA as an External +Authorization service for the Envoy proxy. We'll do this by: -## Steps +* Running a local Kubernetes cluster +* Creating a simple authorization policy in Rego and serving it via the Bundle API +* Deploying a sample application with Envoy and OPA sidecars +* Run some sample requests to see the policy in action -### 1. Start Minikube +Note that other than the HTTP client and bundle server, all components +are co-located in the same pod. -```bash -minikube start +## Running a local Kubernetes cluster + +To start a local Kubernetes cluster to run our demo, we'll be using +[kind](https://kind.sigs.k8s.io/). In order to use the `kind` command, +you'll need to have Docker installed on your machine. Running +`docker info` is the easiest way to check if Docker is installed and +running. + +You should see output simil + +```shell +$ docker info +Client: + ... + +Server: + ... +``` + +If the above command shows information for both the client and server, +then Docker is installed and running. + +{{< info >}} +If you haven't used `kind` before, you can find installation instructions +in the [project documentation](https://kind.sigs.k8s.io/#installation-and-usage). +{{}} + +Create a cluster with the following command: + +```shell +$ kind create cluster --name opa-envoy --image kindest/node:v1.27.3 +Creating cluster "opa-envoy" ... + ✓ Ensuring node image (kindest/node:v1.27.3) 🖼 + ✓ Preparing nodes 📦 + ✓ Writing configuration 📜 + ✓ Starting control-plane 🕹️ + ✓ Installing CNI 🔌 + ✓ Installing StorageClass 💾 +... +``` + +Once the cluster is created, make sure your `kubectl` context is set to connect +to the new cluster: + +```shell +$ kubectl cluster-info --context kind-opa-envoy +Kubernetes control plane is running at ... +CoreDNS is running at ... +... +``` + +Listing the cluster nodes, should show something like this: + +```shell +$ kubectl get nodes +NAME STATUS ROLES AGE VERSION +opa-envoy-control-plane Ready control-plane 2m35s v1.27.3 +``` + +## Creating & Serving our Policy Bundle + +This tutorial assumes you have some Rego knowledge, in summary the policy below does the following: + +* Checks that the JWT token is valid +* Checks that the action is allowed based on the token payload `role` and the request path +* Guests have read-only access to the `/people` endpoint, admins can create users too as long as the + name is not the same as the admin's name. + +```rego +# policy.rego +package envoy.authz + +import future.keywords.if + +import input.attributes.request.http as http_request + +default allow := false + +allow if { + is_token_valid + action_allowed +} + +is_token_valid if { + token.valid + now := time.now_ns() / 1000000000 + token.payload.nbf <= now + now < token.payload.exp +} + +action_allowed if { + http_request.method == "GET" + token.payload.role == "guest" + glob.match("/people", ["/"], http_request.path) +} + +action_allowed if { + http_request.method == "GET" + token.payload.role == "admin" + glob.match("/people", ["/"], http_request.path) +} + +action_allowed if { + http_request.method == "POST" + token.payload.role == "admin" + glob.match("/people", ["/"], http_request.path) + lower(input.parsed_body.firstname) != base64url.decode(token.payload.sub) +} + +token := {"valid": valid, "payload": payload} if { + [_, encoded] := split(http_request.headers.authorization, " ") + [valid, _, payload] := io.jwt.decode_verify(encoded, {"secret": "secret"}) +} +``` + +Create a file called `policy.rego` with the above content and store it in a ConfigMap: + +```shell +kubectl create configmap authz-policy --from-file policy.rego ``` -### 2. Create ConfigMap containing configuration for Envoy +Now that the policy is stored in a ConfigMap, we can spin up an HTTP server to make it +available to as a Bundle to OPA when it's making decisions for our application: + +```yaml +# bundle-server.yaml +--- +apiVersion: apps/v1 +kind: Deployment +metadata: + name: bundle-server + labels: + app: bundle-server +spec: + replicas: 1 + selector: + matchLabels: + app: bundle-server + template: + metadata: + labels: + app: bundle-server + spec: + initContainers: + - name: opa-builder + image: openpolicyagent/opa:latest + args: + - "build" + - "--bundle" + - "/opt/policy/" + - "--output" + - "/opt/output/bundle.tar.gz" + volumeMounts: + - name: index + mountPath: /opt/output/ + - name: policy + mountPath: /opt/policy/ + containers: + - name: bundle-server + image: nginx:1.25 + ports: + - containerPort: 80 + name: http + volumeMounts: + - name: index + mountPath: /usr/share/nginx/html + volumes: + - name: index + emptyDir: {} + - name: policy + configMap: + name: authz-policy +--- +apiVersion: v1 +kind: Service +metadata: + name: bundle-server +spec: + selector: + app: bundle-server + ports: + - protocol: TCP + port: 80 + targetPort: http +``` + +Create a file called `bundle-server.yaml` with the above content and apply it to the cluster: + +```shell +kubectl apply -f bundle-server.yaml +``` + +Once the deployment is running, we can check that the bundle is available by running: + +```shell +kubectl port-forward service/bundle-server 8080:80 +``` + +Before checking that the bundle has been generated correctly and is available to download: + +```shell +$ curl -I localhost:8080/bundle.tar.gz +HTTP/1.1 200 OK +... +``` + +You may now exit the port-forwarding session, the bundle server will only be accessed +from inside the cluster from now on. + +## Deploying an application with Envoy and OPA sidecars + +In this tutorial, we are manually configuring the Envoy proxy sidecar to intermediate +HTTP traffic from clients and our application. Envoy will consult OPA to +make authorization decisions for each request by sending `CheckRequest` messages over +a gRPC connection. -The Envoy configuration below defines an external authorization filter -`envoy.ext_authz` for a gRPC authorization server. +We will use the following Envoy configuration to achieve this. In summary, this +configures Envoy to: -Save the configuration as **envoy.yaml**: +* Listen on port `8000` for HTTP traffic +* Consult OPA for authorization decisions at 127.0.0.1:9191 & deny failing requests +* Forward requests to the application at 127.0.0.1:8080 if ok. ```yaml +# envoy.yaml static_resources: listeners: - address: @@ -70,6 +287,8 @@ static_resources: stat_prefix: ext_authz timeout: 0.5s - name: envoy.filters.http.router + typed_config: + "@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router clusters: - name: service connect_timeout: 0.25s @@ -103,134 +322,25 @@ layered_runtime: global_downstream_max_connections: 50000 ``` -Create the ConfigMap: - -```bash -kubectl create configmap proxy-config --from-file envoy.yaml -``` - -### 3. Define a OPA policy - -The following OPA policy restricts access to the `/people` endpoint exposed by -our sample app: - -* Alice is granted a **guest** role and can perform a `GET` request to `/people`. -* Bob is granted an **admin** role and can perform a `GET` and `POST` request to `/people`. - -The policy also restricts an `admin` user, in this case `bob` from creating an -employee with the same `firstname` as himself. - -**policy.rego** - -```live:example:module:openable -package envoy.authz - -import future.keywords - -import input.attributes.request.http as http_request - -default allow := false - -allow if { - is_token_valid - action_allowed -} - -is_token_valid if { - token.valid - now := time.now_ns() / 1000000000 - token.payload.nbf <= now - now < token.payload.exp -} - -action_allowed if { - http_request.method == "GET" - token.payload.role == "guest" - glob.match("/people", ["/"], http_request.path) -} - -action_allowed if { - http_request.method == "GET" - token.payload.role == "admin" - glob.match("/people", ["/"], http_request.path) -} - -action_allowed if { - http_request.method == "POST" - token.payload.role == "admin" - glob.match("/people", ["/"], http_request.path) - lower(input.parsed_body.firstname) != base64url.decode(token.payload.sub) -} - -token := {"valid": valid, "payload": payload} if { - [_, encoded] := split(http_request.headers.authorization, " ") - [valid, _, payload] := io.jwt.decode_verify(encoded, {"secret": "secret"}) -} -``` - -Then, build an OPA bundle. +Create a `ConfigMap` containing the above configuration by running: ```shell -opa build policy.rego -``` - -In the next step, OPA is configured to query for the `data.envoy.authz.allow` -decision. If the response is `true` the operation is allowed, otherwise the -operation is denied. Sample input received by OPA is shown below: - -```live:example:query:hidden -data.envoy.authz.allow -``` - -```live:example:input -{ - "attributes": { - "request": { - "http": { - "method": "GET", - "path": "/people", - "headers": { - "authorization": "Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJyb2xlIjoiZ3Vlc3QiLCJzdWIiOiJZV3hwWTJVPSIsIm5iZiI6MTUxNDg1MTEzOSwiZXhwIjoxNjQxMDgxNTM5fQ.K5DnnbbIOspRbpCr2IKXE9cPVatGOCBrBQobQmBmaeU" - } - } - } - } -} -``` - -With the input value above, the answer is: - -```live:example:output -``` - -An example of the complete input received by OPA can be seen [here](https://github.com/open-policy-agent/opa-envoy-plugin#example-input). - -### 4. Publish OPA Bundle - -We will now serve the OPA bundle created in the previous step using Nginx. - -```bash -docker run --rm --name bundle-server -d -p 8888:80 -v ${PWD}:/usr/share/nginx/html:ro nginx:latest +kubectl create configmap proxy-config --from-file envoy.yaml ``` -The above command will start a Nginx server running on port `8888` on your host and act as a bundle server. - -### 5. Create App Deployment with OPA and Envoy sidecars - -Our deployment contains a sample Go app which provides information about -employees in a company. It exposes a `/people` endpoint to `get` and `create` -employees. More information can on the app be found -[here](https://github.com/ashutosh-narkar/go-test-server). - -OPA is started with a configuration that sets the listening address of Envoy -External Authorization gRPC server and specifies the name of the policy decision -to query. OPA will also periodically download the policy bundle from the local Nginx server -configured in the previous step. More information on the configuration options can be found -[here](https://github.com/open-policy-agent/opa-envoy-plugin#configuration). - -Save the deployment as **deployment.yaml**: +Our application will be configured using a `Deployment` and `Service`. +There are a few things to note: +* the pods have an `initContainer` that configures the `iptables` rules to + redirect traffic to the Envoy proxy. +* the `demo-test-server` container is a simple user store using in-memory state. +* the `envoy` container is configured to use the `proxy-config` `ConfigMap` we + created earlier. +* The OPA container is configured to download policy bundles from + the in-cluster bundle server (`bundle-server.default.svc.cluster.local`). +* The OPA license key must be set. We show how to do this in the next step. ```yaml +# app.yaml kind: Deployment apiVersion: apps/v1 metadata: @@ -251,172 +361,205 @@ spec: - name: proxy-init image: openpolicyagent/proxy_init:v8 # Configure the iptables bootstrap script to redirect traffic to the - # Envoy proxy on port 8000, specify that Envoy will be running as user - # 1111, and that we want to exclude port 8282 from the proxy for the - # OPA health checks. These values must match up with the configuration - # defined below for the "envoy" and "opa" containers. + # Envoy proxy on port 8000. Envoy will be running as 1111, and port + # 8282 will be excluded to support OPA health checks. args: ["-p", "8000", "-u", "1111", "-w", "8282"] securityContext: capabilities: add: - - NET_ADMIN + - NET_ADMIN runAsNonRoot: false runAsUser: 0 containers: - - name: app - image: openpolicyagent/demo-test-server:v1 - ports: - - containerPort: 8080 - - name: envoy - image: envoyproxy/envoy:v1.20.0 - volumeMounts: - - readOnly: true - mountPath: /config - name: proxy-config - args: - - "envoy" - - "--config-path" - - "/config/envoy.yaml" - env: - - name: ENVOY_UID - value: "1111" - - name: opa - # Note: openpolicyagent/opa:latest-envoy is created by retagging - # the latest released image of OPA-Envoy. - image: openpolicyagent/opa:{{< current_opa_envoy_docker_version >}} - args: - - "run" - - "--server" - - "--addr=localhost:8181" - - "--diagnostic-addr=0.0.0.0:8282" - - "--set=services.default.url=http://host.minikube.internal:8888" - - "--set=bundles.default.resource=bundle.tar.gz" - - "--set=plugins.envoy_ext_authz_grpc.addr=:9191" - - "--set=plugins.envoy_ext_authz_grpc.path=envoy/authz/allow" - - "--set=decision_logs.console=true" - - "--set=status.console=true" - - "--ignore=.*" - livenessProbe: - httpGet: - path: /health?plugins - scheme: HTTP - port: 8282 - initialDelaySeconds: 5 - periodSeconds: 5 - readinessProbe: - httpGet: - path: /health?plugins - scheme: HTTP - port: 8282 - initialDelaySeconds: 5 - periodSeconds: 5 + - name: app + image: openpolicyagent/demo-test-server:v1 + ports: + - containerPort: 8080 + - name: envoy + image: envoyproxy/envoy:v1.26.3 + volumeMounts: + - readOnly: true + mountPath: /config + name: proxy-config + args: + - "envoy" + - "--config-path" + - "/config/envoy.yaml" + env: + - name: ENVOY_UID + value: "1111" + - name: opa + image: openpolicyagent/opa:latest-envoy + args: + - "run" + - "--server" + - "--addr=localhost:8181" + - "--diagnostic-addr=0.0.0.0:8282" + - "--set=services.default.url=http://bundle-server" + - "--set=bundles.default.resource=bundle.tar.gz" + - "--set=plugins.envoy_ext_authz_grpc.addr=:9191" + - "--set=plugins.envoy_ext_authz_grpc.path=envoy/authz/allow" + - "--set=decision_logs.console=true" + - "--set=status.console=true" + - "--ignore=.*" + livenessProbe: + httpGet: + path: /health?plugins + scheme: HTTP + port: 8282 + initialDelaySeconds: 5 + periodSeconds: 5 + readinessProbe: + httpGet: + path: /health?plugins + scheme: HTTP + port: 8282 + initialDelaySeconds: 1 + periodSeconds: 3 volumes: - - name: proxy-config - configMap: - name: proxy-config -``` - -```bash -kubectl apply -f deployment.yaml + - name: proxy-config + configMap: + name: proxy-config +--- +apiVersion: v1 +kind: Service +metadata: + name: example-app +spec: + selector: + app: example-app + ports: + - protocol: TCP + port: 80 + targetPort: 8080 ``` -Check that the Pod shows `3/3` containers `READY` the `STATUS` as `Running`: - -```bash -kubectl get pod +Deploy the application and Kubernetes Service to the cluster with: -NAME READY STATUS RESTARTS AGE -example-app-67c644b9cb-bbqgh 3/3 Running 0 8s +```shell +kubectl apply -f app.yaml ``` -> The `proxy-init` container installs iptables rules to redirect all container - traffic through the Envoy proxy sidecar. More information can be found - [here](https://github.com/open-policy-agent/contrib/tree/main/envoy_iptables). +Check that everything is working by listing the pod (make sure that +all three pods are running ok). +```shell +$ kubectl get pods +NAME READY STATUS RESTARTS AGE +bundle-server-5d7bfffdb6-bgn86 1/1 Running 0 1m +example-app-74b4bc88-5d4wh 3/3 Running 0 1m +``` -### 6. Create a Service to expose HTTP server +## See the Policy in Action -In a second terminal, start a [minikube tunnel](https://minikube.sigs.k8s.io/docs/handbook/accessing/#using-minikube-tunnel) to allow for use of the `LoadBalancer` service type. +Run a shell inside the cluster to use for testing. We will use this in-cluster +shell for the rest of the tutorial. -```bash -minikube tunnel +```shell +kubectl run curl --restart=Never -it --rm --image curlimages/curl:8.1.2 -- sh ``` -In the first terminal, create a `LoadBalancer` service for the deployment. +Set two tokens for two users, Alice and Bob with different permissions. +As defined by our policy: -```bash -kubectl expose deployment example-app --type=LoadBalancer --name=example-app-service --port=8080 +```shell +export ALICE_TOKEN="eyJhbGciOiAiSFMyNTYiLCAidHlwIjogIkpXVCJ9.eyJleHAiOiAyMjQxMDgxNTM5LCAibmJmIjogMTUxNDg1MTEzOSwgInJvbGUiOiAiZ3Vlc3QiLCAic3ViIjogIllXeHBZMlU9In0.Uk5hgUqMuUfDLvBLnlXMD0-X53aM_Hlziqg3vhOsCc8" +export BOB_TOKEN="eyJhbGciOiAiSFMyNTYiLCAidHlwIjogIkpXVCJ9.eyJleHAiOiAyMjQxMDgxNTM5LCAibmJmIjogMTUxNDg1MTEzOSwgInJvbGUiOiAiYWRtaW4iLCAic3ViIjogIlltOWkifQ.5qsm7rRTvqFHAgiB6evX0a_hWnGbWquZC0HImVQPQo8" ``` -Check that the Service shows an `EXTERNAL-IP`: +### Listing People -```bash -kubectl get service example-app-service +Send a request to list people. This should succeed for both Alice and Bob. -NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE -example-app-service LoadBalancer 10.109.64.199 10.109.64.199 8080:32170/TCP 5s +```shell +curl -i -H "Authorization: Bearer $ALICE_TOKEN" http://example-app/people ``` - -Set the `SERVICE_URL` environment variable to the service's IP/port. - -**minikube:** - -```bash -export SERVICE_HOST=$(kubectl get service example-app-service -o jsonpath='{.status.loadBalancer.ingress[0].ip}') -export SERVICE_URL=$SERVICE_HOST:8080 -echo $SERVICE_URL +``` +HTTP/1.1 200 OK +content-type: application/json +date: Tue, 18 Jul 2023 15:22:25 GMT +content-length: 96 +x-envoy-upstream-service-time: 14 +server: envoy + +[{"id":"1","firstname":"John","lastname":"Doe"},{"id":"2","firstname":"Jane","lastname":"Doe"}] ``` -**minikube (example):** +And for Bob: +```shell +curl -i -H "Authorization: Bearer $BOB_TOKEN" http://example-app/people +``` ``` -10.109.64.199:8080 +HTTP/1.1 200 OK +...omitted... ``` -### 7. Exercise the OPA policy +### Creating People -For convenience, we’ll want to store Alice's and Bob's tokens in environment variables. +Send a request to create a new user. This should fail for Alice but not Bob: -```bash -export ALICE_TOKEN="eyJhbGciOiAiSFMyNTYiLCAidHlwIjogIkpXVCJ9.eyJleHAiOiAyMjQxMDgxNTM5LCAibmJmIjogMTUxNDg1MTEzOSwgInJvbGUiOiAiZ3Vlc3QiLCAic3ViIjogIllXeHBZMlU9In0.Uk5hgUqMuUfDLvBLnlXMD0-X53aM_Hlziqg3vhOsCc8" -export BOB_TOKEN="eyJhbGciOiAiSFMyNTYiLCAidHlwIjogIkpXVCJ9.eyJleHAiOiAyMjQxMDgxNTM5LCAibmJmIjogMTUxNDg1MTEzOSwgInJvbGUiOiAiYWRtaW4iLCAic3ViIjogIlltOWkifQ.5qsm7rRTvqFHAgiB6evX0a_hWnGbWquZC0HImVQPQo8" +```shell +curl -i -H "Authorization: Bearer $ALICE_TOKEN" \ + -d '{"firstname":"Foo", "lastname":"Bar"}' -H "Content-Type: application/json" \ + -X POST http://example-app/people ``` +``` +HTTP/1.1 403 Forbidden +date: Tue, 18 Jul 2023 15:25:28 GMT +server: envoy +content-length: 0 +``` +And for Bob, the request is permitted and the user is saved with an ID -Check that `Alice` can get employees **but cannot** create one. - -```bash -curl -i -H "Authorization: Bearer $ALICE_TOKEN" http://$SERVICE_URL/people -curl -i -H "Authorization: Bearer $ALICE_TOKEN" -d '{"firstname":"Charlie", "lastname":"OPA"}' -H "Content-Type: application/json" -X POST http://$SERVICE_URL/people +```shell +curl -i -H "Authorization: Bearer $BOB_TOKEN" \ + -d '{"firstname":"Foo", "lastname":"Bar"}' -H "Content-Type: application/json" \ + -X POST http://example-app/people +``` ``` +HTTP/1.1 200 OK +content-type: application/json +date: Tue, 18 Jul 2023 15:28:20 GMT +content-length: 51 +x-envoy-upstream-service-time: 11 +server: envoy + +{"id":"498081","firstname":"Foo","lastname":"Bar"} +``` + +### Creating People: Conflict -Check that `Bob` can get employees and also create one. +Our policy also blocks users from creating users with the same name, test that +functionality with this request: -```bash -curl -i -H "Authorization: Bearer $BOB_TOKEN" http://$SERVICE_URL/people -curl -i -H "Authorization: Bearer $BOB_TOKEN" -d '{"firstname":"Charlie", "lastname":"Opa"}' -H "Content-Type: application/json" -X POST http://$SERVICE_URL/people +```shell +curl -i -H "Authorization: Bearer $BOB_TOKEN" \ + -d '{"firstname":"Bob", "lastname":"Bar"}' -H "Content-Type: application/json" \ + -X POST http://example-app/people +``` +``` +HTTP/1.1 403 Forbidden +date: Tue, 18 Jul 2023 15:31:48 GMT +server: envoy +content-length: 0 ``` -Check that `Bob` **cannot** create an employee with the same firstname as himself. +## Shutting Down -```bash -curl -i -H "Authorization: Bearer $BOB_TOKEN" -d '{"firstname":"Bob", "lastname":"Rego"}' -H "Content-Type: application/json" -X POST http://$SERVICE_URL/people -``` +Exit the in-cluster shell by typing `exit`. -To remove the kubernetes resources created during this tutorial please use the following commands. -```bash -kubectl delete service example-app-service -kubectl delete deployment example-app -kubectl delete configmap proxy-config -``` +Delete the cluster by running: -To remove the bundle server run: -```bash -docker rm -f bundle-server +```shell +$ kind delete cluster --name opa-envoy +Deleting cluster "opa-envoy" ... +Deleted nodes: ["opa-envoy-control-plane"] ``` ## Wrap Up -Congratulations for finishing the tutorial ! +Congratulations on finishing the tutorial ! This tutorial showed how to use OPA as an External authorization service to enforce custom policies by leveraging Envoy’s External authorization filter.