Added network policy to radar applications #107

keyvaann · 2023-11-09T16:49:23Z

Solves RADAR-base/RADAR-Kubernetes#239

Unified applications labels

Now all applications will have same label key. Following applications have their labels standardized:

management-portal
radar-appserver
radar-backend
radar-fitbit-connector
radar-gateway
radar-integration
radar-output
radar-push-endpoint
radar-rest-sources-authorizer
radar-rest-sources-backend
radar-s3-connector
radar-upload-connect-backend
radar-upload-connect-frontend
radar-upload-source-connector
s3-proxy

Added network policies

app-config

Accepts incoming requests from: nginx
Allowed to send requests to: management-portal, postgresql

app-config-frontend

Accepts incoming requests from: nginx
Allowed to send requests to: none

catalog-server

Accepts incoming requests from: none
Allowed to send requests to: internet, kafka, schema-registry

management-portal

Accepts incoming requests from: nginx, app-config, radar-appserver, radar-fitbit-connector, radar-gateway, radar-integration, radar-push-endpoint, radar-rest-sources-backend, radar-upload-connect-backend, radar-upload-source-connector
Allowed to send requests to: internet, catalog-server, postgresql

radar-appserver

Accepts incoming requests from: nginx
Allowed to send requests to: internet, management-portal, postgresql

radar-backend

Accepts incoming requests from: none
Allowed to send requests to: internet, zookeeper, kafka, schema-registry

radar-fitbit-connector

Accepts incoming requests from: none
Allowed to send requests to: internet, zookeeper, kafka, schema-registry, radar-rest-sources-backend, management-portal

radar-gateway

Accepts incoming requests from: nginx
Allowed to send requests to: kafka, schema-registry, management-portal

radar-home

Accepts incoming requests from: nginx
Allowed to send requests to: none

radar-integration

Accepts incoming requests from: nginx
Allowed to send requests to: internet, management-portal

radar-jdbc-connector

Accepts incoming requests from: none
Allowed to send requests to: kafka, schema-registry, timescaledb-postgresql

radar-jdbc-connector

Accepts incoming requests from: none
Allowed to send requests to: kafka, schema-registry, timescaledb-postgresql

radar-output

Accepts incoming requests from: none
Allowed to send requests to: internet, minio, redis

radar-push-endpoint

Accepts incoming requests from: nginx
Allowed to send requests to: internet, kafka, schema-registry, redis, radar-rest-sources-backend, management-portal

radar-rest-sources-authorizer

Accepts incoming requests from: nginx
Allowed to send requests to: none

radar-rest-sources-backend

Accepts incoming requests from: nginx, management-portal, radar-push-endpoint
Allowed to send requests to: postgres, redis, management-portal

radar-s3-connector

Accepts incoming requests from: none
Allowed to send requests to: internet, kafka, schema-registry, catalog-server, minio

radar-upload-connect-backend

Accepts incoming requests from: nginx
Allowed to send requests to: internet, postgres, management-portal

radar-upload-connect-frontend

Accepts incoming requests from: nginx
Allowed to send requests to: none

radar-upload-source-connector

Accepts incoming requests from: none
Allowed to send requests to: internet, kafka, zookeeper, schema-registry, management-portal, minio

s3-proxy

Accepts incoming requests from: radar-output, radar-s3-connector, radar-upload-source-connector
Allowed to send requests to: internet, minio

velero-s3-deployment

Accepts incoming requests from: none
Allowed to send requests to: internet, minio

* main: Update appserver chart Fixed radar-connector env

github-actions · 2023-11-09T16:49:37Z

Great PR! Please pay attention to the following items before merging:

Files matching charts/*/values.yaml:

Is the PR adding a new container? Please reviewer, add it to the models (internal process)
Is the PR adding a new parameter? Please, ensure it’s documented in the README.md

This is an automatically generated QA checklist based on modified files.

Bdegraaf1234

Looks good to me

yatharthranjan · 2023-11-22T14:19:35Z

How will these play with using external managed services? For instance we use

kafka -> Managed Service for Kafka
postgres -> AWS RDS postgres
minio -> AWS S3

keyvaann · 2023-11-22T16:01:36Z

How will these play with using external managed services? For instance we use

kafka -> Managed Service for Kafka

postgres -> AWS RDS postgres

minio -> AWS S3

The relevant components have access to internet so they can send requests to external managed services.

yatharthranjan · 2023-11-23T12:11:19Z

Some of the managed services we have are in the same VPC as the k8s nodes (not on the internet). I assume it will be fine with those too?

keyvaann · 2023-11-23T12:29:19Z

The current default is blocking access to all private IPs, can you check if your managed resources have a public IP address to a private one? You can also add an exception to your external services in the network policy definition or disable them altogether.

yatharthranjan

looks good. Just a question on blocking of private IPs when egress to all internet is enabled.
Also, i don't see a value for enabling/disabling theses new policies (if we wanted to keep it as it is)

yatharthranjan · 2023-11-23T12:13:57Z

charts/catalog-server/values.yaml

+        except:
+        - 10.0.0.0/8
+        - 192.168.0.0/16
+        - 172.16.0.0/20


why are we blocking these private/local IPs?

The policies will be enabled by default. If you want to disable them you can change the networkpolicy value in production.yaml to {}.
The IPs are being blocked to prevent access to other pods and services in local network.

Should we also accept incoming requests from management portal here? Since management portal needs the catalog server for source types?

Yes! Good catch! Thank you!

yatharthranjan

Do we also need to allow access for monitoring and logging components?

yatharthranjan · 2023-11-29T15:56:14Z

charts/catalog-server/values.yaml

+          kubernetes.io/metadata.name: '{{ .Release.Namespace }}'
+      podSelector:
+        matchLabels:
+          app.kubernetes.io/name: radar-s3-connector


what about other connectors?

It seems only only these two applications connect to catalog-server:

-> grep -r 'catalog-server' --exclude-dir=catalog-server charts charts/management-portal/README.md:89:| catalogue_server | string | `"catalog-server"` | Hostname of the catalogue-server | charts/management-portal/values.yaml:189:catalogue_server: catalog-server charts/radar-s3-connector/README.md:68:| catalogServer.url | string | `"http://catalog-server:9010"` | Catalog server URL | charts/radar-s3-connector/values.yaml:117: url: http://catalog-server:9010

yatharthranjan · 2023-11-29T15:58:55Z

charts/management-portal/values.yaml

+          kubernetes.io/metadata.name: '{{ .Release.Namespace }}'
+      podSelector:
+        matchLabels:
+          app.kubernetes.io/name: '{{ .Values.catalogue_server }}'


check spelling of catalog

Yes catalog server is spelled like this in MP configuration

keyvaann · 2023-11-29T18:44:50Z

Do we also need to allow access for monitoring and logging components?

We have servicemonitor in a few applications but I don't know if we actually monitor them in prometheus, so they're not added here. Logging components doesn't connect to the pods so there is no need to mention it.

yatharthranjan

LGTM, thanks.

keyvaann added 2 commits November 9, 2023 17:46

Added network policy to RADAR-Base applications

a638e5b

Merge branch 'main' into networkpolicy

93d5452

* main: Update appserver chart Fixed radar-connector env

keyvaann added 2 commits November 9, 2023 17:54

Added labels to radar-push-endpoint

a7546bd

Enabled network policy for kube-prometheus-stack

2c446d9

keyvaann requested review from yatharthranjan, mpgxvii, peyman-mohtashami and Bdegraaf1234 November 15, 2023 11:35

Bdegraaf1234 approved these changes Nov 15, 2023

View reviewed changes

yatharthranjan reviewed Nov 23, 2023

View reviewed changes

Allow requests from MP and S3 connector to catalog-server

7a2e375

yatharthranjan reviewed Nov 29, 2023

View reviewed changes

keyvaann mentioned this pull request Nov 30, 2023

[radar-oura-connector] Add new radar-oura-connector chart #104

Merged

3 tasks

yatharthranjan approved these changes Nov 30, 2023

View reviewed changes

keyvaann merged commit d0ad32b into main Dec 1, 2023
4 checks passed

keyvaann deleted the networkpolicy branch December 1, 2023 10:10

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Added network policy to radar applications #107

Added network policy to radar applications #107

keyvaann commented Nov 9, 2023 •

edited

Loading

github-actions bot commented Nov 9, 2023 •

edited

Loading

Bdegraaf1234 left a comment

yatharthranjan commented Nov 22, 2023 •

edited

Loading

keyvaann commented Nov 22, 2023

yatharthranjan commented Nov 23, 2023

keyvaann commented Nov 23, 2023

yatharthranjan left a comment •

edited

Loading

yatharthranjan Nov 23, 2023

keyvaann Nov 23, 2023

mpgxvii Nov 24, 2023 •

edited

Loading

keyvaann Nov 24, 2023

yatharthranjan left a comment

yatharthranjan Nov 29, 2023

keyvaann Nov 29, 2023

yatharthranjan Nov 29, 2023

keyvaann Nov 29, 2023

keyvaann commented Nov 29, 2023

yatharthranjan left a comment

Added network policy to radar applications #107

Added network policy to radar applications #107

Conversation

keyvaann commented Nov 9, 2023 • edited Loading

Unified applications labels

Added network policies

app-config

app-config-frontend

catalog-server

management-portal

radar-appserver

radar-backend

radar-fitbit-connector

radar-gateway

radar-home

radar-integration

radar-jdbc-connector

radar-jdbc-connector

radar-output

radar-push-endpoint

radar-rest-sources-authorizer

radar-rest-sources-backend

radar-s3-connector

radar-upload-connect-backend

radar-upload-connect-frontend

radar-upload-source-connector

s3-proxy

velero-s3-deployment

github-actions bot commented Nov 9, 2023 • edited Loading

Bdegraaf1234 left a comment

Choose a reason for hiding this comment

yatharthranjan commented Nov 22, 2023 • edited Loading

keyvaann commented Nov 22, 2023

yatharthranjan commented Nov 23, 2023

keyvaann commented Nov 23, 2023

yatharthranjan left a comment • edited Loading

Choose a reason for hiding this comment

yatharthranjan Nov 23, 2023

Choose a reason for hiding this comment

keyvaann Nov 23, 2023

Choose a reason for hiding this comment

mpgxvii Nov 24, 2023 • edited Loading

Choose a reason for hiding this comment

keyvaann Nov 24, 2023

Choose a reason for hiding this comment

yatharthranjan left a comment

Choose a reason for hiding this comment

yatharthranjan Nov 29, 2023

Choose a reason for hiding this comment

keyvaann Nov 29, 2023

Choose a reason for hiding this comment

yatharthranjan Nov 29, 2023

Choose a reason for hiding this comment

keyvaann Nov 29, 2023

Choose a reason for hiding this comment

keyvaann commented Nov 29, 2023

yatharthranjan left a comment

Choose a reason for hiding this comment

keyvaann commented Nov 9, 2023 •

edited

Loading

github-actions bot commented Nov 9, 2023 •

edited

Loading

yatharthranjan commented Nov 22, 2023 •

edited

Loading

yatharthranjan left a comment •

edited

Loading

mpgxvii Nov 24, 2023 •

edited

Loading