Fail when "error(s) occurred while compiling module(s)"

[yuzhouliu9]

Running OPA 0.54.0 and kube-mgmt v8.3.0

We frequently see errors like these from kube-mgmt

level=info msg="Added policy opa/opa-istio-whitelist-rego-policy/opa-istio-whitelist.rego, err=code invalid_parameter: error(s) occurred while compiling module(s)"

It is a problem from our .rego that we will address.

But the issue is that kube-mgmt container continues on even with the error. The rules we've defined are not enforced on this OPA pod.
Is there a way to fail the container when this error occurs?

[eshepelyuk]

there is no way to fail kube-mgmt container, whatever you mean with fail.

[yuzhouliu9]

The behavior we observe is that when the invalid_parameter error occurs, the .rego defined is not enforced. Thus defeating the purpose of having OPA.

kube-mgmt container should restart or error. This is a better signal than just logging this error.

[eshepelyuk]

if kube-mgmt pod restarts on such error, it will re apply the same incorrect rego from the same ConfigMap and will receive the same error.

So, what is the purpose of restart ?

[yuzhouliu9]

It will signal to the operator something is wrong immediately.

Think of it from a user perspective:

1. A single error line occurs in kube-mgmt container. OPA rules for that .rego is not enforced.
2. OPA pod is ready, starts serving requests. Lots of logs generated indicating all is well, and that single log is buried.
3. Then one day a request that was supposed to be denied by OPA is allowed in. The operator has to look through all the OPA logs to find that one line from 60 days ago when kube-mgmt started to find the cause.

This is what happened in our case.

[eshepelyuk]

• kube-mgmt watches ConfigMaps and applies their content to running OPA instance
• if error happens, kube-mgmt adds an openpolicyagent.org/kube-mgmt-status annotation to ConfigMap and logs the error
• if you want to react quickly on incorrect OPAs, you should configure your observability tools\CD pipelines to monitor ConfigMap containing OPA sources for the value of status annotation

[yuzhouliu9]

If there are multiple OPA pods, does openpolicyagent.org/kube-mgmt-status annotation get overwritten by newer pods?

[eshepelyuk]

If there are multiple OPA pods, does openpolicyagent.org/kube-mgmt-status annotation get overwritten by newer pods?

kube mgmt doesn't support scaling opa pods
#194

[yuzhouliu9]

We deploy multiple replica OPA pods with opa and kube-mgmt container 1:1.

I was not aware this is unsupported. Not supporting multiple replicas seems like a huge gap for something that's supposed to be kubernetes native?

It seems almost impossible to me that there are production use cases with a single OPA pod.

[eshepelyuk]

This is OSS
You're always welcome to address the issue by providing PR.