Strimzi
Replies: 6 comments 4 replies
-
|
Maybe you can format the log properly to make it readable? You should probably also share the full log and not just single exception. Maybe you should also explain how your API server works. Why does it close the connection exactly in this place and not in the many steps before? Also, keep in mind that we do not support Kubernetes 1.22 anymore. |
Beta Was this translation helpful? Give feedback.
0 replies
-
|
Triaged on the community call on 2.5.2024: There does not seem to be a log describing the issue and itis not clear if it affects supported environments. This should be converted to discussion and we can continue there. |
Beta Was this translation helpful? Give feedback.
0 replies
-
|
hey @scholzj I can confirm that this is still an issue. I have plenty of logs :) |
Beta Was this translation helpful? Give feedback.
0 replies
-
|
Beta Was this translation helpful? Give feedback.
0 replies
-
|
These errors appear randomly and fires some alerts at our argoCD. Steps to reproduce
Expected behavior
Strimzi version Kubernetes version Installation method
Infrastructure
Configuration files and logs Strimzi config is pretty much standard Additional context
|
Beta Was this translation helpful? Give feedback.
4 replies
-
|
So, I'm not sure if Strimzi can do something about this. From the log, the Kubernetes client gets the error from the Kubernetes API server and raises it to use. What Strimzi can do at its best is to handle it differently. But from the more complete log you shared it seems to handle it well -> it fails the ongoing reconciliation and the next one is started later and passes. Not sure what else we could do. TBH, I have no idea what the GOAWAY even means. I guess you can ask the Fabric8 Kubernetes client if they ever saw it and know what it means or if they see anything better they could do about it. But the exception comes from Java itself, so it might be something that needs to be fixed in Java itself. |
Beta Was this translation helpful? Give feedback.
-
|
@scholzj First of all. Thank you very much for the fast response. I really appreciate it! GOAWAY is an http2 signal sent by a server to indicate that it will no longer accept new streams (requests) on the current connection. It doesn't close the connection immediately but instead allows any ongoing streams to finish processing before the connection is fully terminated. Probably kubernetes control plane finished the connection with strimzi and its indicating that a new one must be created if something else is needed |
Beta Was this translation helpful? Give feedback.
-
|
As @oudb mentioned, I think that strimzi should do a retry in the same step and not fail and try again. |
Beta Was this translation helpful? Give feedback.
-
|
If retry is really the right approach, I would expect it to be done already in the Kubernetes client. I do not think we want to handle this in Strimzi on our level where it needs to be done separately for every single type of call. |
Beta Was this translation helpful? Give feedback.
-
|
This behavior is also visible with the KafkaRebalance resource... they turn to "NotReady" status because strimzi receives a GOAWAY from the kubernetes API. A minor inconvenience in my case but I thought it could be a good addition to this thread. |
Beta Was this translation helpful? Give feedback.
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
Bug Description
when the k8s apiserver use--goaway-chance flag, apiserver will randomly close the client http2 connection(GOAWAY)，but fabric8 client do not retry. this make kafka reconciliation failed。 the failed log is：
[2024-04-22 17:15:14]
content: 2024-04-22 17:15:14 WARN VertxUtil:103 - Reconciliation #2947(timer) Kafka(strimzi-kafka/kafka35t3p0m8406): Caught exception while waiting for Pods resource kafka35t3p0m8406-1077-3000-3000 in namespace strimzi-kafka to get ready
-23
[2024-04-22 17:15:14]
content: io.fabric8.kubernetes.client.KubernetesClientException: Operation: [get] for kind: [Pod] with name: [kafka35t3p0m8406-1077-3000-3000] in namespace: [strimzi-kafka] failed.
-22
[2024-04-22 17:15:14]
content: at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:159) ~[io.fabric8.kubernetes-client-api-6.9.2.jar:?]
-21
[2024-04-22 17:15:14]
content: at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.requireFromServer(BaseOperation.java:194) ~[io.fabric8.kubernetes-client-6.9.2.jar:?]
-20
[2024-04-22 17:15:14]
content: at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.get(BaseOperation.java:148) ~[io.fabric8.kubernetes-client-6.9.2.jar:?]
-19
[2024-04-22 17:15:14]
content: at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.isReady(BaseOperation.java:912) ~[io.fabric8.kubernetes-client-6.9.2.jar:?]
-18
[2024-04-22 17:15:14]
content: at io.strimzi.operator.common.operator.resource.AbstractReadyNamespacedResourceOperator.isReady(AbstractReadyNamespacedResourceOperator.java:67) ~[io.strimzi.operator-common-0.39.0-0.0.2.jar:0.39.0-0.0.2]
-17
[2024-04-22 17:15:14]
content: at io.strimzi.operator.common.operator.resource.AbstractNamespacedResourceOperator.lambda$waitFor$7(AbstractNamespacedResourceOperator.java:393) ~[io.strimzi.operator-common-0.39.0-0.0.2.jar:0.39.0-0.0.2]
-16
[2024-04-22 17:15:14]
content: at io.strimzi.operator.common.VertxUtil$1.lambda$handle$0(VertxUtil.java:101) ~[io.strimzi.operator-common-0.39.0-0.0.2.jar:0.39.0-0.0.2]
-15
[2024-04-22 17:15:14]
content: at io.vertx.core.impl.ContextImpl.lambda$executeBlocking$0(ContextImpl.java:177) ~[io.vertx.vertx-core-4.5.0.jar:4.5.0]
-14
[2024-04-22 17:15:14]
content: at io.vertx.core.impl.ContextInternal.dispatch(ContextInternal.java:276) ~[io.vertx.vertx-core-4.5.0.jar:4.5.0]
-13
[2024-04-22 17:15:14]
content: at io.vertx.core.impl.ContextImpl.lambda$internalExecuteBlocking$2(ContextImpl.java:209) ~[io.vertx.vertx-core-4.5.0.jar:4.5.0]
-12
[2024-04-22 17:15:14]
content: at io.vertx.core.impl.TaskQueue.run(TaskQueue.java:76) ~[io.vertx.vertx-core-4.5.0.jar:4.5.0]
-11
[2024-04-22 17:15:14]
content: at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
-10
[2024-04-22 17:15:14]
content: at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
-9
[2024-04-22 17:15:14]
content: at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[io.netty.netty-common-4.1.100.Final.jar:4.1.100.Final]
-8
[2024-04-22 17:15:14]
content: at java.lang.Thread.run(Thread.java:840) ~[?:?]
-7
[2024-04-22 17:15:14]
content: Caused by: java.io.IOException: /192.168.176.105:55338: GOAWAY received
-6
[2024-04-22 17:15:14]
content: at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.waitForResult(OperationSupport.java:504) ~[io.fabric8.kubernetes-client-6.9.2.jar:?]
-5
[2024-04-22 17:15:14]
content: at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.handleResponse(OperationSupport.java:524) ~[io.fabric8.kubernetes-client-6.9.2.jar:?]
-4
[2024-04-22 17:15:14]
content: at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.handleGet(OperationSupport.java:467) ~[io.fabric8.kubernetes-client-6.9.2.jar:?]
-3
[2024-04-22 17:15:14]
content: at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.handleGet(BaseOperation.java:791) ~[io.fabric8.kubernetes-client-6.9.2.jar:?]
-2
[2024-04-22 17:15:14]
content: at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.requireFromServer(BaseOperation.java:192) ~[io.fabric8.kubernetes-client-6.9.2.jar:?]
-1
[2024-04-22 17:15:14]
content: ... 13 more
0
[2024-04-22 17:15:14]
content: Caused by: java.io.IOException: /192.168.176.105:55338: GOAWAY received
+1
Steps to reproduce
No response
Expected behavior
No response
Strimzi version
0.39.0
Kubernetes version
Kubernetes 1.22
Installation method
No response
Infrastructure
No response
Configuration files and logs
No response
Additional context
No response
Beta Was this translation helpful? Give feedback.
All reactions