Handle the failure due to reaching the servlet capacity when getting user tasks

[tinaselenge]

Type of change

Select the type of your PR

• Bugfix

Description

If failed to get user tasks due to reaching the servlet capacity, it will no longer transition the KafkaRebalance status to NotReady. Instead, KafkaRebalance status will not be changed and in the next reconciliation, it will retry getting the user tasks again. The operator will also report a warning message.

Resolves #10704

Checklist

Please go through this checklist and make sure all applicable tasks have been done

• Write tests
• Make sure all tests pass
• Update documentation
• Check RBAC rights for Kubernetes / OpenShift roles
• Try your changes from Pod inside your Kubernetes and OpenShift cluster, not just locally
• Reference relevant issue(s) and close them after merging
• Update CHANGELOG.md
• Supply screenshots for visual changes, such as Grafana dashboards

[scholzj]

/azp run build

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[katheris]

Couple of small suggestions, but otherwise LGTM

[ppatierno]

/azp run regression

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[ppatierno]

LGTM. Thanks!

[tinaselenge]

Thank you all for reviewing the PR. I pushed an update addressing the suggestions from @katheris.

[tinaselenge]

@ppatierno can you please kick off the build and regression tests again?

[im-konge]

/azp run build

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[im-konge]

/azp run regression

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[ppatierno]

@tinaselenge I see some failures on regression while one could not be involved, can you double check the failure on testAutoKafkaRebalanceScaleUpScaleDown because this PR is about tweaking the rebalance operator, please? Thanks!

[tinaselenge]

Looks like testAutoKafkaRebalanceScaleUpScaleDown was added recently and did not yet exist in my branch. I rebased to main and then ran this test locally which then passed. I'm also running KafkaRollerST#testKafkaDoesNotRollsWhenTopicIsUnderReplicated locally just in case, as this test had timed out during regression run but it shouldn't be related to the changes anyway.

[ppatierno]

I ran the failed jobs again, let's see.

[ppatierno]

Regression was ok. Going to merge this one.

[tinaselenge]

Thank you @ppatierno !