Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

The throughput of the consumer side is only about 10M #522

Open
JuddOslv opened this issue Aug 22, 2024 · 0 comments
Open

The throughput of the consumer side is only about 10M #522

JuddOslv opened this issue Aug 22, 2024 · 0 comments

Comments

@JuddOslv
Copy link

Describe the bug
The throughput of the consumer side is only about 10M. Whether it is to modify the queuesize of the consumer side or to add more consumer nodes, it is of no use.

To Reproduce

  1. Install pulsar to k8s cluster through helm and test normal production and consumption functions
  2. Modify some configuration parameters in helm chart and some initialization scripts, such as the authorization information of pulsar manager user
  3. It looks better after deployment. The pulsar manager background interface can be opened. Except for a few functions with problems, everything looks good
  4. Start developing consumer server code and prepare for testing
  5. The local environment runs in standalone mode, which is very fast. Each batch of 2,000 messages can be obtained, and the next batch can be obtained immediately
  6. But it doesn't work when deployed online. The production end can write more than 100MB, but the consumer end is only about 10M

Expected behavior
The consumer end is expanded from 10 to more than 100, and the consumption throughput has not changed. Modify the consumer end code such as receiverQueueSize and maxNumMessages, and just print the number of messages obtained on the consumer end. Without any business logic, the throughput still cannot be increased

Screenshots
image

Desktop (please complete the following information):
broker: 8c16g 100g ssd
proxy: 8c16g 100g ssd
bookie: 8c32g 1.5T essd * 3

Additional context

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant