Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow other queues to be fed #10

Open
DiegoPino opened this issue Nov 3, 2020 · 1 comment
Open

Allow other queues to be fed #10

DiegoPino opened this issue Nov 3, 2020 · 1 comment
Assignees
Labels
API flavor endpoints Quick access to internals via APIs enhancement New feature or request Future Unknown but hopeful about what comes queue FIFO

Comments

@DiegoPino
Copy link
Member

What is this?

This came from a friendly talk I had today. For larger deployments the internal Drupal queues may not be scaleable enough, or said differently, we may have macro queues feeding the Drupal Worker driven ones. The need is simple to accomplish. I will add a queue service that can be extended via plugins to feed others/too/instead and then we can figure out if an AWS/rabbit to even a RocksDB one can be added as recipient for pending processes as external services.

@DiegoPino DiegoPino self-assigned this Nov 3, 2020
@DiegoPino DiegoPino added enhancement New feature or request API flavor endpoints Quick access to internals via APIs queue FIFO labels Nov 3, 2020
@DiegoPino
Copy link
Member Author

Note for myself @DiegoPino:
REDIS is bad for large persisting Queues. In an experimental setting, with slow ingesting queues because the site is not public and Cron was set to every few days, the SBR (e.g OCR) queue grew to 500K from a total of 30K objects (15 pages per PDF?). That in memory (redis) became 11 Gbytes.

@DiegoPino DiegoPino added the Future Unknown but hopeful about what comes label Sep 26, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
API flavor endpoints Quick access to internals via APIs enhancement New feature or request Future Unknown but hopeful about what comes queue FIFO
Projects
None yet
Development

No branches or pull requests

1 participant