aiodiskqueue

Persistent queue for Python AsyncIO.

Description

This library provides a persistent FIFO queue for Python AsyncIO:

• Queue content persist a process restart
• Feature parity with Python's asyncio.Queue
• Similar API to Python's asyncio.Queue
• Sane logging
• Type hints
• Fully tested
• Supports different storage engines and can be extended with custom storage engines

Usage

Here is a basic example on how to use the queue:

import asyncio
from aiodiskqueue import Queue

async def main():
    q = await Queue.create("example_queue.sqlite")
    await q.put("some item")
    item = await q.get()
    print(item)

asyncio.run(main())

Please see the examples folder for more usage examples.

Installation

You can install this library directly from PyPI with the following command:

pip install aiodiskqueue

Logging

The name of the logger for all logging by this library is: aiodiskqueue.

Storage Engines

aiodiskqueue support different storage engines. The default engine is DbmEngine.

We measured the throughput for a typical load scenario (5 producers, 1 consumer) with each storage engine:

• DbmEngine: Consistent throughput at low and high volumes and about 3 x faster then Sqlite
• PickledList: Very fast at low volumes, but does not scale well
• SqliteEngine: Consistent throughput at low and high volumes. Relatively slow.

The scripts for running the measurements and generating this chart can be found in the measurements folder.