Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Client side caching refactoring #3350

Merged
merged 89 commits into from
Sep 27, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
89 commits
Select commit Hold shift + click to select a range
629d3db
Restructure client side caching code
gerzse Jul 19, 2024
48607e9
Temporary refactor
vladvildanov Aug 6, 2024
2c0c812
Finished CacheProxyConnection implementation, added comments
vladvildanov Aug 7, 2024
a734303
Added test cases and scheduler dependency
vladvildanov Aug 7, 2024
339735c
Added support for RedisCluster and multi-threaded test cases
vladvildanov Aug 8, 2024
88f7e54
Added support for BlockingConnectionPool
vladvildanov Aug 8, 2024
936be84
Fixed docker-compose command
vladvildanov Aug 8, 2024
eb95bd3
Revert port changes
vladvildanov Aug 8, 2024
2c50adc
Initial take on Sentinel support
vladvildanov Aug 9, 2024
6f582a0
Remove keys option after usage
vladvildanov Aug 9, 2024
59fe379
Added condition to remove keys entry on async
vladvildanov Aug 9, 2024
eaeef12
Added same keys entry removal in pipeline
vladvildanov Aug 9, 2024
33f656e
Added caching support for Sentinel
vladvildanov Aug 12, 2024
e77cb60
Added locking when accesing cache object
vladvildanov Aug 13, 2024
2a14e13
Rmoved keys option from options
vladvildanov Aug 13, 2024
64fb176
Removed redundant entities
vladvildanov Aug 15, 2024
47e6c7a
Added cache support for SSLConnection
vladvildanov Aug 16, 2024
eec44bd
Moved ssl argument handling to cover cluster case
vladvildanov Aug 16, 2024
5422955
Revert local test changes
vladvildanov Aug 16, 2024
41190d7
Fixed bug with missing async operator
vladvildanov Aug 16, 2024
6146d13
Revert accidental changes
vladvildanov Aug 16, 2024
d184d6b
Added API to return cache object
vladvildanov Aug 19, 2024
fe124e1
Added eviction policy configuration
vladvildanov Aug 19, 2024
21778be
Added mark to skip test on cluster
vladvildanov Aug 19, 2024
0eb018b
Merge branch 'master' into client-side-caching-improvements
vladvildanov Aug 19, 2024
19c8f35
Removed test case that makes no sense
vladvildanov Aug 20, 2024
3d7f84b
Merge branch 'client-side-caching-improvements' of github.com:redis/r…
vladvildanov Aug 20, 2024
7edb46b
Skip tests in RESP2
vladvildanov Aug 20, 2024
e6ebab6
Added scheduler to dev_requirements
vladvildanov Aug 20, 2024
852c36f
Codestyle changes
vladvildanov Aug 20, 2024
726803a
Fixed characters per line restriction
vladvildanov Aug 20, 2024
bed4d73
Fixed line length
vladvildanov Aug 20, 2024
a78690a
Removed blank lines in imports
vladvildanov Aug 20, 2024
c685248
Fixed imports codestyle
vladvildanov Aug 20, 2024
65bd5af
Added CacheInterface abstraction
vladvildanov Aug 21, 2024
9e9b68c
Removed redundant references
vladvildanov Aug 21, 2024
ba94215
Moved hardcoded values to constants, restricted dependency versions
vladvildanov Aug 21, 2024
2572167
Changed defaults to correct values
vladvildanov Aug 21, 2024
74499a9
Added custom background scheduler, added unit testing
vladvildanov Aug 23, 2024
da32b9e
Codestyle changes
vladvildanov Aug 23, 2024
36de29b
Updated RESP2 restriction
vladvildanov Aug 27, 2024
b023619
Cahnged typing to more generic
vladvildanov Aug 27, 2024
9fe36f8
Restrict pytest-asyncio version to 0.23
vladvildanov Aug 27, 2024
8a2a01e
Added upper version limit
vladvildanov Aug 27, 2024
7b50300
Removed usntable multithreaded tests
vladvildanov Aug 27, 2024
73cb068
Removed more flacky multithreaded tests
vladvildanov Aug 27, 2024
ad7e977
Fixed issue with Sentinel killing healthcheck thread before execution
vladvildanov Aug 27, 2024
c106873
Removed cachetools dependency, added custom cache implementation
vladvildanov Aug 29, 2024
d78c5d3
Updated test cases
vladvildanov Aug 29, 2024
9cac761
Updated typings
vladvildanov Aug 29, 2024
43af6ee
Updated types
vladvildanov Aug 29, 2024
fa1a431
Revert changes
vladvildanov Aug 29, 2024
0ffc298
Removed use_cache, make health_check configurable, removed retry logi…
vladvildanov Aug 30, 2024
4459dd6
Revert test skip
vladvildanov Aug 30, 2024
6e522e1
Merge branch 'master' into client-side-caching-improvements
vladvildanov Sep 2, 2024
9fb5aa2
Added documentation and codestyle fixes
vladvildanov Sep 2, 2024
13a9b2d
Merge branch 'client-side-caching-improvements' of github.com:redis/r…
vladvildanov Sep 2, 2024
01e405e
Updated excluded wordlist
vladvildanov Sep 2, 2024
6b61f1a
Merge branch 'master' into client-side-caching-improvements
vladvildanov Sep 2, 2024
9ddc9c6
Merge branch 'master' into client-side-caching-improvements
vladvildanov Sep 3, 2024
b5a8133
Added health_check thread cancelling in BlockingPool
vladvildanov Sep 3, 2024
471c7cb
Merge branch 'client-side-caching-improvements' of github.com:redis/r…
vladvildanov Sep 3, 2024
face34a
Merge branch 'master' into client-side-caching-improvements
vladvildanov Sep 4, 2024
4035ce6
Revert argument rename, extended documentation
vladvildanov Sep 5, 2024
b434efd
Merge branch 'client-side-caching-improvements' of github.com:redis/r…
vladvildanov Sep 5, 2024
6c47c64
Updated NodesManager to create shared cache between all nodes
vladvildanov Sep 9, 2024
c47d4aa
Codestyle fixes
vladvildanov Sep 9, 2024
fd361a7
Updated docs
vladvildanov Sep 9, 2024
97ebebf
Added version restrictions
vladvildanov Sep 10, 2024
6387a86
Added missing property getter
vladvildanov Sep 10, 2024
097d92e
Updated Redis server version
vladvildanov Sep 10, 2024
c2a4edf
Skip on long exception message
vladvildanov Sep 10, 2024
c110207
Removed keys entry as it's csc specific
vladvildanov Sep 12, 2024
e3980b6
Merge branch 'master' into client-side-caching-improvements
vladvildanov Sep 12, 2024
ffff100
Updated exception message for CSC
vladvildanov Sep 12, 2024
f684739
Merge branch 'client-side-caching-improvements' of github.com:redis/r…
vladvildanov Sep 12, 2024
9852b78
Updated condition by adding server name check
vladvildanov Sep 12, 2024
97abacd
Added test coverage for decoded responses
vladvildanov Sep 13, 2024
20f3851
Codestyle changes
vladvildanov Sep 13, 2024
2d8cf27
Removed background healthcheck, use connection reference approach ins…
vladvildanov Sep 19, 2024
1f6876a
Removed unused imports
vladvildanov Sep 19, 2024
30717aa
Fixed broken tests
vladvildanov Sep 19, 2024
98bf72d
Codestyle changes
vladvildanov Sep 19, 2024
cdf8504
Fixed additional broken tests
vladvildanov Sep 19, 2024
7340aad
Codestyle changes
vladvildanov Sep 19, 2024
c36219f
Increased timer to avoid flackiness
vladvildanov Sep 19, 2024
6d63a59
Restrict tests cause of PyPy
vladvildanov Sep 19, 2024
18a4d3d
Codestyle changes
vladvildanov Sep 19, 2024
94ddab1
Updated docs, convert getters function to properties, added dataclasses
vladvildanov Sep 27, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .github/wordlist.txt
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
APM
ARGV
BFCommands
CacheImpl
CFCommands
CMSCommands
ClusterNode
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/integration.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ env:
CODECOV_TOKEN: ${{ secrets.CODECOV_TOKEN }}
# this speeds up coverage with Python 3.12: https://github.com/nedbat/coveragepy/issues/1665
COVERAGE_CORE: sysmon
REDIS_IMAGE: redis:7.4-rc2
REDIS_IMAGE: redis:latest
REDIS_STACK_IMAGE: redis/redis-stack-server:latest

jobs:
Expand Down
1 change: 0 additions & 1 deletion dev_requirements.txt
Original file line number Diff line number Diff line change
@@ -1,5 +1,4 @@
black==24.3.0
cachetools
click==8.0.4
flake8-isort
flake8
Expand Down
2 changes: 0 additions & 2 deletions docs/examples/connection_examples.ipynb
Original file line number Diff line number Diff line change
Expand Up @@ -69,9 +69,7 @@
},
{
"cell_type": "markdown",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"### By default this library uses the RESP 2 protocol. To enable RESP3, set protocol=3."
]
Expand Down
32 changes: 32 additions & 0 deletions docs/resp3_features.rst
Original file line number Diff line number Diff line change
Expand Up @@ -67,3 +67,35 @@ This means that should you want to perform something, on a given push notificati
>> p = r.pubsub(push_handler_func=our_func)

In the example above, upon receipt of a push notification, rather than log the message, in the case where specific text occurs, an IOError is raised. This example, highlights how one could start implementing a customized message handler.

Client-side caching
-------------------

Client-side caching is a technique used to create high performance services.
It utilizes the memory on application servers, typically separate from the database nodes, to cache a subset of the data directly on the application side.
For more information please check `official Redis documentation <https://redis.io/docs/latest/develop/use/client-side-caching/>`_.
Please notice that this feature only available with RESP3 protocol enabled in sync client only. Supported in standalone, Cluster and Sentinel clients.

Basic usage:

Enable caching with default configuration:

.. code:: python

>>> import redis
>>> from redis.cache import CacheConfig
>>> r = redis.Redis(host='localhost', port=6379, protocol=3, cache_config=CacheConfig())

The same interface applies to Redis Cluster and Sentinel.

Enable caching with custom cache implementation:

.. code:: python

>>> import redis
>>> from foo.bar import CacheImpl
>>> r = redis.Redis(host='localhost', port=6379, protocol=3, cache=CacheImpl())

CacheImpl should implement a `CacheInterface` specified in `redis.cache` package.

More comprehensive documentation soon will be available at `official Redis documentation <https://redis.io/docs/latest/>`_.
Loading
Loading