Skip to content

Commit

Permalink
v1.4.0
Browse files Browse the repository at this point in the history
---------
2023-07-13:
    - check for killed child processes (for instance by OOMKiller)
    - change dill requirements for windows
    - require minimum python 3.8
    - remove python 3.7 tests
    - introduce PEP517 packaging standard
    - introduce pyproject.toml build-system
    - remove mypy.ini
    - remove pytest.ini
    - remove setup.cfg
    - remove setup.py
    - remove .bettercodehub.yml
    - remove .travis.yml
    - update black config
    - clean ./tests/test_cli.py
    - add codeql badge
    - move 3rd_party_stubs outside the src directory to ``./.3rd_party_stubs``
    - add pypy 3.10 tests
    - add python 3.12-dev tests
  • Loading branch information
bitranox committed Jul 14, 2023
1 parent b71e9d4 commit 530574d
Show file tree
Hide file tree
Showing 3 changed files with 82 additions and 32 deletions.
39 changes: 28 additions & 11 deletions .docs/description.rst
Original file line number Diff line number Diff line change
@@ -1,16 +1,33 @@
there are many timeout decorators out there - that one focuses on correctness when using with Classes, methods,
class methods, static methods and so on, preserving also the traceback information for Pycharm debugging.
There are several timeout decorators available, but the one mentioned here
focuses on ensuring correctness when used with classes, methods, class methods,
static methods, etc. It also preserves traceback information for PyCharm debugging.

There is also a powerful eval function, it allows to read the desired timeout value even from Class attributes.
Additionally, there is a powerful eval function that allows reading
the desired timeout value even from class attributes.

There are two timeout strategies implemented, the ubiquitous method using "Signals" and the second using Multiprocessing.
Using "Signals" is slick and lean, but there are nasty caveats, please check section `Caveats using Signals`_
Two timeout strategies have been implemented:
one using "Signals" and the other using "Multiprocessing".

The default strategy is therefore using Multiprocessing, but You can also use Signals, You have been warned !
Signals
-------

Due to the lack of signals on Windows, or for threaded functions (in a subthread) where signals cant be used, Your only choice is Multiprocessing,
this is set automatically.
The "Signals" strategy (for POSIX Systems) is elegant and efficient,
but it has some important caveats which should be reviewed
in the `Caveats using Signals`_ section.

Under Windows the decorated function and results needs to be pickable.
For that purpose we use "multiprocess" and "dill" instead of "multiprocessing" and "pickle", in order to be able to use this decorator on more sophisticated objects.
Communication to the subprocess is done via "multiprocess.pipe" instead of "queue", which is faster and might also work on Amazon AWS.

Multiprocessing
---------------

The default strategy is to use Multiprocessing

- on Windows, due to the lack of signals, this is only available choice, which is enforced automatically
- signals (on POSIX) can not be used in a subthread, therefore multiprocessing is enforced in such cases

When using a subprocess many types from multiprocessing need to be pickable so that child processes can use them.
Therefore we use "dill" instead of "pickle" and "multiprocess" instead of "multiprocessing".

dill extends python’s pickle module for serializing and de-serializing python objects to the majority of the built-in python types

Communication with the subprocess is done via "multiprocess.pipe" instead of "queue",
which offers improved speed and may also work on Amazon AWS.
18 changes: 13 additions & 5 deletions .docs/parts/general_recommendations.rst
Original file line number Diff line number Diff line change
@@ -1,11 +1,19 @@
General Recommendations
-----------------------
dont sprincle Your code with timeouts. Just use it were absolutely necessary, for instance when reading or writing a file. And do that on the lowest
abstraction level possible to avoid unwanted side effects (Exceptions caught by some other code, non pickable functions or arguments, and so on, but not TOO
low. Remember that forking the program takes some time (when use multiprocessing).
Minimize the excessive use of timeouts in your code and reserve their usage for essential cases.

Implement timeouts at the lowest possible abstraction level to prevent unintended
consequences such as exceptions being caught by unrelated code or non-pickable
functions and arguments.

Avoid incorporating the Timeout Decorator within a loop that iterates multiple times,
as this can result in considerable time overhead,
especially on Windows systems, when utilizing multiprocessing.

Whenever feasible, leverage the existing built-in timeouts already provided by the functions
and libraries you utilize. These built-in timeouts can handle most situations effectively.
Only resort to use this Timeout Decorator as a last resort when all other alternatives have been exhausted.

Most functions and libraries You call, they HAVE already some timeouts. use those. This Timeout Decorator should be only the last ressort, if everything else
fails.

BAD EXAMPLE (Pseudocode) - lets assume the write to the database fails sometimes for unknown reasons, and "hangs"

Expand Down
57 changes: 41 additions & 16 deletions README.rst
Original file line number Diff line number Diff line change
Expand Up @@ -51,22 +51,39 @@ Version v1.4.0 as of 2023-07-14 see `Changelog`_
:target: https://pypi.org/project/wrapt-timeout-decorator/
:alt: PyPI - Downloads

there are many timeout decorators out there - that one focuses on correctness when using with Classes, methods,
class methods, static methods and so on, preserving also the traceback information for Pycharm debugging.
There are several timeout decorators available, but the one mentioned here
focuses on ensuring correctness when used with classes, methods, class methods,
static methods, etc. It also preserves traceback information for PyCharm debugging.

There is also a powerful eval function, it allows to read the desired timeout value even from Class attributes.
Additionally, there is a powerful eval function that allows reading
the desired timeout value even from class attributes.

There are two timeout strategies implemented, the ubiquitous method using "Signals" and the second using Multiprocessing.
Using "Signals" is slick and lean, but there are nasty caveats, please check section `Caveats using Signals`_
Two timeout strategies have been implemented:
one using "Signals" and the other using "Multiprocessing".

The default strategy is therefore using Multiprocessing, but You can also use Signals, You have been warned !
Signals
-------

The "Signals" strategy (for POSIX Systems) is elegant and efficient,
but it has some important caveats which should be reviewed
in the `Caveats using Signals`_ section.


Multiprocessing
---------------

The default strategy is to use Multiprocessing

Due to the lack of signals on Windows, or for threaded functions (in a subthread) where signals cant be used, Your only choice is Multiprocessing,
this is set automatically.
- on Windows, due to the lack of signals, this is only available choice, which is enforced automatically
- signals (on POSIX) can not be used in a subthread, therefore multiprocessing is enforced in such cases

Under Windows the decorated function and results needs to be pickable.
For that purpose we use "multiprocess" and "dill" instead of "multiprocessing" and "pickle", in order to be able to use this decorator on more sophisticated objects.
Communication to the subprocess is done via "multiprocess.pipe" instead of "queue", which is faster and might also work on Amazon AWS.
When using a subprocess many types from multiprocessing need to be pickable so that child processes can use them.
Therefore we use "dill" instead of "pickle" and "multiprocess" instead of "multiprocessing".

dill extends python’s pickle module for serializing and de-serializing python objects to the majority of the built-in python types

Communication with the subprocess is done via "multiprocess.pipe" instead of "queue",
which offers improved speed and may also work on Amazon AWS.

----

Expand Down Expand Up @@ -124,12 +141,20 @@ Usage
General Recommendations
-----------------------
dont sprincle Your code with timeouts. Just use it were absolutely necessary, for instance when reading or writing a file. And do that on the lowest
abstraction level possible to avoid unwanted side effects (Exceptions caught by some other code, non pickable functions or arguments, and so on, but not TOO
low. Remember that forking the program takes some time (when use multiprocessing).
Minimize the excessive use of timeouts in your code and reserve their usage for essential cases.

Implement timeouts at the lowest possible abstraction level to prevent unintended
consequences such as exceptions being caught by unrelated code or non-pickable
functions and arguments.

Avoid incorporating the Timeout Decorator within a loop that iterates multiple times,
as this can result in considerable time overhead,
especially on Windows systems, when utilizing multiprocessing.

Whenever feasible, leverage the existing built-in timeouts already provided by the functions
and libraries you utilize. These built-in timeouts can handle most situations effectively.
Only resort to use this Timeout Decorator as a last resort when all other alternatives have been exhausted.

Most functions and libraries You call, they HAVE already some timeouts. use those. This Timeout Decorator should be only the last ressort, if everything else
fails.

BAD EXAMPLE (Pseudocode) - lets assume the write to the database fails sometimes for unknown reasons, and "hangs"

Expand Down

0 comments on commit 530574d

Please sign in to comment.