Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Loosen typeguard upper bound to allow major version 4 #2812

Closed
wants to merge 15 commits into from

Conversation

benclifford
Copy link
Collaborator

This is most immediately driven by installing parsl on top of the lsst_distrib stack which currently has a recent typeguard.

Type of change

  • Code maintentance/cleanup

@benclifford
Copy link
Collaborator Author

This needs a Pytest upgrade first, which I'll do in a different PR

@benclifford benclifford marked this pull request as draft July 12, 2023 08:13
@benclifford
Copy link
Collaborator Author

I fixed up a test to look for typeguards new TypeCheckError, but that now fails on typeguard 2.x (which is not tested by CI as it's installing 4.x):

      assert isinstance(e, (typeguard.TypeCheckError, TypeError, perror.BadStdStreamFile)), "Exception is wrong type"

E AttributeError: module 'typeguard' has no attribute 'TypeCheckError'

@benclifford
Copy link
Collaborator Author

That's a concern for me mostly because typeguard>2 is relatively new and I'm wary of increasing the minimum bound there...

@benclifford
Copy link
Collaborator Author

got this in CI - will rerun and see what happens

PYTHONPATH=/tmp/cctools/lib/python3.8/site-packages  pytest parsl/tests/ -k "not cleannet and not issue363" --config parsl/tests/configs/workqueue_ex.py --random-order --durations 10
============================= test session starts ==============================
platform linux -- Python 3.11.4, pytest-7.4.0, pluggy-1.2.0
Using --random-order-bucket=module
Using --random-order-seed=977312

rootdir: /home/runner/work/parsl/parsl/parsl/tests
configfile: pytest.ini
plugins: random-order-1.1.0, typeguard-4.0.0, cov-4.1.0
collected 268 items / 30 deselected / 2 skipped / 238 selected

parsl/tests/test_error_handling/test_retry_handler.py s                  [  0%]
parsl/tests/test_flux.py ssssssssss                                      [  4%]
parsl/tests/test_error_handling/test_htex_missing_worker.py s            [  5%]
parsl/tests/test_python_apps/test_memoize_joinapp.py .                   [  5%]
parsl/tests/test_python_apps/test_futures.py ....                        [  7%]
parsl/tests/test_python_apps/test_type5.py ..                            [  7%]
parsl/tests/test_checkpointing/test_python_checkpoint_2.py s             [  8%]
parsl/tests/test_staging/test_docs_2.py s                                [  8%]
parsl/tests/test_docs/test_workflow3.py .                                [  9%]
parsl/tests/test_docs/test_from_slides.py .                              [  9%]
parsl/tests/test_checkpointing/test_regression_232.py ss                 [ 10%]
parsl/tests/site_tests/test_provider.py s                                [ 10%]
parsl/tests/test_docs/test_workflow2.py s                                [ 11%]
parsl/tests/test_bash_apps/test_keyword_overlaps.py ..                   [ 12%]
parsl/tests/test_python_apps/test_simple.py ..                           [ 13%]
parsl/tests/test_error_handling/test_wrap_with_logs.py ss                [ 13%]
parsl/tests/test_staging/test_elaborate_noop_file.py ss                  [ 14%]
parsl/tests/sites/test_affinity.py s                                     [ 15%]
parsl/tests/test_thread_parallelism.py ss                                [ 15%]
parsl/tests/test_staging/test_staging_https_in_task.py s                 [ 16%]
parsl/tests/test_regression/test_1606_wait_for_current_tasks.py s        [ 16%]
parsl/tests/test_python_apps/test_memoize_bad_id_for_memo.py ....        [ 18%]
parsl/tests/test_error_handling/test_resource_spec.py .                  [ 18%]
parsl/tests/test_regression/test_854.py ss                               [ 19%]
parsl/tests/test_python_apps/test_pipeline.py ....                       [ 21%]
parsl/tests/test_serialization/test_2555_caching_deserializer.py s       [ 21%]
parsl/tests/test_staging/test_1316.py ss                                 [ 22%]
parsl/tests/test_python_apps/test_overview.py .                          [ 23%]
parsl/tests/test_scaling/test_scale_down.py s                            [ 23%]
parsl/tests/test_python_apps/test_join.py .........                      [ 27%]
parsl/tests/test_python_apps/test_timeout.py .....F

=================================== FAILURES ===================================
_____________________________ test_timeout[0.005] ______________________________

timeout_dur = 0.005

    @pytest.mark.parametrize("timeout_dur", (0.005, 0.01, 0.1))
    def test_timeout(timeout_dur):
>       timeout(kernel, timeout_dur)()

parsl/tests/test_python_apps/test_timeout.py:20: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
parsl/app/python.py:31: in wrapper
    with AutoCancelTimer(seconds, inject_exception, args=[thread]):
parsl/utils.py:371: in __enter__
    self.start()
/opt/hostedtoolcache/Python/3.11.4/x64/lib/python3.11/threading.py:962: in start
    self._started.wait()
/opt/hostedtoolcache/Python/3.11.4/x64/lib/python3.11/threading.py:622: in wait
    signaled = self._cond.wait(timeout)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <Condition(<unlocked _thread.lock object at 0x7efcb0cad180>, 0)>
timeout = None

    def wait(self, timeout=None):
        """Wait until notified or until a timeout occurs.
    
        If the calling thread has not acquired the lock when this method is
        called, a RuntimeError is raised.
    
        This method releases the underlying lock, and then blocks until it is
        awakened by a notify() or notify_all() call for the same condition
        variable in another thread, or until the optional timeout occurs. Once
        awakened or timed out, it re-acquires the lock and returns.
    
        When the timeout argument is present and not None, it should be a
        floating point number specifying a timeout for the operation in seconds
        (or fractions thereof).
    
        When the underlying lock is an RLock, it is not released using its
        release() method, since this may not actually unlock the lock when it
        was acquired multiple times recursively. Instead, an internal interface
        of the RLock class is used, which really unlocks it even when it has
        been recursively acquired several times. Another internal interface is
        then used to restore the recursion level when the lock is reacquired.
    
        """
        if not self._is_owned():
            raise RuntimeError("cannot wait on un-acquired lock")
        waiter = _allocate_lock()
        waiter.acquire()
        self._waiters.append(waiter)
        saved_state = self._release_save()
        gotit = False
        try:    # restore state no matter what (e.g., KeyboardInterrupt)
            if timeout is None:
>               waiter.acquire()
E               parsl.app.errors.AppTimeout

/opt/hostedtoolcache/Python/3.11.4/x64/lib/python3.11/threading.py:320: AppTimeout
---------------------------- Captured log teardown -----------------------------
INFO     parsl.dataflow.dflow:dflow.py:1145 Waiting for all remaining tasks to complete
INFO     parsl.dataflow.dflow:dflow.py:1158 All remaining tasks completed
=============================== warnings summary ===============================
parsl/executors/workqueue/executor.py:46
  /home/runner/work/parsl/parsl/parsl/executors/workqueue/executor.py:46: DeprecationWarning: 'import work_queue' is deprecated. Please instead use: 'import ndcctools.work_queue'
    import work_queue as wq

test_docs/test_workflow3.py::test_parallel_for
  /opt/hostedtoolcache/Python/3.11.4/x64/lib/python3.11/site-packages/_pytest/python.py:198: PytestReturnNotNoneWarning: Expected None, but test_docs/test_workflow3.py::test_parallel_for returned [1, 2, 2, 1], which will be an error in a future version of pytest.  Did you mean to use `assert` instead of `return`?
    warnings.warn(

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
============================= slowest 10 durations =============================
8.08s call     test_python_apps/test_simple.py::test_slow_increment
8.04s call     test_python_apps/test_simple.py::test_increment
6.04s call     test_python_apps/test_join.py::test_combine
6.03s call     test_python_apps/test_pipeline.py::test_increment_slow[3]
6.03s call     test_python_apps/test_pipeline.py::test_increment[3]
4.09s call     test_python_apps/test_futures.py::test_fut_case_3
4.05s call     test_python_apps/test_overview.py::test_1
4.04s call     test_python_apps/test_futures.py::test_fut_case_4
4.03s call     test_docs/test_from_slides.py::test_slides
4.03s call     test_python_apps/test_type5.py::test_func_linked_to_multiple_previous_results
=========================== short test summary info ============================
SKIPPED [1] parsl/tests/configs/ec2_single_node.py:30: 'public_ip' not configured in user_opts.py
SKIPPED [1] parsl/tests/configs/local_threads_globus.py:14: 'globus' not configured in user_opts.py
SKIPPED [1] parsl/tests/test_error_handling/test_retry_handler.py:41: intended for local config
SKIPPED [1] parsl/tests/test_flux.py:47: Flux not available, test will fail
SKIPPED [1] parsl/tests/test_flux.py:120: intended for local config
SKIPPED [1] parsl/tests/test_flux.py:108: intended for local config
SKIPPED [1] parsl/tests/test_flux.py:165: intended for local config
SKIPPED [1] parsl/tests/test_flux.py:152: intended for local config
SKIPPED [1] parsl/tests/test_flux.py:58: Flux not available, test will fail
SKIPPED [1] parsl/tests/test_flux.py:34: Flux not available, test will fail
SKIPPED [1] parsl/tests/test_flux.py:92: intended for local config
SKIPPED [1] parsl/tests/test_flux.py:71: Flux not available, test will fail
SKIPPED [1] parsl/tests/test_flux.py:135: intended for local config
SKIPPED [1] parsl/tests/test_error_handling/test_htex_missing_worker.py:27: intended for local config
SKIPPED [1] parsl/tests/test_checkpointing/test_python_checkpoint_2.py:24: intended for local config
SKIPPED [1] parsl/tests/test_staging/test_docs_2.py:11: intended for local config
SKIPPED [1] parsl/tests/test_checkpointing/test_regression_232.py:16: fails on Travis in pytest
SKIPPED [1] parsl/tests/test_checkpointing/test_regression_232.py:57: fails on Travis in pytest
SKIPPED [1] parsl/tests/site_tests/test_provider.py:20: This test cannot run on sites which cannot be identified by site_config_selector
SKIPPED [1] parsl/tests/test_docs/test_workflow2.py:19: fails with pytest+xdist
SKIPPED [1] parsl/tests/test_error_handling/test_wrap_with_logs.py:27: intended for local config
SKIPPED [1] parsl/tests/test_error_handling/test_wrap_with_logs.py:17: intended for local config
SKIPPED [1] parsl/tests/test_staging/test_elaborate_noop_file.py:64: intended for local config
SKIPPED [1] parsl/tests/test_staging/test_elaborate_noop_file.py:34: intended for local config
SKIPPED [1] parsl/tests/sites/test_affinity.py:43: intended for local config
SKIPPED [1] parsl/tests/test_thread_parallelism.py:24: fails intermittently in pytest
SKIPPED [1] parsl/tests/test_thread_parallelism.py:45: fails intermittently in pytest
SKIPPED [1] parsl/tests/test_staging/test_staging_https_in_task.py:19: intended for local config
SKIPPED [1] parsl/tests/test_regression/test_1606_wait_for_current_tasks.py:15: intended for local config
SKIPPED [1] parsl/tests/test_regression/test_854.py:39: intended for local config
SKIPPED [1] parsl/tests/test_regression/test_854.py:19: intended for local config
SKIPPED [1] parsl/tests/test_serialization/test_2555_caching_deserializer.py:12: intended for local config
SKIPPED [1] parsl/tests/test_staging/test_1316.py:43: intended for local config
SKIPPED [1] parsl/tests/test_staging/test_1316.py:66: intended for local config
SKIPPED [1] parsl/tests/test_scaling/test_scale_down.py:63: intended for local config
!!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!
= 1 failed, 37 passed, 35 skipped, 30 deselected, 2 warnings in 101.26s (0:01:41) =
make: *** [Makefile:73: wqex_local_test] Error 1
Error: Process completed with exit code 2.

khk-globus added a commit that referenced this pull request Jul 21, 2023
As seen in CI at least once, it's possible that the noop() function takes too
long to complete.  Address by:

- increasing the timeout
- removing the now-deemed-extraneous other parameterizations
- and verifying the test pre-condition first.  ("We expect to get a result!")

Reference problem observed: #2812 (comment)
benclifford pushed a commit that referenced this pull request Jul 21, 2023
As seen in CI at least once, it's possible that the noop() function takes too
long to complete.  Address by:

- increasing the timeout
- removing the now-deemed-extraneous other parameterizations
- and verifying the test pre-condition first.  ("We expect to get a result!")

Reference problem observed: #2812 (comment)
@benclifford
Copy link
Collaborator Author

typeguard 4.0.1, released on 27th July, introduces new complications ... see agronholm/typeguard#372

I think this is a typeguard bug.

benclifford added a commit that referenced this pull request Nov 13, 2023
commit fa66de4
Merge: 46f0a5f c39700b
Author: Ben Clifford <[email protected]>
Date:   Thu Jul 20 08:15:14 2023 +0000

    Merge remote-tracking branch 'origin/master' into benc-typeguard-4

commit 46f0a5f
Author: Ben Clifford <[email protected]>
Date:   Wed Jul 19 15:10:34 2023 +0000

    Remote unused from __future__ import

commit b8e5cb9
Author: Ben Clifford <[email protected]>
Date:   Wed Jul 19 14:23:51 2023 +0000

    Upgrade typeguard, because TypeCheckError is new in typeguard 4

commit b452efd
Author: Ben Clifford <[email protected]>
Date:   Wed Jul 19 13:02:32 2023 +0000

    Expect a different type error in tests with typeguard 4.0.0

commit 177f1e4
Merge: 70e0879 b9f2ca3
Author: Ben Clifford <[email protected]>
Date:   Wed Jul 19 12:54:16 2023 +0000

    Merge remote-tracking branch 'origin/master' into benc-typeguard-4

commit 70e0879
Merge: acde393 a001fe2
Author: Ben Clifford <[email protected]>
Date:   Tue Jul 18 15:00:08 2023 +0200

    Merge branch 'master' into benc-typeguard-4

commit acde393
Author: Ben Clifford <[email protected]>
Date:   Wed Jul 12 08:03:59 2023 +0000

    Loosen typeguard upper bound to allow major version 4
@benclifford
Copy link
Collaborator Author

reworked this into #3044

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant