tests: add test for multiple interrupts and tasks #1347
eyurtsevAnnotations
1 warning and 2 notices
|
benchmark
ubuntu-latest pipelines will use ubuntu-24.04 soon. For more details, see https://github.com/actions/runner-images/issues/10636
|
|
Benchmark results:
libs/langgraph/tests/test_pregel.py#L1
.........................................
fanout_to_subgraph_10x: Mean +- std dev: 61.5 ms +- 1.2 ms
.........................................
fanout_to_subgraph_10x_sync: Mean +- std dev: 53.5 ms +- 0.7 ms
.........................................
fanout_to_subgraph_10x_checkpoint: Mean +- std dev: 74.9 ms +- 0.8 ms
.........................................
fanout_to_subgraph_10x_checkpoint_sync: Mean +- std dev: 95.8 ms +- 1.6 ms
.........................................
fanout_to_subgraph_100x: Mean +- std dev: 604 ms +- 22 ms
.........................................
fanout_to_subgraph_100x_sync: Mean +- std dev: 522 ms +- 12 ms
.........................................
fanout_to_subgraph_100x_checkpoint: Mean +- std dev: 749 ms +- 10 ms
.........................................
fanout_to_subgraph_100x_checkpoint_sync: Mean +- std dev: 963 ms +- 16 ms
.........................................
react_agent_10x: Mean +- std dev: 30.8 ms +- 0.7 ms
.........................................
react_agent_10x_sync: Mean +- std dev: 22.9 ms +- 0.3 ms
.........................................
react_agent_10x_checkpoint: Mean +- std dev: 38.2 ms +- 0.7 ms
.........................................
react_agent_10x_checkpoint_sync: Mean +- std dev: 36.9 ms +- 0.3 ms
.........................................
react_agent_100x: Mean +- std dev: 343 ms +- 6 ms
.........................................
react_agent_100x_sync: Mean +- std dev: 273 ms +- 3 ms
.........................................
react_agent_100x_checkpoint: Mean +- std dev: 636 ms +- 6 ms
.........................................
react_agent_100x_checkpoint_sync: Mean +- std dev: 620 ms +- 6 ms
.........................................
wide_state_25x300: Mean +- std dev: 23.3 ms +- 0.5 ms
.........................................
wide_state_25x300_sync: Mean +- std dev: 15.3 ms +- 0.1 ms
.........................................
wide_state_25x300_checkpoint: Mean +- std dev: 249 ms +- 13 ms
.........................................
wide_state_25x300_checkpoint_sync: Mean +- std dev: 245 ms +- 13 ms
.........................................
wide_state_15x600: Mean +- std dev: 27.4 ms +- 0.6 ms
.........................................
wide_state_15x600_sync: Mean +- std dev: 17.8 ms +- 0.2 ms
.........................................
wide_state_15x600_checkpoint: Mean +- std dev: 429 ms +- 14 ms
.........................................
wide_state_15x600_checkpoint_sync: Mean +- std dev: 424 ms +- 13 ms
.........................................
wide_state_9x1200: Mean +- std dev: 27.2 ms +- 0.5 ms
.........................................
wide_state_9x1200_sync: Mean +- std dev: 17.8 ms +- 0.1 ms
.........................................
wide_state_9x1200_checkpoint: Mean +- std dev: 278 ms +- 13 ms
.........................................
wide_state_9x1200_checkpoint_sync: Mean +- std dev: 274 ms +- 12 ms
|
|
Comparison against main:
libs/langgraph/tests/test_pregel.py#L1
+-----------------------------------------+---------+-----------------------+
| Benchmark | main | changes |
+=========================================+=========+=======================+
| fanout_to_subgraph_100x | 631 ms | 604 ms: 1.04x faster |
+-----------------------------------------+---------+-----------------------+
| fanout_to_subgraph_100x_checkpoint | 776 ms | 749 ms: 1.04x faster |
+-----------------------------------------+---------+-----------------------+
| react_agent_100x_checkpoint_sync | 641 ms | 620 ms: 1.03x faster |
+-----------------------------------------+---------+-----------------------+
| react_agent_100x_checkpoint | 652 ms | 636 ms: 1.02x faster |
+-----------------------------------------+---------+-----------------------+
| wide_state_25x300_checkpoint_sync | 250 ms | 245 ms: 1.02x faster |
+-----------------------------------------+---------+-----------------------+
| fanout_to_subgraph_10x_checkpoint_sync | 97.7 ms | 95.8 ms: 1.02x faster |
+-----------------------------------------+---------+-----------------------+
| wide_state_9x1200_checkpoint_sync | 279 ms | 274 ms: 1.02x faster |
+-----------------------------------------+---------+-----------------------+
| fanout_to_subgraph_100x_checkpoint_sync | 978 ms | 963 ms: 1.02x faster |
+-----------------------------------------+---------+-----------------------+
| wide_state_15x600_checkpoint_sync | 430 ms | 424 ms: 1.01x faster |
+-----------------------------------------+---------+-----------------------+
| react_agent_10x_sync | 23.1 ms | 22.9 ms: 1.01x faster |
+-----------------------------------------+---------+-----------------------+
| wide_state_15x600_sync | 17.9 ms | 17.8 ms: 1.01x faster |
+-----------------------------------------+---------+-----------------------+
| react_agent_10x_checkpoint | 38.5 ms | 38.2 ms: 1.01x faster |
+-----------------------------------------+---------+-----------------------+
| react_agent_100x_sync | 275 ms | 273 ms: 1.01x faster |
+-----------------------------------------+---------+-----------------------+
| wide_state_9x1200_sync | 17.9 ms | 17.8 ms: 1.00x faster |
+-----------------------------------------+---------+-----------------------+
| wide_state_25x300_sync | 15.4 ms | 15.3 ms: 1.00x faster |
+-----------------------------------------+---------+-----------------------+
| react_agent_10x_checkpoint_sync | 37.0 ms | 36.9 ms: 1.00x faster |
+-----------------------------------------+---------+-----------------------+
| Geometric mean | (ref) | 1.01x faster |
+-----------------------------------------+---------+-----------------------+
Benchmark hidden because not significant (12): wide_state_9x1200_checkpoint, wide_state_25x300_checkpoint, wide_state_15x600_checkpoint, fanout_to_subgraph_100x_sync, react_agent_100x, fanout_to_subgraph_10x, wide_state_9x1200, wide_state_15x600, wide_state_25x300, react_agent_10x, fanout_to_subgraph_10x_checkpoint, fanout_to_subgraph_10x_sync
|