HyperQueue 0.20.0
New features
-
It is now possible to dynamically submit new tasks into an existing job (we call this concept "Open jobs").
See Open jobs documentation -
Worker streaming. Before, you could stream task stderr/stdout to the server over the network using the
--log
parameter ofhq submit
.
This approach had various issues and was not scalable. Therefore, we have replaced this functionality with worker streaming,
where the streaming of task output to a set of files on disk is performed by workers instead.
This new streaming approach creates more files than original solution (where it was always one file per job),
but the number of files stays small and independent on the number of executed tasks.
The new architecture also allows parallel I/O writing and storing of multiple job streams in one stream handle.
You can use worker streaming using the--stream
parameter ofhq submit
. Check out the documentation for more information. -
Optimization of journal size
-
Tasks' crash counters are not increased when worker is stopped by
hq worker stop
or by time limit.
Removed
- Because worker streaming fully replaces original streaming, the original server streaming was removed.
For most cases, you can rename--log
to--stream
andhq log
tohq output-log
. See the docs for more details.
Fixes
- HQ should no longer crash while printing job info when a failed task does not have any workers
attached (#731).
Note
- Dashboard still not enabled in this version
Artifact summary:
- hq-v0.20.0-*: Main HyperQueue build containing the
hq
binary. Download this archive to
use HyperQueue from the command line. - hyperqueue-0.20.0-*: Wheel containing the
hyperqueue
package with HyperQueue Python
bindings.