You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Hello, I am using Dask with IBM LSF cluster, I noted that the current LSF Cluster implementation does not give the ability to start an LSF job on multiple hosts, which can be achieved by using the LSF blaunch command.
Modification proposal
The current way of starting multiple workers on one job that has multiple tasks (-n bsub option) is by setting the --nworkers option of distributed.cli.dask_worker to the number of tasks. Could it be a better option to use instead the blaunch command provided by LSF to run a command on each requested task ? In this way the workers will be dispatched on the different hosts provided by LSF and the span[hosts=1] will not be required anymore.
Here a test code I made by subclassing LSFJob class to try using blaunch.
Hello, I am using Dask with IBM LSF cluster, I noted that the current LSF Cluster implementation does not give the ability to start an LSF job on multiple hosts, which can be achieved by using the LSF
blaunch
command.Modification proposal
The current way of starting multiple workers on one job that has multiple tasks (
-n
bsub option) is by setting the--nworkers
option ofdistributed.cli.dask_worker
to the number of tasks. Could it be a better option to use instead theblaunch
command provided by LSF to run a command on each requested task ? In this way the workers will be dispatched on the different hosts provided by LSF and thespan[hosts=1]
will not be required anymore.Here a test code I made by subclassing LSFJob class to try using
blaunch
.This code is not usable as is, but I think the changes will only be limited to the constructor of the
LSFJob
class.Is this an interesting idea ?
The text was updated successfully, but these errors were encountered: