Using dask with iris

[DamienIrving]

One major new feature of Iris 2.0.0 was dask integration.

To activate dask in your scripts, add the following:

import dask
dask.set_options(get=dask.get)

This puts you in sequential mode. If you want to try to profit from speedup through parallelisation you might try:

import dask
import dask.multiprocessing
dask.set_options(get=dask.multiprocessing.get)

which would give you parallelisation based on multiprocessing instead of multithreading that should not suffer from the same problem.

[DamienIrving]

In a notebook you can run:

from dask.distributed import Client, progress, LocalCluster

client = LocalCluster() 
client

It will then give you a URL to click to watch all the cores being used on your machine.

This is essentially the method used in the pangeo example notebooks with xarray - I really need to try a MemoryError-inducing task using both xarray and iris on the NCI virtual desktop environment.

[DamienIrving]

There's also some pangeo teaching materials here that cover dask:
https://github.com/jmunroe/pangeo-tutorial-c3dis-2019

The setup for using it on your own machine was:

from dask.distributed import LocalCluster
from dask.distributed import Client

cluster = LocalCluster(n_workers=4)
cluster

client = Client(cluster)
client