Dask tests

[rsignell-usgs]

As requested here by @mrocklin: pangeo-data/pangeo#75 (comment)

• Try XArray + Dask locally on the HSDS data to verify that it can be accessed concurrently from multiple threads
• Try XArray + Dask.distributed locally on the HSDS data to verify that the h5pyd objects can survive being serialized
• Try everything on a distributed cluster using KubeCluster and then look at the performance of scalable computing
• Try this all again on a cluster on S3, where presumably we would expect 100-200MB/s network access from each node.

[rsignell-usgs]

@jreadey , do you think you might be able to take a stab at these?

[jreadey]

@rsignell-usgs - yes I think so, but may need to do a bit of self-education on Dask.

What is KubeCluster?

[mrocklin]

KubeCluster is what we use on pangeo.pydata.org (which you might consider trying out) to launch dask on kubernetes. It comes from https://github.com/yuvipanda/daskernetes

…

On Mon, Jan 15, 2018 at 11:51 PM, John Readey ***@***.***> wrote: @rsignell-usgs <https://github.com/rsignell-usgs> - yes I think so, but may need to do a bit of self-education on Dask. What is KubeCluster? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#51 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AASszHCNRBK4-8m7SGTVe4FXK5KhbVNCks5tLCrBgaJpZM4Re7DA> .

[mrocklin]

Note that we're currently deploying from this branch: dask/dask-kubernetes#22

…

On Tue, Jan 16, 2018 at 9:00 AM, Rich Signell ***@***.***> wrote: Looks like KubeCluster is part of daskernetes: https://github.com/yuvipanda/daskernetes/blob/master/ daskernetes/tests/test_core.py#L6 This is used in the pangeo xarray-data.ipynb example, so for me, it's http://pangeo.pydata.org/user/rsignell-usgs/notebooks/ examples/xarray-data.ipynb (I can't find the github repo for this notebook!) — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#51 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AASszLQPyVhZymY0yV5r4a_XGGvJde4Sks5tLKttgaJpZM4Re7DA> .