Raising Environment Error on Single Node Clusters

[benrutter]

This addresses #39 - would appreciate some feedback as I'm not sure if this is a great idea or not.

Currently, if you start client = dask_databricks.client() on a single node databricks cluster, it'll allow you to, but then hang indefinitely if you try to carry out any tasks (makes sense since there are no workers to do them).

My thinking is it would be handy to throw an error if we can since if we're on a single-node cluster we already know that any attempted use of the dask_databricks client is going to end badly.

I wanted to implement this a little like this:

if os.getenv("NUMNODES") == "1":
    raise EnvironmentError(
        "You appear to be running dask-databricks on a "
        "single-node cluster. Dask requires at least one worker node "
        "in order to function as expected."
    )

But databricks doesn't appear to have an environment variable like "NUMNODES" - it does have a "MASTER" environment variable, which for single node clusters tells you the core working as the driver (such as local[4]) and in distributed clusters gives you the tcp url.

I've used that to implement it like this:

if os.getenv("MASTER") and "local[" in os.getenv("MASTER"):

My thinking is that this is a pretty conservative check (owing to the fact that databricks could hypothetically change this implementation detail at some point, and I can't find any documentation of it). If there is no MASTER variable, nothing will get raised, so the only possible false positive is if local[4] is set as MASTER in a multi-node cluster which I don't think is possible?

It's still a little hackier than I'd of liked though, but does boost the user-friendliness for beginners. Any thoughts?

[jacobtomlinson]

Many thanks for raising this. The hackiness is a shame, but I agree it's better than the alternative experience. I left a couple of small comments.

[benrutter]

Thanks for the review!