Upload folder implementation.

[JohnMoutafis]

Upload a local folder, utilizing the upload_file method.

• Respects the uploading folder structure by default and recreates it in the destination.
• Ingests the files into a given Group, or creates a new one form the folder name

[thetorpedodog]

Why are we treating Amazon URIs specially here?

[JohnMoutafis]

I am now checking if the name contains :// will that suffice?

[thetorpedodog]

why are we turning the exception here into a string? it strips away any useful information that might have been included.

[JohnMoutafis]

I prefer the result to be serializable.

[Shelnutt2]

Do we have a design doc for this? There seems to be miss understanding of what the goals here are. Lets get this cleared up in a design doc before we continue iterating here,

cc: @antalakas

[JohnMoutafis]

@Shelnutt2 I have added you in the relevant ticket

[Shelnutt2]

There is also a recursive=True flag available, I think this might be needed to list the whole hierarchy below?

[JohnMoutafis]

The recursive vfs.ls flag lists every file AND folder in the input_uri.
The upload process does the recursion internally creating a Group for every sub-folder and then uploading the folder level listed files in it.

[Shelnutt2]

@JohnMoutafis can you check the performance of the current implementation?

[JohnMoutafis]

Using the following snippet:

def vfs_ls_recursive(uri):
    vfs = tiledb.VFS()
    files = []
    for fname in vfs.ls(uri):
        files.append(fname)
        if vfs.is_dir(fname):
            files += vfs_ls_recursive(fname)
    return files

to time the implementation's manual recursion against vfs.ls(<folder URI>, recursive=True) and vfs.ls_recursive on a local folder containing sub-folders, got the following results:

• vfs_ls_recursive: 1.18 ms ± 24.4 µs per loop (mean ± std. dev. of 100 runs, 1,000 loops each)
• vfs.ls(<folder URI>, recursive=True): 1.95 ms ± 121 µs per loop (mean ± std. dev. of 100 runs, 1,000 loops each)
• vfs.ls_recursive: 1.96 ms ± 35.4 µs per loop (mean ± std. dev. of 100 runs, 1,000 loops each)

Timings were collected using IPython's %timeit for 100 runs of 1000 loops per run.
The solutions seem comparable.