Slim down the repository for cloning by identifying big files.

[michaelJwilson]

Cloning and Binder can take a while. Identify opportunities to slim down or repackage the repository.

[cylammarco]

Minimal work, purely for faster cloning: Use git clone -–depth 1 to avoid cloning the history of the repository

More work: this can get rid of the big files in the older commits, basically rewritting history: BFG repo cleaner. You still need to identify the existing big files for improving cloning speed.

[michaelJwilson]

Agreed, numbering above (1) and (2).

For (1), relies on big files being in the history, but not the current repo. Is this true? Related question: we have an existing Dockerfile that seemingly overrides the environment.yml on Binder and results in a slower build. I paused the Docker builds for this reason. Ideally, we'd clean up to the Docker file to implement --depth and get a fast, tailored build.

For (2), yeh I was thinking BFG - great name! I'm agnostic on rewriting the history. I'd be surprised if the big files didn't have an equivalent that wasn't much smaller.

Thanks for the input!

[cylammarco]

git is very inefficient in versioning Jupyter notebooks, so it can generate a lot of big files very quickly.

[michaelJwilson]

Reviving this,

git clone -–depth 1

+1

also building from minimal_environment (which needs updated to include the requests package, at least). Question is, how do we tell binder to do this? I've totally forgotten, if I ever knew.

[michaelJwilson]

As pointed out by @cylammarco, some merge of this and --depth

https://discourse.jupyter.org/t/how-to-reduce-mybinder-org-repository-startup-time/4956

Including apt-get install of texlive.