Free Up Storage Space to Support Complex Builds

[hafenkran]

This Pull Request introduces a small change aimed at freeing up storage space to accommodate more complex builds, such as the BigQuery extension. The idea, its benefits, and the problem it addresses are detailed in Issue #127.

The proposed changes add an extra step in the build process that removes cached docker images that are not used.

[samansmink]

@hafenkran do you have any idea how much space this saves?

[hafenkran]

@samansmink about 4gb if I'm not mistaken. If you also delete the tool cache dir, it will save 12-13gb (see my message in the thread). I use the second approach here right now. If you want, I can add those steps as well

[samansmink]

Okay thats pretty good indeed! let's add it.

Im not sure if there are side effects to clearing the /opt/hostedtoolcache directory? would this slow down other parts of the job?

[samansmink]

I'll just merge this one, feel free to open another PR if you feel the other step also makes sense

[hafenkran]

Sorry for the late response—been super busy today and just got a chance to check this out.

To answer your question: since you're using a Docker-based build now, there are no dependencies on the /opt/hostedtoolcache directory. Here’s what’s in that directory:

1.6G /opt/hostedtoolcache/Python
5.1G /opt/hostedtoolcache/CodeQL
664M /opt/hostedtoolcache/PyPy
20K /opt/hostedtoolcache/Java_Temurin-Hotspot_jdk
769M /opt/hostedtoolcache/go
123M /opt/hostedtoolcache/Ruby
432M /opt/hostedtoolcache/node
8.5G /opt/hostedtoolcache

No other steps in your workflow rely on the hostedtoolcache. I’ve adapted your workflow as much as possible, and all steps combined (including the Docker build but excluding the actual extension build + test) just take around 4 minutes, with no "bigger" step except the Docker build. Therefore, I would conclude that clearing the directory should not cause any slow downs.

I'll take another look at the current build, and if necessary, I'll create another PR.

[samansmink]

@hafenkran ah no rush! I just thought I might as well just merge it already

I'll take another look at the current build, and if necessary, I'll create another PR.

Thanks a lot! I'm sure you will not be the only one running into size limitations eventually