Resuming or continuing a (Py)BADS run

[lacerbi]

Moving here a question asked by @pfebrer:

Hi, I'm testing your package and it seems to be working amazingly for my use case :)

However, I have two questions regarding how BADS can be initialized:

1. If a previous BADS run has stopped for some reason, is there a way to continue it from that point?
2. More generally, if I have a known set of points of the function to optimize, is there some way I can feed them into BADS, maybe as initial points?

Thanks!

[lacerbi]

1. The official approach we recognize is to start a new optimization, passing the final point as initial point x0. Resuming a terminated optimization is something we considered, but might not yield the results you expect, for a number of reasons (e.g., if BADS has terminated, in most situation continuing the optimization won't do much).
2. In principle this could be done, but it is not implemented, again for a number of reasons. The presence of additional points not sampled by BADS may or may not help, and in fact it might also hinder the optimization process (I have been playing around with this for the MATLAB version, years ago). Perhaps we could add it as an advanced option "at your own risk", but not likely to be implemented soon.

Tagging @GurjeetSinghSangra in case he has any additional comment.

[pfebrer]

Thanks @lacerbi ! I posted initially in the python version because I thought it could be a python specific thing, e.g. keeping a python object with the state of the optimizer, and then storing it to disk somehow. But I guess this concept can also be realised in MATLAB.

1. In our case each function evaluation is relatively expensive and is done through system calls and then parsing an output file (the function is a fortran program). This system calls are done in an HPC environment, sending jobs to different nodes. So there can be several unexpected issues like network failures, job time limit, filesystem errors, etc... Of course we can implement a check and restart for all of them, but it feels less scary if you can simply resume the optimizer in case of an unexpected issue.

1(b).

The official approach we recognize is to start a new optimization, passing the final point as initial point x0.

I was actually thinking if that would be helpful or not. It was hard to wrap my mind around it because in pure bayesian optimizers one point doesn't mean much, so they would need to resample all the space around them again. But perhaps in BADS it is enough for the optimizer to continue the search without much recomputing(?).

2. Ok, yes I thought that might be the case. I had the intuition that these points would fall out of the grid and I didn't know if they could be handled by BADS out of the box.

[lacerbi]

I posted initially in the python version because I thought it could be a python specific thing, e.g. keeping a python object with the state of the optimizer, and then storing it to disk somehow.

You're right - technically speaking, in Python it wouldn't be hard for us to allow the user to continue an optimization with the current bads object (at least for PyBADS). In fact, it is a feature that we have implemented in other tools (PyVBMC), and we discussed with @GurjeetSinghSangra whether to have it in PyBADS, but I frankly don't remember if we left it in (at least unofficially).

I was actually thinking if that would be helpful or not. It was hard to wrap my mind around it because in pure bayesian optimizers one point doesn't mean much, so they would need to resample all the space around them again. But perhaps in BADS it is enough for the optimizer to continue the search without much recomputing(?).

BADS is a semi-local optimization method (unlike pure Bayesian optimization which has claims of globality). This has both pros and cons. If you start BADS in a good region, it will likely stick around there without the need for sampling outside. Unless perhaps your objective function is very noisy.