Transfer learning benchmarking (continuous vs. discrete)

[sgbaird]

Continuing from #257 (comment) per request by @AVHopp. As @AVHopp mentioned, the basic transfer learning example is set up with the user in mind.

For a benchmarking equivalent of this example, do you have a strong preference towards keeping it a discrete search space, or are you OK with the benchmark using a continuous search space?

[AVHopp]

I can see points for both. In my opinion, the analytical botorch function should be modeled as a continuous space, since it is a continuous function. For a discrete TL benchmark (which we should have), I think it would be more reasonable to also have something that is "naturally discrete".

The only issue that we might currently face is that there is an issue with using our internal simulation tool when doing Transfer Learning in a continuous space as the TaskParameter needs to be handled with care/some hacky stuff needs to be done. However, when not using this functionality but just do a "manual" simulation, this is not an issue, and this is already on the agenda for the next internal discussion.

[AdrianSosic]

Agree, let's simply use what is most natural / serves best as a benchmark. Plus, we should make sure that the benchmarks broadly cover different scenarios, i.e. there should be both (naturally) discrete and continuous spaces included, and perhaps also hybrid ones.

Regarding the "hacky" thing that @AVHopp mentioned: I think you can safely ignore this when setting up the benchmarks. We'll simply push a PR that fixes it (the issue was about reading in data from the lookup, which failed in the presence of TaskParameters, but pretty much only because the entire lookup mechanism was not cleanly implemented. A fix is already WIP)