SMAC time budget walltime vs CPU time

[thijssnelleman]

Hi,

I am currently implementing SMAC3 2.2.0 into my software, and want to use both Walltime and CPU time budgets (Not at the same time persay). However, when investigating the package, I noted that cputime_limit is rarely used and could only find and update in smac/main/smbo.py as the following:

@property def remaining_cputime(self) -> float: """Subtracts the target function running budget with the used time.""" return self._scenario.cputime_limit - self._used_target_function_walltime

However, the self._used_target_function_walltime is used there and there seems to be no self.used_target_function_cputime.

As we currently do our CPU time measurements ourselves, it would be no problem to have these be delivered by the target_function, but currently only the cost can be returned by that.

Would it be possible to have the target function also declare how much CPU time / Wall clock time it used?

Thanks!

Best,

Thijs

[benjamc]

Hi Thijs,
yes, you can add additional information. You can explicitly add time but also other information in the TrialValue object.
See the ask and tell example.
Here is the TrialValue defined.
Are you using SMAC in ask and tell mode?

[thijssnelleman]

Thanks for the quick reply! Looks like I should use Ask and tell mode. Does the as/tell mechanic also optimize the model, or is that only done when using the Facade in .optimize mode?

[benjamc]

At the end of tell (or before ask?), the model is trained. Everything works as usual.

[thijssnelleman]

Then as a last question: TrialValue does not distinguish CPU time from Wallclock in order to update SMAC's budgets. Would it be possible to add this into the package? With some assistance, I could perhaps create a PR for that.

[benjamc]

Yes, currently CPU time (or any time passed to TrialValue) means wallclock time.
Happy to assist in a PR! Although I would need a refresher why you need to have to different times and why it is not the same.:)
When you create a PR, branch off the dev branch.

[thijssnelleman]

Sure! I will send you a slack message with my email adress so we can set up a meeting to discuss