[Question] Algorithm configuration for a small config space - too few evaluations

[graidl]

We intend to use SMAC to optimize only few discrete parameters with small domains of a stochastic metaheuristic using the AlgorithmConfigurationFacade. Our issue is that SMAC stops far too early.

To be a bit more specific:

Let the config space bet

config_space = ConfigurationSpace({
        "y": (1, 3), 
        "z":["opt1", "opt2"],
    })

Thus, we have just 3*2=6 possible configurations. Moreover, let us use just one instance.

SMAC stops already after about 26 algorithm evaluations with the message Could not return a new configuration after 16 retries, despite we set n_trials to 2000 and deterministic to False, and the metaheuristic returned quite different values for the same configurations.

Even though the search space for SMAC is very small, clearly a larger number of evaluations would be needed to conclude with some statistical significance that one configuration is best.

What am I missing?

AlgorithmConfigurationFacade sets max_config_calls in the intensifier to 2000, so this cannot be the issue.

Note also that the table in https://automl.github.io/SMAC3/main/3_getting_started.html says that the AlgorithmConfigurationFacade uses the Hyperband Intensifier, which, however I don't think is true.