[Question] Sum of ensemble weights is not equal to 1. How can that happen?

[kefa23]

Hi everyone,
I'm currently working with AutoSklearn2.0 and I faced the issue that the sum of my ensemble weights is not equal to 1.
If I understood well, the sum should always be equal to 1.

Here is my output for leaderboard(ensemble_only=True) and sprint_statistics():

      rank  ensemble_weight      type     cost   duration                                                          

78 1 0.08 extra_trees 0.35250 12.887410
80 2 0.04 random_forest 0.38000 3.531165
27 3 0.08 extra_trees 0.38125 11.443674
73 4 0.02 extra_trees 0.38250 3.550254
35 5 0.02 random_forest 0.38500 3.836283
112 6 0.02 extra_trees 0.38500 10.285409
63 7 0.02 random_forest 0.38750 3.349284
107 8 0.04 extra_trees 0.39250 4.052376
110 9 0.04 random_forest 0.40125 4.926213
85 10 0.02 random_forest 0.40250 4.314672
116 11 0.06 extra_trees 0.40625 5.042522
108 12 0.04 extra_trees 0.41000 4.623906
101 13 0.06 passive_aggressive 0.41500 5.175897

auto-sklearn results:
Dataset name: 7d71fbd4-badd-11ee-a319-00155df7031a
Metric: accuracy
Best validation score: 0.647500
Number of target algorithm runs: 132
Number of successful target algorithm runs: 108
Number of crashed target algorithm runs: 23
Number of target algorithms that exceeded the time limit: 1
Number of target algorithms that exceeded the memory limit: 0

Please help :)

[kefa23]

Update:
When I call get_models_with_weights(), more models are returned and the sum of the model weights equals 1 like that.
But I'm still confused why leaderboard(ensemble_only=True) and get_models_with_weights() return different things.

[eddiebergman]

Hi @kefa23,

I'm not sure and it sounds like a bug. I will keep this mind in the new reimplementation version whenever that is released! I apologise that I can't give you any more then that.

The bug likely stems from the fact that there's at least two sources of truths for config ids due to AutoSklearn's own config tracking and using SMAC's tracking (SMAC being the underlying Hyperparameter optimization tool).

Best,
Eddie