Not using all instances?

[ketildanielsen]

After 5000 of n_trials=20000, it has used only 3 of the 18 instances provided. My "scenario.json" confirms all 18 are in the "instances" part, each having 3 features in the "instance_features" part.

Everything else seems to run very well, but I want SMAC to expose the target algorithm to all 18 instances!

Any parameter I should use? Appreciate any advice. The code is a modification of one of the examples:

if __name__ == "__main__":
    model = MySearch ()
    scenario = Scenario(model.configspace,
        deterministic=False,  
        instances=instance_filenames,   # list of 18 file names 
        instance_features=instance_features,   # dict 18 items, each a list of 3 integer features
        seed=-1, 
        n_workers=20, 
        n_trials=20000)

    intensifier = HyperparameterOptimizationFacade.get_intensifier(
        scenario,
        max_config_calls=1,  # We basically use one seed per config only
    )

    smac = HyperparameterOptimizationFacade(
        scenario,
        model.train,
        intensifier=None, 
        overwrite=True,
        logging_level=20,
    )

    incumbent = smac.optimize()

[ketildanielsen]

Update: By setting intensifier=intensifier and max_config_calls to 18, they (all 18) are now used. Very unevenly, not round robin order. In the beginning, it seems to select 5-6 most of the time, then (slowly) the rest are brought in. Anyway, an improvement.

[benjamc]

Hi ketildanielsen,
this is due to the intensification process. During that process we need to estimate whether a new, challenging configuration (called challenger) performs better than the current best incumbent configuration. In order to become the new best, the challenger needs to be at least as good and better on all instances the incumbent is evaluated on.
If we would evaluate each configuration on all instances we would waste evaluations for unpromising challengers. The intensification therefore aims to distribute the compute budget such that new configurations can be tried and configurations are evaluated on many instances.
You can read further about it in the original SMAC paper in section 3.
If you have any further questions, please let us know.