Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(rng): Ensure reproduciblity through construction and reset #84

Merged
merged 3 commits into from
Jul 3, 2024

Conversation

eddiebergman
Copy link
Contributor

@eddiebergman eddiebergman commented May 7, 2024

Heyo,

This address issue #83 which seemed a bit more deep-rooted than I thought, i.e. not just seeding the configspace.
The main change is to never share the rng object itself but rather pass around seeds generated from it, making things like reset() much easier to deal with. Also prevents having to save and store random state object internals.

Tested with the following script, which should probably be made into a full-fledged test for reproducibility.

from __future__ import annotations

from ConfigSpace import ConfigurationSpace

from dehb import DEHB

cs = ConfigurationSpace({"a": (1.0, 10.0)})


run1 = []
run2 = []
run3 = []
run_4 = []
run_list = run1


def f(config, fidelity):
    run_list.append((config["a"], fidelity))
    return {"fitness": config["a"] ** 2, "cost": 1.0, "info": {}}


run_list = run1
D = DEHB(cs, f=f, seed=1, min_fidelity=1, max_fidelity=100, n_workers=1, output_path="test1")
D.run(10)

run_list = run2
D2 = DEHB(cs, f=f, seed=1, min_fidelity=1, max_fidelity=100, n_workers=1, output_path="test2")
D2.run(10)

run_list = run3
D.reset()
D.run(10)

# Run it again to make sure its different than run3
run_list = run_4
D.run(10)


print(run1)
print(run2)
print(run3)
print(run_4)

@Bronzila Bronzila linked an issue May 7, 2024 that may be closed by this pull request
@Bronzila Bronzila changed the base branch from master to development May 10, 2024 14:24
@Bronzila
Copy link
Collaborator

Bronzila commented May 10, 2024

Hey,
I just took some time to look over the changes and they look great! There is however a little extra work needed in order to properly allow restarting. I will try to incorporate these chaneges over the weekend :)

PS: I've redirected the PR to development, but will make sure that it will be part of the next release, thanks for your time :)

@eddiebergman
Copy link
Contributor Author

Nice, sorry I didn't know where to develop off or include. Feel free to close this PR and just integrate the changes into whatever new version you make if it's easier!

@Bronzila Bronzila merged commit f2ee1a9 into development Jul 3, 2024
@Bronzila Bronzila deleted the fix-seeding branch August 12, 2024 08:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Bug] Seeding is not applied to configspace
2 participants