Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Move allow_* flags to Campaign #423

Merged
merged 43 commits into from
Jan 28, 2025
Merged

Move allow_* flags to Campaign #423

merged 43 commits into from
Jan 28, 2025

Conversation

AdrianSosic
Copy link
Collaborator

@AdrianSosic AdrianSosic commented Nov 9, 2024 •
edited
Loading

This PR builds upon #412 and finalizes the metadata migration from search spaces / recommenders to Campaign, making Campaign the only stateful class:

  • All allow_* flags are now passed directly to Campaign, strengthening the role of Campaign as the class for metadata handling / tracking the progress of an experimentation path.
  • A corresponding deprecation mechanism is set into place.
  • The documentation gets updated.
  • allow_repeated_recommendations is renamed to allow_recommending_already_recommended because i) the original name was imprecise in that it also suggested that repetitions are disallowed within a batch and ii) the new follows the same pattern as the other two flags.
  • The flags are now context-aware, i.e. setting flags in contexts where they have no effect now raises an error. This avoids surprises on the user side, which have been reported multiple times.
  • Replaces the private class AnnotatedSubspaceDiscrete with a new private class FilteredSubspaceDiscrete taking over the original role but without the necessity of being metadata aware.

Coming next

The changes introduced here and in #423 bring significant improvements from a user interface perspective in that:

  • Metadata handling now happens entirely behind the scenes
  • No more fiddling with internal low-level objects (like setting Boolean values in metadata dataframes) is necessary to control candidate generation is necessary. Instead, there is now an interface that enables control via user-level objects such as Constraint objects and candidates dataframes.
  • The pandas part of the current discrete search space implementation is less entangled in the code base

This paves the way for upcoming enhancements:

  • A SubspaceDiscreteProtocol is already in sight, which allows seamless integration of other backends (polars, databases, etc) and will help us complete the ongoing polars transition.
  • An improved user interface for manipulating existing state spaces a la SubspaceDiscrete.filter(constraints), which can also become the backbone of the current constraints logic

@AdrianSosic AdrianSosic self-assigned this Nov 9, 2024
@AdrianSosic AdrianSosic mentioned this pull request Nov 9, 2024
Merged
@AdrianSosic AdrianSosic force-pushed the refactor/allow_flags branch 3 times, most recently from 2ccb684 to f79b557 Compare November 14, 2024 07:14
@AdrianSosic AdrianSosic added the enhancement Expand / change existing functionality label Nov 21, 2024
@AdrianSosic AdrianSosic marked this pull request as ready for review November 21, 2024 08:36
AdrianSosic added a commit that referenced this pull request Nov 22, 2024
@AdrianSosic
Immutable searchspace (#412)
98cb4ea 
Fixes #371 by making `SubspaceDiscrete` stateless.

### Current Approach
* The `SubspaceDiscrete.metadata` attribute gets deprecated and the
responsibility of metadata handling is shifted to `Campaign`
* The new mechanism is not yet final (see out of scope below) but
designed in a way that allows to implement upcoming changes in a
non-breaking manner. In particular:
* The metadata handling mechanism is redesigned in that the actual
metadata representation is completely hidden from the user, i.e.
campaign manages the data in form of private attributes. This is to
avoid further lock-in into `pandas` as our search space backend and
prepares for future search space improvements by abstracting away the
specific implementation details, enabling us to easily offer other
backends (polars, databases, etc) in the future.
* The `allow_*` flags are not yet migrated to the `Campaign` class, but
the `AnnotatedSubspaceDiscrete` allows to migrate them in a follow-up PR
(#423) without causing much friction
* A new user-facing method `Campaign.toggle_discrete_candidates` now
allows convenient and dynamic control over the discrete candidate set,
avoiding any fiddling with the backend dataframes and index
manipulations. The positive effect can be seen in the much cleaner code
parts of the simulation package.

### Out of scope / (potentially) coming next
* Migration of `allow_*` flags in order to make `Campaign` the unique
place where the concept of metadata exists, i.e. campaigns will be the
only stateful objects. A PR taking care of this should follow soon
because the `get_candidates` signature of `SubspaceDiscrete` currently
makes not much sense, as it expects these flags in a context where
metadata does not exist.
* Once the flags are migrated, the `AnnotatedSubspaceDiscrete` might
become obsolete since the `Campaign` class can then theoretically filter
down the space before passing it to the recommender. This however
requires an efficient implementation that does not cause unnecessary
dataframe copies.
* Actually turning the state space classes `frozen`. There a few other
things that should be addressed at the same time (i.e. general cleanup
of the classes).
AVHopp
AVHopp reviewed Nov 25, 2024
View reviewed changes
Copy link
Collaborator

@AVHopp AVHopp left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Very brief set of initial comments. More to come.

CHANGELOG.md Outdated Show resolved Hide resolved
CHANGELOG.md Outdated Show resolved Hide resolved
baybe/utils/basic.py Show resolved Hide resolved
AVHopp
AVHopp reviewed Nov 26, 2024
View reviewed changes
Copy link
Collaborator

@AVHopp AVHopp left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, just some minor questions/comments

baybe/searchspace/_filtered.py Outdated Show resolved Hide resolved
baybe/searchspace/_filtered.py Outdated Show resolved Hide resolved
tests/conftest.py Outdated Show resolved Hide resolved
tests/test_campaign.py Outdated Show resolved Hide resolved
Scienfitz
Scienfitz approved these changes Nov 26, 2024
View reviewed changes
baybe/recommenders/pure/base.py Show resolved Hide resolved
baybe/recommenders/pure/nonpredictive/base.py Outdated Show resolved Hide resolved
baybe/searchspace/_filtered.py Show resolved Hide resolved
baybe/campaign.py Show resolved Hide resolved
docs/userguide/async.md Show resolved Hide resolved
tests/conftest.py Show resolved Hide resolved
tests/test_deprecations.py Outdated Show resolved Hide resolved
@AdrianSosic AdrianSosic force-pushed the refactor/allow_flags branch 3 times, most recently from 048d5eb to e366077 Compare November 26, 2024 22:23
AVHopp
AVHopp approved these changes Nov 27, 2024
View reviewed changes
Copy link
Collaborator

@AVHopp AVHopp left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since most if my wishes have been implemented, here you go with your approve :)

@AdrianSosic AdrianSosic mentioned this pull request Dec 18, 2024
Open
@AdrianSosic AdrianSosic mentioned this pull request Jan 10, 2025
Merged
AdrianSosic added a commit that referenced this pull request Jan 22, 2025
@AdrianSosic
Simplify and extend meta recommender logic (#457)
af3f86b 
This PR refines the semantics of Meta recommenders and extends their
use. It was motivated by two problems:
* First one arising in #423 where `pending_experiments` should only be
passed to `BayesianRecommenders`. Unfortunately, with the current meta
recommender interface, it is not straightforward (or impossible?) to
identify what exactly the next recommender will be – which indicates a
suboptimal design.
* Second: fixes #420 

The important things changed:
* `MetaRecommender`s now have a much simpler logic, i.e. they only
contain a `select_recommender`, which *always* just returns the
recommender appropriate for the context of the call. Previous
statefulness, which was only relevant to sequential recommenders, has
been moved to the corresponding classes.
* They now have an `is_stateful` Boolean property
* They now have an `get_non_meta_recommender` method for extracting the
appropriate context-specific non-meta recommender from the hierarchy
* They can now be composed of arbitrary other meta recommenders. The
previous restriction that they need to be composed of pure recommenders
was an unnecessary limitation. Allowing other meta recommenders as
building blocks can indeed be indeed useful, for example, using a
two-phase recommender where the second phase uses an adaptive (e.g.
batch-size dependent) meta recommender.
* ~~Added the new `BatchSizeAdaptiveMetaRecommender`, which can be
useful, e.g. to avoid too costly optimizations for large batch sizes.~~
* `TwoPhaseMetaRecommender` now has an explicit `remain_switch` option
to clarify its behavior.
@AdrianSosic
Mention possible relaxations when not enough candidates remain
0163fb4
@AdrianSosic
Consider both Boolean values for flag test
939e85f
@AdrianSosic
Ignore missing sphinx reference
0dbc54d
@AdrianSosic
Add BotorchRecommender to deprecation test
4bc15f4
@AdrianSosic
Disallow pending points for non-predictive recommenders
13adba9
@AdrianSosic
Rename mask attribute to mask_keep
b543755
@AdrianSosic
Add getting recommendations user guide
6761ed3
@AdrianSosic
Add FAQ
7c490c4
@AdrianSosic
Clear recommendation cache when toggling candidates
a73ea3a
@AdrianSosic
Set allow_recommending_already_recommended to True by default
38cad68
@AdrianSosic
Fix handling of pending experiments in naive recommender
0cc2240
@AdrianSosic
Add sphinx-design dependency
414f9dc
@AdrianSosic
Use dropdowns in FAQ
cf22ab9
@AdrianSosic
Add FAQ entry on difference between recommendations and measurements
037c596
@AdrianSosic
Add admonition mentioning statefulness of meta recommenders
a8ca172
@AdrianSosic
Refine admonitions on campaign candidate generation
136857d
@AdrianSosic
Explain difference between permanent and dynamic exclusion
2874f58
@AdrianSosic
Fix user guide link
30d5b89
@AdrianSosic
Fix admonition heading
572a58d
@AdrianSosic
Highlight that active_values is an experimental feature
f60e81c
@AdrianSosic AdrianSosic merged commit a885b7a into main Jan 28, 2025
9 of 11 checks passed
@AdrianSosic AdrianSosic deleted the refactor/allow_flags branch January 28, 2025 10:33
@AdrianSosic AdrianSosic mentioned this pull request Jan 29, 2025
Merged
AdrianSosic added a commit that referenced this pull request Jan 29, 2025
@AdrianSosic
Reset flag default value (#471)
fb6e7d8 
Reset the default of `allow_recommending_already_recommended` back to
`False`, which was forgotten in #423.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@AVHopp AVHopp AVHopp approved these changes

@Scienfitz Scienfitz Scienfitz approved these changes

Assignees

@AdrianSosic AdrianSosic

Labels
enhancement Expand / change existing functionality refactor
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@AdrianSosic @Scienfitz @AVHopp