Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[WIP] Added a MultiStore to DataServer #502

Draft
wants to merge 6 commits into
base: main
Choose a base branch
from

Conversation

sivonxay
Copy link
Contributor

@sivonxay sivonxay commented May 18, 2023

When running large numbers of calculations simultaneously, rlaunch multi takes care to only create one MongoClient instance and take advantage of connection pooling to prevent large numbers of concurrent connections to the mongodb.

However, when a user would like data to be saved to another database or collection, it is typically achieved by creating maggma Stores within a Firetask and inserting the document (for example jobflow's JobFiretask). This results in poor control over the number of connections made to a database and the potential for an unlimited number of connections.

To solve this, I have implemented a class which keeps track of the Stores in use and allows individual fireworks to share these Stores. This extends the DataServer used to share a LaunchPad, by registering an additional callable MultiStore. No other fireworks code utilizes this functionality, but it is required that the main process keeps track of it and therefore it cannot be put in another code.

Because I have put this class in maggma, this does require fireworks to have maggma as a dependency. I could also put this class in jobflow, but jobflow (and implicitly maggma) would need to be dependencies.

These are the pull requests for maggma and jobflow

@sivonxay sivonxay changed the title Added a MultiStore to DataServer [WIP] Added a MultiStore to DataServer May 18, 2023
@janosh
Copy link
Member

janosh commented May 18, 2023

maggma is not a default dep of Fireworks (yet) causing tests to fail:

CI error
/opt/conda/envs/test-environment/lib/python3.7/importlib/__init__.py:127: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
fireworks/core/tests/test_tracker.py:20: in <module>
    from fireworks.features.multi_launcher import launch_multiprocess
fireworks/features/multi_launcher.py:23: in <module>
    from maggma.stores.shared_stores import MultiStore
E   ModuleNotFoundError: No module named 'maggma'

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants