Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

stop using global config for non-parallel generation #15

Open
mivanit opened this issue Sep 4, 2023 · 0 comments
Open

stop using global config for non-parallel generation #15

mivanit opened this issue Sep 4, 2023 · 0 comments
Labels
code-quality code quality improvement

Comments

@mivanit
Copy link
Member

mivanit commented Sep 4, 2023

This was just kind of a bad decision on my part. To simplify/optimize parallel generation, in MazeDataset.generate() we:

  • run _maze_gen_init_worker(config: MazeDatasetConfig), which initializes a global _GLOBAL_WORKER_CONFIG
  • imap the _generate_maze_helper(index: int) function

this is done to avoid passing the whole config around, and it might not be that bad of an idea for parallel generation, but this is done even when the generation is done on a single thread.

Two options:

  • if passing the config does not have that much of an overhead, then just pass it
  • don't use a global worker init for the non-parallel generation
  • use functools.partial instead (I think I originally tried to use a lambda but that of course does not pickle) and remove the need for a global worker config

I'm partial to the last one, since it feels cleanest.

@mivanit mivanit added the code-quality code quality improvement label Sep 4, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
code-quality code quality improvement
Projects
None yet
Development

No branches or pull requests

1 participant