Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Grid search increasing memory usage #242

Open
nickcorona opened this issue Apr 19, 2019 · 1 comment · May be fixed by #253
Open

Grid search increasing memory usage #242

nickcorona opened this issue Apr 19, 2019 · 1 comment · May be fixed by #253
Labels
bug

Comments

@nickcorona
Copy link

nickcorona commented Apr 19, 2019 •
edited
Loading

The grid search feature doesn't clear memory on each iteration. This means if you're doing a very large grid search, you can run into memory issues very easily and lose all progress. Started with about 10% memory usage, and it keeps increasing with each iteration.

Screenshot from 2019-04-19 16-34-54
@dswah
Copy link
Owner

dswah commented Jul 16, 2019

@nickcorona ahh i see this problem too!
For me it occurs when the search-space is very large. What about you?

I believe this is because pygam tries to pre-determine every point in the search-space before starting the gridsearch.

The problem is that for a n-dimensional search space with m points per dimension, we can quickly get a huge array of points in n-d space: m**n.

I think correct way to create the search space would be with a generator.

Your thoughts?

@dswah dswah added the bug label Jul 16, 2019
@dswah dswah linked a pull request Jul 16, 2019 that will close this issue
Open
2 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

replace combine with itertools product dswah/pyGAM
2 participants
@nickcorona @dswah