Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

revisit memory fix for fastmultigather #482

Open
ctb opened this issue Oct 17, 2024 · 0 comments
Open

revisit memory fix for fastmultigather #482

ctb opened this issue Oct 17, 2024 · 0 comments

Comments

@ctb
Copy link
Collaborator

ctb commented Oct 17, 2024

this commit in #430 deallocates the duplicate memory used for against_collection here. Unfortunately, the RSS does briefly spike to 2x because of this. We should switch to using into_par_iter() per luiz. From slack:

luizirber you want into_par_iter and into_iter
13h
luizirber
par_iter and iter return references, into_* returns values
13h
luizirber

   pub fn into_par_iter(&self) -> impl IndexedParallelIterator<Item = (&Collection, Idx, &Record)> {
        // first create a Vec of all triples (Collection, Idx, Record)
        self
            .collections
            .into_par_iter() // CTB: are we loading things into memory here? No...
            .flat_map(|c| c.into_iter().map(move |(_idx, record)| (c, _idx, record)))
    }

13h
luizirber
or something close to that?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant