Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Need to lower memory requirements in distortionCorrect #429

Open
KathleenLabrie opened this issue Mar 15, 2023 · 0 comments
Open

Need to lower memory requirements in distortionCorrect #429

KathleenLabrie opened this issue Mar 15, 2023 · 0 comments
Labels
enhancement Performance or usage improvement severity-routine Needs to be fixed urgency-low Do when we have a breather
Milestone

Comments

@KathleenLabrie
Copy link
Contributor

For GMOS LS, distortionCorrect triples the required memory. There might be a way to do better

From Chris - What is going on right now.

distortionCorrect() for GMOS (which I assume is what we're talking about) also mosaics the frames. The input frames are 12 separate extensions so the first thing it has to do is tile the individual CCDs. Since numpy requires that arrays be contiguous in memory this doubles the memory. Then it has to create the mosaicked frames, which is a tripling of the memory. The problem after that is probably one of fragmentation. The mosaicked frames are the largest python objects and it may be that the only place to put them is on top of the heap because the space freed up isn't big enough to be reused by these objects. Having said that, because that primitive creates new AD objects (adoutputs), the input ADs (adinputs) don't go out of scope until the primitive ends so when you mosaic the first AD the memory of its input isn't freed up immediately.

From Chris - A possible solution to look into

It might help if, instead of creating a new adoutputs list each output AD was poked back into the adinputs list to replace the thing it was created from, doing something like replacing the loop with for i, ad in enumerate(adinputs) and instead of adoutputs.append(ad_out) having adinputs[i] = ad_out. This will set the reference count to that object to zero and the memory should get freed by the garbage collector.

@KathleenLabrie KathleenLabrie added enhancement Performance or usage improvement severity-routine Needs to be fixed urgency-low Do when we have a breather labels Mar 15, 2023
@KathleenLabrie KathleenLabrie added this to the v3.2.0 milestone Mar 15, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement Performance or usage improvement severity-routine Needs to be fixed urgency-low Do when we have a breather
Projects
None yet
Development

No branches or pull requests

1 participant