Crash When Generating Multiple Rooms

[GavinZhengOI]

Describe the bug

As I mentioned in #326 , Infinigen crashed while generating an indoor scene with multiple rooms containing objects.

Steps to Reproduce

python -m infinigen.datagen.manage_jobs --output_folder outputs/huge_dataset_nooverride \
--num_scenes 5 --pipeline_configs local_256GB.gin monocular.gin blender_gt.gin indoor_background_configs.gin \ 
--pipeline_overrides get_cmd.driver_script='infinigen_examples.generate_indoors' manage_datagen_jobs.num_concurrent=16

This will run tens of hours and then crash.

What version of the code were you using?

v1.8.1

commit 88fb49cde0bbca401601d05d672b31d28e9b45cb (HEAD -> main, origin/main, origin/HEAD)
Merge: 126a41eb ef50641f
Author: Alex Raistrick <[email protected]>
Date:   Fri Aug 23 13:01:23 2024 -0400

    Merge pull request #252 from princeton-vl/develop
    
    v1.8.1

What command did you run?

python -m infinigen.datagen.manage_jobs --output_folder outputs/huge_dataset_nooverride \
--num_scenes 5 --pipeline_configs local_256GB.gin monocular.gin blender_gt.gin indoor_background_configs.gin \ 
--pipeline_overrides get_cmd.driver_script='infinigen_examples.generate_indoors' manage_datagen_jobs.num_concurrent=16

What are your FULL output logs?

crash_summaries.txt

First Sample

92139964677_0_log.out.txt
92139964677_0_log.err.txt

Second Sample

80530831448_0_log.out.txt
80530831448_0_log.err.txt

If this is your first time running Infinigen, what are the full install logs?**

Not the first time running. If turn on --configs singleroom.gin and --overrides compose_indoors.restrict_single_supported_roomtype=True, it does not lead to a crash.

Platform

• OS & OS Version: Ubuntu 24.04.1 LTS
• GPU: 4090
• GPU Driver Version: 535
• RAM (GB): 64GB

Additional context

I suspect this might be a memory leak issue. I noticed that swap usage was over 85%, while memory utilization was only around 50%. I think it's possible that memory is leaking during execution and eventually consuming all available memory. I've just installed Prometheus on my system and started another execution with the same configuration to monitor memory usage. I'll update you once I have the results.

[GavinZhengOI]

It seems like there are some kind of memory leak. I'm using python -m infinigen.datagen.manage_jobs --output_folder outputs/single_data_test --num_scenes 1 --pipeline_configs local_64GB.gin monocular.gin blender_gt.gin indoor_background_configs.gin --pipeline_overrides get_cmd.driver_script='infinigen_examples.generate_indoors' manage_datagen_jobs.num_concurrent=16 running single job, and it takes more than 55G of ram gradually then crashed.