How to shut down cuboro cleanly and release all resources

[blooop]

I have a class which sets up curobo with a robot, does some tasks and then shuts down. I then load another robot into this class and perform more actions. I have been starting and stopping this script during debugging, but have noticed that after about 20 minutes curobo hangs during loading.

  kinematics_fused_cu not found, JIT compiling...
  geom_cu binary not found, jit compiling...
  lbfgs_step_cu not found, JIT compiling...
  line_search_cu not found, JIT compiling...
  tensor_step_cu not found, jit compiling...
  CODE HANGS HERE

I have found a solution is to rebuild the docker image without the docker cache, but this takes a long time and is not ideal.

I have tried releasing curobo resources explicilty, but this has not helped;

    def shutdown_curobo(self) -> None:
        del self.kin_model
        del self.mpc
        del self.motion_gen
        del self.ik_solver

Is there a more explicit way to release all curobo gpu resources?

[balakumar-s]

I need some clarification:

1. By shut down do you mean that you stop the container and start a new container to run on a new robot?
2. From your description, are the kernels being JIT compiled every time you call a new script?

If you are not starting a new container every time but only running a new python script for every robot, you don't have to do anything extra as python will take care of clearing resources.

If you are not exiting the python script but recreating instances of curobo, you can call torch.cuda.empty_cache() between instance creations to force pytorch to clear cuda cache. This is not typically needed.

If you can provide a minimal example, that would also help in narrowing down the issue. We have had no issues in creating new instance of curobo inside a python script.

[blooop]

Thanks, that is helpful.

Its complicated for me to produce a minimal example, but I think I think I am closing in on the source of the problem and should have a workaround soon. I will update here when I have a confirmed solution, and it should also fix this issue here:

#13

[blooop]

The problem was with how I was creating the wheel of curobo. installing via the normal instructions fixed the problem.