Hardware recommendations

[chuanli11]

First, i have to admit that I know little about gym and its associated tools like Gazebo etc. However recently I was asked for hardware recommendations for doing reinforcement learning. And the terms gym, Gazebo and ROS poped up.

I did some research and contacted a few authors of the related papers (such as 1), and was adviced to seek for help from this community.

Would be very grateful if someone can recommend the ideal hardware for reinforcement learning in the environment that involves gym + Gazebo + ROS. In particular,

• Is it true that CPU matters the most for simulation
• CPU core speed v.s. core number
• Is it true that GPU is only useful when deep neural networks is used for the learning part
• Do GPU memory / system memory matter?

Thanks

[diegoferigo]

Hi @chuanli11, welcome!

First, i have to admit that I know little about gym and its associated tools like Gazebo etc. However recently I was asked for hardware recommendations for doing reinforcement learning. And the terms gym, Gazebo and ROS poped up.

I did some research and contacted a few authors of the related papers (such as 1), and was adviced to seek for help from this community.

Your requirements are a bit vague and I can only imagine what you want to achieve. I guess your aim is to train a policy in simulation and then deploy it with minimal effort to a ROS-based robot. This project can maybe help you in the first phase, where you need to feed samples to your preferred RL algorithm. Being based on Ignition Gazebo (note: we don't use ROS at all) you can use all the resources of this ecosystem like SDF support and extend the simulation with the plugin system the Gazebo provides. There are also other options like pybullet, the recently open-sourced (going to be) mujoco, or ISAAC gym from Nvidia. The big advantage that you get with Ignition Gazebo is that sooner or later those physics engine could get integrated, and if you base your code on Gazebo, switching backend is just matter of a single line changed. Though, right now the only supported physics engine is DART, which might or might not fit your needs.

Would be very grateful if someone can recommend the ideal hardware for reinforcement learning in the environment that involves gym + Gazebo + ROS. In particular,

• Is it true that CPU matters the most for simulation

Definitely yes, simulations run entirely on CPU and if you want to parallelize them, a multiprocess setup is recommended. In our experiments we used Rllib, part of the ray project, but any other framework like stable-baselines or similar would work similarly.

• CPU core speed v.s. core number

We don't have enough data to get a really exhaustive response to this. Running multiple simulations in parallel (e.g. with Rllib) requires many cores, this is a fact. Then, the need of sampling in parallel from a big number of environments depends on the RL algorithm you select. This is to say that, if you use algorithms that do not require a lot of parallel environments (that need many core) it's better having less cores but faster. So I guess the reply is, as often, it depends.

• Is it true that GPU is only useful when deep neural networks is used for the learning part

Yes. Most of the RL frameworks allow sampling from simulators, that typically run on CPU as detailed above, and then move the collected samples to GPU where a NN is optimized. This is the only usage of the GPU. If your setup does not involve big samples (for example, you don't have images in the observations) or your NN are not very deep, using the GPU to optimize the NN becomes less important, you could also chose to do everything in the CPU without suffering from a massive performance penalty.

• Do GPU memory / system memory matter?

Again, if the NNs are not very deep, and their inputs don't have a big size, the GPU usage is very very low compared to CPU.

[chuanli11]

Hi, @diegoferigo Your answer is gold. Thank you!

Can you give one example RL algorithm that has "the need of sampling in parallel from a big number of environments"?

Also wonder if ISAAC gym from Nvidia has been trending up, so GPU may play a more significant role in the future.

[diegoferigo]

Hi, @diegoferigo Your answer is gold. Thank you!

Can you give one example RL algorithm that has "the need of sampling in parallel from a big number of environments"?

You can have a look to the Rllib Algorithms for a good overview, where you also find all the corresponding papers.

Also wonder if ISAAC gym from Nvidia has been trending up, so GPU may play a more significant role in the future.

Yes, it will. I wanted to keep the overview quite simple providing more traditional methods, but you're right saying that in few selected cases (like NVIDIA-Omniverse/IsaacGymEnvs, google/brax, google-research/tiny-differentiable-simulator, etc) the GPU becomes a significant requirement. After these years, I convinced myself that a RL pipeline that can run entirely in GPU has a big advantage over any other method, but yet there isn't any established solution (though, any of those I just listed are very good candidates! It's worth keeping an eye on them and, even better, trying to develop some small example to test their functionality and limitations).

[chuanli11]

Thanks for the insights!