Policy output actions jump to max values

[veczieds]

Hi!
Ive tried implementing DDPG from spinningup in the competition script just for my own research. And ive been getting weird results. After some training updates the policy return action tensor values jump to max values (-1 or 1). And they just stay like that without much change. From time to time they might switch from -1 to 1 for example but still stay maxed out. I cant seem to find the reason for that. If someone has any ideas what ive done wrong with the code, would love a tip how to fix it. Thank you!

ive added the code in txt file. Didnt change much from the original script except the MyActorModule class a bit and SacTrainingAgent train part. P.S. the problem isnt the added noise in MyActorModule. the values jump to max even without it
DDPGt.txt

[veczieds]

Ive realized my mistake by doing torch.clamp instead of torch.tanh. but still the values jump close to max, about 0.99 with little deviation from them after that happens

[veczieds]

kinda fixed it by moving the noise to act() part, but still seems like not the right solution. Atleast now the noise was strong enough to get the agent to explore

[veczieds]

As always, as soon as you create a post about problem, you find the solutions :D. Admins can delete this discussion :)

[yannbouteiller]

Haha glad you could nail it, can you share the solution in case people end up here with a similar issue please ? :)

[veczieds]

sure. I didnt share it because the car wasnt able to complete my simple track, but it got close to it. It just converged to a bad solution because of the weak exploration i think. A few warnings for people who might look at the code: the algorithm might still be wrong, cause i cant validate it and the code is littered with useless stuff left over from the original competition script.

DDPGt.txt