How are the inputs given to Trackmania with such precision/accuracy?

[HulettD]

Hello,

A few friends and I often share records with each other to see who can do the best on a given map. On longer maps, sometimes we will share No-Respawn times instead, since it can be difficult to consistently complete each checkpoint. We used this repo for a project in our introductory machine learning class and were inspired to create something of our own.

Our idea was basically to create a tool that can turn our No-Respawn times into actual times. We created a keylogger in python to track arrow key press/releases and record them to a json file, which went fine. We then created a program to read the json and simulate those recorded keys being pressed. However, it was extremely inconsistent and inaccurate. After switching to a C++ implementation and optimizing as much as we could figure out, things started to look more feasible. Our program finished the track sometimes at least, but often would stray from the path, and still was wildly inconsistent.

Which leads to our question. We've looked through the code in this repository but were having trouble identifying where/how inputs are actually sent to trackmania, nonetheless how they are so precise/accurate. We were hoping someone smarter than ourselves could shed some light on the issue.

If our request is too tangential feel free to remove it, it's just that we're running out of ideas and this was the only Trackmania 2020 tool we could find to try to learn from

[eliegoudout]

Hello,

I only answer as a mere follower of the repo, not an expert on the matter.
I think there is a fundamental difference between your case and the one of tmrl.

When trying to replicate inputs, you face the problem of synchronisation, i.e. "at exactly which moment should every key press be made". Even the slightest errors can accumulate and lead to breaking behavior. Fun fact: at one point around 2006, Nadeo themselves had this problem, and some "press forward" maps were broken because the game was too chaotic, not consistent enough.

On the other hand, in tmrl, the input in constantly reevaluated according to observed state, and there is no precise replication consideration at play. In this way, it is very robust, as even some time warping or input lag should be managed (even though producing different runs) by the AI.

In conclusion, sadly, I don't think that your answer is to be found in this repo's code yet, because I don't think the problem of synchronization in this context is even relevant.

Maybe @yannbouteiller can confirm that later.

Sincerely.

[yannbouteiller]

Exact. tmrl, as the name implies, computes input via RL (Reinforcement Learning), which is an AI technique that we use to design a "policy". In the context of RL, a policy is a function (a deep neural network in practice) to which you feed the state of the game (screenshots, speed, gear, rpm...) and that outputs actions (joystick commands) that are relevant to this state. Thus, there is no synchronization at play : the policy adapts to the state in real-time.

Note that, if you want your approach to work (and I think that in TrackMania it can work because people do this kind of things to design the perfect run), you need to record the input at each game frame, and send these exact inputs at each corresponding frame when your algorithm is playing.

As @eliegoudout pointed out, this is very far from what we do in tmrl though. What we do here is basically real-world robotics, we just use TrackMania as a fun and safe proxy for advances in that direction.