UnityTimeOutException: The Unity environment took too long to respond.

[wookayin]

I am using obstacle tower 1.2 binary and mlagents-envs-0.6.2, under a Linux environment.

In [1]: from obstacle_tower_env import ObstacleTowerEnv

In [2]: ObstacleTowerEnv.ALLOWED_VERSIONS
Out[2]: ['1', '1.1', '1.2']

In [3]: env = ObstacleTowerEnv('./ObstacleTower/obstacletower.x86_64', retro=False, realtime_mode=False)
Found path: ./ObstacleTower/obstacletower.x86_64
Mono path[0] = './ObstacleTower/obstacletower_Data/Managed'
Mono config path = './ObstacleTower/obstacletower_Data/MonoBleedingEdge/etc'
Preloaded 'ScreenSelector.so'
Preloaded 'libgrpc_csharp_ext.x64.so'
Logging to $HOME/.config/unity3d/Unity Technologies/ObstacleTower/Player.log

I get the following error:

UnityTimeOutException: The Unity environment took too long to respond. Make sure that :
         The environment does not need user interaction to launch
         The Academy and the External Brain(s) are attached to objects in the Scene
         The environment and the Python interface have compatible versions.

I am correctly using the obstacletower 1.2 binary, and it was a head-less mode.

2f6c7ca247c98cdb27776bfde28037a36d1280a4  obstacletower.x86_64

Related Issue?

I checked #44 #46 but they were about Windows or before 1.2 release. I had tried out v1.0 long time ago, and I remember there was no problem at that time.

Executing the binary from the console (with strace) -- nothing follows after some system calls.

... (many lines) ...

munmap(0x7f6f1ac54000, 92100)           = 0
munmap(0x7f6ee85f5000, 2109768)         = 0
munmap(0x7f6ee83ef000, 2118024)         = 0
open("./ObstacleTower/obstacletower_Data/Resources/UnityPlayer.png", O_RDONLY) = 4
fstat(4, {st_mode=S_IFREG|0644, st_size=24711, ...}) = 0
brk(0x415c000)                          = 0x415c000
read(4, "\211PNG\r\n\32\n\0\0\0\rIHDR\0\0\0\200\0\0\0\200\10\2\0\0\0L\\\366"..., 1048576) = 24711

What should I check for this issue?

[awjuliani]

Hi @wookayin

Are you running this in a headless environment? Or does your linux system have a display?

[GrachevArtem]

Hi @awjuliani

I also have the same problem.
It works fine on the local machine. But when I run it on a server I get the same issue.
Do I understand correctly that I need to set up Xvfb server first?

I did it and set option docker_training=True and False, but still have this problem.

[wookayin]

It happened in a head-less mode, but previously I was able to run the environments in head-less mode if I remember correctly, and shouldn't it be the case?. I will try having display servers running.

[awjuliani]

Hi @wookayin and @GrachevArtem

In order to run Obstacle Tower on a headless Linux machine you will either need to run xvfb on that machine (and you can use our obstacle-tower-challenge repo/docker image to do that https://github.com/Unity-Technologies/obstacle-tower-challenge), or run a virtual x-server. We have instructions for installing a virtual x server as part of our GCP/Dopamine guide: https://github.com/Unity-Technologies/obstacle-tower-env/blob/master/examples/gcp_training.md

[twoleggedeye]

Hi I have the same issue here, I'm trying to run it in docker and getting same exception both with xvfb and with virtualgl.

(base) root@dl0:~/root/obstacle-tower-challenge# xvfb-run python run.py
Found path: /root/root/obstacle-tower-challenge/./ObstacleTower/obstacletower.x86_64
Mono path[0] = '/root/root/obstacle-tower-challenge/./ObstacleTower/obstacletower_Data/Managed'
Mono config path = '/root/root/obstacle-tower-challenge/./ObstacleTower/obstacletower_Data/MonoBleedingEdge/etc'
Preloaded 'ScreenSelector.so'
Preloaded 'libgrpc_csharp_ext.x64.so'
Display 0 'screen': 640x480 (primary device).
Logging to /root/.config/unity3d/Unity Technologies/ObstacleTower/Player.log
Traceback (most recent call last):
  File "run.py", line 26, in <module>
    env = ObstacleTowerEnv(args.environment_filename, docker_training=args.docker_training)
  File "/opt/conda/lib/python3.7/site-packages/obstacle_tower_env.py", line 45, in __init__
    timeout_wait=timeout_wait)
  File "/opt/conda/lib/python3.7/site-packages/mlagents_envs/environment.py", line 69, in __init__
    aca_params = self.send_academy_parameters(rl_init_parameters_in)
  File "/opt/conda/lib/python3.7/site-packages/mlagents_envs/environment.py", line 491, in send_academy_parameters
    return self.communicator.initialize(inputs).rl_initialization_output
  File "/opt/conda/lib/python3.7/site-packages/mlagents_envs/rpc_communicator.py", line 80, in initialize
    "The Unity environment took too long to respond. Make sure that :\n"
mlagents_envs.exception.UnityTimeOutException: The Unity environment took too long to respond. Make sure that :
	 The environment does not need user interaction to launch
	 The Academy and the External Brain(s) are attached to objects in the Scene
	 The environment and the Python interface have compatible versions.

[awjuliani]

Hi @twoleggedeye

One potential reason for this may be that it takes longer than 30 seconds for the Unity environment to initialize and handshake with python. Our python api comes with a timeout_wait parameter which can be increased. Could try try with a larger value for this variable, and let us know if that resolves the issue?

[twoleggedeye]

@awjuliani I increased timeout_wait and now able to run it with xvfb, thank you! However it doesn't run with virtualgl. Seems it tries to initialize audio driver which is obviously missing on the headless server.

Found path: /root/root/obstacle-tower-challenge/./ObstacleTower/obstacletower.x86_64
Mono path[0] = '/root/root/obstacle-tower-challenge/./ObstacleTower/obstacletower_Data/Managed'
Mono config path = '/root/root/obstacle-tower-challenge/./ObstacleTower/obstacletower_Data/MonoBleedingEdge/etc'
Preloaded 'ScreenSelector.so'
Preloaded 'libgrpc_csharp_ext.x64.so'
Display 0 'TurboVNC': 1240x900 (primary device).
Logging to /root/.config/unity3d/Unity Technologies/ObstacleTower/Player.log
ALSA lib confmisc.c:768:(parse_card) cannot find card '0'
ALSA lib conf.c:4292:(_snd_config_evaluate) function snd_func_card_driver returned error: No such file or directory
ALSA lib confmisc.c:392:(snd_func_concat) error evaluating strings
ALSA lib conf.c:4292:(_snd_config_evaluate) function snd_func_concat returned error: No such file or directory
ALSA lib confmisc.c:1251:(snd_func_refer) error evaluating name
ALSA lib conf.c:4292:(_snd_config_evaluate) function snd_func_refer returned error: No such file or directory
ALSA lib conf.c:4771:(snd_config_expand) Evaluate error: No such file or directory
ALSA lib pcm.c:2266:(snd_pcm_open_noupdate) Unknown PCM default
ALSA lib confmisc.c:768:(parse_card) cannot find card '0'
ALSA lib conf.c:4292:(_snd_config_evaluate) function snd_func_card_driver returned error: No such file or directory
ALSA lib confmisc.c:392:(snd_func_concat) error evaluating strings
ALSA lib conf.c:4292:(_snd_config_evaluate) function snd_func_concat returned error: No such file or directory
ALSA lib confmisc.c:1251:(snd_func_refer) error evaluating name
ALSA lib conf.c:4292:(_snd_config_evaluate) function snd_func_refer returned error: No such file or directory
ALSA lib conf.c:4771:(snd_config_expand) Evaluate error: No such file or directory
ALSA lib pcm.c:2266:(snd_pcm_open_noupdate) Unknown PCM default
X Error of failed request:  BadMatch (invalid parameter attributes)
  Major opcode of failed request:  72 (X_PutImage)
  Serial number of failed request:  112
  Current serial number in output stream:  113

[awjuliani]

Hi @twoleggedeye,

Unfortunately we don't support virtualGL right now with Unity / Obstacle Tower. If you'd like to use a GPU rendering solution, we recommend running a virtual x-server. https://github.com/Unity-Technologies/obstacle-tower-env/blob/master/examples/gcp_training.md#set-up-xserver

[twoleggedeye]

@awjuliani,
ok, thanks!