Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Tip for troubleshooting rserver startup problems #120

Open
agordon opened this issue Dec 22, 2021 · 2 comments
Open

Tip for troubleshooting rserver startup problems #120

agordon opened this issue Dec 22, 2021 · 2 comments
Labels

Comments

@agordon
Copy link

agordon commented Dec 22, 2021

Hello,

Thanks for the awesome work on jupyter-resession-proxy.

After many hours of troubleshooting and debugging, perhaps I can save some time for future users:

-1-
The rserver/rsession process requires access to some system-wide directories.
If using a JupyterHub configuration with SystemdSpawner, and isolating paths - that could prove to be problematic.

Specifically, if your jupyterhub_config.py has something like this:

c.JupyterHub.spawner_class = 'systemdspawner.SystemdSpawner'
c.SystemdSpawner.isolate_tmp = True
c.SystemdSpawner.isolate_devices = True
c.SystemdSpawner.disable_user_sudo = True
c.SystemdSpawner.readonly_paths = ['/']
c.SystemdSpawner.readwrite_paths = ['/home/{USERNAME}','/var/run']
c.SystemdSpawner.dynamic_users = False
c.SystemdSpawner.cpu_limit = 1

The readonly_paths =['/'] will cause the rserver/rsession process to fail to start.
Similarly, it seems the rserver/rsession needs read access to /proc and read/write access to /var/run and /var/log
Any attempts to hide/block these folders will again result in rserver/rsession process to fail.
For example, the following will cause it to fail:

c.SystemdSpawner.unit_extra_properties = {
        'InaccessiblePaths' : [
             '/proc/',
             '/var/log',
             '/root/',
             '/sys/',
             '/var/www'
             ]
        }

(If you wonder why would anyone use such restrictive settings, it is because they add one small additional layer of protection on a shared system, and most jupyter kernels work just fine with them).

-2-
When encountering jupyterhub+rstudio start up problems (the dreaded 404 or "timeout errors),
the first place to look is in the logs. If starting jupyterhub from systemd, then journalctl -xe will show the error messages.
In my case it was completely unhelpful rserver[12354]: ... Error 13: permission denied but without showing which file access is denied.

If you need to escalate the troubleshooting and see exactly what is being access/denied, you can replace the rserver binary with a wrapper script, like so:

cd /usr/lib/rstudio-server/bin
mv rserver rserver.bin
cat<<'EOF'>rserver
#!/bin/sh
log=$(mktemp /tmp/rserver.strace.XXXXXXX.log)
echo "=== RSERVER DEBUG, strace log file: $log ==="
echo "args: $@"
echo "=== env: "
env
echo ===

strace -f -e trace=file -o "$log" /usr/lib/rstudio-server/bin/rserver.bin "$@"
EOF

Then, the journalctl log will show the parameters used to start rserver, and the name of the temporary log file.
In that log file (/tmp/rserver.strace.XXXXXX.log), grep for EPERM and EACCES to find the blocked files.
NOTE: if the strace log file is stored in /tmp, ensure that c.SystemdSpawner.isolate_tmp is set to False in the jupyterhub config file during the troubleshooting session - otherwise it will disappear.

This is certainly a low-level technical method, but if all else fails, it'll provide some hints as to what the problem is.

@agordon agordon added the bug label Dec 22, 2021
@welcome
Copy link

welcome bot commented Dec 22, 2021

Thank you for opening your first issue in this project! Engagement like this is essential for open source projects! 🤗

If you haven't done so already, check out Jupyter's Code of Conduct. Also, please try to follow the issue template as it helps other other community members to contribute more effectively.
welcome
You can meet the other Jovyans by joining our Discourse forum. There is also an intro thread there where you can stop by and say Hi! 👋

Welcome to the Jupyter community! 🎉

@ryanlovett
Copy link
Collaborator

Thanks @agordon ! I've definitely debugged rserver the way you did in the second section. :) Instead of moving the executables, you can put the wrapper script somewhere earlier in PATH than rserver.

I feel like this should be preserved in documentation somehow. Perhaps the first section should be put in README.md and the second section should be put in CONTRIBUTING.md? If so, care to submit a PR?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

2 participants