netbox services not starting after update

[Sylvain420]

Hello,
I have upgraded from version 3.1.9 to the latest version and the services do not start anymore.
I don't understand why in the logs I see python3.7 when I should see python3.8.
I think my problem comes from there but I don't know how to correct it.
Do you have an idea?
Thanks

Unit netbox.service entered failed state.
Jan 31 14:57:07 prol-plapp013 systemd: netbox.service failed.
Jan 31 14:57:12 prol-plapp013 systemd: netbox.service holdoff time over, scheduling restart.
Jan 31 14:57:12 prol-plapp013 systemd: Stopped Netbox - IP address management (IPAM) and data center infrastructure management (DCIM) tool.
Jan 31 14:57:12 prol-plapp013 systemd: Started Netbox - IP address management (IPAM) and data center infrastructure management (DCIM) tool.
Jan 31 14:57:12 prol-plapp013 gunicorn: [2023-01-31 14:57:12 +0100] [32352] [INFO] Starting gunicorn 20.1.0
Jan 31 14:57:12 prol-plapp013 gunicorn: [2023-01-31 14:57:12 +0100] [32352] [INFO] Listening at: http://127.0.0.1:8001 (32352)
Jan 31 14:57:12 prol-plapp013 gunicorn: [2023-01-31 14:57:12 +0100] [32352] [INFO] Using worker: sync
Jan 31 14:57:12 prol-plapp013 gunicorn: [2023-01-31 14:57:12 +0100] [32355] [INFO] Booting worker with pid: 32355
Jan 31 14:57:12 prol-plapp013 gunicorn: [2023-01-31 14:57:12 +0100] [32356] [INFO] Booting worker with pid: 32356
Jan 31 14:57:12 prol-plapp013 gunicorn: [2023-01-31 14:57:12 +0100] [32357] [INFO] Booting worker with pid: 32357
Jan 31 14:57:13 prol-plapp013 gunicorn: [2023-01-31 14:57:13 +0100] [32355] [ERROR] Exception in worker process
Jan 31 14:57:13 prol-plapp013 gunicorn: Traceback (most recent call last):
Jan 31 14:57:13 prol-plapp013 gunicorn: File "/usr/local/lib/python3.7/site-packages/gunicorn/arbiter.py", line 589, in spawn_worker
Jan 31 14:57:13 prol-plapp013 gunicorn: worker.init_process()
Jan 31 14:57:13 prol-plapp013 gunicorn: File "/usr/local/lib/python3.7/site-packages/gunicorn/workers/base.py", line 134, in init_process
Jan 31 14:57:13 prol-plapp013 gunicorn: self.load_wsgi()
Jan 31 14:57:13 prol-plapp013 gunicorn: File "/usr/local/lib/python3.7/site-packages/gunicorn/workers/base.py", line 146, in load_wsgi
Jan 31 14:57:13 prol-plapp013 gunicorn: self.wsgi = self.app.wsgi()
Jan 31 14:57:13 prol-plapp013 gunicorn: File "/usr/local/lib/python3.7/site-packages/gunicorn/app/base.py", line 67, in wsgi
Jan 31 14:57:13 prol-plapp013 gunicorn: self.callable = self.load()
Jan 31 14:57:13 prol-plapp013 gunicorn: File "/usr/local/lib/python3.7/site-packages/gunicorn/app/wsgiapp.py", line 58, in load
Jan 31 14:57:13 prol-plapp013 gunicorn: return self.load_wsgiapp()
Jan 31 14:57:13 prol-plapp013 gunicorn: File "/usr/local/lib/python3.7/site-packages/gunicorn/app/wsgiapp.py", line 48, in load_wsgiapp
Jan 31 14:57:13 prol-plapp013 gunicorn: return util.import_app(self.app_uri)
Jan 31 14:57:13 prol-plapp013 gunicorn: File "/usr/local/lib/python3.7/site-packages/gunicorn/util.py", line 359, in import_app
Jan 31 14:57:13 prol-plapp013 gunicorn: mod = importlib.import_module(module)
Jan 31 14:57:13 prol-plapp013 gunicorn: File "/usr/local/lib/python3.7/importlib/__init__.py", line 127, in import_module
Jan 31 14:57:13 prol-plapp013 gunicorn: return _bootstrap._gcd_import(name[level:], package, level)
Jan 31 14:57:13 prol-plapp013 gunicorn: File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
Jan 31 14:57:13 prol-plapp013 gunicorn: File "<frozen importlib._bootstrap>", line 983, in _find_and_load
Jan 31 14:57:13 prol-plapp013 gunicorn: File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
Jan 31 14:57:13 prol-plapp013 gunicorn: File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
Jan 31 14:57:13 prol-plapp013 gunicorn: File "<frozen importlib._bootstrap_external>", line 728, in exec_module
Jan 31 14:57:13 prol-plapp013 gunicorn: File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
Jan 31 14:57:13 prol-plapp013 gunicorn: File "/opt/netbox/netbox/netbox/wsgi.py", line 7, in <module>
Jan 31 14:57:13 prol-plapp013 gunicorn: application = get_wsgi_application()
Jan 31 14:57:13 prol-plapp013 gunicorn: File "/usr/local/lib/python3.7/site-packages/django/core/wsgi.py", line 12, in get_wsgi_application
Jan 31 14:57:13 prol-plapp013 gunicorn: django.setup(set_prefix=False)
Jan 31 14:57:13 prol-plapp013 gunicorn: File "/usr/local/lib/python3.7/site-packages/django/__init__.py", line 19, in setup
Jan 31 14:57:13 prol-plapp013 gunicorn: configure_logging(settings.LOGGING_CONFIG, settings.LOGGING)
Jan 31 14:57:13 prol-plapp013 gunicorn: File "/usr/local/lib/python3.7/site-packages/django/conf/__init__.py", line 82, in __getattr__
Jan 31 14:57:13 prol-plapp013 gunicorn: self._setup(name)
Jan 31 14:57:13 prol-plapp013 gunicorn: File "/usr/local/lib/python3.7/site-packages/django/conf/__init__.py", line 69, in _setup
Jan 31 14:57:13 prol-plapp013 gunicorn: self._wrapped = Settings(settings_module)
Jan 31 14:57:13 prol-plapp013 gunicorn: File "/usr/local/lib/python3.7/site-packages/django/conf/__init__.py", line 170, in __init__
Jan 31 14:57:13 prol-plapp013 gunicorn: mod = importlib.import_module(self.SETTINGS_MODULE)
Jan 31 14:57:13 prol-plapp013 gunicorn: File "/usr/local/lib/python3.7/importlib/__init__.py", line 127, in import_module
Jan 31 14:57:13 prol-plapp013 gunicorn: return _bootstrap._gcd_import(name[level:], package, level)
Jan 31 14:57:13 prol-plapp013 gunicorn: File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
Jan 31 14:57:13 prol-plapp013 gunicorn: File "<frozen importlib._bootstrap>", line 983, in _find_and_load
Jan 31 14:57:13 prol-plapp013 gunicorn: File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
Jan 31 14:57:13 prol-plapp013 gunicorn: File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
Jan 31 14:57:13 prol-plapp013 gunicorn: File "<frozen importlib._bootstrap_external>", line 728, in exec_module
Jan 31 14:57:13 prol-plapp013 gunicorn: File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
Jan 31 14:57:13 prol-plapp013 gunicorn: File "/opt/netbox/netbox/netbox/settings.py", line 11, in <module>

[candlerb]

You are right that Netbox 3.4 requires python3.8 or later.

What OS are you running? And where did you install python from?

The version of python will be whatever the default python3 version is at the time you ran ./upgrade.sh. To force a different version, you have to tell the script which one to use:

PYTHON=python3.8 ./upgrade.sh

P.S. Please could you edit your post and put three backticks (```) on a line of their own, before and after the log dump, so it formats correctly.

[Sylvain420]

Hello,
My OS is a Centos 7.9. Python 3.8 is installed in the directory /usr/local/bin/python3.8
I run the update with this command :

sudo PYTHON=/usr/local/bin/python3.8 ./upgrade.sh

[candlerb]

Check the contents of /etc/systemd/system/netbox.service, and ensure it says

ExecStart=/opt/netbox/venv/bin/gunicorn --pid /var/tmp/netbox.pid --pythonpath /opt/netbox/netbox --config /opt/netbox/gunicorn.py netbox.wsgi

i.e. there's no reference to a globally-installed version of gunicorn. It must be the one inside the virtualenv.

That's what I think is going wrong: you appear to be running /usr/local/lib/python3.7/site-packages/gunicorn/..., possibly as a link from /usr/local/bin/gunicorn, instead of the one inside netbox's virtualenv.

[Sylvain420]

Thank you. netbox.services is started now but i have a problem with the netbox-rq. It don't start.

journalctl -eu netbox-rq
févr. 01 13:59:51 prol-plapp013 systemd[1]: netbox-rq.service: main process exited, code=exited, status=217/USER
févr. 01 13:59:51 prol-plapp013 systemd[1]: Unit netbox-rq.service entered failed state.
févr. 01 13:59:51 prol-plapp013 systemd[1]: netbox-rq.service failed.
févr. 01 14:00:21 prol-plapp013 systemd[1]: netbox-rq.service holdoff time over, scheduling restart.
févr. 01 14:00:21 prol-plapp013 systemd[1]: Stopped NetBox Request Queue Worker.
févr. 01 14:00:21 prol-plapp013 systemd[1]: Started NetBox Request Queue Worker.
févr. 01 14:00:21 prol-plapp013 systemd[1]: netbox-rq.service: main process exited, code=exited, status=217/USER
févr. 01 14:00:21 prol-plapp013 systemd[1]: Unit netbox-rq.service entered failed state.
févr. 01 14:00:21 prol-plapp013 systemd[1]: netbox-rq.service failed.

*************************************
cat /opt/netbox/netbox/manage.py
#!/usr/bin/env python
import os
import sys

if __name__ == "__main__":
    os.environ.setdefault("DJANGO_SETTINGS_MODULE", "netbox.settings")
    from django.core.management import execute_from_command_line
    execute_from_command_line(sys.argv)
~
************************************
cat /etc/systemd/system/netbox-rq.service
[Unit]
Description=NetBox Request Queue Worker
Documentation=https://netbox.readthedocs.io/en/stable/
After=network-online.target
Wants=network-online.target

[Service]
Type=simple

User=netbox
Group=netbox
WorkingDirectory=/opt/netbox

ExecStart=/opt/netbox/venv/bin/python3 /opt/netbox/netbox/manage.py rqworker high default low
Restart=on-failure
RestartSec=30
PrivateTmp=true

[Install]
WantedBy=multi-user.target

[candlerb]

Check journalctl -eu netbox-rq for more errors.

Check that /opt/netbox/venv/bin/python3 -V works as expected (gives a 3.8.x version)

But error 217 from systemd normally implies that the user doesn't exist. In /etc/systemd/system/netbox-rq.service you have:

User=netbox
Group=netbox

How does that compare to /etc/systemd/system/netbox.service ? What does id netbox show?

See https://www.freedesktop.org/software/systemd/man/systemd.exec.html

217: EXIT_USER: Failed to determine or change user credentials, or to set up user namespacing. See User=/PrivateUsers= above.

[Sylvain420]

I am making progress. Both services are now started.
But I get this error when I go to the web interface:

name 'NBLDAPBackend' is not defined

In the file /opt/netbox/netbox/netbox/authentication.py i have this :

obj = NBLDAPBackend()

What should be filled in ?

journalctl -eu netbox

févr. 03 13:37:45 prol-plapp013 gunicorn[12620]: return SafeString("".join([node.render_annotated(context) for node in self]))
févr. 03 13:37:45 prol-plapp013 gunicorn[12620]: File "/opt/netbox/venv/lib/python3.8/site-packages/django/template/base.py", line 966, in render_annotated
févr. 03 13:37:45 prol-plapp013 gunicorn[12620]: return self.render(context)
févr. 03 13:37:45 prol-plapp013 gunicorn[12620]: File "/opt/netbox/venv/lib/python3.8/site-packages/django/template/library.py", line 258, in render
févr. 03 13:37:45 prol-plapp013 gunicorn[12620]: _dict = self.func(*resolved_args, **resolved_kwargs)
févr. 03 13:37:45 prol-plapp013 gunicorn[12620]: File "/opt/netbox/netbox/utilities/templatetags/navigation.py", line 25, in nav
févr. 03 13:37:45 prol-plapp013 gunicorn[12620]: if user.has_perms(item.permissions):
févr. 03 13:37:45 prol-plapp013 gunicorn[12620]: File "/opt/netbox/venv/lib/python3.8/site-packages/django/contrib/auth/models.py", line 485, in has_perms
févr. 03 13:37:45 prol-plapp013 gunicorn[12620]: return all(self.has_perm(perm, obj) for perm in perm_list)
févr. 03 13:37:45 prol-plapp013 gunicorn[12620]: File "/opt/netbox/venv/lib/python3.8/site-packages/django/contrib/auth/models.py", line 485, in <genexpr>
févr. 03 13:37:45 prol-plapp013 gunicorn[12620]: return all(self.has_perm(perm, obj) for perm in perm_list)
févr. 03 13:37:45 prol-plapp013 gunicorn[12620]: File "/opt/netbox/venv/lib/python3.8/site-packages/django/contrib/auth/models.py", line 480, in has_perm
févr. 03 13:37:45 prol-plapp013 gunicorn[12620]: return _user_has_perm(self, perm, obj=obj)
févr. 03 13:37:45 prol-plapp013 gunicorn[12620]: File "/opt/netbox/venv/lib/python3.8/site-packages/django/contrib/auth/models.py", line 216, in _user_has_perm
févr. 03 13:37:45 prol-plapp013 gunicorn[12620]: for backend in auth.get_backends():
févr. 03 13:37:45 prol-plapp013 gunicorn[12620]: File "/opt/netbox/venv/lib/python3.8/site-packages/django/contrib/auth/__init__.py", line 38, in get_backends
févr. 03 13:37:45 prol-plapp013 gunicorn[12620]: return _get_backends(return_tuples=False)
févr. 03 13:37:45 prol-plapp013 gunicorn[12620]: File "/opt/netbox/venv/lib/python3.8/site-packages/django/contrib/auth/__init__.py", line 27, in _get_backends
févr. 03 13:37:45 prol-plapp013 gunicorn[12620]: backend = load_backend(backend_path)
févr. 03 13:37:45 prol-plapp013 gunicorn[12620]: File "/opt/netbox/venv/lib/python3.8/site-packages/django/contrib/auth/__init__.py", line 21, in load_backend
févr. 03 13:37:45 prol-plapp013 gunicorn[12620]: return import_string(path)()
févr. 03 13:37:45 prol-plapp013 gunicorn[12620]: File "/opt/netbox/netbox/netbox/authentication.py", line 342, in __new__
févr. 03 13:37:45 prol-plapp013 gunicorn[12620]: obj = NBLDAPBackend()
févr. 03 13:37:45 prol-plapp013 gunicorn[12620]: NameError: name 'NBLDAPBackend' is not defined

[candlerb]

If you look earlier in that file:

# Create a new instance of django-auth-ldap's LDAPBackend with our own ObjectPermissions
try:
    from django_auth_ldap.backend import LDAPBackend as LDAPBackend_

    class NBLDAPBackend(ObjectPermissionMixin, LDAPBackend_):
        def get_permission_filter(self, user_obj):
            permission_filter = super().get_permission_filter(user_obj)
            if (self.settings.FIND_GROUP_PERMS and
                    hasattr(user_obj, "ldap_user") and
                    hasattr(user_obj.ldap_user, "group_names")):
                permission_filter = permission_filter | Q(groups__name__in=user_obj.ldap_user.group_names)
            return permission_filter
except ModuleNotFoundError:
    pass

That is: the class NBLDAPBackend is defined within this file, but only if it was able to import django_auth_ldap.backend

To fix this, you need to follow the instructions here, especially this step:

sudo sh -c "echo 'django-auth-ldap' >> /opt/netbox/local_requirements.txt"

If you do that, then whenever you re-run ./upgrade.sh, the django-auth-ldap module will be installed.

[Sylvain420]

It's ok now. Thanks a lot for your help !

[Dasags]

Hello,

when you have a permission problem and you have a log "/var/log/netbox" check these permissions, i have the same problem, and i have change the permissions of these and restart the service and now it is fine to work.
Ubuntu 24.04.1 LTS
Netbox 4.2.1
Python 3.12

BS