x264 not working

[thexperiments]

xrdp version

devel

Detailed xrdp version, build options

xrdp 0.10.80
  A Remote Desktop Protocol Server.
  Copyright (C) 2004-2024 Jay Sorg, Neutrino Labs, and all contributors.
  See https://github.com/neutrinolabs/xrdp for more information.

  Configure options:
      --prefix=/usr
      --sysconfdir=/etc
      --localstatedir=/var
      --sbindir=/usr/bin
      --libexecdir=/usr/lib
      --with-systemdsystemdunitdir=/usr/lib/systemd/system
      --enable-jpeg
      --enable-tjpeg
      --enable-fuse
      --enable-opus
      --enable-rfxcodec
      --enable-mp3lame
      --enable-pixman
      --enable-vsock
      --enable-x264
      CFLAGS=-march=x86-64 -mtune=generic -O2 -pipe -fno-plt -fexceptions         -Wp,-D_FORTIFY_SOURCE=3 -Wformat -Werror=format-security         -fstack-clash-protection -fcf-protection         -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -g -ffile-prefix-map=/home/thex/xrdp-devel-git/src=/usr/src/debug/xrdp-devel-git -flto=auto
      LDFLAGS=-Wl,-O1 -Wl,--sort-common -Wl,--as-needed -Wl,-z,relro -Wl,-z,now          -Wl,-z,pack-relative-relocs -flto=auto
      CXXFLAGS=-march=x86-64 -mtune=generic -O2 -pipe -fno-plt -fexceptions         -Wp,-D_FORTIFY_SOURCE=3 -Wformat -Werror=format-security         -fstack-clash-protection -fcf-protection         -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -Wp,-D_GLIBCXX_ASSERTIONS -g -ffile-prefix-map=/home/thex/xrdp-devel-git/src=/usr/src/debug/xrdp-devel-git -flto=auto

  Compiled with OpenSSL 3.3.1 4 Jun 2024

Operating system & version

arch

Installation method

other

Which backend do you use?

xorgxrdp

What desktop environment do you use?

KDE

Environment xrdp running on

VM with Nvidia 1080 TI via HyperV Partitioning

What's your client?

HyperV Enhanced session / Windows 11 RDP client

Area(s) with issue?

Performance

Steps to reproduce

• Connect to XRDP server
• Observe output of sudo less /var/log/xrdp.log | grep xrdp_encoder
  • [INFO ] xrdp_encoder_create: starting gfx rfx pro codec session

✔️ Expected Behavior

Session uses h264

❌ Actual Behavior

session is using rfx pro instead of h264

Anything else?

Using AUR

• xorgxrdp-devel-git*
• xrdp-devel-git (+ patch to configure vsock and x264)

Can't find any references to XRDP_USE_HELPER in the inis or code.
Are there any other magic things I need to configure or is there simply some code missing still?

Wondering if there is something wrong with the flags to not fall into this:

xrdp/xrdp/xrdp_mm.c

Line 1412 in 1d927ad

best_h264_index = index;

[2024-06-29T23:53:54.674+0200] [INFO ] xrdp_egfx_create: error 0 channel_id 1
[2024-06-29T23:53:54.679+0200] [INFO ] xrdp_mm_egfx_caps_advertise:
[2024-06-29T23:53:54.683+0200] [INFO ]   version 0x00080004 flags 0x00000000 (index: 0)
[2024-06-29T23:53:54.687+0200] [INFO ]   version 0x00080105 flags 0x00000000 (index: 1)
[2024-06-29T23:53:54.692+0200] [INFO ]   version 0x000a0002 flags 0x00000000 (index: 2)
[2024-06-29T23:53:54.696+0200] [INFO ]   version 0x000a0200 flags 0x00000000 (index: 3)
[2024-06-29T23:53:54.701+0200] [INFO ]   version 0x000a0301 flags 0x00000000 (index: 4)
[2024-06-29T23:53:54.705+0200] [INFO ]   version 0x000a0400 flags 0x00000000 (index: 5)
[2024-06-29T23:53:54.710+0200] [INFO ]   version 0x000a0502 flags 0x00000000 (index: 6)
[2024-06-29T23:53:54.714+0200] [INFO ]   version 0x000a0600 flags 0x00000000 (index: 7)
[2024-06-29T23:53:54.718+0200] [INFO ]   version 0x000a0701 flags 0x00000000 (index: 8)
[2024-06-29T23:53:54.723+0200] [INFO ]   replying version 0x000a0701 flags 0x00000000
[2024-06-29T23:53:54.727+0200] [INFO ] xrdp_mm_egfx_caps_advertise: xrdp_egfx_send_capsconfirm error 0 best_index 8
[2024-06-29T23:53:54.731+0200] [INFO ] xrdp_egfx_reset_graphics:
[2024-06-29T23:53:54.736+0200] [INFO ] xrdp_egfx_reset_graphics: width 3840 height 2160 monitorcount 1
[2024-06-29T23:53:54.740+0200] [INFO ] xrdp_mm_egfx_caps_advertise: xrdp_egfx_send_reset_graphics error 0 monitorCount 0
[2024-06-29T23:53:54.744+0200] [INFO ] xrdp_mm_egfx_create_surfaces: map surface_id 0 left 0 top 0 width 3840 height 2160
[2024-06-29T23:53:54.749+0200] [INFO ] xrdp_encoder_create: starting gfx rfx pro codec session

[matt335672]

@thexperiments - H.264 isn't yet supported in v0.10.x. From the release notes

GFX H.264/AVC 444 mode and hardware-accelerated encoding are not supported in this version yet.

[thexperiments]

I know, but arch is building from the devel branch sources thus I was hoping it is in there already. Is there a branch I can use to test it?

[matt335672]

There's nothing for this in devel yet I'm afraid.

What will become the next version of xrdp is documented on this wiki page. v0.10.x was a big release graphics-wise, and we're still working on the issues that are being reported against it.

[thexperiments]

Ok thanks, understand that.
I don't have to much time but if there is something I can do to push this forward let me know. Is there a PR for the missing code? As I understand there was some fork where this was already working and to me it looked like most of the changes from there already made it to the devel branch.

[matt335672]

@Nexarian - is there any way you can make use of @thexperiments offer above? I'll admit I'm not fully up to speed on where you and Jay have gotten to with this.

[Nexarian]

x264 was merged into devel awhile ago? I'll take a look

[Hiero32]

I built and installed devel on Ubuntu 24.04/VirtualBox.

$ xrdp -v
xrdp 0.10.80
  A Remote Desktop Protocol Server.
  Copyright (C) 2004-2024 Jay Sorg, Neutrino Labs, and all contributors.
  See https://github.com/neutrinolabs/xrdp for more information.

  Configure options:
      --enable-fuse
      --enable-jpeg
      --enable-rfxcodec
      --enable-mp3lame
      --enable-vsock
      --enable-pixman
      --enable-x264

  Compiled with OpenSSL 3.0.13 30 Jan 2024

Accessed from both Windows 10 (mstsc.exe) and Windows 11.

$ sudo cat /var/log/xrdp.log | grep xrdp_encoder
[2024-07-04T21:02:37.109+0900] [INFO ] xrdp_encoder_create: starting h264 codec session gfx
[2024-07-04T21:02:38.994+0900] [INFO ] xrdp_encoder_x264_encode: x264_encoder_open rv 0x71d404306e70 for 2560 height 1600

It looks working.
I played back youtube. Bandwidth is less than 50% compared to no h264.

[Nexarian]

@thexperiments: @Hiero32's testing matches mine. Can you try again on a clean VM?

[thexperiments]

Interesting, what config are you using?
As said I could not find the flags/config mentioned anywhere in the sources.
Is it necessary to set the flags? Is there anything else to configure?

(damn always posting from wrong account)

[Nexarian]

You need to build with --enable-x264. I'm not sure if you need --enable-rfxcodec as well.

Try this gist to install it.

[thexperiments]

I have both as can be seen in the initial issue. I was thinking about any config for xorg or similar. I can't see anything in the gist that I currently don't have... can't easily apply it as I'm on arch with KDE

[thexperiments]

Looks like both clients I was using do not support h264... investigating...

[2024-07-04T19:03:27.831+0200] [INFO ] xrdp_caps_process_codecs: NSCodec(CA8D1BB9-000F-154F-589F-AE2D1A87E2D6), codec id [1], properties len [3]
[2024-07-04T19:03:27.836+0200] [INFO ] xrdp_caps_process_codecs: Image RemoteFX(2744CCD4-9D8A-4E74-803C-0ECBEEA19C54), codec id [5], properties len [49]

[thexperiments]

it becomes more mysterious, debugging with GDB now

Thread 6 "xrdp" hit Breakpoint 2, xrdp_mm_egfx_caps_advertise (user=0x77cb4e6ca110, caps_count=9, versions=<optimized out>, flagss=<optimized out>)
    at /usr/src/debug/xrdp-devel-git/xrdp-devel-git/xrdp/xrdp_mm.c:1380
1380            version = ver_flags[index].version;
(gdb) print best_pro_index
$1 = <optimized out>
(gdb) print best_h264_index
$2 = <optimized out>

[Nexarian]

You have to recompile XRDP with --enable-all-debug

[thexperiments]

Really wired, build log tells me it is compiled with the right configure resulting in right defines...

cat build.log |grep -i x264
checking for XRDP_X264... yes
configure: running /bin/sh ./configure --disable-option-checking '--prefix=/usr'  '--sysconfdir=/etc' '--localstatedir=/var' '--sbindir=/usr/bin' '--libexecdir=/usr/lib' '--with-systemdsystemdunitdir=/usr/lib/systemd/system' '--enable-jpeg' '--enable-tjpeg' '--enable-fuse' '--enable-opus' '--enable-rfxcodec' '--enable-mp3lame' '--enable-pixman' '--enable-vsock' '--enable-x264' 'CFLAGS=-march=x86-64 -mtune=generic -O2 -pipe -fno-plt -fexceptions         -Wp,-D_FORTIFY_SOURCE=3 -Wformat -Werror=format-security         -fstack-clash-protection -fcf-protection         -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -g -ffile-prefix-map=/home/thex/xrdp-devel-git/src=/usr/src/debug/xrdp-devel-git -flto=auto' 'LDFLAGS=-Wl,-O1 -Wl,--sort-common -Wl,--as-needed -Wl,-z,relro -Wl,-z,now          -Wl,-z,pack-relative-relocs -flto=auto' 'CXXFLAGS=-march=x86-64 -mtune=generic -O2 -pipe -fno-plt -fexceptions         -Wp,-D_FORTIFY_SOURCE=3 -Wformat -Werror=format-security         -fstack-clash-protection -fcf-protection         -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -Wp,-D_GLIBCXX_ASSERTIONS -g -ffile-prefix-map=/home/thex/xrdp-devel-git/src=/usr/src/debug/xrdp-devel-git -flto=auto' --cache-file=/dev/null --srcdir=.
configure: running /bin/sh ./configure --disable-option-checking '--prefix=/usr'  '--sysconfdir=/etc' '--localstatedir=/var' '--sbindir=/usr/bin' '--libexecdir=/usr/lib' '--with-systemdsystemdunitdir=/usr/lib/systemd/system' '--enable-jpeg' '--enable-tjpeg' '--enable-fuse' '--enable-opus' '--enable-rfxcodec' '--enable-mp3lame' '--enable-pixman' '--enable-vsock' '--enable-x264' 'CFLAGS=-march=x86-64 -mtune=generic -O2 -pipe -fno-plt -fexceptions         -Wp,-D_FORTIFY_SOURCE=3 -Wformat -Werror=format-security         -fstack-clash-protection -fcf-protection         -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -g -ffile-prefix-map=/home/thex/xrdp-devel-git/src=/usr/src/debug/xrdp-devel-git -flto=auto' 'LDFLAGS=-Wl,-O1 -Wl,--sort-common -Wl,--as-needed -Wl,-z,relro -Wl,-z,now          -Wl,-z,pack-relative-relocs -flto=auto' 'CXXFLAGS=-march=x86-64 -mtune=generic -O2 -pipe -fno-plt -fexceptions         -Wp,-D_FORTIFY_SOURCE=3 -Wformat -Werror=format-security         -fstack-clash-protection -fcf-protection         -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -Wp,-D_GLIBCXX_ASSERTIONS -g -ffile-prefix-map=/home/thex/xrdp-devel-git/src=/usr/src/debug/xrdp-devel-git -flto=auto' --cache-file=/dev/null --srcdir=.
  x264                    yes
gcc -DHAVE_CONFIG_H -I. -I..  -DXRDP_CFG_PATH=\"/etc/xrdp\" -DXRDP_SBIN_PATH=\"/usr/bin\" -DXRDP_SHARE_PATH=\"/usr/share/xrdp\" -DXRDP_PID_PATH=\"/var/run\" -DXRDP_MODULE_PATH=\"/usr/lib/xrdp\" -DXRDP_SOCKET_ROOT_PATH=\"/var/run/xrdp\" -I.. -I../common -I../libipm -I../libxrdp -I../third_party -I../third_party/tomlc99 -DXRDP_RFXCODEC -I../librfxcodec/include -DXRDP_X264 -DX264_API_IMPORTS -DXRDP_PIXMAN -I/usr/include/pixman-1 -DXRDP_PAINTER -I../libpainter/include   -march=x86-64 -mtune=generic -O2 -pipe -fno-plt -fexceptions         -Wp,-D_FORTIFY_SOURCE=3 -Wformat -Werror=format-security         -fstack-clash-protection -fcf-protection         -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -g -ffile-prefix-map=/home/thex/xrdp-devel-git/src=/usr/src/debug/xrdp-devel-git -flto=auto -Wall -Wwrite-strings -Wmissing-prototypes -Werror -MT xrdp.o -MD -MP -MF .deps/xrdp.Tpo -c -o xrdp.o xrdp.c
/bin/sh ../libtool  --tag=CC   --mode=link gcc  -march=x86-64 -mtune=generic -O2 -pipe -fno-plt -fexceptions         -Wp,-D_FORTIFY_SOURCE=3 -Wformat -Werror=format-security         -fstack-clash-protection -fcf-protection         -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -g -ffile-prefix-map=/home/thex/xrdp-devel-git/src=/usr/src/debug/xrdp-devel-git -flto=auto -Wall -Wwrite-strings -Wmissing-prototypes -Werror  -Wl,-O1 -Wl,--sort-common -Wl,--as-needed -Wl,-z,relro -Wl,-z,now          -Wl,-z,pack-relative-relocs -flto=auto -o xrdp funcs.o lang.o xrdp.o xrdp_bitmap.o xrdp_bitmap_load.o xrdp_bitmap_common.o xrdp_cache.o xrdp_encoder.o xrdp_font.o xrdp_listen.o xrdp_login_wnd.o xrdp_mm.o xrdp_painter.o xrdp_process.o xrdp_region.o xrdp_egfx.o xrdp_wm.o xrdp_main_utils.o xrdp_encoder_x264.o ../common/libcommon.la ../libipm/libipm.la ../libxrdp/libxrdp.la ../third_party/tomlc99/libtoml.la  ../librfxcodec/src/.libs/librfxencode.a -lx264 -lpixman-1 ../libpainter/src/.libs/libpainter.a
libtool: link: gcc -march=x86-64 -mtune=generic -O2 -pipe -fno-plt -fexceptions -Wp,-D_FORTIFY_SOURCE=3 -Wformat -Werror=format-security -fstack-clash-protection -fcf-protection -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -g -ffile-prefix-map=/home/thex/xrdp-devel-git/src=/usr/src/debug/xrdp-devel-git -flto=auto -Wall -Wwrite-strings -Wmissing-prototypes -Werror -Wl,-O1 -Wl,--sort-common -Wl,--as-needed -Wl,-z -Wl,relro -Wl,-z -Wl,now -Wl,-z -Wl,pack-relative-relocs -flto=auto -o .libs/xrdp funcs.o lang.o xrdp.o xrdp_bitmap.o xrdp_bitmap_load.o xrdp_bitmap_common.o xrdp_cache.o xrdp_encoder.o xrdp_font.o xrdp_listen.o xrdp_login_wnd.o xrdp_mm.o xrdp_painter.o xrdp_process.o xrdp_region.o xrdp_egfx.o xrdp_wm.o xrdp_main_utils.o xrdp_encoder_x264.o  ../common/.libs/libcommon.so ../libipm/.libs/libipm.so ../libxrdp/.libs/libxrdp.so /home/thex/xrdp-devel-git/src/xrdp-devel-git/common/.libs/libcommon.so -lpthread -lssl -lcrypto -lturbojpeg ../third_party/tomlc99/.libs/libtoml.so ../librfxcodec/src/.libs/librfxencode.a -lx264 -lpixman-1 ../libpainter/src/.libs/libpainter.a -Wl,-rpath -Wl,/usr/lib/xrdp

[thexperiments]

also after building with --enable--all-debug I still can't set a breakpoint to xrdp_mm.c:1414 it jumps to 1418 (next thing that is not optimized away)

[Nexarian]

You may have to change the -O2 to -O0, this might be a bug in the compile flags.

[matt335672]

If you're on Ubuntu, you might want to add -fvar-tracking as well. I found without that, it was challenging to examine variables.

[Hiero32]

I built and installed xrdp-devel-git and xorgxrdp-devel-git on arch linux. desktop is gnome.

h264 is working.
Bandwidth for youtube is roughly same as on Ubuntu 24.04.

$ sudo cat /var/log/xrdp.log |grep xrdp_encoder
[2024-07-06T10:51:49.419+0900] [INFO ] xrdp_encoder_create: starting h264 codec session gfx
[2024-07-06T10:51:52.412+0900] [INFO ] xrdp_encoder_x264_encode: x264_encoder_open rv 0x73fafc000be0 for width 2560 height 1600

I just added --enable-vsock and --enable-x264 to PKGBUILD of xrdp-devel-git.
build() in PKGBUILD is as follows.

build() {
  cd $pkgname
  ./configure --prefix=/usr \
              --sysconfdir=/etc \
              --localstatedir=/var \
              --sbindir=/usr/bin \
	      --libexecdir=/usr/lib \
              --with-systemdsystemdunitdir=/usr/lib/systemd/system \
              --enable-jpeg \
              --enable-tjpeg \
              --enable-fuse \
	      --enable-opus \
	      --enable-rfxcodec \
	      --enable-mp3lame \
	      --enable-pixman \
              --enable-vsock \
              --enable-x264

  # Fight unused direct deps
  sed -i -e 's/ -shared / -Wl,-O1,--as-needed\0 /g' -e 's/    if test "$export_dynamic" = yes && test -n "$export_dynamic_flag_spec"; then/      func_append compile_command " -Wl,-O1,--as-needed"\n      func_append finalize_command " -Wl,-O1,--as-needed"\n\0/' libtool
  make V=0
}

[thexperiments]

It started working after changing the optimization yesterday night.
However i need to investigate more as there was something broken and I hat to completely reinstall xrdp and xorgrdp. (thus not sure if it was only the optimization)

I see GPU use (although I think h264 is still software right) but the performance is still meh maybe 5-10 FPS with quite some latency, any hints at what performance to expect. Connection is local to a VM

[thexperiments]

Switched back to vsock and performance is a bit better. Had switched to tcp earlier to try clients from differnt machines