macOS Monterey stays on login window after entering credentials

[paulb39]

I am trying to connect from Windows 10 to my mac. Following the wiki I was able to build and install xrdp on the mac.

I am able to connect to the Mac via the remote desktop client, but after I add my credentials, the username/password page disappears and I am left with a teal background.

Steps:
Restart mac, when at login page SSH in to run:
sudo /usr/local/sbin/xrdp
sudo /usr/local/sbin/xrdp-sesman

Disconnect from SSH and try to to remote in from Windows machine

Expected results:
After selecting Xorg profile and entering credentials, mac desktop should be shown

Actual results:
After entering credentials I am stuck on the login window with a teal background.

Logs:
[20220319-23:54:44] [INFO ] ++ created session (access granted): username paul, ip 192.168.0.14:11178 - socket: 11
[20220319-23:54:44] [INFO ] starting Xorg session...
[20220319-23:54:44] [DEBUG] Did not find a running X server at /tmp/.X11-unix/X10
[20220319-23:54:44] [DEBUG] Did not find a running X server at /tmp/.X10-lock
[20220319-23:54:44] [DEBUG] Closed socket 12 (AF_INET 0.0.0.0:5910)
[20220319-23:54:44] [DEBUG] Did not find a running X server at 5910
[20220319-23:54:44] [DEBUG] Closed socket 12 (AF_INET 0.0.0.0:6010)
[20220319-23:54:44] [DEBUG] Did not find a running X server at 6010
[20220319-23:54:44] [DEBUG] Closed socket 12 (AF_INET 0.0.0.0:6210)
[20220319-23:54:44] [DEBUG] Did not find a running X server at 6210
[20220319-23:54:44] [DEBUG] Did not find a running X server at /tmp/.xrdp/xrdp_chansrv_socket_10
[20220319-23:54:44] [DEBUG] Did not find a running X server at /tmp/.xrdp/xrdp_chansrv_audio_out_socket_10
[20220319-23:54:44] [DEBUG] Did not find a running X server at /tmp/.xrdp/xrdp_chansrv_audio_in_socket_10
[20220319-23:54:44] [DEBUG] Did not find a running X server at /tmp/.xrdp/xrdpapi_10
[20220319-23:54:44] [INFO ] Starting session: session_pid 774, display :10.0, width 4480, height 1440, bpp 24, client ip 192.168.0.14:11178 - socket: 11, user name paul
[20220319-23:54:44] [ERROR] sesman_data_in: scp_process_msg failed
[20220319-23:54:44] [INFO ] [session start] (display 10): calling auth_start_session from pid 774
[20220319-23:54:44] [ERROR] sesman_main_loop: trans_check_wait_objs failed, removing trans
[20220319-23:54:44] [DEBUG] Closed socket 11 (AF_INET 127.0.0.1:3350)

[20220319-23:54:44] [INFO ] xrdp_wm_log_msg: login successful for user paul on display 10
[20220319-23:54:44] [INFO ] login successful for user paul on display 10
[20220319-23:54:44] [INFO ] loaded module 'libxup.dylib' ok, interface size 10296, version 4
[20220319-23:54:44] [DEBUG] xrdp_wm_log_msg: started connecting
[20220319-23:54:44] [INFO ] started connecting
[20220319-23:54:44] [INFO ] lib_mod_connect: connecting via UNIX socket
[20220319-23:54:47] [DEBUG] Closed socket 24 (AF_UNIX)

xrdp and sesman configs:
https://gist.github.com/paulb39/41bfd36b2eaedca06f5800bdc868ca6d
https://gist.github.com/paulb39/e4c36ab447eb42928c75dc2e9a6c9c9f

Note that I know this issue is common in Linux and the solution is to not be logged in when trying to connect, but I get the same result if I connect when the mac is on the login page or is already logged in.

Also note that I can use the vnc profile no problem.. but vnc sucks which is why I am trying to get xorg working.

[matt335672]

Hi @paulb39

You'll need a copy of Xorg on your system, along with the xorgxrdp module. The idea is that sesman starts Xorg with a particular config file which loads the xorgxrdp module. This can then communicate with xrdp.

[paulb39]

Hey, The wiki has us install XQuartz - is that what you mean for Xorg? The wiki also has us compile xorgxrdp and I have that installed. One thing I noticed is the sesman config has this set: param=/usr/lib/xorg/Xorg
but I don't have that on my system and am not sure what that should be set to

[matt335672]

I have no idea whether xorgxrdp will work with XQuartz - I don't have a Mac.

Set sesman to run XQuartz instead of Xorg. It will either work or it wont'.

[paulb39]

Updated config:
param=/opt/X11/bin/Xorg

I get a different error now:
[20220321-16:19:06] [INFO ] Socket 11: AF_INET connection received from 127.0.0.1 port 49170
[20220321-16:19:06] [DEBUG] session_get_bydata: search policy 0 U paul W 4480 H 1440 bpp 24 T 3 IP 192.168.0.14:4124 - socket: 11
[20220321-16:19:06] [DEBUG] session_get_bydata: try 0x14e61b510 U paul W 4480 H 1440 bpp 24 T 3 IP 192.168.0.14:4000 - socket: 11
[20220321-16:19:06] [INFO ] ++ reconnected session: username paul, display :10.0, session_pid 488, ip 192.168.0.14:4124 - socket: 11
[20220321-16:19:06] [ERROR] sesman_data_in: scp_process_msg failed
[20220321-16:19:06] [ERROR] sesman_main_loop: trans_check_wait_objs failed, removing trans
[20220321-16:19:06] [DEBUG] Closed socket 11 (AF_INET 127.0.0.1:3350)
[20220321-16:19:06] [INFO ] Starting session reconnection script on display 10: /etc/xrdp/reconnectwm.sh
[20220321-16:19:06] [DEBUG] Calling exec (excutable: /etc/xrdp/reconnectwm.sh, arguments: /etc/xrdp/reconnectwm.sh )
[20220321-16:19:06] [DEBUG] receiving SIGCHLD
[20220321-16:19:06] [INFO ] Process 1194 has exited

scp_process_msg failed isn't really helpful, any idea how I can look into this more?

[matt335672]

That's from a reconnect - sesman thinks a session is already running which I don't think is the case.

I think the scp_process_msg error is expected, as xrdp disconnects from sesman when it receives the session details. In other words, that sesman log looks quite normal for a reconnect.

Suggest stopping any remote sessions and restarting sesman (a reboot will obviously do it), and then seeing what's in the log.

[paulb39]

Well this is very frustrating, I seem to have a catch-22 situation.

For the life of me I can't get launchd to work so that xrdp and xrdp-sesman run at the login screen. Regardless if I could get that to work, the main problem seems to be I can't get the X server to run until I log in.

If on the login screen I SSH in and do
sudo /opt/X11/libexec/launchd_startx /opt/X11/bin/startx /opt/X11/bin/Xquartz
or
startx -- -quartz
I get:
xinit: giving up
xinit: unable to connect to X server: Connection refused
xinit: server error

So if I try to remote in I get the xrd-sesman error "Did not find a running X server"
Only way I can get that error to go away is to log in and start the Xquartz application. But then if I try to remote in I get the "Starting session reconnection script on display 10" message and just the teal background.

Has no one ever been able to get Xorg to work on OSX?

[matt335672]

What happens when you run this command (which is the command sesman would use):-

/opt/X11/bin/Xquartz :10 -auth .Xauthority -config xrdp/xorg.conf -noreset -nolisten tcp -logfile .xorgxrdp.%s.log

it might give us a few clues as to whether this can be got working or not.

[paulb39]

Getting closer.
Running that command crashes Xquartz immediately with "Unrecognized option -config"
Xquartz documentation seems non existent, so not sure how to pass it a config.

My xrdp-sesman config has:

param=/opt/X11/bin/Xorg
; Leave the rest paramaters as-is unless you understand what will happen.
param=-config
param=xrdp/xorg.conf
param=-noreset
param=-nolisten
param=tcp
param=-logfile
param=.xorgxrdp.%s.log

So I am guessing you wanted me to run Xorg instead of Xquartz?

If I run: sudo /opt/X11/bin/Xorg :10 -auth .Xauthority -config xrdp/xorg.conf -noreset -nolisten tcp -logfile .xorgxrdp.%s.log
it fails- and here is the log: https://gist.github.com/paulb39/276f04569869b2702877467dc99d8ce4

It fails to load modules:
libxorgxrdp.so
xrdpdev_drv.so
xrdpmouse_drv.so

and ends with:
Fatal server error:
[3100101.320] (EE) no screens found(EE)
[3100101.320] (EE)

In the xrdp.ini I had to update lib=libxup.so to lib=libxup.dylib, so I am not sure why it is trying to load .so file - from my limited understanding .so files are for linux

[matt335672]

Xorg allows a user config to be specified. We use that to load the xorgxrdp module. You'll need to find out a way to do that to even try this with Xquartz

Your log file tells you that Xorg has at least found the modules, and why they aren't being loaded:-

Failed to load /opt/X11/lib/xorg/modules/libxorgxrdp.so: dlopen(/opt/X11/lib/xorg/modules/libxorgxrdp.so, 0x0009): tried: '/opt/X11/lib/xorg/modules/libxorgxrdp.so' (code signature in <7635EB4A-9FFE-3877-BD99-D20BB1D387C9> '/opt/X11/lib/xorg/modules/libxorgxrdp.so' not valid for use in process: mapped file has no Team ID and is not a platform binary (signed with custom identity or adhoc?))

To get around this, the little searching I've done tells me you'll either need an Apple developer ID or you'll need to disable library validation. Again, that's not an area I can help you with I'm afraid.

[paulb39]

So disabling library validation let me get past those error messages. I ran csrutill disable in recovery console. This would probably have been a better way of doing it, but doing it system wide was easier.

Now I get a new error:
[3200463.402] (EE) Cannot run in framebuffer mode. Please specify busIDs for all framebuffer devices

Researching it looks like this is from the device config being wrong:
Section "Device"
Identifier "Video Card (xrdpdev)"
Driver "xrdpdev"
Option "DRMDevice" "/dev/dri/renderD128"
Option "DRI3" "1"
EndSection

What sucks if I can't figure out what that section should be. I was hoping to find some guide with commands I could run to get the correct values, but I can't find anything helpful.

The best I could find to get info about my system is:
system_profiler SPDisplaysDataType
Graphics/Displays:
Apple M1 Max:
Chipset Model: Apple M1 Max
Type: GPU
Bus: Built-In
Total Number of Cores: 32
Vendor: Apple (0x106b)
Metal Family: Supported, Metal GPUFamily Apple 7
Displays:
Color LCD:
Display Type: Built-in Liquid Retina XDR Display
Resolution: 3456 x 2234 Retina
Main Display: Yes
Mirror: Off
Online: Yes
Automatically Adjust Brightness: Yes
Connection Type: Internal

But that's not really helpful, I have no idea how to find out what the BusID should be.
I tried:
Section "Device"
Identifier "Video Card (xrdpdev)"
Driver "xrdpdev"
Option "DRMDevice" "/dev/dri/renderD128"
Option "DRI3" "1"
BusID "0x01"
EndSection

But I get the same error. I tried "Internal" as well since its just using the integrated graphics, but I still get the error.

[matt335672]

What does the whole log look like?

[matt335672]

See also #2194, if you haven't already!

[paulb39]

Full log: https://gist.github.com/paulb39/0bd2967fb9f198096ce4386a7820ff10

[matt335672]

Thanks.

I don't think the PCI ID of your graphics card is relevant here. In the configuration we need, the xrdpdev driver is the main graphics card - there isn't anything else.

Solving this is probably going to need stepping through the X server and figuring out why the check in xf86PostProbe() is triggering on the Mac, and not on a standard PC.

[paulb39]

Hey, I am happy to look into this, but I'm a C#/java dev and a little out of my element. I looked at the xorgxrdp and xrdp repos but couldn't find xf86PostProbe() anywhere, I also checked the XQuartz repo but didn't see it there either. What do I need to step through?

[Nexarian]

This repo is what you're looking for: https://gitlab.freedesktop.org/xorg/xserver

As to coming from C#/Java -- Me too! It took some work, but I did eventually learn to work in the XRDP code base. It's incredibly challenging, and many days you want to bang your head against the wall/throw things, and give up! I've gotten frustrated and stopped working on this for several months, but the desire to have solid remote desktop on Linux OSes keeps drawing me back. It's a space that is in need of talented motivated devs. If you want to ask further questions on the Gitter, we're happy to help.

I learned a lot of what I know the hard way. I think getting this to work on Mac OS could be a really fun challenge.

[matt335672]

@paulb39 - what exactly is the Xorg you are using?

Is it an Apple special built to run with XQuartz, or is it a completely separate build of Xorg?

If it's the former, you could be having trouble because of this code:-

https://gitlab.freedesktop.org/xorg/xserver/-/blob/master/hw/xfree86/common/xf86Bus.c#L557-L574

If neither XSERVER_PLATFORM_BUS or XSERVER_LIBPCIACCESS are defined, the error condition will always trigger with xorgxrdp. That's the bit of code you'd need to step through.

[paulb39]

So this has been extremely frustrating.

My Xorg is located /opt/X11/bin, I believe its from XQuartz.

What values should those env variables be? As a quick test I set XSERVER_PLATFORM_BUS to true with an export command, no change, I then cleared it and did XSERVER_LIBPCIACCESS=true, but no change there either.

I am trying to make sure I can compile the project first, (using instructions from https://www.xquartz.org/Developer-Info.html).
but it fails on:
./configure --prefix=/opt/X11 --disable-dependency-tracking --enable-maintainer-mode --with-apple-application-name=XQuartz --with-bundle-id-prefix=org.xquartz

The logs showed this: ld: warning: The i386 architecture is deprecated for macOS (remove from the Xcode build setting: ARCHS)

After some googling it looks like it is because they removed the ability to compile 32 bit projects in Xcode 10 (thanks Apple!)
The work around is to download Xcode 9.4 and set it as default. Now I get a new error:

configure:5172: checking whether the C compiler works
configure:5194: gcc -Wall -O0 -ggdb3 -arch i386 -arch x86_64 -pipe -Wall -O0 -ggdb3 -arch i386 -arch x86_64 -pipe conftest.c >&5
xcrun: error: unable to load libxcrun (dlopen(/Users/paul/Downloads/Xcode.app/Contents/Developer/usr/lib/libxcrun.dylib, 0x0005): tried: '/Users/paul/Downloads/Xcode.app/Contents/Developer/usr/lib/libxcrun.dylib' (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64e'))).

I have no idea how to get past that. If I try to open the 9.4 version of Xcode it says it can't run on Monterey, so maybe that's the problem?

[matt335672]

They aren't environment variables - they are macros set at compilation time.

If you can build the X server, you can at least disable the check completely - that might be interesting.

I can't help you much with the Apple compilation side, except to make a few comments on the above:-

1. You're not trying to build an i386 project. It looks (to me) like there might be a better fix for your first problem, which is to remove references to i386 (and possibly x86_64) from the project.
2. The second problem looks like an architectural mismatch between the dynamic library, and your compiler.

[paulb39]

Okay, so with some modification I was able to compile the xorg-server project, but I am really confused.

I ran:
export ACLOCAL="aclocal -I /opt/X11/share/aclocal -I /opt/homebrew/share/aclocal"
export PKG_CONFIG_PATH="/opt/X11/share/pkgconfig:/opt/X11/lib/pkgconfig"
./configure --prefix=/opt/X12 --disable-dependency-tracking --with-apple-application-name=XQuartz --with-bundle-id-prefix=org.xquartz
make
sudo make install

And that compiles fine, but it only made /opt/X12/bin/Xquartz
Looking at the code you need me to debug, I realized that the file lives under /xfree86
If I cd into /xfree86 and do make, I get:
xf86Configure.c:32:10: fatal error: 'xf86_OSlib.h' file not found

Any idea how I can compile xfree86?

[matt335672]

Sorry - no. You'll need to ask in a Mac forum I think. Sounds like progress though.

[paulb39]

Finally had some time to look at this again. I noticed I had the missing file under /opt/X11/include/xorg, so I moved it to the project folder and tried to make again. Still fails: error log

I noticed the project also allows meson for building, so I installed that build system and tried it. Fails as well. build log.

I can't get passed this.

[paulb39]

Posted on their gitlab, xfree86 can't be compiled on a Mac: https://gitlab.freedesktop.org/xorg/xserver/-/issues/1348