Deadzones and sensitivity do not match USB driver settings

[ghost]

Version of xpadneo

Commit 28f5034

Severity / Impact

Xbox One Wireless controller responds differently when connected via USB or Bluetooth. This is due to default parameters such as flatness or fuzz set to different values when the device is connected using Bluetooth and xpadneo.

Describe the bug

The default parameters flatness: 4096 and fuzz: 255 are different from the USB driver's default values flatness: 128 and fuzz: 16. Refer to this comment from the related xow project: medusalix/xow#105 (comment)

Steps to Reproduce

Connect gamepad either via USB or using xpadneo and head to https://gamepad-tester.com/ to notice how radically different it reacts to precise input movements.

Expected behavior

The gamepad should feel precise and smooth using xpadneo, in the same way it does when using the USB drivers.

Screenshots/Gifs

System information

# uname -a
Linux nebula 5.7.8-200.fc32.x86_64 #1 SMP Thu Jul 9 14:34:51 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

# xxd -c20 -g1 /sys/module/hid_xpadneo/drivers/hid:xpadneo/0005:045E:*/report_descriptor | tee >(cksum)
00000000: 05 01 09 05 a1 01 85 01 09 01 a1 00 09 30 09 31 15 00 27 ff  .............0.1..'.
00000014: ff 00 00 95 02 75 10 81 02 c0 09 01 a1 00 09 33 09 34 15 00  .....u.........3.4..
00000028: 27 ff ff 00 00 95 02 75 10 81 02 c0 05 01 09 32 15 00 26 ff  '......u.......2..&.
0000003c: 03 95 01 75 0a 81 02 15 00 25 00 75 06 95 01 81 03 05 01 09  ...u.....%.u........
00000050: 35 15 00 26 ff 03 95 01 75 0a 81 02 15 00 25 00 75 06 95 01  5..&....u.....%.u...
00000064: 81 03 05 01 09 39 15 01 25 08 35 00 46 3b 01 66 14 00 75 04  .....9..%.5.F;.f..u.
00000078: 95 01 81 42 75 04 95 01 15 00 25 00 35 00 45 00 65 00 81 03  ...Bu.....%.5.E.e...
0000008c: 05 09 19 01 29 0b 15 00 25 01 75 01 95 0b 81 02 15 00 25 00  ....)...%.u.......%.
000000a0: 75 01 95 05 81 03 05 0c 0a 24 02 15 00 25 01 95 01 75 01 81  u........$...%...u..
000000b4: 02 15 00 25 00 75 07 95 01 81 03 05 0c 09 01 85 02 a1 01 05  ...%.u..............
000000c8: 0c 0a 23 02 15 00 25 01 95 01 75 01 81 02 15 00 25 00 75 07  ..#...%...u.....%.u.
000000dc: 95 01 81 03 c0 05 0f 09 21 85 03 a1 02 09 97 15 00 25 01 75  ........!........%.u
000000f0: 04 95 01 91 02 15 00 25 00 75 04 95 01 91 03 09 70 15 00 25  .......%.u......p..%
00000104: 64 75 08 95 04 91 02 09 50 66 01 10 55 0e 15 00 26 ff 00 75  du......Pf..U...&..u
00000118: 08 95 01 91 02 09 a7 15 00 26 ff 00 75 08 95 01 91 02 65 00  .........&..u.....e.
0000012c: 55 00 09 7c 15 00 26 ff 00 75 08 95 01 91 02 c0 05 06 09 20  U..|..&..u......... 
00000140: 85 04 15 00 26 ff 00 75 08 95 01 81 02 c0                    ....&..u......
1461450476 1558

Controller and Bluetooth information

xpadneo-btmon.txt
xpadneo-dmesg.txt
xpadneo-lsusb.txt

[kakra]

I'll test this during weekend... Seems like a nice enhancement from your description.

[kakra]

@a-kring Here's something to play with: https://github.com/kakra/xpadneo/tree/rework-deadzones-radial

As outlined in medusalix/xow#105 it would be wrong to set the flatness to a low value without implementing a dead zone as this introduces centering problems in joydev which leads to phantom movement in such applications, as you already found in the HTML5 gamepad demo: Sticks do not center as expected.

The current implementation in this branch is a bit flawed as it does cubic scaling, so the sticks now actually move in a square instead of a circle (which may not be too bad as you can reach the max values now at 45° angles). Also there are rounding errors as you can see from the small bumps around the 0° and 90° outer range, also there may occur erratic values within the dead zone and I'm not yet sure where that comes from. But it should be good enough for a test in games.

[kakra]

@a-kring Here's a different one which implements a classical axial dead zone but implements it as a smooth exit from the dead zone instead of jumping out of it: https://github.com/kakra/xpadneo/tree/rework-deadzones-smoothaxial

[ghost]

I've been fighting to get the Bluetooth controller recognized and I just realized that ERTM is not disabled automatically by your branch.

[kakra]

Hmm yes maybe... I'm using a patch for L2CAP to work around this. But following the normal install procedure it should install the ERTM setting.

[ghost]

I followed the normal install procedure. It doesn't add the disable_ertm option to the bluetooth module.

[kakra]

I've updated the branches:

https://github.com/kakra/xpadneo/tree/kakra/rework-deadzones:
Uses a small deadline and smaller fuzz I found by looking at values in evtest (as these are not rescaled around a deadzone).

https://github.com/kakra/xpadneo/tree/kakra/rework-deadzones-radial:
Implements the radial dead zone from some comments above and has some small flaws.

https://github.com/kakra/xpadneo/tree/kakra/rework-deadzones-smoothaxial:
Implements a smooth axial exit from the dead zones. But I looked at the joydev implementation, it does actually the same, so this is useless.

https://github.com/kakra/xpadneo/tree/kakra/rework-deadzones-xpadlike:
Very tiny flatness value, reasonable fuzz value. The dead zones are too small on a worn controller, joydev shows erratic values at the center point and could introduce phantom movement, as games expect joydev to do proper dead zones.

I also looked at the Wine source. The SDL bus implementation sets the dead zone to 0 by default, so you don't need to fiddle with registry settings as long as your games use the Windows HID layer for joystick input (xinput games do). Games using dinput may be a different story as they may pick the Linux joydev joysticks.

I didn't investigate SDL yet. Since Wine uses SDL, the observed dead zones may come from there. SDL does the dumbest thing: It copied the code from Linux joydev.c and it applies the dead zones, no matter if it is already corrected by joydev, if at all, it should only do this for evdev devices. And then it even didn't copy it correctly, it just compares "if in deadzone then 0", so it jumps in and out of the dead zone.

Wine MAY apply a dead zone and does it by rescaling the values - like joydev. So in the worst case, you may see three dead zones applied, and SDL does it in a dumb way. But I'm a bit clueless how Wine decides whether and when to apply a dead zone.

[ghost]

So far I tested this branch, which is a bit better than default, but it is still imprecise. I circle around my targets without hitting them, but it is closer. I could see myself getting used to it if I had to. I'll install the radial branch and restart to test.

[ghost]

I think it is due to a difference in philosophy. Game and input library developers seem to expect raw output from the device. This seems to be what Steam assumes, and what the official Xbox One gamepad drivers Microsoft do. Also it is what is typically done on full-scale joysticks. Since the driver developer cannot know how the application intends to use the device, it is best to just expose it, warts and all. It is then up to the software developer to work out the best approach for their application. What the driver should probably do is get rid of atypical noise generated by the device, say a very jittery sensor or a faulty button contact that may lead to register multiple presses when only one was intended. The stick being slightly off-center in its neutral position is far from atypical for analog input devices like joysticks or gamepads, and it seems that SDL developers do not expect this to be "corrected" by the driver.

The ThinkPad index finger pointing device that is usually found in the middle of laptop keyboards does something remarkable. It constantly monitors and removes drift, re-calibrating itself to the point that if you find yourself applying constant force in a particular direction for more than a few seconds, you'll see that the device starts moving in the opposite direction of its own accord when you let it go, at least for a few seconds as it re-calibrates itself again.

You can get very sophisticated and find a good elegant way to deal with re-centering the stick at zero while causing minimal pain to users, but games and input libraries do not expect the driver to solve this, and will add its own corrections on top of yours, which will make matters worse, unless perhaps you use the Stadia controller approach, which is effective and doesn't conflict with ad-hoc solutions added by user space libraries and software.

By the way, I tested this branch. I was expecting to love it, but it turned out to be the least usable so far. I guess linearity does matter. I'll try smoothaxial now.

[kakra]

I've looked at Wine more intense: There seems no way that it applies a dead zone unless a game explicitly requested it - or you've put the default value in the registry. Otherwise it just passes plain values as received from the lower layers. winejoystick.sys seems the only driver that reads from joydev - and that is probably only used by some old games. Such games will see a dead zone applied by joydev. I think we can ignore that.

Modern games use xinput or dinput - and this - at least in current Proton version - seems to read from SDL or evdev directly. In the case of SDL it will see dead zones applied, in the case of reading evdev directly, it will see raw values with no dead zones applied. Since you are clearly seeing a dead zone that should not be there, the problem path is SDL in Wine. I'll look into possibilities of instructing SDL to not apply a dead zone. Then we could suggest a patch to Proton to ignore SDL dead zones.

I'm currently somewhat convinced that we should not patch this in the driver - and the approach of xpad is probably wrong (and they did it probably for the same reason as you). We can shrink the dead zone, tho. I would apply such a patch - but not for a value of 128 because that clearly messes with joydev.

Funny enough: Only the Linux version of SDL handles dead zones... Of all the things, that's the system that may handle dead zones in the driver already. :-( But it's probably for that matter: Games ported from joydev API to SDL will just work the same. SDL seems to have no way of telling it that we want unfiltered values...

[ghost]

I tried smoothaxial and it is pretty good. It is almost as good as xpad in terms of precision, with the added benefit of having the center mostly at zero. I'm still going to use my own forked version that uses xpad parametres, because it feels pretty much exactly as it does in Windows, which is just right for this particular game.

It's interesting that SDL on Linux is the only version that adds a dead zone, especially observing that Windows drivers do not impose any. I will venture to guess that a discussion not unlike this one happened already, where a hardware driver developer at Microsoft may have had a version of the driver that had dead zones, a game studio developer complained that their beta players didn't like how their gamepad responded, while another game studio developer may have complained earlier about the thumb stick drifting instead of being properly centered when the user wasn't providing any input. I suppose the answer to everything ended up being: the driver gives you raw output, and as a game developer you are expected to use Xinput and parametrize it accordingly. The information about dead zones in that link seems to suggest this.

[kakra]

Since the driver developer cannot know how the application intends to use the device, it is best to just expose it, warts and all. It is then up to the software developer to work out the best approach for their application. What the driver should probably do is get rid of atypical noise generated by the device, say a very jittery sensor or a faulty button contact that may lead to register multiple presses when only one was intended.

This is actually the case when reading from evdev: The fuzz value filters noise, and that's it. You get values without dead zone applied. The problem is Wine using SDL as an intermediate layer. It should (and can) read evdev directly. But on further inspection, only dinput seems to do this. It seems you could set Enable SDL=0 somewhere in the registry to disable SDL. Wine should then use evdev for game controllers. You may have more luck with that as this makes the Wine input event reader completely ignore dead zones in any layer, I guess. But that also seems to stop the device from being shown as a xinput-compatible controller... The situation is bad.

I think we may end up with a driver option in xpadneo to use either 0 or default for flatness but hard-coding 128 is wrong.

[kakra]

I've been fighting to get the Bluetooth controller recognized and I just realized that ERTM is not disabled automatically by your branch.

The code for this is in place:

ERTM_OVERRIDE="/etc/modprobe.d/99-xpadneo-bluetooth.conf"

if [ "$(readlink "${ERTM_OVERRIDE}")" = "/dev/null" ]; then
	echo "Not disabling ERTM, local override in place..."
elif [ -L "${ERTM_OVERRIDE}" ]; then
	echo >&2 "WARNING: '${ERTM_OVERRIDE}' is an arbitrary symlink, this is not supported."
	echo >&2 "Skipping ERTM fixup, you're expected to manage the options on your own..."
else
	echo "Disabling ERTM permanently (requires reboot)..."
	echo "options bluetooth disable_ertm=y" >"${ERTM_OVERRIDE}"
	echo "HINT: If you want to prevent this in the future, run 'ln -snf /dev/null ${ERTM_OVERRIDE}'."

	echo "Disabling ERTM instantly..."
	echo "Y" >/sys/module/bluetooth/parameters/disable_ertm
fi

It should create /etc/modprobe.d/99-xpadneo-bluetooth.conf unless that's a symlink.

Did you install with dkms? What's the output?

[ghost]

After enduring some warfare against DKMS, I managed to get it all working with your branch. I confirm that this works very well. I do not see the option to disable dead zones in the output of ./configure.sh, but I did change it directly in /etc/modprobe.d/99-xpadneo-options.conf and it is working as intended.

By the way, the new default parameters set to less aggressive values also improve the experience without dead zones fully disabled.

Thank you for this work. It was very interesting to go deep into low-level system stuff to see what was going on. I guess I should have a word with SDL/Wine/Proton developers such that they can tune their libraries to disable deadzones for a better experience with Windows games running on Linux. They can override these driver parameters directly from there, right?

Update: I missed some of your posts above. I see that if Wine updates to use evdev directly, they can do whatever they want with the dead zones, including ignoring them completely for Windows games that do not expect them to be set by the driver. That's actually a good idea.

[kakra]

I confirm that this works very well. I do not see the option to disable dead zones in the output of ./configure.sh

That's because I'm lazy. Feel free to open a PR. I do not particularly like how this script was designed - it duplicates a lot of stuff.

By the way, the new default parameters set to less aggressive values also improve the experience without dead zones fully disabled.

Yes, that was the intention: To make a better OOTB experience without compromising compatibility with, e.g., joydev. I think the xpad driver goes a bit too far with setting a dead zone of 128 - tho they probably did that on purpose because of the multi-layer dead zone calculations. But that's not fully fixing it and makes it worse at other places.

I guess I should have a word with SDL/Wine/Proton developers such that they can tune their libraries to disable deadzones for a better experience with Windows games running on Linux.

May be a good idea. Andrew from the Proton project may probably be the best contact to discuss this as much (most? all?) of the work of modern Wine joystick driver is by him.

They can override these driver parameters directly from there, right?

Yes and no: There's no parameter to modify but they can modify the code to read evdev directly instead of going through SDL, that works around getting values that passed through a dead zone filter.

I see that if Wine updates to use evdev directly, they can do whatever they want with the dead zones, including ignoring them completely for Windows games that do not expect them to be set by the driver.

Exactly. Reading from evdev directly bypasses the dead zones. I don't know if SDL actually adds something valuable that they couldn't do directly with evdev - except for the mappings that SDL provides. But maybe it is possible to use SDL just for detecting the mappings and then opening the device directly. A better solution would probably if SDL had an API call for disabling dead zone handling. If they also want to fix this for the Wine driver that uses the joydev API, that probably needs some bigger changes and reworks in Wine, in the way that calibration data would be passed down to the Linux device and Wine itself no longer does the calculations. But I suppose that only affects older games, where most of them that would require precious don't have sane gamepad support anyways.

I played some Borderland 3 now with dead zones disabled, and there's a perceivable difference, especially when the dynamic auto-aim doesn't kick in. But it's not a huge difference for that game. Overall it feels more smooth which I attribute to the reduced fuzz value - this involved the usual stick movement outside of the dead zone, and aiming on weapon zoom is easier. Maybe I should implement the "duplicate reports filter" I'm having in mind: When using Bluetooth, the controller repeats the same HID report multiple times, not sure if this messes with the effectiveness of the fuzz filter.

So thanks for the suggestion. I'll try some more games but I feel the PR can be merged now.

[ghost]

Yeah, I like the pull request the way it is. Feel free.

I do not get any perceivable issues from duplicate Bluetooth reports when playing Elite Dangerous. It feels as butter-smooth as wired or Windows with dead zones disabled. It really makes the game more enjoyable. I'll have to test another game where I was enduring pain caused by these dead zones: Doom Eternal. I more or less fixed the problem naively by using the Steam gamepad support, but I don't like it as it seems to add plenty of input lag on Linux. It may be just a perception issue though. I'll have to check again tomorrow.

Cheers

[kakra]

I more or less fixed the problem naively by using the Steam gamepad support, but I don't like it as it seems to add plenty of input lag on Linux.

Yeah, I always disable that, too, because it feels strange. May actually be due to the lag, I never could nail it to why I don't like it. But it's probably the lag. Also, some games seem to go crazy about whether you're using a mouse/keyboard or a gamepad, and constantly switch modes and on-screen symbols.

[kakra]

FYI: ValveSoftware/wine#94

[kakra]

I proposed a patch for SDL which enables Proton to easily fix this for all drivers and games, without any change to drivers:
https://bugzilla.libsdl.org/show_bug.cgi?id=5241