Cluster and DPD modes

[arpruss]

Do you by any chance know the returned data format for the IR camera in the clustering or the DPD modes? I've been looking at it, and while I'm expecting to see coordinates of clusters or blobs, the hex data doesn't look like it's encoding coordinates (I'm expecting coordinates to be something like the 0-319 and 0-239 ranges.)

An example of data in clustering mode that I'm seeing is:
318c2e0000000000002af86e0b000000000000000000000000000000000000000000000000000000000000000000000000030006d937022439000001007eeb71424408441f00004d000000ef0000540200400ea000390039000200030065ff5d7bc333d66a7d01ef017600ff00aef66bfa6f4e3e379f019f012200df007c527387557cc91f7e01eb01ba00be00ffbb5fcfde03237eeb01b30162004900f5e7ebefef578f57f501bd001f00bf00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000b203df

I can tell that the 0x06 in position 51 is the mode (0x06 = clustering), and the next byte is the fragment number. I am guessing that the 0x01 in position 59 may be the noise level, and that soon after that come the clusters, 32 bytes per cluster, but I can't figure out what that data is.

Any hints would be appreciated. Is there maybe some sort of encryption here?

[arpruss]

I've figured out the clustering mode, I think. The data starts at offset 61, and it is a sequence of 16-byte items, one per cluster, each consisting of 8 littleendian uint16 entries:
average brightness
number of pixels in cluster
center of mass x-coordinate in units of 1/64 pixel
center of mass y-coordinate in units of 1/64 pixel
x start
x end
y start
y end

It would be really cool if visualizing the clusters got added to the toolkit.

[arpruss]

I figured out the DPD mode. It is also a sequence of clusters, in the same format as in clustering mode, starting at offset 61. However, there is an extra byte inserted after every third cluster. Thus, the first entry starts at 61, the second at 61+16, the third at 61+32, but the fourth is at 61+49, the fifth at 61+49+16, the sixth at 61+49+32, the seventh at 61+49+49, and so on.

Maybe the extra padding byte carries some information sometimes, but I think I've only seen 00 for it. It's a strange padding!

[CTCaer]

Hey, have in mind that JC toolkit is in indefinite hiatus.
I may sometime start cleaning it up again and improving it but dunno when.

Just wanted to chime in since you did actual research on the protocol!

So don't fret if you don't get an answer or I won't work again with jctoolkit right away.
Documenting these and having a PoC in PRs are still very good for the general public until then.

About that, most of my latest but old research is in the fork that I used to make PRs for the main dekunukem repo https://github.com/CTCaer/Nintendo_Switch_Reverse_Engineering.
Internally I used to have notes for the extra IR modes but dunno where they are after 6+ years.
Btw, nintendo added more IR modes/options since then. So it's good to also document on what jc firmware you are.

[arpruss]

Thanks for the response!

I will try to remember to document the firmware when I get the joycon back from Nintendo (today I sent it for drift repair).

[arpruss]

BTW, do you happen to have a recommendation for a good library for communicating with the joycon over BT in python? I have a project which turns a Wiimote into a lightgun for retro emulation (e.g., Duck Hunt), using four IR LEDs around the TV, and I am looking to move the project to a joycon for greater precision.

[CTCaer]

it's the same lib for everything; hidapi

btw it's better to use a compiled lang for such things instead of python to reduce latency and cpu usage.

I also wonder if there's an accessory that makes joycon more ergonomic to hold with IR camera at front.

[arpruss]

I am thinking of 3D printing a holder for the joycon to attach to a wooden grip, with the joycon upside down so that the analog stick could be used as a trigger.

[CTCaer]

@arpruss btw if you put that code around WaitForSingleObject for hid_read_timeout in hid.c and enable the console at the bottom of jctool.cpp, what times do you get for spi dumping?

char time[32];
struct timespec ts, te;
timespec_get(&ts, TIME_UTC);

res = WaitForSingleObject(ev, milliseconds);

timespec_get(&te, TIME_UTC);
sprintf(time, "%d\n", te.tv_nsec / 1000 - ts.tv_nsec / 1000);
printf(time);

I want to see if my old bt4 adapter is bad or windows is just bad in usb-bt hid latencies.
I see 28ms myself and it shows in wireshark. 14ms for the hci packet received event and 14ms for the packet.
Where it should be only 14-15ms (1x ssr).

[arpruss]

I can check once I get my right joycon back from Nintendo (I only own one; our Switch is a salvage unit with a broken screen and it didn't come with joycons).

[arpruss]

Do you happen to know how to get the joycon into the mode where it's forwarding strain data from the ringcon? Someone on the web says that there is a mode for that, and then the ringcon data replaces the gyro data.

[arpruss]

@arpruss btw if you put that code around WaitForSingleObject for hid_read_timeout in hid.c and enable the console at the bottom of jctool.cpp, what times do you get for spi dumping?

char time[32];
struct timespec ts, te;
timespec_get(&ts, TIME_UTC);

res = WaitForSingleObject(ev, milliseconds);

timespec_get(&te, TIME_UTC);
sprintf(time, "%d\n", te.tv_nsec / 1000 - ts.tv_nsec / 1000);
printf(time);

I want to see if my old bt4 adapter is bad or windows is just bad in usb-bt hid latencies. I see 28ms myself and it shows in wireshark. 14ms for the hci packet received event and 14ms for the packet. Where it should be only 14-15ms (1x ssr).

I'm seeing about 28000.

[CTCaer]

Do you happen to know how to get the joycon into the mode where it's forwarding strain data from the ringcon? Someone on the web says that there is a mode for that, and then the ringcon data replaces the gyro data.

subcmd: 3 arg: 0x32 to enable it
this enables attachment input report (ringcon and spaceship accessories)
0x12 as output report id to send stuff to attachment.

typedef struct _attachment_output_report_t
{
	u8 id; // report id 0x12
	u8 pkt_id;
	u8 rumble[8];
	u8 data[38]; // attachment data.
} attachment_output_report_t;

typedef struct _attachment_input_report_t
{
	// Start - Same as 0x30/0x31.
	u8 id;     // report id 0x32
	u8 pkt_id; // latency timer
	u8 info;   // conn_info/power_info
	u8 btn_right;
	u8 btn_shared;
	u8 btn_left;
	u8 stick_left[3];
	u8 stick_right[3];
	u8 vib_decider;
	u8 sixaxis_acc0[6];
	u8 sixaxis_gyro0[6];
	u8 sixaxis_acc1[6];
	u8 sixaxis_gyro1[6];
	u8 sixaxis_acc2[6];
	u8 sixaxis_gyro2[6];
	// End - Same as 0x30/0x31.

	// Same size as 0x31.
	u8 data[313]; // attachment data.
} attachment_input_report_t;

I'm seeing about 28000.

Yeah, dunno why it takes 14ms for event and 14ms for the data. That issue doesn't exist for reports that are pushed constantly like 0x30/0x31/etc.

EDIT: btw you won't get something valid if you don't configure attachement.

relevant subcmds:
0x28	attachment write
0x29	attachment read
0x2A	attachment enable
0x2B	attachment info

0x58	ext dev write
0x59	ext dev info
0x5A	ext dev polling enable
0x5B	ext dev polling disable
0x5C	ext dev input format config
0x5D	??
0x60	internal attachment write
0x61	??
0x62	internal attachment Enable
0x63	internal attachment polling enable
0x64	internal attachment polling disable

subcmd with data acks:
0xA8 (0x28)	attachment data
0xA9 (0x29)	attachment info
0xAA (0x2A)	internal attachment read
0xD8 (0x58)	ext dev read
0xD9 (0x59)	ext dev info

[arpruss]

Do you happen to know what registers control sensitivity for clustering and pointing modes?

[CTCaer]

I can't remember at all. And I can't find my notes but only what I pushed in github.
https://github.com/CTCaer/Nintendo_Switch_Reverse_Engineering/blob/ir-nfc/mcu_ir_registers_map.md

https://github.com/CTCaer/Nintendo_Switch_Reverse_Engineering/blob/ir-nfc/mcu_ir_nfc_notes.md
https://github.com/CTCaer/Nintendo_Switch_Reverse_Engineering/blob/ir-nfc/mcu_ir_notes.md

You can eventually find them out by trial and error.
Just add a debug textbox where you can send custom cfg, while it's live.

[arpruss]

Register 0x121 seems to control some sort of a threshold for pointing mode. 0=most sensitive, 7=least sensitive.

[CTCaer]

I'm assuming that's page 1, register 0x21.
Make that clear and continue to use that format (0x121).