iNav Quadcopter became stuck in quasi-armed state - Reproducible

[FrankPetrilli]

Current Behavior

Using a build only slightly branched from HEAD, my quad became stuck in a quasi-ARM mode multiple times, and I'm able to reproduce the issue.

Steps to Reproduce

1. Build from c108199 with the following patch applied - see this issue for context:

diff --git a/src/main/target/MATEKF405SE/target.c b/src/main/target/MATEKF405SE/target.c
index 1ab3ec5c7..109f2047b 100644
--- a/src/main/target/MATEKF405SE/target.c
+++ b/src/main/target/MATEKF405SE/target.c
@@ -36,7 +36,7 @@ const timerHardware_t timerHardware[] = {
     DEF_TIM(TIM2,  CH1, PA15, TIM_USE_LED,   0, 0), //2812LED  D(1,5,3)

     DEF_TIM(TIM9,  CH2, PA3,  TIM_USE_PPM,   0, 0), //RX2
-    DEF_TIM(TIM5,  CH3, PA2,  TIM_USE_ANY,   0, 0), //TX2  softserial1_Tx
+    //DEF_TIM(TIM5,  CH3, PA2,  TIM_USE_ANY,   0, 0), //TX2  softserial1_Tx
 };

 const int timerHardwareCount = sizeof(timerHardware) / sizeof(timerHardware[0]);

2. Flash a MATEK F405-SE with the firmware just built
3. Apply the following config diff: https://pastebin.com/shw42Gdc
4. Toggle the ARM control back and forth

Expected behavior

ARM flightmode to behave according to the position of the ranges applied, especially to disarm when commanded.

Observed behavior

ARM flightmode engages sometimes when switch is toggled. When it does, it doesn't toggle off, and flight controls are limited to only throttle - roll/pitch/yaw controls do not function, and PID control seems inoperative - the craft does not respond to changes of its attitude. The "Receiver" page shows all input controls moving as expected.

This image shows the craft armed with the control channel 5 clearly set to a disarmed position. When a battery was installed and the issue reproduced, the motors did begin to spin.

In attempting to fix the problem I did the following while the motors spun, all of which had no effect on the problem at hand.

• Toggling the ARM control
• Toggling an alternative flightmode control
• Adjusting all four main axes (only throttle control worked)
• Turning off the controller to cause a failsafe
• Unplugging the FC from USB
• Opening the "Outputs" page and attempting to toggle the "Motor Test Mode" toggle. It was off, and would not change.
• Rebooting the FC

Eventually, the problem stopped itself with seemingly no impetus. I attempted to reproduce the issue and was able to using the procedure above.

Additional information

The F405-SE is attached to a SPM4650 on TX2 via SRXL2. The git diff above disabled softserial on this port, which was interfering with the use of the port for SRXL2.

This same hardware configuration works flawlessly in Betaflight.

• FC Board name and vendor: INAV/MATEKF405SE
• INAV version string: 2.6.0 Nov 12 2020 / 20:29:43 (c108199)

[issue-label-bot[bot]]

Issue-Label Bot is automatically applying the label BUG to this issue, with a confidence of 0.99. Please mark this comment with 👍 or 👎 to give our bot feedback!

Links: app homepage, dashboard and code for this bot.

[stronnag]

• Why not just disable softserial FEATURE -SOFTSERIAL
• if you remove the s/serial resouce you should probably also remove the target.h definitions

[FrankPetrilli]

@stronnag

I've reproduced at head. Once again, armed with the arm switch disengaged. Image.

Please take another look.

[stronnag]

It is a feature that you can arm (even if arming is disabled due to e.g. "nav unsafe") by toggling the arm switch rapidly.
If I configure a quadcopter with your settings (less the FC/RX specific values), I can force arming (as expected) but can't reproduce any inability to disarm. My machine always disarms on command.

• Can you reproduce the case with either of the core developer preview builds? Please try them both.
  • http://seyrsnys.myzen.co.uk/inav_dev/ (used for my tests)
  • https://github.com/DzikuVx/inav/releases/tag/2.6.0-Alfa2
• Can you reproduce the case with any other RX (i.e. other than SRXL2) ?
• Can you reproduce the case with any other FC (if you have one) ?

[digitalentity]

Two main questions here IMO are:

• Is the issue reproducible without the patch mentioned in the issue description? Build configuration with the softserial enabled in the target.h but not having a valid timer mapping is not normal and may have unexpected side-effects.
• If the issue is reproducible with unmodified c108199 - does it reproduce with a different RX (S.BUS would be preferred as most tested driver)

[digitalentity]

Possible leads:

• Disarm requires the uninterrupted arming switch in OFF position for 250ms. Uninterrupted = consistently in one state, no RX dropouts and no failsafe
• An RX signal loss shorter than 700ms will not trigger a failsafe
• An RX signal loss will reset the disarm timer

If an RX driver (SRXL in this case) ends up in a state when the rxSignalReceived is flapping (easiest way by flapping the RX_FRAME_FAILSAFE bit) and the period of flapping is below 250ms - this will effectively disable a possibility to disarm by switch.

To test this hypothesis you can configure the KILLSWITCH mode which is designed to act instantly on just one RC frame with channel in the killswitch position. It's a dangerous mode as a glitch in RC can instantly disarm a machine, but for testing if my theory is correct it's suited very well.

I'm not at all familiar with SRXL2 code, but it's interrupt-driven and therefore prone to race conditions.

[FrankPetrilli]

Answers to pending questions:

• Can you reproduce the case with either of the core developer preview builds? Please try them both.
  http://seyrsnys.myzen.co.uk/inav_dev/ was used for the reproduction in my comment here. I'll attempt the other build source in just a bit
• Can you reproduce the case with any other RX / FC?
  Presently, my only receiver & FC pair is the SRXL2 and F405-SE. I have some FrSky receivers and a Matek F722-SE arriving in just a few days, so I can attempt a reproduction there.
• Is the issue reproducible without the patch mentioned in the issue description? Build configuration with the softserial enabled in the target.h but not having a valid timer mapping is not normal and may have unexpected side-effects.
  Affirmative. In this screenshot, I'm running a build from myzen.co.uk.

Since I'm able to reproduce with firmware built at c108199 by myzen.co.uk, SRXL2 is the only thing I'm doing out of the ordinary. I could see this being the likely cause. My plan of action is as follows:

• Attempt with https://github.com/DzikuVx/inav/releases/tag/2.6.0-Alfa2 to confirm it's not build-related
• Configure KILLSWITCH mode on an alternative input and test during a stuck-armed event to test @digitalentity's theory on race conditions
• After arrival of some parts in the mail
  • Attempt with SRXL2 on a F722-SE
  • Attempt with S.BUS on F405-SE
  • Attempt with S.BUS on F722-SE

[stronnag]

Since I'm able to reproduce with firmware built at c108199 by myzen.co.uk, SRXL2 is the only thing I'm doing out of the ordinary.

Thanks, that's vanilla master, albeit GCC 10.20, Alpha2 or RC1 will be 9.2 (as your cli diff) so it's probably not a compiler issue.

• Attempt with SRXL2 on a F722-SE
• Attempt with S.BUS on F405-SE
• Attempt with S.BUS on F722-SE

Also tested SBUS on F4, again, no problem, so SXRL2 looks increasingly like the prime suspect.

[digitalentity]

Agreed, we're narrowing down on this. Tests with KILLSWITCH and/or an S.BUS RX would narrow this down to SRXL2 driver, if which case we'll probably need to loop in @MiguelFAlvarez or procure an SRXL2 hardware to debug the driver.

[FrankPetrilli]

Good news, some parts arrived early!

I configured my F722 with the same setup and the newest release from myzen.co.uk INAV/MATEKF722SE 2.6.0 Nov 15 2020 / 18:10:48 (3ce5833f). I was able to force an arm by toggling back and forth 10 times, as expected. Disarm was non-functional, and the craft stayed armed with the arm switch toggled off. The killswitch worked as expected, and forced the craft to disarm.

I have an S.BUS receiver in my possession, but unfortunately the transmitter for it is still in the mail (estimated delivery Tuesday US/Pacific).

So here's where we stand:

• Custom softserial build - eliminated, myzen.co.uk builds exhibit the same behavior
• F405 - eliminated, F722 exhibits the same behavior
• KILLSWITCH worked, pointing to a race condition / inability to hold the ARM switch low for 250ms in the SRXL2 code

I can provide more information on Tuesday when the transmitter arrives. Until then, please let me know if there's anything else needed. Thanks!

[MiguelFAlvarez]

I tested this with 2.6.0 Alfa2 on a Revo. Using an iX20 with a SPM4651T
Pasted your diff, but disabled mag and baro to get past the Hardware Health check.
I appear to be able to arm/disarm without the need for the killswitch feature. This is a bare FC, so I'm just looking at the pwm pulses and LED status changing to determine arm/disarm state.

@FrankPetrilli What radio are you using (I believe I saw DX6i in another thread? Is it DSM2 or DSMX?), and what firmware version are you using on the receiver? I believe there was a receiver bug fix at some point this year that addressed an issue with 22ms only radios. Might want to update your receiver it if it doesn't have the latest version already.

[digitalentity]

@MiguelFAlvarez what was the bug? Did it cause the receiver to report failsafe intermittently? If not - we should probably work around the problem is the SRXL2 driver. WDYT?

[MiguelFAlvarez]

I don't quite remember. I didn't work on the fix for it, but I think it was more related to handling telemetry properly.
Either way, it sounds like betaflight works fine. The only thing I changed on here vs that is polling the uart idle status instead of using the idle interrupt to set it. Not sure that should have any significant impact.
I'd also ask if the failsafe bit flapping makes sense to reset this 250ms counter that you mentioned? This will surely be a likely case, aside from a driver bug, if a disarm is requested for any reason at a further range. What are the odds that the the arming channel changed accidentally and is set back before 250ms elapse?

[digitalentity]

Thing here is that we can't trust channel values if the receiver is reporting that it's not receiving valid signal. Therefore we want the consistent DISARM command to make sure it is actually sent by the receiver.

[digitalentity]

Betaflight might not care about signal consistency. It might suffer from the same receiver bug, it just doesn't affect the UX.

INAV didn't care about signal consistency either several versions ago, which caused flood of complaints about disarm when flying somewhere out there. As a mitigation the current logic was implemented.

@FrankPetrilli if you are willing to help debugging this further, I can build you several firmware variants that would collect various data about what we're getting from the receiver.

[MiguelFAlvarez]

As far as SRXL2 goes, there are specific packet ID's for an actual channel packet, and another ID for FailsafeChannelData. In inav and betaflight, we only process the channel data for RF packet channel data. If the packet is FailsafeChannelData, the channel values are not processed, and we set the RX_FRAME_FAILSAFE bit. This implementation is to give inav full control of how it wants to handle failsafes.
As far as frame losses go during a non-failsafe state, the receiver just reports a normal ChannelData packet, but with no channel data. This sets the RX_FRAME_DROPPED bit.
This implementation should prevent any of the receivers own failsafe channel values ever getting processed by inav, which I assume may have been the cause of some of the false disarm reports you're mentioning?

If there is a receiver bug causing false frame drop or failsafe bits getting set, I'm not sure how we would fix this in the inav srxl2 driver other than ignoring the 250ms timer for this protocol specifically. We could have it skip frame drop reports, but that specifically seems like a bandage to another issue.

[FrankPetrilli]

@MiguelFAlvarez I'm running a slightly-dated DSM2 DX6i, so 22ms is a distinct possibility. As for the receiver firmware version, I'm not seeing an easy way to check, or update. The cable to do so appears to cost as much as the receiver did itself, and wouldn't arrive until the end of the month. Do you know if it's possible to perform the update through some other hardware? I have plenty of microcontrollers lying around that could speak whatever protocol is required.

@digitalentity I've compiled a version for my F722 with some plain LOG_D statements around every operation where globalResult |= RX_FRAME_FAILSAFE. I'll run that today and let you know what I find - if you want to collect any other data, I'll be happy to flash whatever's needed.

Thanks, all!

[MiguelFAlvarez]

@FrankPetrilli Unfortunately its not possible to flash through other hardware. The programmer/updater doesn't haven the capability to communicate with generic com ports, though we'd like to have this feature at some point to support updating through FC's and such. It's just currently not a priority.