Debugging in general

[rahlhar]

While I will outline a specific issue being faced, I'm really mostly interested in knowing how to debug a barebones project like this.

Should we use ndk-gdb somehow? (From initial testing I get the feeling this is not the approach.)
It throws an error Failed to retrieve application ABI from Android.mk.

Or perhaps running am from an adb shell, presumably using set-debug-app and -w to wait for a debugger; then using plain old gdb with gdbserver.

Other?

I just wanted to get some advice if possible before I potentially go digging down some unnecessary rabbit holes.

---

Currently just trying to get the example code working, so I can then poke around and develop a good understanding of how things are working.

Sideloading onto a device using Android 10 (API 29).

Can get the demo to startup on the device, I see a couple of seconds of content, then it crashes.

logcat yields the following (with the errors singled out).
The standout error being:

08-03 05:41:34.628   869   869 E Layer   : [Surface(name=AppWindowToken{3e549b token=Token{abc50aa ActivityRecord{da98795 u0 org.foo.quicktest/android.app.NativeActivity t235}}})/@0xdffaabd - animation-leash#0] No local sync point found

08-03 05:41:34.022  5387  5411 I ActivityTaskManager: START u0 {act=android.intent.action.MAIN cat=[android.intent.category.LAUNCHER] flg=0x10200000 cmp=org.foo.quicktest/android.app.NativeActivity bnds=[439,1146][640,1409]} from uid 10132
08-03 05:41:34.087  5387  5466 I ActivityManager: Start proc 14945:org.foo.quicktest/u0a204 for activity {org.foo.quicktest/android.app.NativeActivity}
08-03 05:41:34.095 14945 14945 I chatty  : uid=10204(org.foo.quicktest) expire 6 lines
08-03 05:41:34.237 14945 14968 I chatty  : uid=10204(org.foo.quicktest) expire 31 lines
08-03 05:41:34.269 14945 14969 I chatty  : uid=10204(org.foo.quicktest) expire 5 lines
08-03 05:41:34.285  5387  5464 I ActivityTaskManager: Displayed org.foo.quicktest/android.app.NativeActivity: +262ms
08-03 05:41:34.451 14945 15001 I chatty  : uid=10204(org.foo.quicktest) expire 1 line
08-03 05:41:34.471  5387  5466 I ActivityManager: Start proc 15014:com.android.webview:sandboxed_process0:org.chromium.content.app.SandboxedProcessService0:0/u0i7 for  {org.foo.quicktest/org.chromium.content.app.SandboxedProcessService0:0}
08-03 05:41:34.498 14945 14998 I chatty  : uid=10204(org.foo.quicktest) expire 2 lines
08-03 05:41:34.570 14945 15000 I chatty  : uid=10204(org.foo.quicktest) expire 21 lines

08-03 05:41:34.628   869   869 E Layer   : [Surface(name=AppWindowToken{3e549b token=Token{abc50aa ActivityRecord{da98795 u0 org.foo.quicktest/android.app.NativeActivity t235}}})/@0xdffaabd - animation-leash#0] No local sync point found

08-03 05:41:34.673 14945 14981 I chatty  : uid=10204(org.foo.quicktest) expire 254 lines
08-03 05:41:34.955 15062 15062 I chatty  : uid=10204(org.foo.quicktest) /system/bin/app_process64 identical 1 line
08-03 05:41:35.195 14945 14981 F libc    : Fatal signal 6 (SIGABRT), code -6 (SI_TKILL) in tid 14981 (Thread-3), pid 14945 (.foo.quicktest)
08-03 05:41:35.345 15077 15077 F DEBUG   : pid: 14945, tid: 14981, name: Thread-3  >>> org.foo.quicktest <<<
08-03 05:41:35.351 15077 15077 F DEBUG   :       #08 pc 00000000000096a4  /data/app/org.foo.quicktest-z63IAQZUKcj_BhprsVvQ5A==/lib/arm64/libquicktest.so
08-03 05:41:35.685  5387 15080 W ActivityTaskManager:   Force finishing activity org.foo.quicktest/android.app.NativeActivity
08-03 05:41:35.742  5387  5880 W InputDispatcher: channel '127ffac org.foo.quicktest/android.app.NativeActivity (server)' ~ Consumer closed input channel or an error occurred.  events=0x9

08-03 05:41:35.743  5387  5880 E InputDispatcher: channel '127ffac org.foo.quicktest/android.app.NativeActivity (server)' ~ Channel is unrecoverably broken and will be disposed!

08-03 05:41:35.745  5387 10078 I ActivityManager: Process org.foo.quicktest (pid 14945) has died: fore TOP 
08-03 05:41:35.746  5387 12904 I WindowManager: WIN DEATH: Window{127ffac u0 org.foo.quicktest/android.app.NativeActivity}
08-03 05:41:35.746  5387 12904 W InputDispatcher: Attempted to unregister already unregistered input channel '127ffac org.foo.quicktest/android.app.NativeActivity (server)'

[cnlohr]

You can use addr2line to figure out the line of code this is crashing on:

#08 pc 00000000000096a4  /data/app/org.foo.quicktest-z63IAQZUKcj_BhprsVvQ5A==/lib/arm64/libquicktest.so

But normally you get a much more sophisticated call stack, I am surprised it's so spartan.

Also, typically people who use this just use printf/tombstones to debug, I don't know if anyone's gotten gdb working, but a guide would be cool.

[rahlhar]

Cheers. I'll try that out.

Will do some more digging as well. If I can get gdb working, I'll definitely write something up.

[cnlohr]

Please do!

[rahlhar]

After some playing around trying to get debug symbols - had some partial luck - it looks like the crash is referencing test.c:161.

But that doesn't seem to make much sense.

Also, I say partial luck because initially addr2line couldn't determine anything until I realised I needed to tweak the Makefile to compile a debug build of the .so.

But while it gives some more info, it seems wrong. I'm likely messing something up.

[rahlhar]

Since then I've been playing with the ndk's ndk-gdb.py.
Got some success with attaching to the process both with lldb and gdb.

Have just now managed to get a simplified shell script version (just for gdb) working.
It's partially automated, but relies more on some well defined environment vars than the code on which it's based.

It definitely still needs some work. I'll upload it soon in the hope anyone else will poke at it, maybe make some suggestions.

I'm now back to struggling with actually debugging the problem that started this journey.
But I'm poorly experienced when it comes to debugging threaded coded, and I still haven't managed to track the source of the crash.

[cnlohr]

Well (1) Please write it up, and include it to the readme.
and (2) What is your problem?

[rahlhar]

Ok, I've pushed a branch to here.
I've not yet added to the readme, but the comments in ndk-gdb.sh should hopefully explain things for now.

After tweaking a few environment vars it should (I hope) be as simple as running:

$ make push
$ ./ndk-gdb.sh

As for my actual problem, I'm not exactly certain beyond the cnfgtest app crashing after a second or two (SIGABRT).

I've as yet been unable to actually pin down what exactly is raising the signal, so have no idea what the root cause might be.

[rahlhar]

I'm only now learning about what you meant earlier by tombstone debugging.
Also about not being too restrictive when running logcat.
Sorry about that; you were right that there was a lot of detail missing from my initial log output.

This is the freshest tombstone.

[rahlhar]

I also should have mentioned that while the library name is changed, the code for cnfgtest is otherwise unchanged.

[cnlohr]

1. I feel like I've seen this before. Like someone else who's fought with this before re: debugging and Android. I am sending it to them.
2. It would be cool if you could undo some of your changes, and make sure for release, cflags is the same, a lot of care went into those decisions. But, for debug, it would be really cool to get a PR.

[rahlhar]

1. Much appreciated. Cheers.
2. Ack. Will separate into two branches.

[rahlhar]

For now I've force pushed the removal of the Makefile changes to my gdb-support branch.

If I'm not mistaken though I did set it up so that if BUILD=RELEASE then CFLAGS would end up being exactly as in your original.

But I should not have included those changes in this branch anyway.

Before submitting a PR, I should probably clean up the bash script a bit.

For one, I'm not keen on the duplication of values that are already in the Makefile.
Should I call the script via a Makefile recipe instead so it can use the env vars set there?

[rahlhar]

Regarding the Makefile changes - I was wrong to say "exactly" above.
If the order and/or duplication of certain values was crucial then I failed in that regard.

[cnlohr]

Order/duplication doesn't matter - any plans to open a proper PR here?

[rahlhar]

Yes. Wanted to clear some things up first, but I suppose that can be done before the PR gets merged.

[dreua]

PR is #87