X86-64 kernel backtrace #4

deepfire · 2016-12-23T14:14:17Z

This optionally provides a backtrace mechanism on x86-64:

governed by Config.stack_trace
looks like this:

FATAL: stack smashed.
    call trace (rbp: fffffe0003a45c80, stack: fffffe0003a45dc0-fffffe0003a41dc0, text: fffffe00030c3000-fffffe00031f43da):
fffffe00031ddaf7 __stack_chk_fail + 0x17
fffffe00031ce5e4 dump_idt + 0x294
fffffe00031ced1e gdb_arch_read_byte + 0x15e
         31cf841 arch_init + 0x5b1
kernel 0 PANIC! finally reached __stack_chk_fail()

barrelfish-bot · 2016-12-23T14:14:19Z

Can one of the admins verify this patch?

Signed-off-by: Kosyrev Serge <[email protected]>

antiguru · 2017-01-11T08:03:30Z

hake/Config.hs.template

@@ -367,7 +371,7 @@ config_arrakismon = False
 defines :: [RuleToken]
 defines = [ Str ("-D" ++ d) | d <- [
             if microbenchmarks then "CONFIG_MICROBENCHMARKS" else "",
-             if trace then "CONFIG_TRACE" else "",
+             if stack_trace then "CONFIG_KERNEL_STACK_TRACE" else "",


Shouldn't delete trace/CONFIG_TRACE

@antiguru, I have welded in a patch that rectifies this travesty, sorry!

antiguru · 2017-01-11T08:06:28Z

kernel/include/arch/x86/debug.h

 void debug_vaddr_identify(lvaddr_t pml4, lvaddr_t vaddr);

+void debug_sort_dynsyms (struct Elf64_Sym *dynsyms, int n);
+void debug_setup_stackwalker (uint64_t stack_top, uint64_t stack_bottom, uint64_t text_start, uint64_t text_end, struct Elf64_Sym *dynsyms, char *dynstr, int nsyms);


It's better to use more specific types instead of uint64_t, for example lpaddr_t for local physical addresses or lvaddr_t for virtual addresses. We might not do it correctly all the time but at least it gives a hint on how that address could be accessed.

Addressed this in additional patches..

antiguru · 2017-01-11T08:06:59Z

kernel/include/arch/x86/x86.h

@@ -62,6 +62,20 @@

 #ifndef __ASSEMBLER__

+static inline uint64_t __attribute__((always_inline)) rdrip(void)


I guess this should have a return type of lvaddr_t.

Addressed this in additional patches..

antiguru · 2017-01-11T08:13:44Z

kernel/qsort.c

@@ -0,0 +1,266 @@
+/*


An alternative of copying newlib's qsort.c here would be to use the one in /lib/newlib/newlib/libc/search/qsort.c directly. I'm undecided what makes more sense.

Please, do tell if you come to a decision!

I initially thought there was no precedent to this kind of external inclusion of out-of-kernel-directory sources, but kernel/Hakefile references ../usr/drivers/cpuboot/init_ap_x86_64.S.

Not sure if this constitutes a significant point..

antiguru

Thanks for sending us the pull request. It seems like a valuable addition to the kernel. I added some comments in-line where I see potential room for improvement.
Could the code be unified for all architectures, especially ARMv8?

simu · 2017-01-11T08:27:46Z

@barrelfish-bot ok to test

Signed-off-by: Kosyrev Serge <[email protected]>

..doesn't work due to: NixOS/nixpkgs#21374 Signed-off-by: Kosyrev Serge <[email protected]>

Signed-off-by: Kosyrev Serge <[email protected]>

…bols missing from .dynsyms Signed-off-by: Kosyrev Serge <[email protected]>

Signed-off-by: Kosyrev Serge <[email protected]>

…r code review Signed-off-by: Kosyrev Serge <[email protected]>

…er review Signed-off-by: Kosyrev Serge <[email protected]>

deepfire · 2017-01-12T15:30:49Z

An extra note ought to be made -- perhaps even documented.

The backtrace mechanism currently depends on the .dynsyms section, which only carries globally-visible symbols, which naturally excludes functions declared as static.
This is somewhat painful, and so had to be worked around by making the stack analysis more precise -- that is, it now considers inter-symbol gaps, and reports when a stack return address points to such.

A more advanced solution would be to promote a more complete symbol section, in a format amenable to runtime parsing, to be marked as ALLOC and LOAD.

Since, apparently (I'd be happy to be wrong!), ld is unable to mark .debug_* sections as ALLOC and LOAD, an objcopy pass is required -- but that is more invasive than I could venture for without reaching for consensus on whether the thing is needed at all.

So, please, tell me what all of this makes you think..

deepfire · 2017-01-12T15:37:46Z

On unification. The code can be generalised, indeed. It would take one more pass, though..

deepfire · 2017-01-13T00:05:07Z

Do I get the interpretation of the test suite right -- namely that the boot on physical hardware has failed?
I see the following:

relinking /home/netos/tftpboot/menu.lst.10.110.4.15 to harness/nos5_harness/menu.lst
DEPRECATED reboot
executing /home/netos/tools/bin/rackpower -r nos5
Host key verification failed.
Warning: rackpower -r nos5 failed
DEPRECATED get_output
[0:00:00] [Enter `^Ec?' for help]
[0:00:00] [no, [email protected] is attached]
[0:06:00] [Error: boot timed out, retrying...]
DEPRECATED reboot
executing /home/netos/tools/bin/rackpower -r nos5
Host key verification failed.
Warning: rackpower -r nos5 failed
[0:12:00] [Error: boot timed out, retrying...]
DEPRECATED reboot
executing /home/netos/tools/bin/rackpower -r nos5
Host key verification failed.
Warning: rackpower -r nos5 failed
[0:18:00] [Error: boot timed out, retry limit reached]
timeout encountered in collect_data
harness: cleanup test
DEPRECATED get_tftp_dir
DEPRECATED shutdown
executing /home/netos/tools/bin/rackpower -d nos5
Host key verification failed.
Warning: rackpower -d nos5 failed
DEPRECATED unlock
quitting console process (12659)
removing /home/netos/tftpboot/harness/nos5_harness
Error: Timeout while running test (attempting to continue)
magic string "root (nd)" or "Kernel starting at address" not found, assuming no garbage in output

0/1 tests passed
Failed tests:
 * perfmontest

simu · 2017-01-13T08:30:11Z

@deepfire Looks like the machine didn't come up, this does not say anything about the code under test.

deepfire force-pushed the x86-64-backtrace branch from 06d371d to 7e2a887 Compare December 23, 2016 14:32

deepfire added 6 commits December 23, 2016 17:37

x86_64 | paging: expose the PDE.PS bit and the leaf-ness test

6fb3df2

Signed-off-by: Kosyrev Serge <[email protected]>

x86: add rdrip() and rdrsp()

b25cee3

Signed-off-by: Kosyrev Serge <[email protected]>

x86_64 | mem_to_local_phys: fix docstring

a1aae48

Signed-off-by: Kosyrev Serge <[email protected]>

x86_64 | kernel | debug: recognize non-4k pages in debug_vaddr_identify

c9a85f5

Signed-off-by: Kosyrev Serge <[email protected]>

kernel | hake: drop an apparently obsolete comment

fe1bf2b

Signed-off-by: Kosyrev Serge <[email protected]>

kernel | x86-64: provide _start_kernel_text

a06227c

Signed-off-by: Kosyrev Serge <[email protected]>

antiguru reviewed Jan 11, 2017

View reviewed changes

deepfire force-pushed the x86-64-backtrace branch from eba42c4 to 39cf69c Compare January 12, 2017 13:41

deepfire and others added 15 commits January 12, 2017 17:15

hake: add a config option for stack trace functionality

f3023ff

Signed-off-by: Kosyrev Serge <[email protected]>

kernel | x86: provide all symbols in .dynsyms, if Config.stack_trace

f362eb1

Signed-off-by: Kosyrev Serge <[email protected]>

kernel | x86: compile with frame pointer, if Config.stack_trace

1943e38

Signed-off-by: Kosyrev Serge <[email protected]>

kernel: add qsort.c, if Config.stack_trace

db4d8ca

Signed-off-by: Kosyrev Serge <[email protected]>

kernel | x86-64 | debug: backtrace implementation

92cdb5e

Signed-off-by: Kosyrev Serge <[email protected]>

kernel | x86-64 | init: enable stack tracing, if Config.stack_trace

9e293c2

Signed-off-by: Kosyrev Serge <[email protected]>

kernel | x86: print backtrace on __stack_chk_fail()

9d2a528

Signed-off-by: Kosyrev Serge <[email protected]>

nix: provisionally switch to gdb-multitarget

26eaa3b

..doesn't work due to: NixOS/nixpkgs#21374 Signed-off-by: Kosyrev Serge <[email protected]>

qemu-wrapper: use gdb-multiarch where available, fall back to plain gdb

701e047

Signed-off-by: Kosyrev Serge <[email protected]>

kernel | x86_64 | init: relocate dynsyms later

c23c34f

Signed-off-by: Kosyrev Serge <[email protected]>

x86 | debug: qemu_debug_puts()

8b4662b

Signed-off-by: Kosyrev Serge <[email protected]>

x86 | debug: qemu_debug_printf ()

c78e6bc

Signed-off-by: Kosyrev Serge <[email protected]>

kernel | x86_64 | debug: IDT dumping facility

25010f0

Signed-off-by: Kosyrev Serge <[email protected]>

kernel | x86_64 | stackwalker: print the address delta

333db58

Signed-off-by: Kosyrev Serge <[email protected]>

kernel | x86_64 | stackwalker: avoid misattributions due to local sym…

2dfa4f0

…bols missing from .dynsyms Signed-off-by: Kosyrev Serge <[email protected]>

deepfire and others added 4 commits January 12, 2017 17:15

qemu-wrapper: write QEMU debug port output to debugcon.out

86db9bd

Signed-off-by: Kosyrev Serge <[email protected]>

qemu-wrapper: --display option to allow QEMU show target display

3ee9d31

Signed-off-by: Kosyrev Serge <[email protected]>

kernel | x86_64 | stackwalker: use more specific address types, as pe…

ba95644

…r code review Signed-off-by: Kosyrev Serge <[email protected]>

kernel | x86.h: use more specific address types for rdr{i,s}p(), as p…

5040d46

…er review Signed-off-by: Kosyrev Serge <[email protected]>

deepfire force-pushed the x86-64-backtrace branch from 4cad1eb to 5040d46 Compare January 12, 2017 14:15

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

X86-64 kernel backtrace #4

X86-64 kernel backtrace #4

deepfire commented Dec 23, 2016

barrelfish-bot commented Dec 23, 2016

antiguru Jan 11, 2017

deepfire Jan 12, 2017

antiguru Jan 11, 2017

deepfire Jan 12, 2017

antiguru Jan 11, 2017

deepfire Jan 12, 2017

antiguru Jan 11, 2017

deepfire Jan 12, 2017

deepfire Jan 12, 2017

antiguru left a comment

simu commented Jan 11, 2017

deepfire commented Jan 12, 2017

deepfire commented Jan 12, 2017

deepfire commented Jan 13, 2017

simu commented Jan 13, 2017

		@@ -62,6 +62,20 @@

		#ifndef __ASSEMBLER__

		static inline uint64_t __attribute__((always_inline)) rdrip(void)

X86-64 kernel backtrace #4

Are you sure you want to change the base?

X86-64 kernel backtrace #4

Conversation

deepfire commented Dec 23, 2016

barrelfish-bot commented Dec 23, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

antiguru left a comment

Choose a reason for hiding this comment

simu commented Jan 11, 2017

deepfire commented Jan 12, 2017

deepfire commented Jan 12, 2017

deepfire commented Jan 13, 2017

simu commented Jan 13, 2017