Crash logging

[Razish]

This would greatly benefit developers and server owners alike, being able to track down the cause of an obscure or intermittent crash.
Listing the type of crash, loaded modules, dumping registers, disassembling the code at the crash location and stack backtrace with debug symbol lookup, server-specific information such as players connected, map, etc
Works on Windows and Linux.
May need permission from @Lrns123 aka BobaFett, I originally adapted this from JKG =o
Example crash

@exidl also had Google Breakpad with automatic cURL, but I'm not sure how suitable that is for live environments.

[aisouard]

Breakpad will just create it's own signal handler to catch them, then write a minidump file in a temporary folder containing the stack backtrace, loaded modules and registers. At this moment, it crashes while sending the minidump via HTTP, because a call to NET_Sleep on "select" occurs in the meantime.

But yours looks better ! Do we have to extract the debug symbols and place them into a specific folder with yours ?

[eezstreet]

afaik, yes

Date: Sat, 20 Apr 2013 23:17:29 -0700
From: [email protected]
To: [email protected]
Subject: Re: [OpenJK] Crash logging (#141)

Breakpad will just create it's own signal handler to catch them, then write a minidump file in a temporary folder containing the stack backtrace, loaded modules and registers. At this moment, it crashes while send the minidump via HTTP, because a call to NET_Sleep on "select" occurs in the meantime.

But yours looks better ! Do we have to extract the debug symbols and place them into a specific folder with yours ?

—
Reply to this email directly or view it on GitHub.

[Razish]

Slight issue with my exception handler is it's meant for 32 bit disassembly =]

[ensiform]

Why even need anyway just add signal handling?

[Razish]

It's the same thing, I just had debug symbol lookup and disassembly for crashes as-well.

[xycaleth]

Minidumps for Windows might be a good idea :D

[xycaleth]

Adding this to milestone 2. Will be useful for debugging crashes.

[mrwonko]

Minidumps are very useful, but only if you also have a symbol server. Otherwise you'll have trouble finding the right debug symbols for the build.

There are a couple of articles on symbol servers on Random ASCII.

If a program might crash, and if you want to be able to investigate the crash then put the symbols on the symbol server.

You'll also want a source server. At least that's what it was called at one point, when it used Perforce in the backend, I don't know if it's the right term in this context. Basically Visual Studio can automatically fetch the source code revision associated with a crash, and I'm sure by now it supports Git, but this probably needs to be set up.

Initially the focus should be on Windows since that's the largest user base.

[ensiform]

Google breakpad could be a good choice for this but would need the symbols hosted on the buildbot or something.

[mrwonko]

One thing I just noticed is that pdb files can be compressed nicely with gzip. Some old OpenJK build I had lying around went down to 1/4 (40MB to 10MB) while my template-heavy compiler/vm went from 60MB to a staggering 3MB.

That's useful because gzip is also commonly used to compress HTTP traffic, and nginx for example supports the usage of precompressed .gz files, decompressing them on the fly if the client doesn't support it. (A symbol server is just an HTTP server with a specific directory layout, to my knowledge.) So that's useful for saving space.

I'd offer space on my server but with a 200GB limit (60+GB of which are already filled with the jk3files backup) I'm not sure I could support 10MB/commit...

[mrwonko]

Actually at 10MB per commit per platform this isn't really all that big; buildbot has about 500 builds in total so far, which comes out at 15GB - I can offer that much. And old symbols can always be purged and just kept in an offline backup.

[deepy]

If space is an issue, we could always have buildbot upload the symbols somewhere else, buildbot is quite flexible.

[mrwonko]

Yeah, but where? It should be some publicly accessible server, and I have one at mrwonko.de, which could very well be used for this. (Just need to adjust the buildbot build scripts to upload it there.)

[deepy]

We can start at pretty much anywhere, I could host some at xd.cm in case all the alternatives fail.

[deepy]

pubkey for buildbot:

ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA1DJyxhLAT4kzx9VkQIJpirXPr+3XueD+sTQbiqA4WzV7uhAaB7Td4kIWLEjHDng8cJkKLScquSiFDnwUqfkzMjqP/+z03fDvQaVODeG0+fe4i73Nqgv1ZyxqMRhA27ylDWddc7rEg7uG74ApWl2/IFLyaKcbkDNdBkckgHMCj1/WAYklsoP+XsgJLcEcaT9HZCcjPA4d/j625AZ1mkbIuxs6vaSAzkguxcf9d9AxZ7HvoXfEH9oBR8f1bAaOT8G1GZFBiMzCLLgCM19AJvCaGxSeMJwO/BNORuPEWFu8VHdWdMA+/Nt8U3jopkRN1m4RyKhAFy4EKiOmxlyDICzREw== [email protected]

[dionrhys]

Why not just put the .pdb files into the release archives on builds.openjk.org? There wouldn't need to be any extra hosted infrastructure.

[mrwonko]

Because then it (and the right revision of the source code) would not automatically be retrieved when you load a minidump. Also causes unnecessary traffic.

[dionrhys]

👌

[mrwonko]

I started looking into symbol servers. 89ea9c3 adds tools to determine the directory to save .exe/.pdb files on the symbol server; usually Microsoft's symstore would be used for this, but I don't think that can upload them using scp to my Linux server.

Still, without a source server that's of limited use; you still need the right revision of the source to be able to effectively analyze crashdumps. Setting one up is not exactly straightforward though...

[mrwonko]

Just a quick note since I'm closing the tab - SourceServer-GitExtension may be of use for setting up a source server/source indexing (a configuration that tells VisualStudio how to download the source code belonging to a given binary/pdb) - see also Source Indexing is Underused Awesomeness.