Howto: Get access to core dumps from GitHub Actions runs

Sometimes your software segfaults when running tests in CI. That's great! Better it fail in CI than in production. But then you need to debug that segfault, in which case a coredump would be very helpful.

Alternatives:

• SSH into your running workflow

Getting coredumps

But how do you get a coredump to your computer?

In order to get coredumps you can debug, you need to:

1. Upload the binary that generated that coredump.
2. Make sure the coredump was stored to disk on segfaults and the like.
3. Upload the coredump even though a CI step has failed.

To ensure core dumps are stored to disk you need to:

1. Override /proc/sys/kernel/core_pattern so core dumps are written to file instead passed to apport (the default on Ubuntu).
2. Run ulimit -c unlimited in the relevant step to ensure core dumps get written to disk.

This repository demonstrates how to do this; see the Workflow definition to see the necessary steps. The coredumps will end up as downloadable artifacts in the results of each relevant GitHub Actions runs.

Debugging the resulting cores

The program will have been compiled on Ubuntu (whatever version you configured in GHA), so some of the shared libraries might not be in the right location unless you use a matching Docker image. However, given the original executable you can at least see your own code. After downloading executable.zip and cores.zip from the GHA artifacts to the directory with the source code:

$ unzip executable.zip
Archive:  executable.zip
  inflating: example
$ unzip cores.zip
Archive:  cores.zip
  inflating: example.1603.1657733715
$ gdb ./example -core ./example.1603.*
...
Core was generated by `./example'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x0000563a1b2b1190 in main () at main.c:6
6         string[0] = 'b';
(gdb) 

If your code is a shared library, and it is in the current directory, you can load it into gdb like so:

(gdb) set solib-search-path .
(gdb) sharedlibrary library-downloaded-from-gha.so

This repository is sponsored by Sciagraph, a performance observability service for Python batch jobs.

And, yes, sometimes my code segfaults in CI. That's why I run tests in CI, to catch bugs in advance before they're released!