You can use the core with GDB in many ways, but passing parameters which is to be passed to the executable to GDB is not the way to use the core file. This could also be the reason you got that error. You can use the core file in the following ways:
gdb <executable> <core-file> or gdb <executable> -c <core-file> or
gdb <executable>
...
(gdb) core <core-file>
When using the core file you don't have to pass arguments. The crash scenario is shown in GDB (checked with GDB version 7.1 on Ubuntu).
For example:
$ ./crash -p param1 -o param2
Segmentation fault (core dumped)
$ gdb ./crash core
GNU gdb (GDB) 7.1-ubuntu
...
Core was generated by `./crash -p param1 -o param2'. <<<<< See this line shows crash scenario
Program terminated with signal 11, Segmentation fault.
#0 __strlen_ia32 () at ../sysdeps/i386/i686/multiarch/../../i586/strlen.S:99
99 ../sysdeps/i386/i686/multiarch/../../i586/strlen.S: No such file or directory.
in ../sysdeps/i386/i686/multiarch/../../i586/strlen.S
(gdb)
If you want to pass parameters to the executable to be debugged in GDB, use --args.
For example:
$ gdb --args ./crash -p param1 -o param2
GNU gdb (GDB) 7.1-ubuntu
...
(gdb) r
Starting program: /home/@@@@/crash -p param1 -o param2
Program received signal SIGSEGV, Segmentation fault.
__strlen_ia32 () at ../sysdeps/i386/i686/multiarch/../../i586/strlen.S:99
99 ../sysdeps/i386/i686/multiarch/../../i586/strlen.S: No such file or directory.
in ../sysdeps/i386/i686/multiarch/../../i586/strlen.S
(gdb)
Man pages will be helpful to see other GDB options.
Most useful commands are:
bt(backtrace)info locals(show values of local variables)info registers(show values of CPU registers)frame X(change to stack frame X)upanddown(navigate in the stack frame (call chain))
You can use the core with GDB in many ways, but passing parameters which is to be passed to the executable to GDB is not the way to use the core file. This could also be the reason you got that error. You can use the core file in the following ways:
gdb <executable> <core-file> or gdb <executable> -c <core-file> or
gdb <executable>
...
(gdb) core <core-file>
When using the core file you don't have to pass arguments. The crash scenario is shown in GDB (checked with GDB version 7.1 on Ubuntu).
For example:
$ ./crash -p param1 -o param2
Segmentation fault (core dumped)
$ gdb ./crash core
GNU gdb (GDB) 7.1-ubuntu
...
Core was generated by `./crash -p param1 -o param2'. <<<<< See this line shows crash scenario
Program terminated with signal 11, Segmentation fault.
#0 __strlen_ia32 () at ../sysdeps/i386/i686/multiarch/../../i586/strlen.S:99
99 ../sysdeps/i386/i686/multiarch/../../i586/strlen.S: No such file or directory.
in ../sysdeps/i386/i686/multiarch/../../i586/strlen.S
(gdb)
If you want to pass parameters to the executable to be debugged in GDB, use --args.
For example:
$ gdb --args ./crash -p param1 -o param2
GNU gdb (GDB) 7.1-ubuntu
...
(gdb) r
Starting program: /home/@@@@/crash -p param1 -o param2
Program received signal SIGSEGV, Segmentation fault.
__strlen_ia32 () at ../sysdeps/i386/i686/multiarch/../../i586/strlen.S:99
99 ../sysdeps/i386/i686/multiarch/../../i586/strlen.S: No such file or directory.
in ../sysdeps/i386/i686/multiarch/../../i586/strlen.S
(gdb)
Man pages will be helpful to see other GDB options.
Most useful commands are:
bt(backtrace)info locals(show values of local variables)info registers(show values of CPU registers)frame X(change to stack frame X)upanddown(navigate in the stack frame (call chain))
Simple usage of GDB, to debug coredump files:
gdb <executable_path> <coredump_file_path>
A coredump file for a "process" gets created as a "core.pid" file.
After you get inside the GDB prompt (on execution of the above command), type:
...
(gdb) where
This will get you with the information, of the stack, where you can analayze the cause of the crash/fault. Other command, for the same purposes is:
...
(gdb) bt full
This is the same as above. By convention, it lists the whole stack information (which ultimately leads to the crash location).
You just need a binary (with debugging symbols included) that is identical to the one that generated the core dump file. Then you can run gdb path/to/the/binary path/to/the/core/dump/file to debug it.
When it starts up, you can use bt (for backtrace) to get a stack trace from the time of the crash. In the backtrace, each function invocation is given a number. You can use frame number (replacing number with the corresponding number in the stack trace) to select a particular stack frame.
You can then use list to see code around that function, and info locals to see the local variables. You can also use print name_of_variable (replacing "name_of_variable" with a variable name) to see its value.
Typing help within GDB will give you a prompt that will let you see additional commands.
Steps to debug coredump using GDB:
Some generic help:
gdb start GDB, with no debugging les
gdb program begin debugging program
gdb program core debug coredump core produced by program
gdb --help describe command line options
First of all, find the directory where the corefile is generated.
Then use
ls -ltrcommand in the directory to find the latest generated corefile.To load the corefile use
gdb binary path of corefileThis will load the corefile.
Then you can get the information using the
btcommand.For a detailed backtrace use
bt full.To print the variables, use
print variable-nameorp variable-nameTo get any help on GDB, use the
helpoption or useapropos search-topicUse
frame frame-numberto go to the desired frame number.Use
up nanddown ncommands to select frame n frames up and select frame n frames down respectively.To stop GDB, use
quitorq.
Videos
Tested in Ubuntu 20.04.
1. Enable core files
First off, run ulimit -c to see what the max allowed size is for core files on your system. On Ubuntu 20.04 for me, mine returns 0, which means no core file can be created.
ulimit --help shows the meaning of -c:
-c the maximum size of core files created
So, set the allowed core file size to unlimited, as shown below. Note that I think this only applies to the one terminal you run this in, and I do not think it's persistent across reboots, so you have to run this each time you want core files to be created, and in each terminal you want it to work in:
# set max core dump file size to unlimited
ulimit -c unlimited
# verify it is now set to "unlimited"
ulimit -c
That's it! Now, run your program and if it crashes and does a "core dump" it will dump the core as a core file into the same directory you were in when you called the executable. The name of the file is simply "core".
UPDATE: wait, where are the core files again?
I tested all of my code and examples here, and they work for me. However, if you do not get a core file locally as described above, apparently you are not alone. So, try the following:
- (As of Ubuntu 21.10 and 22.04): Look inside
/var/lib/apport/coredump, as explained in this other answer by @guyr here. - Look for answers here:
- Where do I find the core dump in ubuntu 16.04LTS?. Possibilities (or commands to run) include:
/var/crash/cat /var/log/apport.log
- [Lots of votes on this one!] Stack Overflow: Core dumped, but core file is not in the current directory?
- systemd may be causing some problems here
- Where do I find the core dump in ubuntu 16.04LTS?. Possibilities (or commands to run) include:
- If your core dump files don't work like mine, leave a comment with what version of Linux you have (ex: Ubuntu 22.04), what kernel version you have (run
cat /proc/version), and where your core dump files are.- Here are my own answers: on Linux Ubuntu 18.04 and 20.04,
ulimit -c unlimitedcauses core dump files to appear right in the dir where I am. I have to manually delete old ones for new ones to be created. On my Ubuntu 18.04 machine,cat /proc/versionshowsLinux version 4.15.0-194-generic (buildd@lcy02-amd64-052) (gcc version 7.5.0 (Ubuntu 7.5.0-3ubuntu1~18.04)).
- Here are my own answers: on Linux Ubuntu 18.04 and 20.04,
2. View the backtrace in gdb
You should have built your C or C++ program with debug symbols on, in order to see useful information in your core file. Without debug symbols, you can only see the addresses of the functions called, not the actual names or line numbers.
In gcc, use -ggdb -O0 to turn on debug symbols optimized for the gdb GNU debugger. You can also use -g -O0, -g3 -O0, etc, but -ggdb -O0 is best. Do we really need optimization level 0 (-O0) for this? Yes, yes we do. See my answer here: Stack Overflow: What's the difference between a compiler's -O0 option and -Og option?
Example build and run commands in C and C++: so, your full build and run commands in C or C++ might look like this:
# C build and run command for "hello_world.c"
gcc -Wall -Wextra -Werror -ggdb -O0 -std=gnu17 -o hello_world hello_world.c \
&& ./hello_world
# C++ build and run command for "hello_world.c"
g++ -Wall -Wextra -Werror -ggdb -O0 -std=gnu++17 -o hello_world hello_world.c \
&& ./hello_world
Open the core file in gdb like this:
gdb path/to/my/executable path/to/core
Assuming you just ran path/to/my/executable, then the core file will be in the same directory you were just in when the core was dumped, so you can just run this:
gdb path/to/my/executable core
In gdb, view the backtrace (function call stack at the time of the crash) with:
bt
# or (exact same command)
where
# OR (for even more details, such as seeing all arguments to the functions--
# thanks to Peter Cordes in the comments below)
bt full
# For gdb help and details, see:
help bt
# or
help where
IMPORTANT: when a core dump occurs, it does NOT automatically overwrite any pre-existing core file in your current directory with a new one, so you must manually remove the old core file with rm core PRIOR TO generating the new core file when your program crashes, in order to always have the latest core file to analyze.
3. Try it out
- In a terminal, run
sleep 30to start a process sleeping for 30 seconds. - While it is running, press Ctrl + \ to force a core dump. You'll now see a
corefile in the directory you are in. - Since we don't have an executable for this with debugging symbols in it, we will just open up the core file in gdb instead of the executable file with symbols + the core file. So, run
gdb -c coreto open the core file just created by the forced crash. - You'll see this. Notice it knows what command you called (
sleep 30) when the core dump occurred:Core was generated by `sleep 30'. Program terminated with signal SIGQUIT, Quit. #0 0x00007f93ed32d334 in ?? () (gdb) - Run
btorwhereto see the backtrace. You'll see this:(gdb) bt #0 0x00007f93ed32d334 in ?? () #1 0x000000000000000a in ?? () #2 0x00007f93ed2960a5 in ?? () #3 0x0000000000000000 in ?? () (gdb) - Those are the addresses to the functions called on the call stack. If you had debugging symbols on, you'd see a lot more info, including function names and line numbers, like this (pulled from a C program of mine):
#10 0x00007fc1152b8ebf in __printf (format=<optimized out>) at printf.c:33 #11 0x0000562bca17b3eb in fast_malloc (num_bytes=1024) at src/fast_malloc.c:225 #12 0x0000562bca17bb66 in malloc (num_bytes=1024) at src/fast_malloc.c:496
4. Forget about core files and just run the program to the crash point in gdb directly!
As @Peter Cordes states in the comments below, you can also just run the program inside gdb directly, letting it crash there, so you have no need to open up a core file after-the-fact! He stated:
Those GDB commands are not specific to core files, they work any time you're stopped at a breakpoint. If you have a reproducible crash, it's often easier / better to run your program under GDB (like
gdb ./a.out) so GDB will have the process in memory instead of a core file. The main advantage is that you can set a breakpoint or watchpoint somewhere before the thing that crashed, and single-step to see what's happening. Or with GDB's record facilities, you may be able to step backwards and see what led up to the crash, but that can be flaky, slow, and memory-intensive.
As stated above, you should have compiled your program with debugging symbols on and with Optimization Level 0, using -ggdb -O0. See the full example build and run commands in C and C++ above.
Now run the program in gdb:
# Open the executable in gdb
gdb path/to/my/executable
# Run it (if it's still crashing, you'll see it crash)
r
# View the backtrace (call stack)
bt
# Quit when done
q
And if you ever need to manually log the backtrace to a log file to analyze later, you can do so like this (adapted from notes in my eRCaGuy_dotfiles repo here):
set logging file gdb_log.txt
set logging on
set trace-commands on
show logging # prove logging is on
flush
set pretty print on
bt # view the backtrace
set logging off
show logging # prove logging is back off
Done! You've now saved the gdb backtrace in file "gdb_log.txt".
References:
- [the answer I needed is in this question itself] https://stackoverflow.com/questions/2065912/core-dumped-but-core-file-is-not-in-the-current-directory
- https://stackoverflow.com/questions/5115613/core-dump-file-analysis
- https://stackoverflow.com/questions/8305866/how-do-i-analyze-a-programs-core-dump-file-with-gdb-when-it-has-command-line-pa/30524347#30524347
- [very useful info, incl. the Ctrl + \ trick to force a core dump!] https://unix.stackexchange.com/questions/277331/segmentation-fault-core-dumped-to-where-what-is-it-and-why/409776#409776
- [referenced by the answer above] https://unix.stackexchange.com/questions/179998/where-to-search-for-the-core-file-generated-by-the-crash-of-a-linux-application/180004#180004
- [answer is in the question itself] Where do I find the core dump in ubuntu 16.04LTS?
- [my answer] Stack Overflow: What's the difference between a compiler's
-O0option and-Ogoption?
Additional reading to do
- [I STILL NEED TO STUDY & TRY THIS] How to use
LD_PRELOADwithgdb: https://stackoverflow.com/questions/10448254/how-to-use-gdb-with-ld-preload
The location of the core file depends on /proc/sys/kernel/core_pattern
E.g. on Ubuntu 22.04:
cat /proc/sys/kernel/core_pattern
gives:
|/usr/share/apport/apport -p%p -s%s -c%c -d%d -P%P -u%u -g%g -- %E
The magic leading pipe syntax means that the Linux kernel will call the program /usr/share/apport/apport with a bunch of informational arguments and then the apport executable will then handle and store the dump somewhere.
apport in particular is part of Ubuntu's ultra automated error statistics setup, I think Canonical keeps a database of all crashes by users who agree to provide them to help prioritize development https://wiki.ubuntu.com/Apport
If you want to get a simple raw core file in the current directory as Linus intended, you have to instead set it to something like core with:
echo 'core' | sudo tee /proc/sys/kernel/core_pattern
How to change it persistently across reboot: How to permanently edit the core_pattern file?
If you do that and then:
ulimit -c unlimited
the current kernel 5.15.0 dumps files with form:
core.<pid>
e.g.:
core.494536
under the current working directory.
It is also apparently possible to turn off apport with: How do I enable or disable Apport?
sudo systemctl disable apport.service
You can test this e.g. with a minimal C program:
segfault.c
#include <stdlib.h>
int main(void) {
*(int *)0 = 1;
return EXIT_SUCCESS;
}
compile and run with:
gcc -ggdb3 -O0 -pedantic-errors -std=c89 -Wall -Wextra -o segfault.out segfault.c
./segfault.out
How to obtain apport core dumps?
Explained at: https://stackoverflow.com/questions/2065912/core-dumped-but-core-file-is-not-in-the-current-directory/47481884#47481884
Let's learn to use it if you decide to keep Ubuntu's mega opinionated apport system.
Maybe it is the for the best that non programs won't get random core.1234 files randomly created in their directories from time to time.
apport stores core dumps under .crash files under /var/crash/. .crash files are wrappers that contain the core dumps and some more system logs to help Ubuntu devs debug.
If you just run:
./segfault.out
you don't get anything. "Great" default behavior!
But Where do I find the core dump in ubuntu 16.04LTS? explains that we can look for apport logs under:
cat /var/log/apport.log
which now contains entries of type:
ERROR: apport (pid 503174) Sat Nov 26 21:51:47 2022: called for pid 503173, signal 11, core limit 18446744073709551615, dump mode 1
ERROR: apport (pid 503174) Sat Nov 26 21:51:47 2022: ignoring implausibly big core limit, treating as unlimited
ERROR: apport (pid 503174) Sat Nov 26 21:51:47 2022: executable: /home/ciro/segfault.out (command line "./segfault.out")
ERROR: apport (pid 503174) Sat Nov 26 21:51:47 2022: executable does not belong to a package, ignoring
ERROR: apport (pid 503174) Sat Nov 26 21:51:47 2022: writing core dump to core._home_ciro_segfault_out.1000.57a7653e-d57e-4871-ad0d-f7b8d9f5b3b9.503173.5035472 (limit: -1)
and no file is generated as per the ignore.
https://stackoverflow.com/questions/14204961/how-to-change-apport-default-behaviour-for-non-packaged-application-crashes asks how to enable the cores so we try:
mkdir -p ~/.config/apport
printf '[main]
unpackaged=true
' >> ~/.config/apport/settings
and now it works, we have a .crash file under /var/crash:
-rw-r--r-- 1 ciro whoopsie 0 Nov 26 22:09 _home_ciro_segfault.out.1000.upload
-rw-r----- 1 ciro whoopsie 137K Nov 26 22:09 _home_ciro_segfault.out.1000.crash
-rw------- 1 whoopsie whoopsie 37 Nov 26 22:09 _home_ciro_segfault.out.1000.uploaded
We can then extract the core dump with:
apport-unpack /var/crash/_home_ciro_segfault.out.1000.crash segfault
which creates a directory named segfault/ containing a bunch of files split out of the .crash file, including our core dump segfault/CoreDump.
The program does crash however due to a bug, it is pitiful:
Traceback (most recent call last):
File "/usr/bin/apport-unpack", line 77, in <module>
pr.extract_keys(f, bin_keys, dir)
File "/usr/lib/python3/dist-packages/problem_report.py", line 269, in extract_keys
raise ValueError('%s has no binary content' %
ValueError: ['separator'] has no binary content
Report at: https://bugs.launchpad.net/ubuntu/+source/apport/+bug/1889443 But it crashes after generating our CoreDumpso we can successfully run:
gdb segfault.out segfault/CoreDump
There is also a potentially more automated way with:
apport-retrace /var/crash/_home_ciro_segfault.out.1000.crash
but it fails for non packaged programs with:
ERROR: report file does not contain one of the required fields: Package
Tested on Ubuntu 22.04.
How to prevent "this executable already crashed 2 times, ignoring"?
If you keep testing with our minimal ./segfault.out, you will notice that at some point .crash files stop being generated.
cat /var/log/apport.log
then clarifies that:
this executable already crashed 2 times, ignoring
OMG this system was not made with devs in mind!
Looking at at source code from apt source apport does not seem configurable at first sight:
crash_counter = 0
# Create crash report file descriptor for writing the report into
# report_dir
try:
report = '%s/%s.%i.crash' % (apport.fileutils.report_dir, info['ExecutablePath'].replace('/', '_'), pidstat.st_uid)
if os.path.exists(report):
if apport.fileutils.seen_report(report):
# do not flood the logs and the user with repeated crashes
# and make sure the file isn't a FIFO or symlink
fd = os.open(report, os.O_NOFOLLOW | os.O_RDONLY | os.O_NONBLOCK)
st = os.fstat(fd)
if stat.S_ISREG(st.st_mode):
with os.fdopen(fd, 'rb') as f:
crash_counter = apport.fileutils.get_recent_crashes(f)
crash_counter += 1
if crash_counter > 1:
write_user_coredump(
pid, process_start, core_ulimit, coredump_fd
)
error_log('this executable already crashed %i times, ignoring' % crash_counter)
sys.exit(0)
# remove the old file, so that we can create the new one with
# os.O_CREAT|os.O_EXCL
os.unlink(report)
How to prevent apport from uploading/requesting to upload reports for non packaged binaries?
On Ubuntu 22.04 under:
- Settings
- Privacy
- Diagnostics
- Send error reports to Canonical
there are three possible options:
- Never
- Automatic, which seems to mean more precisely: "Always without asking"
- Manual, which in plain English means "Ask before sending" with a popup
I don't see how to prevent upload with the "Automatic" method. And "Manual" will drive you mad with popups. So the only reasonable options are "Never" or disabling apport:
- What is the 'whoopsie' process and how can I remove it?
How to analyze core dumps?
This is very Ubuntu independent, so just go for:
- https://stackoverflow.com/questions/8305866/how-do-i-analyze-a-programs-core-dump-file-with-gdb-when-it-has-command-line-pa
- https://unix.stackexchange.com/questions/89933/how-to-view-core-files-for-debugging-purposes-in-linux
Mozilla rr reverse debugging as the ultimate "core file"
Core files allow you to inspect the stack at break.
But in general what you really need to do is to go back in time to further decide the root failure cause.
The amazing Mozilla rr allows you to do that, at the cost of a larger trace file, and a slight performance hit.
Example at: https://stackoverflow.com/questions/1470434/how-does-reverse-debugging-work/53063242#53063242