Fork me on GitHub
Suzf  Blog

How-To: Debug Crashed Linux Application Core Files Like A Pro

License: Attribution-NonCommercial-ShareAlike 4.0 International

本文出自 Suzf Blog。 如未注明,均为 SUZF.NET 原创。

转载请注明:http://suzf.net/post/319

Core dumps are often used to diagnose or debug errors in Linux or UNIX programs. Core dumps can serve as useful debugging aids for sys admins to find out why Application like Lighttpd, Apache, PHP-CGI or any other program crashed. Many vendors and open source project author requests a core file to troubleshoot a program. A core file is generated when an application program abnormally terminates due to bug, operating system security protection schema, or program simply try to write beyond the area of memory it has allocated, and so on. This article explains how to turn on core file support and track down bugs in programs.

 

Turn On Core File Creation Support

By default most Linux distributions turn off core file creation (at least this is true for RHEL, CentOS, Fedora and Suse Linux). You need to use the ulimit command to configure core files.

See The Current Core File Limits

Type the following command:
# ulimit -c
Sample outputs:   # 0

The output 0 (zero) means core file is not created.

Change Core File Limits

In this example, set the size limit of core files to 75000 bytes:
# ulimit -c 75000

HowTo: Enable Core File Dumps For Application Crashes And Segmentation Faults

Edit /etc/profile file and find line that read as follows to make persistent configuration:

# ulimit -S -c 0 > /dev/null 2>&1

Update it as follows:

# ulimit -c unlimited >/dev/null 2>&1

Save and close the file. Edit /etc/sysctl.conf, enter:
# vi /etc/sysctl.conf
Append the following lines:

kernel.core_uses_pid = 1
kernel.core_pipe_limit = 1
kernel.core_pattern = /tmp/core-%e-%s-%u-%g-%p-%t
fs.suid_dumpable = 2

Save and close the file. Where,

  1. kernel.core_uses_pid = 1 - Appends the coring processes PID to the core file name.
  2. fs.suid_dumpable = 2 - Make sure you get core dumps for setuid programs.
  3. kernel.core_pattern = /tmp/core-%e-%s-%u-%g-%p-%t - When the application terminates abnormally, a core file should appear in the /tmp. The kernel.core_pattern sysctl controls exact location of core file. You can define the core file name with the following template whih can contain % specifiers which are substituted by the following values when a core file is created:
    • %% - A single % character
    • %p - PID of dumped process
    • %u - real UID of dumped process
    • %g - real GID of dumped process
    • %s - number of signal causing dump
    • %t - time of dump (seconds since 0:00h, 1 Jan 1970)
    • %h - hostname (same as ’nodename’ returned by uname(2))
    • %e - executable filename

Finally, enable debugging for all apps, enter (Redhat and friends specific):
# echo "DAEMON_COREFILE_LIMIT='unlimited'" >> /etc/sysconfig/init
Reload the settings in /etc/sysctl.conf by running the following command:
# sysctl -p

How Do I Enable Core Dumping For Specific Deamon?

To enable core dumping for specific deamons, add the following line in the /etc/sysconfig/daemon-file file. In this example, edit /etc/init.d/httpd and add line as follows:

# DAEMON_COREFILE_LIMIT='unlimited'

Please note that DAEMON_COREFILE_LIMIT is Redhat specific, for all other distro add configuration as follows:

ulimit -c unlimited >/dev/null 2>&1
echo /tmp/core-%e-%s-%u-%g-%p-%t > /proc/sys/kernel/core_pattern

 

Save and close the file. Restart / reload httpd:
# /etc/init.d/httpd restart
# su - httpd
$ ulimit -c

Sample outputs:  # unlimited

# Test generation core file

#ps -ef| grep httpd
apache   22997 22989  0 12:16 ?        00:00:00 /usr/sbin/httpd

#kill -s SIGSEGV 22997

or # kill -6 22997

# kill -l
 1) SIGHUP	 2) SIGINT	 3) SIGQUIT	 4) SIGILL	 5) SIGTRAP
 6) SIGABRT	 7) SIGBUS	 8) SIGFPE	 9) SIGKILL	10) SIGUSR1
11) SIGSEGV	12) SIGUSR2	13) SIGPIPE	14) SIGALRM	15) SIGTERM
16) SIGSTKFLT	17) SIGCHLD	18) SIGCONT	19) SIGSTOP	20) SIGTSTP
21) SIGTTIN	22) SIGTTOU	23) SIGURG	24) SIGXCPU	25) SIGXFSZ
26) SIGVTALRM	27) SIGPROF	28) SIGWINCH	29) SIGIO	30) SIGPWR
31) SIGSYS	34) SIGRTMIN	35) SIGRTMIN+1	36) SIGRTMIN+2	37) SIGRTMIN+3
38) SIGRTMIN+4	39) SIGRTMIN+5	40) SIGRTMIN+6	41) SIGRTMIN+7	42) SIGRTMIN+8
43) SIGRTMIN+9	44) SIGRTMIN+10	45) SIGRTMIN+11	46) SIGRTMIN+12	47) SIGRTMIN+13
48) SIGRTMIN+14	49) SIGRTMIN+15	50) SIGRTMAX-14	51) SIGRTMAX-13	52) SIGRTMAX-12
53) SIGRTMAX-11	54) SIGRTMAX-10	55) SIGRTMAX-9	56) SIGRTMAX-8	57) SIGRTMAX-7
58) SIGRTMAX-6	59) SIGRTMAX-5	60) SIGRTMAX-4	61) SIGRTMAX-3	62) SIGRTMAX-2

^_^[13:42:30][[email protected] ~]#ls -lth  /tmp/core*
-rw------- 1 root apache 9.9M Sep 14 12:17 /tmp/core-httpd-11-48-48-22997-1442204248

lol , the core file has created when the process crash .  ^_^

 

Now, you can send core files to vendor or software writes.

How Do I Read Core Files?

You need use the gdb command as follows:
$ gdb /path/to/application /path/to/corefile
See the gdb command man page for more information.

strace command

System administrators, diagnosticians and trouble-shooters will find it invaluable for solving problems with programs for which the source is not readily available since they do not need to be recompiled in order to trace them. This is also useful to submit bug reports to open source developers. See how to use the strace command under Linux to debug the problems.

Recommended readings:

Stay tunned for gdb tutorial which will explains how to use generated core file to track down problem.

Trans from : http://www.cyberciti.biz/tips/linux-core-dumps.html

Jeffrey Modify redistribution

 

「一键投喂 软糖/蛋糕/布丁/牛奶/冰阔乐!」

Suzf Blog

(๑>ڡ<)☆ 谢谢 ~

使用微信扫描二维码完成支付