Use pstack to track threads on Linux

RedHat Linux distros provide a pstack script which can track process’s threads, and the script is like this:

#!/bin/bash

if test $# -ne 1; then
    echo "Usage: `basename $0 .sh` <process-id>" 1>&2
    exit 1
fi

if test ! -r /proc/$1; then
    echo "Process $1 not found." 1>&2
    exit 1
fi

# GDB doesn't allow "thread apply all bt" when the process isn't
# threaded; need to peek at the process to determine if that or the
# simpler "bt" should be used.

backtrace="bt"
if test -d /proc/$1/task ; then
    # Newer kernel; has a task/ directory.
    if test `/bin/ls /proc/$1/task | /usr/bin/wc -l` -gt 1 2>/dev/null ; then
        backtrace="thread apply all bt"
    fi
elif test -f /proc/$1/maps ; then
    # Older kernel; go by it loading libpthread.
    if /bin/grep -e libpthread /proc/$1/maps > /dev/null 2>&1 ; then
        backtrace="thread apply all bt"
    fi
fi

GDB=${GDB:-/usr/bin/gdb}

if $GDB -nx --quiet --batch --readnever > /dev/null 2>&1; then
    readnever=--readnever
else
    readnever=
fi

# Run GDB, strip out unwanted noise.
$GDB --quiet $readnever -nx /proc/$1/exe $1 <<EOF 2>&1 |
$backtrace
EOF
/bin/sed -n \
    -e 's/^(gdb) //' \
    -e '/^#/p' \
    -e '/^Thread/p'

Copy it to Suse, and use it (pstack process_ID):

linux-uibj:/usr/bin # pstack 1487
Thread 2 (Thread 0x7eff7ce91700 (LWP 1489)):
#0  0x00007eff7ea533cd in poll () from /lib64/libc.so.6
#1  0x00007eff7ef86454 in g_main_context_iterate.isra ()
#2  0x00007eff7ef868ba in g_main_loop_run () from /usr/lib64/libglib-2.0.so.0
#3  0x00007eff7f76a6b6 in gdbus_shared_thread_func ()
#4  0x00007eff7efaae15 in g_thread_proxy () from /usr/lib64/libglib-2.0.so.0
#5  0x00007eff7ed260a4 in start_thread () from /lib64/libpthread.so.0
#6  0x00007eff7ea5b7fd in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x7eff7fbfd800 (LWP 1487)):
#0  0x00007eff7ea533cd in poll () from /lib64/libc.so.6
#1  0x00007eff7ef86454 in g_main_context_iterate.isra ()
#2  0x00007eff7ef868ba in g_main_loop_run () from /usr/lib64/libglib-2.0.so.0
#3  0x000000000040ab08 in ?? ()
#4  0x00007eff7e997b05 in __libc_start_main () from /lib64/libc.so.6
#5  0x0000000000405bd6 in ?? ()

How can we resolve ??()? We can utilize gdb command: “gdb --quiet -nx --readnever /proc/$pid/exe $pid“, Take above process ID (1487) as an example:

linux-uibj:/usr/bin # gdb --quiet -nx --readnever /proc/1487/exe 1487
Reading symbols from /proc/1487/exe...(no debugging symbols found)...done.
......
Missing separate debuginfos, use: zypper install gvfs-backends-debuginfo-1.18.3-3.28.x86_64 libgudev-1_0-0-debuginfo-210-44.1.x86_64
......

The gdb will prompt which debuginfo packages are lacked, then install them:

linux-uibj:/usr/bin # zypper install gvfs-backends-debuginfo-1.18.3-3.28.x86_64 libgudev-1_0-0-debuginfo-210-44.1.x86_64

Execute “pstack 1487” again:

linux-uibj:/usr/bin # pstack 1487
Thread 2 (Thread 0x7eff7ce91700 (LWP 1489)):
#0  0x00007eff7ea533cd in poll () from /lib64/libc.so.6
#1  0x00007eff7ef86454 in g_main_context_iterate.isra ()
#2  0x00007eff7ef868ba in g_main_loop_run () from /usr/lib64/libglib-2.0.so.0
#3  0x00007eff7f76a6b6 in gdbus_shared_thread_func ()
#4  0x00007eff7efaae15 in g_thread_proxy () from /usr/lib64/libglib-2.0.so.0
#5  0x00007eff7ed260a4 in start_thread () from /lib64/libpthread.so.0
#6  0x00007eff7ea5b7fd in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x7eff7fbfd800 (LWP 1487)):
#0  0x00007eff7ea533cd in poll () from /lib64/libc.so.6
#1  0x00007eff7ef86454 in g_main_context_iterate.isra ()
#2  0x00007eff7ef868ba in g_main_loop_run () from /usr/lib64/libglib-2.0.so.0
#3  0x000000000040ab08 in g_vfs_proxy_volume_monitor_daemon_main ()
#4  0x00007eff7e997b05 in __libc_start_main () from /lib64/libc.so.6
#5  0x0000000000405bd6 in _start ()

Now all symbols are resolved.

P.S. You should execute the script in root privilege. E.g., modify the script as:

......
sudo $GDB --quiet $readnever -nx /proc/$1/exe $1 <<EOF 2>&1 |
......

Reference:
How to resolve function name through memory address?

 

5 thoughts on “Use pstack to track threads on Linux”

  1. hi could you tell me why pstack do not printf the message,but only printf like below:

    28378: ./deadlock
    (No symbols found in )
    (No symbols found in /lib/i386-linux-gnu/libc.so.6)
    (No symbols found in /lib/ld-linux.so.2)
    crawl: Input/output error
    Error tracing through process 28378
    0xb779fce5: _fini + 0x25d51

  2. HI nanxiao,
    I have logged in with different user but am not able to get any details with the process id but with root user am able to get that. can you help me out. what might be the issue?
    Root user :
    pstack 45773
    #0 0x00007ff581433c03 in select () from /lib64/libc.so.6
    #1 0x00007ff58235bb2c in CBaseSocket::Select(IArchErr*, int, int, int, int, Decl_limo_ssl_st) () from /OnDeck116/BE/Finacle/FC/app/prodbase/exe/arlimoc.so
    #2 0x0000000000408685 in ?? ()
    #3 0x00000000004089f4 in ?? ()
    #4 0x0000000000402f3b in ?? ()
    #5 0x0000000000405bcf in ?? ()
    #6 0x00007ff5813603d5 in __libc_start_main () from /lib64/libc.so.6
    #7 0x0000000000402b69 in ?? ()

    non-root user:
    pstack 45773
    [oracle@FinaclePS oracle]$

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.