Use pstack to track threads on Linux

RedHat Linux distros provide a pstack script which can track process’s threads, and the script is like this:

#!/bin/bash

if test $# -ne 1; then
    echo "Usage: `basename $0 .sh` <process-id>" 1>&2
    exit 1
fi

if test ! -r /proc/$1; then
    echo "Process $1 not found." 1>&2
    exit 1
fi

# GDB doesn't allow "thread apply all bt" when the process isn't
# threaded; need to peek at the process to determine if that or the
# simpler "bt" should be used.

backtrace="bt"
if test -d /proc/$1/task ; then
    # Newer kernel; has a task/ directory.
    if test `/bin/ls /proc/$1/task | /usr/bin/wc -l` -gt 1 2>/dev/null ; then
        backtrace="thread apply all bt"
    fi
elif test -f /proc/$1/maps ; then
    # Older kernel; go by it loading libpthread.
    if /bin/grep -e libpthread /proc/$1/maps > /dev/null 2>&1 ; then
        backtrace="thread apply all bt"
    fi
fi

GDB=${GDB:-/usr/bin/gdb}

if $GDB -nx --quiet --batch --readnever > /dev/null 2>&1; then
    readnever=--readnever
else
    readnever=
fi

# Run GDB, strip out unwanted noise.
$GDB --quiet $readnever -nx /proc/$1/exe $1 <<EOF 2>&1 |
$backtrace
EOF
/bin/sed -n \
    -e 's/^(gdb) //' \
    -e '/^#/p' \
    -e '/^Thread/p'

Copy it to Suse, and use it (pstack process_ID):

linux-uibj:/usr/bin # pstack 1487
Thread 2 (Thread 0x7eff7ce91700 (LWP 1489)):
#0  0x00007eff7ea533cd in poll () from /lib64/libc.so.6
#1  0x00007eff7ef86454 in g_main_context_iterate.isra ()
#2  0x00007eff7ef868ba in g_main_loop_run () from /usr/lib64/libglib-2.0.so.0
#3  0x00007eff7f76a6b6 in gdbus_shared_thread_func ()
#4  0x00007eff7efaae15 in g_thread_proxy () from /usr/lib64/libglib-2.0.so.0
#5  0x00007eff7ed260a4 in start_thread () from /lib64/libpthread.so.0
#6  0x00007eff7ea5b7fd in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x7eff7fbfd800 (LWP 1487)):
#0  0x00007eff7ea533cd in poll () from /lib64/libc.so.6
#1  0x00007eff7ef86454 in g_main_context_iterate.isra ()
#2  0x00007eff7ef868ba in g_main_loop_run () from /usr/lib64/libglib-2.0.so.0
#3  0x000000000040ab08 in ?? ()
#4  0x00007eff7e997b05 in __libc_start_main () from /lib64/libc.so.6
#5  0x0000000000405bd6 in ?? ()

How can we resolve ??()? We can utilize gdb command: “gdb --quiet -nx --readnever /proc/$pid/exe $pid“, Take above process ID (1487) as an example:

linux-uibj:/usr/bin # gdb --quiet -nx --readnever /proc/1487/exe 1487
Reading symbols from /proc/1487/exe...(no debugging symbols found)...done.
......
Missing separate debuginfos, use: zypper install gvfs-backends-debuginfo-1.18.3-3.28.x86_64 libgudev-1_0-0-debuginfo-210-44.1.x86_64
......

The gdb will prompt which debuginfo packages are lacked, then install them:

linux-uibj:/usr/bin # zypper install gvfs-backends-debuginfo-1.18.3-3.28.x86_64 libgudev-1_0-0-debuginfo-210-44.1.x86_64

Execute “pstack 1487” again:

linux-uibj:/usr/bin # pstack 1487
Thread 2 (Thread 0x7eff7ce91700 (LWP 1489)):
#0  0x00007eff7ea533cd in poll () from /lib64/libc.so.6
#1  0x00007eff7ef86454 in g_main_context_iterate.isra ()
#2  0x00007eff7ef868ba in g_main_loop_run () from /usr/lib64/libglib-2.0.so.0
#3  0x00007eff7f76a6b6 in gdbus_shared_thread_func ()
#4  0x00007eff7efaae15 in g_thread_proxy () from /usr/lib64/libglib-2.0.so.0
#5  0x00007eff7ed260a4 in start_thread () from /lib64/libpthread.so.0
#6  0x00007eff7ea5b7fd in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x7eff7fbfd800 (LWP 1487)):
#0  0x00007eff7ea533cd in poll () from /lib64/libc.so.6
#1  0x00007eff7ef86454 in g_main_context_iterate.isra ()
#2  0x00007eff7ef868ba in g_main_loop_run () from /usr/lib64/libglib-2.0.so.0
#3  0x000000000040ab08 in g_vfs_proxy_volume_monitor_daemon_main ()
#4  0x00007eff7e997b05 in __libc_start_main () from /lib64/libc.so.6
#5  0x0000000000405bd6 in _start ()

Now all symbols are resolved.

......
sudo $GDB --quiet $readnever -nx /proc/$1/exe $1 <<EOF 2>&1 |
......

Reference:
How to resolve function name through memory address?。