Nan Xiao's Blog

A system software / performance engineer's home

Tag: gdb (Page 1 of 2)

The tips of using and debugging C++ std::iostream

(1) Be cautious of '\n' and std::endl.

The std::endl will flush the output buffer while '\n' not. For example, in socket programming, if client sends message to server using '\n', like this:

out << "Hello World!" << '\n';
out << "I am coming" << '\n';

The server may still block in reading operation and no data is fetched. So you should use std::endl in this case:

out << "Hello World!" << std::endl;
out << "I am coming" << std::endl;

(2) Use std::ios::rdstate.

std::ios::rdstate is a handy function to check the stream state. You can use it in gdb debugging:

(gdb) p in.rdstate()
$45 = std::_S_goodbit
(gdb) n
350             return in;
(gdb) p in.rdstate()
$46 = std::_S_failbit

Through step-mode, we can see which operation crashes the stream.

(3) Serialize the data into file.

No matter you want to do test or debug issue, dump the data into a file and read it back is a very effective method:

std::ofstream ofs("data.txt");
ofs << output;

std::ifstream ifs("data.txt");
ifs >> input;

The above simple code can verify whether your serialization functions are correct or not. Trust me, it is a very brilliant trouble-shouting std::iostream issue trick, and it saved me just now!

The core dump file in Arch Linux

Arch Linux uses systemd, and the core dump files will be stored in /var/lib/systemd/coredump directory by default.

Let’s look at the following simple program:

int main(void) {
        int *p = 0;
        *p = 1;

Compile and run it:

$ gcc -g -o test test.c
$ ./test
Segmentation fault (core dumped)

Definitely, the program crashes! Check the core dump files:

$ coredumpctl list
TIME                            PID   UID   GID SIG COREFILE EXE
Mon 2017-01-09 16:13:44 SGT    7307  1014  1014  11 present  /home/xiaonan/test
$ ls -alt /var/lib/systemd/coredump/core.test*
-rw-r-----+ 1 root root     22821 Jan  9 16:13 /var/lib/systemd/coredump/core.test.1014.183b4c57ccad464abd2eba2c104a47a8.7307.1483949624000000000000.lz4

According to the file name and time, we can find the program’s corresponding core dump binary. To debug it, we can use “coredumpctl gdb PID” command:

$ coredumpctl gdb 7307
       PID: 7307 (test)
       UID: 1014 (xiaonan)
       GID: 1014 (xiaonan)
    Signal: 11 (SEGV)
 Timestamp: Mon 2017-01-09 16:13:44 SGT (6min ago)
Command Line: ./test
Executable: /home/xiaonan/test
 Control Group: /user.slice/user-1014.slice/session-c4.scope
          Unit: session-c4.scope
         Slice: user-1014.slice
       Session: c4
     Owner UID: 1014 (xiaonan)
       Boot ID: 183b4c57ccad464abd2eba2c104a47a8
    Machine ID: 25671e5feadb4ae4bbe2c9ee6de97d66
      Hostname: supermicro-sys-1028gq-trt
       Storage: /var/lib/systemd/coredump/core.test.1014.183b4c57ccad464abd2eba2c104a47a8.7307.1483949624000000000000.lz4
       Message: Process 7307 (test) of user 1014 dumped core.

            Stack trace of thread 7307:
            #0  0x00000000004004b6 main (test)
            #1  0x00007f67ba722291 __libc_start_main (
            #2  0x00000000004003da _start (test)

GNU gdb (GDB) 7.12
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-pc-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
Find the GDB manual and other documentation resources online at:
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /home/xiaonan/test...done.
[New LWP 7307]
Core was generated by `./test'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x00000000004004b6 in main () at test.c:3
3               *p = 1;
(gdb) bt
#0  0x00000000004004b6 in main () at test.c:3

Attention should be paid. The core dump size is limited to 2GiB, so if it is too large, it will be truncated, and generate the warning like this (Please refer this bug):

BFD: Warning: /var/tmp/coredump-ZrhAM4 is truncated: expected core file size >= 2591764480, found: 2147483648.

GDB and trouble with core dumps;
Core dump.

A brief intro of delve

delve is a debugger developed in Golang and also dedicated to help trouble-shooting Golang programs (Home page is here). Though it is still in pre-1.0 release, I think it is stable enough for daily use. BTW, if you find some bugs, you can report it to developers and help to make delve more stronger! P.S., Albeit fmt.Print buddies are useful in most cases, I strongly recommend you try to usedelve to inspect the internal mechanism of your code, because it can help you know Golang deeper, not just superficial stuff.

Installing delve is very simple: taking Linux platform as an example, it is no different from setting up other Golang projects, just “go get” is enough:

go get 

Now, in $GOPATH/bin, there is an extra dlv executable binary file (Notice: the project is named delve, while the executable file is calleddlv. I even made a foolish mistake when began to use it).Run dlv command, and it will show you a detailed manual of delve:

# dlv
Delve is a source level debugger for Go programs.

Delve enables you to interact with your program by controlling the execution of the process,
evaluating variables, and providing information of thread / goroutine state, CPU register state and more.

The goal of this tool is to provide a simple yet powerful interface for debugging Go programs.

  dlv [command]

Available Commands:
  version     Prints version.
  run         Deprecated command. Use 'debug' instead.
  debug       Compile and begin debugging program.

Let’s check this artificial Hello.go program:

package main

import "fmt"

func main() {
        var s []byte
        s = append(s, []byte("Hello, Debugging!")...)

Use delve to debug it:

# dlv debug Hello.go
Type 'help' for list of commands.
(dlv) help
The following commands are available:
    help (alias: h) ------------- Prints the help message.
    break (alias: b) ------------ Sets a breakpoint.
    trace (alias: t) ------------ Set tracepoint.
    restart (alias: r) ---------- Restart process.
    continue (alias: c) --------- Run until breakpoint or program termination.
    step (alias: s) ------------- Single step through program.
    step-instruction (alias: si)  Single step a single cpu instruction.
    next (alias: n) ------------- Step over to next source line.
    threads --------------------- Print out info for every traced thread.
    thread (alias: tr) ---------- Switch to the specified thread.

If you are familiar with gdb, you will find the commands are very similar, and I promise you can master delve soon.

An interesting thing is that the delve doesn’t provide start command which gdb offers, so you should try to set breakpoints first, then run continue command:

(dlv) b Hello.go:8
Breakpoint 1 set at 0x4011ea for main.main() ./Hello.go:8
(dlv) c
> main.main() ./Hello.go:8 (hits goroutine(1):1 total:1) (PC: 0x4011ea)
     3: import "fmt"
     5: func main() {
     6:         var s []byte
     7:         s = append(s, []byte("Hello, Debugging!")...)
=>   8:         fmt.Println(string(s))
     9: }
(dlv) p s
[]uint8 len: 17, cap: 32, [72,101,108,108,111,44,32,68,101,98,117,103,103,105,110,103,33]
(dlv) goroutines
[4 goroutines]
* Goroutine 1 - User: ./Hello.go:8 main.main (0x4011ea)
  Goroutine 2 - User: /usr/local/go/src/runtime/proc.go:263 runtime.gopark (0x42a153)
  Goroutine 3 - User: /usr/local/go/src/runtime/proc.go:263 runtime.gopark (0x42a153)
  Goroutine 4 - User: /usr/local/go/src/runtime/mfinal.go:144 runtime.runfinq (0x413f80)

Cool! Isn’t it? Now You can observe almost everything you want to know about your program.

Happy Debugging! Happy delving!


The tips of debugging Mesos

In the past week, I was following this tutorial to build a “kubernetes on Mesos” testbed. All went well but the Mesos master always complains following words:

E1228 21:57:13.138357 27257 process.cpp:1911] Failed to shutdown socket with fd 17: Transport endpoint is not connected

Firstly, I tried to get help from Mesos mailing list and stackoverflow, but after other friends can’t give correct answers directly, I knew I must depend on myself. Enduring a tough debugging process, I worked out the root cause. Since I am a newbie of Mesos and C++(Mesos is implemented in C++, and I last time touch C++ was 7 years ago), I think the experiences and tips may also be useful for other novices. So I summarize them as the following words:


When you meet an issue, analyzing log should be the first step. Mesos utilizes the google-glog to generate the logs. And the log format explanation is here:

Log lines have this form:
    Lmmdd hh:mm:ss.uuuuuu threadid file:line] msg...
where the fields are defined as follows:
    L                A single character, representing the log level (eg 'I' for INFO)
    mm               The month (zero padded; ie May is '05')
    dd               The day (zero padded)
    hh:mm:ss.uuuuuu  Time in hours, minutes and fractional seconds
    threadid         The space-padded thread ID as returned by GetTID()
    file             The file name
    line             The line number
    msg              The user-supplied message

So compared with the above words, you can easily understand this log:

E1228 21:57:13.138357 27257 process.cpp:1911] Failed to shutdown socket with fd 17: Transport endpoint is not connected

By default, the Mesos doesn’t output logs generated by VLOG function, and you need to set GLOG_v=m if you want to see the information from VLOG function (Refer this post):

$ sudo GLOG_v=3 ./bin/ --ip= --work_dir=/var/lib/mesos
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1229 22:42:38.818521 11830 process.cpp:2426] Spawned process __gc__@
I1229 22:42:38.818613 11846 process.cpp:2436] Resuming __gc__@ at 2015-12-30 03:42:38.818540032+00:00
I1229 22:42:38.818749 11847 process.cpp:2436] Resuming __gc__@ at 2015-12-30 03:42:38.818712832+00:00
I1229 22:42:38.818802 11844 process.cpp:2436] Resuming help@ at 2015-12-30 03:42:38.818746112+00:00

You can also use LOG function to add logs yourself on suspected locations.

(2) Gdb

If logs can’t save you, it is time for debugger to be your hero. Gdb is no doubt a great tool of debugging C/C++ programs. To use gdb, you should enable --enable-debug configuration option before compiling Mesos:

nan@ubuntu:~/mesos-0.25.0/build$ ../configure --enable-debug

You can set breakpoint on class member function like this:

(gdb) b process::SocketManager::close(int)
Breakpoint 1 at 0x7fd07857c162: file ../../../3rdparty/libprocess/src/process.cpp, line 1849.

You can also make use of “auto-complete” feature of gdb. Input the uncompleted function name:

(gdb) b process::SocketManager::cl  

Then click tab:

(gdb) b process::SocketManager::close(int)
Breakpoint 1 at 0x7fd07857c162: file ../../../3rdparty/libprocess/src/process.cpp, line 1849.

Notice: If the matched symbols are too many, it may hang gdb. So try to reduce the scope as small as possible.

Additionally, since source file names are relative to the directory where the code was compiled (please refer breakpoints in GDB),you can reach the same effect through “b file:line“command :

(gdb) b ../../../3rdparty/libprocess/src/process.cpp:1279
Breakpoint 2 at 0x7fd07857973a: file ../../../3rdparty/libprocess/src/process.cpp, line 1279.
(gdb) c
[Switching to Thread 0x7fd06b9d4700 (LWP 16677)]

Breakpoint 2, process::SocketManager::link_connect (this=0xca1a30, future=..., socket=0x7fd0500026d0, to=...)
    at ../../../3rdparty/libprocess/src/process.cpp:1279
1279      if (future.isDiscarded() || future.isFailed()) {

You can see the breakpoint is set onprocess::SocketManager::link_connect(process::Future<Nothing> const&, process::network::Socket*, process::UPID const&)function.

P.S.:There are also handy out-of-box gdb scripts in build/bin directory:

# ls bin/gdb-mesos-*
bin/  bin/  bin/  bin/

(3) Tcpdump and wireshark

Network packet analyzing tools such as tcpdump and wireshark are essential to diagnose programs which interact with other hosts. E.g., you can use following command to see what come in and out of Mesos master:

sudo tcpdump -A -s 0 'tcp port 5050' -i em1 -w capture.pcap

BTW, my issue is finally fixed by analyzing the following packet:


(4) Pstack script

Personally, I think pstack script is useful when monitoring thread status, and please refer Use pstack to track threads on Linux for detail.

Enjoy debugging!


The tips of using gdb to debug Golang program

Although “GDB does not understand Go programs well.” (from Debugging Go Code with GDB), sometimes gdb is still a useful tool for debugging Golang program. In this post, I will show some small tips.


My OS is Ubuntu 14.04. Launching gdb, it prompts following things:

warning: File "/usr/local/go/src/runtime/" auto-loading has been declined by your `auto-load safe-path' set to "$debugdir:$datadir/auto-load".
To enable execution of this file add
        add-auto-load-safe-path /usr/local/go/src/runtime/
line to your configuration file "/home/nan/.gdbinit".
To completely disable this security protection add
        set auto-load safe-path /
line to your configuration file "/home/nan/.gdbinit".
For more information about this security protection see the
"Auto-loading safe path" section in the GDB manual.  E.g., run from the shell:
        info "(gdb)Auto-loading safe path"

So I add “add-auto-load-safe-path /usr/local/go/src/runtime/” in my .gdbinit file. My preferred .gdbinit file is like this:

$ cat ${HOME}/.gdbinit
add-auto-load-safe-path /usr/local/go/src/runtime/
set confirm off
set print pretty on


If you want to set breakpoint on the start of main function, you should use “b main.main“:

(gdb) b main.main
Breakpoint 1 at 0x4021e0: file /home/nan/kubernetes/_output/local/go/src/, line 26.
(gdb) r
Starting program: /home/nan/kubernetes/_output/local/go/bin/kubectl
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/".
[New Thread 0x7ffff77f6700 (LWP 957)]
[New Thread 0x7ffff6fb5700 (LWP 958)]
[New Thread 0x7ffff5fb3700 (LWP 960)]
[New Thread 0x7ffff67b4700 (LWP 959)]

Breakpoint 1, main.main () at /home/nan/kubernetes/_output/local/go/src/
26      func main() {

Not “b main“, except you want to read mysterious assembly code:

(gdb) b main
Breakpoint 1 at 0x45e830: file /usr/local/go/src/runtime/rt0_linux_amd64.s, line 63.
(gdb) r
Starting program: /home/nan/kubernetes/_output/local/go/bin/kubectl
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/".

Breakpoint 1, main () at /usr/local/go/src/runtime/rt0_linux_amd64.s:63
63              MOVQ    $runtime·rt0_go(SB), AX


If you don’t know the symbol name of a function, you can use readelf tool. E.g.:

$ readelf -s -W /home/nan/kubernetes/_output/local/go/bin/kubectl | grep NewCmdConfig
 14350: 0000000000766da0  3168 FUNC    LOCAL  DEFAULT   14
 14404: 00000000007745d0    48 FUNC    LOCAL  DEFAULT   14

The config.NewCmdConfig.func1 is the function which is defined in NewCmdConfig function:

func NewCmdConfig(pathOptions *PathOptions, out io.Writer) *cobra.Command {
        Run: func(cmd *cobra.Command, args []string) {

Then you can set breakpoint for wanted function:

(gdb) b
Breakpoint 1 at 0x766da0: file /home/nan/kubernetes/_output/local/go/src/, line 63.
(gdb) r
Starting program: /home/nan/kubernetes/_output/local/go/bin/kubectl
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/".
[New Thread 0x7ffff77f6700 (LWP 1416)]
[New Thread 0x7ffff6fb5700 (LWP 1417)]
[New Thread 0x7ffff67b4700 (LWP 1418)]
[New Thread 0x7ffff5fb3700 (LWP 1419)]

Breakpoint 1, (pathOptions=0xc820178b90, out=..., ~r2=0x1)
    at /home/nan/kubernetes/_output/local/go/src/
63      func NewCmdConfig(pathOptions *PathOptions, out io.Writer) *cobra.Command {


Page 1 of 2

Powered by WordPress & Theme by Anders Norén