The tips of using gdb to debug Golang program

Although “GDB does not understand Go programs well.” (from Debugging Go Code with GDB), sometimes gdb is still a useful tool for debugging Golang program. In this post, I will show some small tips.

(1)

My OS is Ubuntu 14.04. Launching gdb, it prompts following things:

......
warning: File "/usr/local/go/src/runtime/runtime-gdb.py" auto-loading has been declined by your `auto-load safe-path' set to "$debugdir:$datadir/auto-load".
To enable execution of this file add
        add-auto-load-safe-path /usr/local/go/src/runtime/runtime-gdb.py
line to your configuration file "/home/nan/.gdbinit".
To completely disable this security protection add
        set auto-load safe-path /
line to your configuration file "/home/nan/.gdbinit".
For more information about this security protection see the
"Auto-loading safe path" section in the GDB manual.  E.g., run from the shell:
        info "(gdb)Auto-loading safe path"
......

So I add “add-auto-load-safe-path /usr/local/go/src/runtime/runtime-gdb.py” in my .gdbinit file. My preferred .gdbinit file is like this:

$ cat ${HOME}/.gdbinit
add-auto-load-safe-path /usr/local/go/src/runtime/runtime-gdb.py
set confirm off
set print pretty on

(2)

If you want to set breakpoint on the start of main function, you should use “b main.main“:

......
(gdb) b main.main
Breakpoint 1 at 0x4021e0: file /home/nan/kubernetes/_output/local/go/src/k8s.io/kubernetes/cmd/kubectl/kubectl.go, line 26.
(gdb) r
Starting program: /home/nan/kubernetes/_output/local/go/bin/kubectl
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[New Thread 0x7ffff77f6700 (LWP 957)]
[New Thread 0x7ffff6fb5700 (LWP 958)]
[New Thread 0x7ffff5fb3700 (LWP 960)]
[New Thread 0x7ffff67b4700 (LWP 959)]

Breakpoint 1, main.main () at /home/nan/kubernetes/_output/local/go/src/k8s.io/kubernetes/cmd/kubectl/kubectl.go:26
26      func main() {
......

Not “b main“, except you want to read mysterious assembly code:

......
(gdb) b main
Breakpoint 1 at 0x45e830: file /usr/local/go/src/runtime/rt0_linux_amd64.s, line 63.
(gdb) r
Starting program: /home/nan/kubernetes/_output/local/go/bin/kubectl
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".

Breakpoint 1, main () at /usr/local/go/src/runtime/rt0_linux_amd64.s:63
63              MOVQ    $runtime·rt0_go(SB), AX
(gdb)
......

(3)

If you don’t know the symbol name of a function, you can use readelf tool. E.g.:

$ readelf -s -W /home/nan/kubernetes/_output/local/go/bin/kubectl | grep NewCmdConfig
 ......
 14350: 0000000000766da0  3168 FUNC    LOCAL  DEFAULT   14 k8s.io/kubernetes/pkg/kubectl/cmd/config.NewCmdConfig
 ......
 14404: 00000000007745d0    48 FUNC    LOCAL  DEFAULT   14 k8s.io/kubernetes/pkg/kubectl/cmd/config.NewCmdConfig.func1

The config.NewCmdConfig.func1 is the function which is defined in NewCmdConfig function:

func NewCmdConfig(pathOptions *PathOptions, out io.Writer) *cobra.Command {
    ......
        Run: func(cmd *cobra.Command, args []string) {
            cmd.Help()
    ......
}

Then you can set breakpoint for wanted function:

......
(gdb) b k8s.io/kubernetes/pkg/kubectl/cmd/config.NewCmdConfig
Breakpoint 1 at 0x766da0: file /home/nan/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/kubectl/cmd/config/config.go, line 63.
(gdb) r
Starting program: /home/nan/kubernetes/_output/local/go/bin/kubectl
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[New Thread 0x7ffff77f6700 (LWP 1416)]
[New Thread 0x7ffff6fb5700 (LWP 1417)]
[New Thread 0x7ffff67b4700 (LWP 1418)]
[New Thread 0x7ffff5fb3700 (LWP 1419)]

Breakpoint 1, k8s.io/kubernetes/pkg/kubectl/cmd/config.NewCmdConfig (pathOptions=0xc820178b90, out=..., ~r2=0x1)
    at /home/nan/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/kubectl/cmd/config/config.go:63
63      func NewCmdConfig(pathOptions *PathOptions, out io.Writer) *cobra.Command {
(gdb)
......

 

Execute “mesos-slave.sh” using root privilege

I have spent the past week to try building “Kubernetes on Mesos” following Getting started with Kubernetes on Mesos document, but “kubectl get node” always reports the node state is not ready:

$ kubectl get node
NAME      LABELS                          STATUS     AGE
ubuntu    kubernetes.io/hostname=ubuntu   NotReady   4h

Today, when executing “./bin/mesos-slave.sh --master=127.0.0.1:5050” command, I notice this log:

W1217 05:52:42.213497 24074 backend.cpp:50] Failed to create 'bind' backend: BindBackend requires root privileges

So I use sudo command to run the ./bin/mesos-slave.sh script again, then the “Kubernetes on Mesos” runs successfully:

 

Capture

Use Intellij IDEA as a Golang IDE

My Intellij IDEA is v15.0.1 community edition, and has been installed with golang-plugin. My Golang workspace is like this:

GOPATH=C:\Work\gocode

(1) Select “Create New Project“:

1

(2) Select “Go“, then “Next“:

2

(3) Select “SDK“, then “Next“:

3

(4) The important step coming! The up setting is for storing IDEA project files, while the down is for Golang workspace, then “Finish“:

4

(5) Now, use Intellij IDEA as a Golang IDE:

5
Enjoy it!

Set proxy for Maven

Sometimes, your Maven needs proxy to download object:

[INFO] Scanning for projects...
Downloading: http://repo.maven.apache.org/maven2/org/apache/apache/11/apache-11.pom
[ERROR] The build could not read 1 project -> [Help 1]
[ERROR]
[ERROR]   The project org.apache.mesos:mesos:0.25.0 (/home/nan/mesos-0.25.0/build/src/java/mesos.pom) has 1 error
[ERROR]     Non-resolvable parent POM: Could not transfer artifact org.apache:apache:pom:11 from/to central (http://repo.maven.apache.org/maven2): Connection to http://repo.maven.apache.org refused and 'parent.relativePath' points at wrong local POM @ line 18, column 11: Connection timed out -> [Help 2]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/ProjectBuildingException
[ERROR] [Help 2] http://cwiki.apache.org/confluence/display/MAVEN/UnresolvableModelException

The method is configuring proxy setting in ${HOME}/.m2/settings.xml:

$ echo ${HOME}
/home/nan
$ cat ${HOME}/.m2/settings.xml
<settings>
        <proxies>
                <proxy>
                        <active>true</active>
                        <protocol>http</protocol>
                        <host>web-proxy.xxxxxx.com</host>
                        <port>8080</port>
                </proxy>
        </proxies>
</settings>

Then it can work!

P.S. My Maven version is 3.0.5.

 

SystemTap needs debug-info package when user-space probing

SystemTap needs debug-info package when user-space probing.E.g.:

 # stap -d /bin/ls --ldd -e 'probe process("ls").function("xmalloc") {print_usyms(ubacktrace())}' -c "ls /"
semantic error: while resolving probe point: identifier 'process' at <input>:1:7
        source: probe process("ls").function("xmalloc") {print_usyms(ubacktrace())}
                      ^

semantic error: no match (similar functions: malloc, calloc, realloc, close, mbrtowc)
Pass 2: analysis failed.  [man error::pass2]

Install coreutils-debuginfo package:

# zypper in coreutils-debuginfo
Loading repository data...
Reading installed packages...
Resolving package dependencies...

The following NEW package is going to be installed:
  coreutils-debuginfo

The following package is not supported by its vendor:
  coreutils-debuginfo

1 new package to install.
Overall download size: 2.1 MiB. Already cached: 0 B. After the operation, additional 18.6 MiB will be used.
Continue? [y/n/? shows all options] (y): y
Retrieving package coreutils-debuginfo-8.22-9.1.x86_64                                                (1/1),   2.1 MiB ( 18.6 MiB unpacked)
Retrieving: coreutils-debuginfo-8.22-9.1.x86_64.rpm ...................................................................[done (105.3 KiB/s)]
Checking for file conflicts: ........................................................................................................[done]
(1/1) Installing: coreutils-debuginfo-8.22-9.1 ......................................................................................[done]

Then execute the stap command again:

# stap -d /bin/ls --ldd -e 'probe process("ls").function("xmalloc") {print_usyms(ubacktrace())}' -c "ls /"
bin  boot  dev  etc  home  lib  lib64  lost+found  mnt  opt  proc  root  run  sbin  selinux  srv  sys  tmp  usr  var
 0x4114a0 : xmalloc+0x0/0x20 [/usr/bin/ls]
 0x411674 : xmemdup+0x14/0x30 [/usr/bin/ls]
 0x40ee4a : clone_quoting_options+0x2a/0x40 [/usr/bin/ls]
 0x403828 : main+0xa58/0x2140 [/usr/bin/ls]
 0x7fad37eefb05 : __libc_start_main+0xf5/0x1c0 [/lib64/libc-2.19.so]
 0x404f39 : _start+0x29/0x30 [/usr/bin/ls]
 0x4114a0 : xmalloc+0x0/0x20 [/usr/bin/ls]
 0x411674 : xmemdup+0x14/0x30 [/usr/bin/ls]
 0x40ee4a : clone_quoting_options+0x2a/0x40 [/usr/bin/ls]
 0x403887 : main+0xab7/0x2140 [/usr/bin/ls]
 0x7fad37eefb05 : __libc_start_main+0xf5/0x1c0 [/lib64/libc-2.19.so]
 0x404f39 : _start+0x29/0x30 [/usr/bin/ls]
 0x4114a0 : xmalloc+0x0/0x20 [/usr/bin/ls]
 0x4039e4 : main+0xc14/0x2140 [/usr/bin/ls]
 0x7fad37eefb05 : __libc_start_main+0xf5/0x1c0 [/lib64/libc-2.19.so]
.....

It works now!