Nan Xiao's Blog

A system software / performance engineer's home

Tag: ebpf

Use pdb to help understand python program

As I have mentioned in Why do I need a debugger?:

(3) Debugger is a good tool to help you understand code.

So when I come across difficulty to understand code in bcc project, I know it is time to resort to pdb, python‘s debugger, to help me.

The thing which confuses me is here:

counts = b.get_table("counts")
for k, v in sorted(counts.items(), key=lambda counts: counts[1].value):
    print("%-16x %-26s %8d" % (k.ip, b.ksym(k.ip), v.value))

From previous code:

BPF_HASH(counts, struct key_t, u64, 256);

It seems the v‘s type is u64, and I can’t figure out why use v.value to fetch its data here.

pdb‘s manual is very succinct and its command is so similar with gdb‘s, so it is no learning curve for me. Just launch a debugging session and set breakpoint at “counts = b.get_table("counts")” line:

# python -m pdb
> /root/Project/bcc/tools/<module>()
-> from __future__ import print_function
(Pdb) b

Start the program and press Ctrl-C after seconds; the breakpoint will be hit:

(Pdb) r
Tracing... Ctrl-C to end.
ADDR             FUNC                          COUNT
> /root/Project/bcc/tools/<module>()
-> counts = b.get_table("counts")

Step into get_table method, and single-step every line. Before leaving method, check the type of keytype and leaftype:

-> counts = b.get_table("counts")
(Pdb) s
> /usr/lib/python3.6/site-packages/bcc/
-> def get_table(self, name, keytype=None, leaftype=None, reducer=None):
(Pdb) n
> /usr/lib/python3.6/site-packages/bcc/
-> map_id = lib.bpf_table_id(self.module, name.encode("ascii"))
(Pdb) p leaf_desc
b'"unsigned long long"'
(Pdb) n
> /usr/lib/python3.6/site-packages/bcc/
-> leaftype = BPF._decode_table_type(json.loads(leaf_desc.decode()))
> /usr/lib/python3.6/site-packages/bcc/
-> return Table(self, map_id, map_fd, keytype, leaftype, reducer=reducer)
(Pdb) p leaftype
<class 'ctypes.c_ulong'>
(Pdb) p keytype
<class 'bcc.key_t'>

Yeah! The magic is here: leaftype‘s type is not pure u64, but ctypes.c_ulong. According to document:

>>> print(i.value)

We should use v.value to get its internal data.

Happy pdbing! Happy python debugging!

Parse BPF_ARRAY macro in bcc

BPF_ARRAY is a very common macro used in bcc scripts. To analyze it, I put all BPF_ARRAY related macros in an example file:

// Changes to the macro require changes in BFrontendAction classes
#define BPF_F_TABLE(_table_type, _key_type, _leaf_type, _name, _max_entries, _flags) \
struct _name##_table_t { \
  _key_type key; \
  _leaf_type leaf; \
  _leaf_type * (*lookup) (_key_type *); \
  _leaf_type * (*lookup_or_init) (_key_type *, _leaf_type *); \
  int (*update) (_key_type *, _leaf_type *); \
  int (*insert) (_key_type *, _leaf_type *); \
  int (*delete) (_key_type *); \
  void (*call) (void *, int index); \
  void (*increment) (_key_type); \
  int (*get_stackid) (void *, u64); \
  _leaf_type data[_max_entries]; \
  int flags; \
}; \
__attribute__((section("maps/" _table_type))) \
struct _name##_table_t _name = { .flags = (_flags) }

#define BPF_TABLE(_table_type, _key_type, _leaf_type, _name, _max_entries) \
BPF_F_TABLE(_table_type, _key_type, _leaf_type, _name, _max_entries, 0);

#define BPF_ARRAY1(_name) \
  BPF_TABLE("array", int, u64, _name, 10240)
#define BPF_ARRAY2(_name, _leaf_type) \
  BPF_TABLE("array", int, _leaf_type, _name, 10240)
#define BPF_ARRAY3(_name, _leaf_type, _size) \
  BPF_TABLE("array", int, _leaf_type, _name, _size)

// helper for default-variable macro function
#define BPF_ARRAYX(_1, _2, _3, NAME, ...) NAME

// Define an array function, some arguments optional
// BPF_ARRAY(name, leaf_type=u64, size=10240)
#define BPF_ARRAY(...) \

enum stat_types {
  S_COUNT = 1,

void main(void)
  BPF_ARRAY(stats, u64, S_MAXSTAT + 1); 

Use gcc -E to preprocess it (I have formatted code to make it clear):

# 1 "test.c"
# 1 "<built-in>"
# 1 "<command-line>"
# 31 "<command-line>"
# 1 "/usr/include/stdc-predef.h" 1 3 4
# 32 "<command-line>" 2
# 1 "test.c"
# 38 "test.c"
enum stat_types {
    S_COUNT = 1,

void main(void)
    struct stats_table_t { 
      int key; 
      u64 leaf; 
      u64 * (*lookup) (int *); 
      u64 * (*lookup_or_init) (int *, u64 *); 
      int (*update) (int *, u64 *); 
      int (*insert) (int *, u64 *); 
      int (*delete) (int *);
      void (*call) (void *, int index); 
      void (*increment) (int); 
      int (*get_stackid) (void *, u64); 
      u64 data[S_MAXSTAT + 1]; 
      int flags; 
    __attribute__((section("maps/" "array"))) struct stats_table_t stats = { .flags = (0) };;

Let’s analyze how the final result is got:


BPF_ARRAY(stats, u64, S_MAXSTAT + 1)    

will be expanded to (__VA_ARGS__ will be replaced by all arguments):

BPF_ARRAYX(stats, u64, S_MAXSTAT + 1, BPF_ARRAY3, BPF_ARRAY2, BPF_ARRAY1)(stats, u64, S_MAXSTAT + 1)

(2) According to:

// helper for default-variable macro function
#define BPF_ARRAYX(_1, _2, _3, NAME, ...) NAME

BPF_ARRAYX(stats, u64, S_MAXSTAT + 1, BPF_ARRAY3, BPF_ARRAY2, BPF_ARRAY1)(stats, u64, S_MAXSTAT + 1) will be replaced by following code:

BPF_ARRAY3(stats, u64, S_MAXSTAT + 1)

We can compared it to BPF_ARRAY1 (both _leaf_type and size have default values) and BPF_ARRAY2 (only size have default value.).

(3) Convert BPF_ARRAY3 to final BPF_F_TABLE is more straightforward, so I won’t drill that down.

Use the same method you can analyze other macros, such as BPF_HASH. Hope this small post can give you a tip!

The anatomy of “Hello World” python program in bcc

The bcc gives a simple Hello World program analysis, but I want to give a more detailed anatomy of it:

from bcc import BPF

BPF(text='int kprobe__sys_clone(void *ctx) { bpf_trace_printk("Hello, World!\\n"); return 0; }').trace_print()


from bcc import BPF  

bcc is a package, and it will be installed in Pyhthon‘s site-packages directory. Take my server as an example:

>>> import bcc
>>> bcc.__path__

BPF is a class defined in file of bcc:

class BPF(object):
# From bpf_prog_type in uapi/linux/bpf.h
    KPROBE = 2

(2) Divide

BPF(text='int kprobe__sys_clone(void *ctx) { bpf_trace_printk("Hello, World!\\n"); return 0; }').trace_print()

into 2 parts:

BPF(text='int kprobe__sys_clone(void *ctx) { bpf_trace_printk("Hello, World!\\n"); return 0; }')




BPF(text='int kprobe__sys_clone(void *ctx) { bpf_trace_printk("Hello, World!\\n"); return 0; }')

This statement will create a BPF object, and the code is simplified as following:

from .libbcc import lib, _CB_TYPE, bcc_symbol, _SYM_CB_TYPE 
class BPF(object):
    def __init__(self, src_file="", hdr_file="", text=None, cb=None, debug=0,
        cflags=[], usdt_contexts=[]):

        if text:
            self.module = lib.bpf_module_create_c_from_string(text.encode("ascii"),
            self.debug, cflags_array, len(cflags_array))



lib = ct.CDLL("", use_errno=True)

We can see lib module is indeed libbcc dynamic library, and bpf_module_create_c_from_string is implemented as this:

void * bpf_module_create_c_from_string(const char *text, unsigned flags, const char *cflags[], int ncflags) {
    auto mod = new ebpf::BPFModule(flags);
    if (mod->load_string(text, cflags, ncflags) != 0) {
        delete mod;
        return nullptr;
    return mod;

We won’t delve it too much, and just know it returns a ebpf module is enough.

For self._trace_autoload() part, we can check _trace_autoload method definition:

def _trace_autoload(self):
    for i in range(0, lib.bpf_num_functions(self.module)):
        func_name = str(lib.bpf_function_name(self.module, i).decode())
        if func_name.startswith("kprobe__"):
            fn = self.load_func(func_name, BPF.KPROBE)
        elif func_name.startswith("kretprobe__"):
            fn = self.load_func(func_name, BPF.KPROBE)
        elif func_name.startswith("tracepoint__"):
            fn = self.load_func(func_name, BPF.TRACEPOINT)
            tp =[len("tracepoint__"):].replace("__", ":")

Based on above code, we can see bcc now supports kprobe, kretprobe and tracepoint. For Hello World example, it actually execute attach_kprobemethod:

def attach_kprobe(self, event="", fn_name="", event_re="",
        pid=-1, cpu=0, group_fd=-1):


    event = str(event)
    fn = self.load_func(fn_name, BPF.KPROBE)
    ev_name = "p_" + event.replace("+", "_").replace(".", "_")
    res = lib.bpf_attach_kprobe(fn.fd, 0, ev_name.encode("ascii"),
            event.encode("ascii"), pid, cpu, group_fd,
            self._reader_cb_impl, ct.cast(id(self), ct.py_object))
    res = ct.cast(res, ct.c_void_p)
    self._add_kprobe(ev_name, res)
    return self

Finally, it will call libbcc‘s bpf_attach_kprobe function. Until now, the kprobe is attached successfully, and attach_kprobe will return a BPF object.



The trace_print method is not too hard:

def trace_print(self, fmt=None):
        while True:
            if fmt:
                fields = self.trace_fields(nonblocking=False)
                if not fields: continue
                line = fmt.format(*fields)
                line = self.trace_readline(nonblocking=False)
    except KeyboardInterrupt:

For fmt=None case, it just calls trace_readline(), which read from trace pipe, and prints it on the stdout:


That’s all, and hope you enjoy bcc and ebpf!

Install bcc on ArchLinux

To install bcc on ArchLinux, firstly you need to setup yaourt from AUR:

$ git clone
$ cd yaourt
$ makepkg -si

Then execute yaourt bcc command:

# yaourt bcc
1 aur/bcc 0.3.0-1 [installed] (17) (2.51)
    BPF Compiler Collection - C library and examples
2 aur/bcc-git v0.1.8.r330.52cd371-1 (2) (0.06)
    BPF Compiler Collection - C library and examples
3 aur/bcc-tools 0.3.0-1 [installed] (17) (2.51)
    BPF Compiler Collection - Tools
4 aur/bcc-tools-git v0.1.8.r330.52cd371-1 (2) (0.06)
    BPF Compiler Collection - Tools

Select the order number of bcc, bcc-tools, python-bcc and python2-bcc, and install them.

Once finished, the bcc would be installed in the directory of /usr/share/bcc:

# ls
examples  man  tools

To facilitate your daily work, you can add man pages and tools in your .bashrc file:


You can also install from source code:

git clone
mkdir bcc/build; cd bcc/build
sudo make install

P.S. To run tools shipped in bcc, you need to install kernel header files:

sudo pacman -S linux-headers

Powered by WordPress & Theme by Anders Norén