Tag Archives: python

Create a symbol link for python after installing it on OpenBSD

Published / by nanxiao / Leave a Comment

After installing Python on FreeBSD:

# pkg_add python
quirks-2.304 signed on 2017-04-02T15:01:33Z
Ambiguous: choose package for python
a       0: <None>
        1: python-2.7.13p0
        2: python-3.4.5p2
        3: python-3.5.2p2
        4: python-3.6.0p0
Your choice: 4

It won’t create a symbol link by default:

# python
ksh: python: not found

So for using it handily, you can create a symbol link yourself:

# cd /usr/local/bin
# ln -s python3.6 python

Use pdb to help understand python program

Published / by nanxiao / 1 Comment on Use pdb to help understand python program

As I have mentioned in Why do I need a debugger?:

(3) Debugger is a good tool to help you understand code.

So when I come across difficulty to understand vfscount.py code in bcc project, I know it is time to resort to pdb, python‘s debugger, to help me.

The thing which confuses me is here:

counts = b.get_table("counts")
for k, v in sorted(counts.items(), key=lambda counts: counts[1].value):
    print("%-16x %-26s %8d" % (k.ip, b.ksym(k.ip), v.value))

From previous code:

BPF_HASH(counts, struct key_t, u64, 256);

It seems the v‘s type is u64, and I can’t figure out why use v.value to fetch its data here.

pdb‘s manual is very succinct and its command is so similar with gdb‘s, so it is no learning curve for me. Just launch a debugging session and set breakpoint at “counts = b.get_table("counts")” line:

# python -m pdb vfscount.py
> /root/Project/bcc/tools/vfscount.py(14)<module>()
-> from __future__ import print_function
(Pdb) b vfscount.py:49

Start the program and press Ctrl-C after seconds; the breakpoint will be hit:

(Pdb) r
Tracing... Ctrl-C to end.
^C
ADDR             FUNC                          COUNT
> /root/Project/bcc/tools/vfscount.py(49)<module>()
-> counts = b.get_table("counts")

Step into get_table method, and single-step every line. Before leaving method, check the type of keytype and leaftype:

-> counts = b.get_table("counts")
(Pdb) s
--Call--
> /usr/lib/python3.6/site-packages/bcc/__init__.py(416)get_table()
-> def get_table(self, name, keytype=None, leaftype=None, reducer=None):
(Pdb) n
> /usr/lib/python3.6/site-packages/bcc/__init__.py(417)get_table()
-> map_id = lib.bpf_table_id(self.module, name.encode("ascii"))
......
(Pdb) p leaf_desc
b'"unsigned long long"'
(Pdb) n
> /usr/lib/python3.6/site-packages/bcc/__init__.py(430)get_table()
-> leaftype = BPF._decode_table_type(json.loads(leaf_desc.decode()))
(Pdb)
> /usr/lib/python3.6/site-packages/bcc/__init__.py(431)get_table()
-> return Table(self, map_id, map_fd, keytype, leaftype, reducer=reducer)
(Pdb) p leaftype
<class 'ctypes.c_ulong'>
(Pdb) p keytype
<class 'bcc.key_t'>

Yeah! The magic is here: leaftype‘s type is not pure u64, but ctypes.c_ulong. According to document:

>>> print(i.value)
42

We should use v.value to get its internal data.

Happy pdbing! Happy python debugging!

The anatomy of “Hello World” python program in bcc

Published / by nanxiao / 2 Comments on The anatomy of “Hello World” python program in bcc

The bcc gives a simple Hello World program analysis, but I want to give a more detailed anatomy of it:

from bcc import BPF

BPF(text='int kprobe__sys_clone(void *ctx) { bpf_trace_printk("Hello, World!\\n"); return 0; }').trace_print()

(1)

from bcc import BPF  

bcc is a package, and it will be installed in Pyhthon‘s site-packages directory. Take my server as an example:

>>> import bcc
>>> bcc.__path__
['/usr/lib/python3.6/site-packages/bcc']

BPF is a class defined in __init__.py file of bcc:

......
class BPF(object):
# From bpf_prog_type in uapi/linux/bpf.h
    SOCKET_FILTER = 1
    KPROBE = 2
......

(2) Divide

BPF(text='int kprobe__sys_clone(void *ctx) { bpf_trace_printk("Hello, World!\\n"); return 0; }').trace_print()

into 2 parts:

BPF(text='int kprobe__sys_clone(void *ctx) { bpf_trace_printk("Hello, World!\\n"); return 0; }')

and

trace_print()

a)

BPF(text='int kprobe__sys_clone(void *ctx) { bpf_trace_printk("Hello, World!\\n"); return 0; }')

This statement will create a BPF object, and the code is simplified as following:

......
from .libbcc import lib, _CB_TYPE, bcc_symbol, _SYM_CB_TYPE 
......
class BPF(object):
......
    def __init__(self, src_file="", hdr_file="", text=None, cb=None, debug=0,
        cflags=[], usdt_contexts=[]):
        ......

        if text:
            self.module = lib.bpf_module_create_c_from_string(text.encode("ascii"),
            self.debug, cflags_array, len(cflags_array))
        else:
            ......

        self._trace_autoload()

Check libbcc.py:

lib = ct.CDLL("libbcc.so.0", use_errno=True)

We can see lib module is indeed libbcc dynamic library, and bpf_module_create_c_from_string is implemented as this:

void * bpf_module_create_c_from_string(const char *text, unsigned flags, const char *cflags[], int ncflags) {
    auto mod = new ebpf::BPFModule(flags);
    if (mod->load_string(text, cflags, ncflags) != 0) {
        delete mod;
        return nullptr;
    }
    return mod;
}

We won’t delve it too much, and just know it returns a ebpf module is enough.

For self._trace_autoload() part, we can check _trace_autoload method definition:

def _trace_autoload(self):
    for i in range(0, lib.bpf_num_functions(self.module)):
        func_name = str(lib.bpf_function_name(self.module, i).decode())
        if func_name.startswith("kprobe__"):
            fn = self.load_func(func_name, BPF.KPROBE)
            self.attach_kprobe(event=fn.name[8:], fn_name=fn.name)
        elif func_name.startswith("kretprobe__"):
            fn = self.load_func(func_name, BPF.KPROBE)
            self.attach_kretprobe(event=fn.name[11:], fn_name=fn.name)
        elif func_name.startswith("tracepoint__"):
            fn = self.load_func(func_name, BPF.TRACEPOINT)
            tp = fn.name[len("tracepoint__"):].replace("__", ":")
            self.attach_tracepoint(tp=tp, fn_name=fn.name)

Based on above code, we can see bcc now supports kprobe, kretprobe and tracepoint. For Hello World example, it actually execute attach_kprobemethod:

def attach_kprobe(self, event="", fn_name="", event_re="",
        pid=-1, cpu=0, group_fd=-1):

    ......

    event = str(event)
    self._check_probe_quota(1)
    fn = self.load_func(fn_name, BPF.KPROBE)
    ev_name = "p_" + event.replace("+", "_").replace(".", "_")
    res = lib.bpf_attach_kprobe(fn.fd, 0, ev_name.encode("ascii"),
            event.encode("ascii"), pid, cpu, group_fd,
            self._reader_cb_impl, ct.cast(id(self), ct.py_object))
    res = ct.cast(res, ct.c_void_p)
    ......
    self._add_kprobe(ev_name, res)
    return self

Finally, it will call libbcc‘s bpf_attach_kprobe function. Until now, the kprobe is attached successfully, and attach_kprobe will return a BPF object.

(b)

BPF(...).trace_print()

The trace_print method is not too hard:

def trace_print(self, fmt=None):
    ......
    try:
        while True:
            if fmt:
                fields = self.trace_fields(nonblocking=False)
                if not fields: continue
                line = fmt.format(*fields)
            else:
                line = self.trace_readline(nonblocking=False)
            print(line)
            sys.stdout.flush()
    except KeyboardInterrupt:
        exit()

For fmt=None case, it just calls trace_readline(), which read from trace pipe, and prints it on the stdout:

print(line)
sys.stdout.flush()

That’s all, and hope you enjoy bcc and ebpf!

Generate carry in Carry-lookahead adder

Published / by nanxiao / Leave a Comment

Recently, I am implementing an 8-bit Carry-lookahead adder. But unfortunately, almost all materials in the Internet only provide the formula to get C1 toC4, so I write a simple python script to generate all carries:

#!/usr/bin/python3

import sys
num = 8

if len(sys.argv) != 1:
    num = int(sys.argv[1])

dict = {'C0': 'C0'}
for i in range(1, num + 1):
    dict['C' + str(i)] = 'G' + str(i - 1) + \
                        '+' + \
                        '+'.join([x + "*P" + str(i - 1) for x in dict['C' + str(i - 1)].split('+')])

for i in range(0, num + 1):
    key = 'C' + str(i)
    print(key, "=", dict[key])

By default, it will generate C1 to C8:

$ ./CarryFormulaInLookAheadAdder.py
C0 = C0
C1 = G0+C0*P0
C2 = G1+G0*P1+C0*P0*P1
C3 = G2+G1*P2+G0*P1*P2+C0*P0*P1*P2
C4 = G3+G2*P3+G1*P2*P3+G0*P1*P2*P3+C0*P0*P1*P2*P3
C5 = G4+G3*P4+G2*P3*P4+G1*P2*P3*P4+G0*P1*P2*P3*P4+C0*P0*P1*P2*P3*P4
C6 = G5+G4*P5+G3*P4*P5+G2*P3*P4*P5+G1*P2*P3*P4*P5+G0*P1*P2*P3*P4*P5+C0*P0*P1*P2*P3*P4*P5
C7 = G6+G5*P6+G4*P5*P6+G3*P4*P5*P6+G2*P3*P4*P5*P6+G1*P2*P3*P4*P5*P6+G0*P1*P2*P3*P4*P5*P6+C0*P0*P1*P2*P3*P4*P5*P6
C8 = G7+G6*P7+G5*P6*P7+G4*P5*P6*P7+G3*P4*P5*P6*P7+G2*P3*P4*P5*P6*P7+G1*P2*P3*P4*P5*P6*P7+G0*P1*P2*P3*P4*P5*P6*P7+C0*P0*P1*P2*P3*P4*P5*P6*P7

But you can pass a parameter to tell how many carries you want:

$ ./CarryFormulaInLookAheadAdder.py 6
C0 = C0
C1 = G0+C0*P0
C2 = G1+G0*P1+C0*P0*P1
C3 = G2+G1*P2+G0*P1*P2+C0*P0*P1*P2
C4 = G3+G2*P3+G1*P2*P3+G0*P1*P2*P3+C0*P0*P1*P2*P3
C5 = G4+G3*P4+G2*P3*P4+G1*P2*P3*P4+G0*P1*P2*P3*P4+C0*P0*P1*P2*P3*P4
C6 = G5+G4*P5+G3*P4*P5+G2*P3*P4*P5+G1*P2*P3*P4*P5+G0*P1*P2*P3*P4*P5+C0*P0*P1*P2*P3*P4*P5

The project is here.