A performance engineering team’s story

It is a story about a performance engineer team: most is genuine, and some is made-up.

Server performance related work requires a broad knowledge and experience: software, hardware, Operating System kernel, and so on. The team members’ background is also a rich variety: embedded system developer, PCB designer, QA, DevOps, compiler researcher, etc.

The team’s daily work is like this:

(1) Since company’s main products are servers, run SPEC benchmark programs and cooperate with other team to get the best result is the most important task. Since only your servers exceed other competitors, the customers are willing to pay money for them.

(2) It is no controversial that Linux has dominated the server market, so being proficient in Linux profiling and tracing tools (from perf,ftrace to BPF) is a compulsory class. Meanwhile, the company also has its own proprietary Unix which is still serving some critical business: bank, telecommunication, etc. The members also need to use some esoteric tools to help diagnosing issues on this proprietary Unix. At the same time, coding is another important task: develop company’s own profiling tools, and contribute to FOSS projects.

(3) Support customers and other teams. E.g., one customer wants to know the Oracle‘s performance on some server models; the other finds that Docker doesn’t run as expected. Another colleague comes to your desk: “When you are free, could you help to check how to optimize this program? Boss isn’t satisfied with it”.

(4) The team encourages sharing. During every weekly meeting, you can introduces a Unix command trick, a debugging skill, like this. Members can have 5% ~ 10% time to do hobby projects, but the prerequisite is work first. This benefit is not free lunch, you should report it in the meeting too.

So what is the point of this article? Nothing, just telling a story, that’s it.

First taste of building cilk program on Arch Linux

I bump into cilk that is much like OpenMP this week, and give it a try on Arch Linux:

(1) Follow the Quick start to build Tapir-Meta:

$ git clone --recursive https://github.com/wsmoses/Tapir-Meta.git
$ cd Tapir-Meta/
$ ./build.sh release
$ source ./setup-env.sh

Please note gcc is the default compiler on Arch Linux, and there is a compile error of using gcc. So I switch to clang:

$ CC=clang CXX=claang++ ./build.sh release

The setup-env.sh is simple, just make Tapir/LLVM compilers as the default ones:

$ which clang
$ which clang++

(2) Build libcilkrts (please refer this issue: “/usr/bin/ld: cannot find -lcilkrts” error).

(3) Write a simple program:

$ cat test.c
#include <unistd.h>
#include <cilk/cilk.h>

int main(void)
        cilk_spawn sleep(100);
        cilk_spawn sleep(100);
        cilk_spawn sleep(100);

        return 0;

Build it:

$ clang -L/home/xiaonan/cilkrts/build/install/lib -fcilkplus test.c
$ ldd a.out
    linux-vdso.so.1 (0x00007ffdccd32000)
    libcilkrts.so.5 => /home/xiaonan/cilkrts/build/install/lib/libcilkrts.so.5 (0x00007f1fe4d60000)
    libgcc_s.so.1 => /usr/lib/libgcc_s.so.1 (0x00007f1fe4b48000)
    libc.so.6 => /usr/lib/libc.so.6 (0x00007f1fe478c000)
    libdl.so.2 => /usr/lib/libdl.so.2 (0x00007f1fe4588000)
    libpthread.so.0 => /usr/lib/libpthread.so.0 (0x00007f1fe436a000)
    libstdc++.so.6 => /usr/lib/libstdc++.so.6 (0x00007f1fe3fe1000)
    libm.so.6 => /usr/lib/libm.so.6 (0x00007f1fe3c4c000)
    /lib64/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007f1fe4f7e000)

From ldd output, we can see it links the ibcilkrts library. Execute the program:

$ ./a.out &
[1] 25530
[[email protected] ~]$ ps -T 25530 | wc -l

We can see a.out spawns many threads.

Learn new technology through writing a tutorial about it

I like to get my feet wet on new technologies, but find if I don’t use it for some time, e.g., several months, I will forget a lot of details, not sure whether other people have the same feeling :-). To let me get a quick refreshment of the technology after a while, I resort to the old school method: writing notes. But one day, I came out a idea: why not try to write a tutorial during studying instead of only recording? So in the past 2 years, Golang 101 hacks and OpenMP Little Book are born. The whole process is really rewarding :

(1) Sometimes you think you have grasped the knowledge, but when you begin to write an article to explain it, you will find there are some points you can’t understand thoroughly. To make your article more clearly, you need to write code to verify it, look for help in the internet, etc. Among this process, you will get a deeper understanding.

(2) Your tutorial can be reviewed by other brilliant engineers who can point out your mistakes, in the meantime, the tutorial can also help others. E.g., I find my tutorial is quoted in stackoverflow’s answer occasionally, and it really encourages me!

(3) Since I am not a native English speaker, creating an English tutorial can also help to improve and practice my English skills. I highly recommend you use English to compose, because that can make your idea shared among the people all over the world!

Based on above points, writing technological tutorial is definitely a win-win process. Why not give a shot? I think you can!

configure script may not check pthread correctly on OpenBSD

I have come into at least 2 times that one project was built well on Linux, while can’t find pthread related definitions on OpenBSD, like this:

../../runtime/cilk-internal.h:39:6: error: unknown type name 'pthread_mutex_t'
     pthread_mutex_t posix;
../../runtime/cilk-internal.h:211:6: error: unknown type name 'pthread_t'
     pthread_t *tid;
../../runtime/cilk-internal.h:216:6: error: unknown type name 'pthread_cond_t'
     pthread_cond_t  waiting_workers_cond;
../../runtime/cilk-internal.h:217:6: error: unknown type name 'pthread_cond_t'
     pthread_cond_t  wakeup_first_worker_cond;
../../runtime/cilk-internal.h:218:6: error: unknown type name 'pthread_cond_t'
     pthread_cond_t  wakeup_other_workers_cond;
../../runtime/cilk-internal.h:219:6: error: unknown type name 'pthread_mutex_t'
     pthread_mutex_t workers_mutex;
../../runtime/cilk-internal.h:220:6: error: unknown type name 'pthread_cond_t'
     pthread_cond_t  workers_done_cond;

The source code is as following:

#include <pthread.h>

While the generated config.h doesn’t define HAVE_PTHREAD macro:

/* Define if you have POSIX threads libraries and header files. */
/* #undef HAVE_PTHREAD */

But in fact, the OpenBSD has provided all support of pthread. So please be aware of this issue.

The difference between const&constexpr objects in C++

In C++, const object means once initialized, its value can’t be changed. The initialization can occur in compile-time:

const auto c = 0;

Or in run-time:

#include <cstdlib>
#include <ctime>

auto func()
    return std::rand() % 10;

int main()
    const auto c = func();

    return 0;

For constexpr objects, you can think they are a subset of const objects, and the constexpr object’s value must be determined in compile-time. Change the declaration of c in above code:

constexpr auto c = func();

The code can’t be compiled because the c‘s value couldn’t be generated during compilation phrase.