The shortcut keys for perf-report command

I am not sure it is only me, but I can’t find any document to introduce the shortcut keys for perf-report command. After executing perf report, press h will show the shortcut keys:

If you want to filter some symbols, press /:

To remove filter, you should press / + ENTER, instead of pressing q/ESC:

Otherwise you will exit perf-report program (Because the filtered symbols screen is actually the main screen when you run perf report):

Leaked socket causes zmq_ctx_term() block forever

I met an issue that zmq_ctx_term() blocks forever:

#0  0x00007ffff33bdddd in poll () from /usr/lib64/libc.so.6
#1  0x00007ffff1519d1a in zmq::signaler_t::wait(int) () from /opt/lib/libzmq.so.5
#2  0x00007ffff1500915 in zmq::mailbox_t::recv(zmq::command_t*, int) () from /opt/lib/libzmq.so.5
#3  0x00007ffff14ef42d in zmq::ctx_t::terminate() () from /opt/lib/libzmq.so.5
......

After debugging, I found the reason is the socket leak which caused by not handling an error condition. So do remember call zmq_close() in every possible path.

AddressSanitizer’s ChunkHeader

Recently, I came across following core dump from libasan:

#0  0x00007fffe76c7387 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:55
#1  0x00007fffe76c8a78 in __GI_abort () at abort.c:90
#2  0x00007ffff74c4582 in __sanitizer::Abort() () from /usr/lib64/libasan.so.6
#3  0x00007ffff74d012c in __sanitizer::Die() () from /usr/lib64/libasan.so.6
#4  0x00007ffff74af63c in __asan::ScopedInErrorReport::~ScopedInErrorReport() () from /usr/lib64/libasan.so.6
#5  0x00007ffff74ad989 in __asan::ReportMallocUsableSizeNotOwned(unsigned long, __sanitizer::BufferedStackTrace*) () from /usr/lib64/libasan.so.6
#6  0x00007ffff7418b82 in __asan::asan_malloc_usable_size(void const*, unsigned long, unsigned long) () from /usr/lib64/libasan.so.6
......;

To debug this issue, I checked libasan source code and found there is a 16-byte ChunkHeader in front of user memory which records the information of the used memory:

class ChunkHeader {
 public:
  atomic_uint8_t chunk_state;
  u8 alloc_type : 2;
  u8 lsan_tag : 2;

  // align < 8 -> 0
  // else      -> log2(min(align, 512)) - 2
  u8 user_requested_alignment_log : 3;

 private:
  u16 user_requested_size_hi;
  u32 user_requested_size_lo;
  atomic_uint64_t alloc_context_id;
  ......
}

By using user_requested_size_hi and user_requested_size_lo, we can calculate how much memory is required, and if it is 0, the above exception will be reported:

uptr asan_malloc_usable_size(const void *ptr, uptr pc, uptr bp) {
  if (!ptr) return 0;
  uptr usable_size = instance.AllocationSize(reinterpret_cast<uptr>(ptr));
  if (flags()->check_malloc_usable_size && (usable_size == 0)) {
    GET_STACK_TRACE_FATAL(pc, bp);
    ReportMallocUsableSizeNotOwned((uptr)ptr, &stack);
  }
  return usable_size;
}

The caveat of thread name length in glibc

Recently, our team met an interesting bug: the process is configured to spawn 16 threads, but only spawns 10 threads in reality. The thread code is like this:

static void *
stat_consumer_thread_run(void *data)
{
    stat_consumer_thread_t *thread = data;
    char thread_name[64];
    snprintf(thread_name, sizeof(thread_name), "stat.consumer.%d",
        thread->id);
    int rc = pthread_setname_np(pthread_self(), thread_name);
    if (rc != 0) {
        return NULL;
    }

    ......
    return NULL;
}

After checking pthread_setname_np manual, we found:

The thread name is a meaningful C language string, whose length is restricted to 16 characters, including the terminating null byte (’\0’).

So thread name is restricted to 16 characters, “stat.consumer.0” ~ “stat.consumer.9” are set successfully, but “stat.consumer.10” ~ “stat.consumer.15” are not, and the corresponding threads are failed to run.

The experience of fixing a memory corruption issue

I came across a program crash last week:

Program terminated with signal 11, Segmentation fault.
#0  0x00007ffff365bd29 in __memcpy_ssse3_back () from /usr/lib64/libc.so.6
#0  0x00007ffff365bd29 in __memcpy_ssse3_back () from /usr/lib64/libc.so.6
#1  0x00007ffff606025c in memcpy (__len=<optimized out>, __src=0x0, __dest=0x0) at /usr/include/bits/string3.h:51
......
#5  0x0000000000000000 in ?? () 

The 5th stack frame address is 0x0000000000000000, and it seems not right. To debug it, get the registers values first:

According to X86_64 architecture, The value in memory address (%rbp) should be previous %rbp value, and the value in memory address (%rbp) + 8 should be return address. Checked these two values, and found they are all 0s, so it means the stack is corrupted.

The next thing to do is dump the memory between %rsp and %rbp, and refer the assembly code of the function at the same time. With this, I can know which memory part doesn’t seem correct, and review code accordingly. Finally I found the root cause and fixed it.

P.S., in optimisation build mode, some functions may be inlined, so please be aware of this caveat.