AddressSanitizer’s ChunkHeader

Recently, I came across following core dump from libasan:

#0  0x00007fffe76c7387 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:55
#1  0x00007fffe76c8a78 in __GI_abort () at abort.c:90
#2  0x00007ffff74c4582 in __sanitizer::Abort() () from /usr/lib64/libasan.so.6
#3  0x00007ffff74d012c in __sanitizer::Die() () from /usr/lib64/libasan.so.6
#4  0x00007ffff74af63c in __asan::ScopedInErrorReport::~ScopedInErrorReport() () from /usr/lib64/libasan.so.6
#5  0x00007ffff74ad989 in __asan::ReportMallocUsableSizeNotOwned(unsigned long, __sanitizer::BufferedStackTrace*) () from /usr/lib64/libasan.so.6
#6  0x00007ffff7418b82 in __asan::asan_malloc_usable_size(void const*, unsigned long, unsigned long) () from /usr/lib64/libasan.so.6
......;

To debug this issue, I checked libasan source code and found there is a 16-byte ChunkHeader in front of user memory which records the information of the used memory:

class ChunkHeader {
 public:
  atomic_uint8_t chunk_state;
  u8 alloc_type : 2;
  u8 lsan_tag : 2;

  // align < 8 -> 0
  // else      -> log2(min(align, 512)) - 2
  u8 user_requested_alignment_log : 3;

 private:
  u16 user_requested_size_hi;
  u32 user_requested_size_lo;
  atomic_uint64_t alloc_context_id;
  ......
}

By using user_requested_size_hi and user_requested_size_lo, we can calculate how much memory is required, and if it is 0, the above exception will be reported:

uptr asan_malloc_usable_size(const void *ptr, uptr pc, uptr bp) {
  if (!ptr) return 0;
  uptr usable_size = instance.AllocationSize(reinterpret_cast<uptr>(ptr));
  if (flags()->check_malloc_usable_size && (usable_size == 0)) {
    GET_STACK_TRACE_FATAL(pc, bp);
    ReportMallocUsableSizeNotOwned((uptr)ptr, &stack);
  }
  return usable_size;
}

Valgrind can’t work with sanitizers together

Valgrind can’t work with sanitizers together. Check following program with explicit memory leak:

# cat memory-leak.c
#include <stdlib.h>
void *p;
int main() {
  p = malloc(7);
  p = 0; // The memory is leaked here.
  return 0;
}

Build it and run with valgrind:

# gcc memory-leak.c -o memory-leak
# valgrind ./memory-leak
==1155== Memcheck, a memory error detector
==1155== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==1155== Using Valgrind-3.16.1 and LibVEX; rerun with -h for copyright info
==1155== Command: ./memory-leak
==1155==
==1155==
==1155== HEAP SUMMARY:
==1155==     in use at exit: 7 bytes in 1 blocks
==1155==   total heap usage: 1 allocs, 0 frees, 7 bytes allocated
==1155==
==1155== LEAK SUMMARY:
==1155==    definitely lost: 7 bytes in 1 blocks
==1155==    indirectly lost: 0 bytes in 0 blocks
==1155==      possibly lost: 0 bytes in 0 blocks
==1155==    still reachable: 0 bytes in 0 blocks
==1155==         suppressed: 0 bytes in 0 blocks
==1155== Rerun with --leak-check=full to see details of leaked memory
==1155==
==1155== For lists of detected and suppressed errors, rerun with: -s
==1155== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)

The memory leak is detected successfully. Build the program with “-fsanitize=address” option and run valgrind again:

# gcc -fsanitize=address memory-leak.c -o memory-leak
# valgrind ./memory-leak
==1193== Memcheck, a memory error detector
==1193== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==1193== Using Valgrind-3.16.1 and LibVEX; rerun with -h for copyright info
==1193== Command: ./memory-leak
==1193==
==1193==ASan runtime does not come first in initial library list; you should either link runtime to your application or manually preload it with LD_PRELOAD.
==1193==
==1193== HEAP SUMMARY:
==1193==     in use at exit: 0 bytes in 0 blocks
==1193==   total heap usage: 0 allocs, 0 frees, 0 bytes allocated
==1193==
==1193== All heap blocks were freed -- no leaks are possible
==1193==
==1193== For lists of detected and suppressed errors, rerun with: -s
==1193== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)

We can see the valgrind can’t work normally.

Reference:
Sourceforge.

A core dump related to jemalloc

Recently I came across a core dump related to jemalloc:

#0  extent_arena_ind_get (extent=0x0) at include/jemalloc/internal/extent_inlines.h:40
#1  je_tcache_bin_flush_small (tsd=tsd@entry=0x7ffff7f717f8, tcache=tcache@entry=0x7ffff7f719e8,
    tbin=tbin@entry=0x7ffff7f71a70, binind=binind@entry=5, rem=<optimized out>) at src/tcache.c:159
#2  0x00007ffff356b97b in je_tcache_event_hard (tsd=tsd@entry=0x7ffff7f717f8,
    tcache=tcache@entry=0x7ffff7f719e8) at src/tcache.c:55
#3  0x00007ffff3512d49 in tcache_event (tcache=<optimized out>, tsd=<optimized out>)
    at include/jemalloc/internal/tcache_inlines.h:37
#4  tcache_dalloc_large (slow_path=<optimized out>, binind=<optimized out>, ptr=<optimized out>,
    tcache=<optimized out>, tsd=<optimized out>) at include/jemalloc/internal/tcache_inlines.h:212
#5  arena_dalloc_large (slow_path=<optimized out>, szind=<optimized out>, tcache=<optimized out>,
    ptr=<optimized out>, tsdn=<optimized out>) at include/jemalloc/internal/arena_inlines_b.h:276
#6  arena_dalloc (slow_path=<optimized out>, alloc_ctx=<optimized out>, tcache=<optimized out>,
    ptr=<optimized out>, tsdn=<optimized out>) at include/jemalloc/internal/arena_inlines_b.h:323
#7  idalloctm (slow_path=<optimized out>, is_internal=<optimized out>, alloc_ctx=<optimized out>,
    tcache=<optimized out>, ptr=<optimized out>, tsdn=<optimized out>)
    at include/jemalloc/internal/jemalloc_internal_inlines_c.h:118
#8  ifree (slow_path=<optimized out>, tcache=<optimized out>, ptr=<optimized out>,
    tsd=<optimized out>) at src/jemalloc.c:2589
#9  je_free_default (ptr=0x7fff2ccf53c0) at src/jemalloc.c:2799

The Sanitizers helped me to find the root cause: a classical “double-free” memory issue. One thing should be noticed is the Sanitizers and jemalloc can’t be used simultaneously because they both intercept memory allocation/free functions. Check following code:

# cat memory-leak.c
#include <stdlib.h>
void *p;
int main() {
  p = malloc(7);
  p = 0; // The memory is leaked here.
  return 0;
}

Build with both Sanitizers and jemalloc:

# gcc -fsanitize=address -g memory-leak.c -L`jemalloc-config --libdir` -Wl,-rpath,`jemalloc-config --libdir` -ljemalloc `jemalloc-config --libs`
# ldd a.out
    linux-vdso.so.1 (0x00007ffdb85a6000)
    libasan.so.6 => /usr/lib/libasan.so.6 (0x00007ffb04605000)
    libjemalloc.so.2 => /usr/lib/libjemalloc.so.2 (0x00007ffb04362000)
    libm.so.6 => /usr/lib/libm.so.6 (0x00007ffb0421d000)
    libstdc++.so.6 => /usr/lib/libstdc++.so.6 (0x00007ffb03fb4000)
    libdl.so.2 => /usr/lib/libdl.so.2 (0x00007ffb03fae000)
    libpthread.so.0 => /usr/lib/libpthread.so.0 (0x00007ffb03f8d000)
    libc.so.6 => /usr/lib/libc.so.6 (0x00007ffb03dc5000)
    librt.so.1 => /usr/lib64/../lib64/librt.so.1 (0x00007ffb03dba000)
    libgcc_s.so.1 => /usr/lib64/../lib64/libgcc_s.so.1 (0x00007ffb03da0000)
    /lib64/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007ffb04fdc000)

Use gdb to debug the program:

......
Breakpoint 2, 0x00007ffff76a90a4 in malloc () from /usr/lib/libasan.so.6
(gdb) bt
#0  0x00007ffff76a90a4 in malloc () from /usr/lib/libasan.so.6
#1  0x0000555555555183 in main () at memory-leak.c:4
......

The program will use functions from Sanitizers instead of jemalloc.

AddressSanitizer’s no_sanitize_address attribute

AddressSanitizer has a no_sanitize_address attribute which can be used to turn off instrumentation. Check following code:

$ cat ../main.c
#include <stdlib.h>
#include <sanitizer/asan_interface.h>

#define ARRAY_SIZE 4

int *array;

void
foo()
{
    array[0] = 1;
}

int
main()
{
    array = malloc(sizeof(int) * ARRAY_SIZE);
    ASAN_POISON_MEMORY_REGION(array, sizeof(int) * ARRAY_SIZE);
    foo();
}

Build and run this program:

$ ./main
=================================================================
==1558==ERROR: AddressSanitizer: use-after-poison on address 0x602000000010 at pc 0x55839733c1b7 bp 0x7ffc102fb320 sp 0x7ffc102fb310
WRITE of size 4 at 0x602000000010 thread T0
    #0 0x55839733c1b6 in foo (/home/nan/code-for-my-blog/2020/05/asan_no_sanitize_address/build/main+0x11b6)
    #1 0x55839733c1f2 in main (/home/nan/code-for-my-blog/2020/05/asan_no_sanitize_address/build/main+0x11f2)
    #2 0x7fe4d79d9dea in __libc_start_main ../csu/libc-start.c:308
    #3 0x55839733c0b9 in _start (/home/nan/code-for-my-blog/2020/05/asan_no_sanitize_address/build/main+0x10b9)

0x602000000010 is located 0 bytes inside of 16-byte region [0x602000000010,0x602000000020)
allocated by thread T0 here:
    #0 0x7fe4d7c824c8 in __interceptor_malloc (/usr/lib/libasan.so.5+0x10c4c8)
    #1 0x55839733c1cd in main (/home/nan/code-for-my-blog/2020/05/asan_no_sanitize_address/build/main+0x11cd)
    #2 0x7fe4d79d9dea in __libc_start_main ../csu/libc-start.c:308

SUMMARY: AddressSanitizer: use-after-poison (/home/nan/code-for-my-blog/2020/05/asan_no_sanitize_address/build/main+0x11b6) in foo
Shadow bytes around the buggy address:
  0x0c047fff7fb0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c047fff7fc0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c047fff7fd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c047fff7fe0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c047fff7ff0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
=>0x0c047fff8000: fa fa[f7]f7 fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c047fff8010: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c047fff8020: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c047fff8030: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c047fff8040: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c047fff8050: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
  Shadow gap:              cc
==1558==ABORTING

Unsurprisingly, AddressSanitizer reports error because the array is poisoned but foo() is trying to modify its first element. Add __attribute__((no_sanitize_address)) for foo():

__attribute__((no_sanitize_address))
void
foo()
{
    array[0] = 1;
}

Run the program again:

$ ./main
$

This time program exits normally.

P.S., the code can be downloaded here.