First taster of Standard C++ Thread Library on OpenBSD

Today I tried Standard C++ Thread Library on OpenBSD, since it requires the compiler to support C++11 standard, and the default c++ only support C++98 (please refer here), so I need to switch to clang++. The program is just a classic “Hello World”:

#include <thread>
#include <iostream>

void hello()
{
    std::cout << "Hello World!\n";
}

int main(void)
{
    std::thread t(hello);
    t.join();
    return 0;
}

Built and run it:

# clang++ -std=c++11 hello.cpp
root:/root/Project# ./a.out
terminate called after throwing an instance of 'std::system_error'
  what():  Enable multithreading to use std::thread: Operation not permitted
Abort trap (core dumped)

Whoops! The program crashed. After reading this post, adding -pthread during compilation fixed this issue:

# clang++ -pthread -std=c++11 hello.cpp
# ./a.out
Hello World!

The anatomy of ldd program on OpenBSD

In the past week, I read the ldd source code on OpenBSD to get a better understanding of how it works. And this post should also be a reference for other*NIX OSs.

The ELF file is divided into 4 categories: relocatable, executable, shared, and core. Only the executable and shared object files may have dynamic object dependencies, so the ldd only check these 2 kinds of ELF file:

(1) Executable.

ldd leverages the LD_TRACE_LOADED_OBJECTS environment variable in fact, and the code is as following:

if (setenv("LD_TRACE_LOADED_OBJECTS", "true", 1) < 0)
    err(1, "setenv(LD_TRACE_LOADED_OBJECTS)");

When LD_TRACE_LOADED_OBJECTS is set to 1 or true, running executable file will show shared objects needed instead of running it, so you even not needldd to check executable file. See the following outputs:

# /usr/bin/ldd
usage: ldd program ...
# LD_TRACE_LOADED_OBJECTS=1 /usr/bin/ldd
        Start            End              Type Open Ref GrpRef Name
        00000b6ac6e00000 00000b6ac7003000 exe  1    0   0      /usr/bin/ldd
        00000b6dbc96c000 00000b6dbcc38000 rlib 0    1   0      /usr/lib/libc.so.89.3
        00000b6d6ad00000 00000b6d6ad00000 rtld 0    1   0      /usr/libexec/ld.so  

(2) Shared object.

The code to print dependencies of shared object is as following:

if (ehdr.e_type == ET_DYN && !interp) {
    if (realpath(name, buf) == NULL) {
        printf("realpath(%s): %s", name,
            strerror(errno));
        fflush(stdout);
        _exit(1);
    }
    dlhandle = dlopen(buf, RTLD_TRACE);
    if (dlhandle == NULL) {
        printf("%s\n", dlerror());
        fflush(stdout);
        _exit(1);
    }
    _exit(0);
}

Why the condition of checking a ELF file is shared object or not is like this:

if (ehdr.e_type == ET_DYN && !interp) {
    ......
}

That’s because the file type of position-independent executable (PIE) is the same as shared object, but normally PIE contains a interpreter program header since it needs dynamic linker to load it while shared object lacks (refer this article). So the above condition will filter PIE file.

The dlopen(buf, RTLD_TRACE) is used to print dynamic object information. And the actual code is like this:

if (_dl_traceld) {
    _dl_show_objects();
    _dl_unload_shlib(object);
    _dl_exit(0);
}

In fact, you can also implement a simple application which outputs dynamic object information for shared object yourself:

#include <dlfcn.h>

int main(int argc, char **argv)
{
    dlopen(argv[1], RTLD_TRACE);
    return 0;
}

Compile and use it to analyze /usr/lib/libssl.so.43.2:

# cc lddshared.c
# ./a.out /usr/lib/libssl.so.43.2
    Start            End              Type Open Ref GrpRef Name
    000010e2df1c5000 000010e2df41a000 dlib 1    0   0      /usr/lib/libssl.so.43.2
    000010e311e3f000 000010e312209000 rlib 0    1   0      /usr/lib/libcrypto.so.41.1

The same as using ldd directly:

# ldd /usr/lib/libssl.so.43.2
/usr/lib/libssl.so.43.2:
    Start            End              Type Open Ref GrpRef Name
    00001d9ffef08000 00001d9fff15d000 dlib 1    0   0      /usr/lib/libssl.so.43.2
    00001d9ff1431000 00001d9ff17fb000 rlib 0    1   0      /usr/lib/libcrypto.so.41.1

Through the studying of ldd source code, I also get many by-products: such as knowledge of ELF file, linking and loading, etc. So diving into code is a really good method to learn *NIX deeper!

The analysis of gRPC communication mode

The route_guide program is a very good tutorial to learn gRPC, I will use it as an example to analyze the gRPC programming.

Prologue:
Compile the program with debug information, so it is easy to use gdb to manipulate them. Modify Makefile:

CXXFLAGS += -std=c++11 -g

(1) Start route_guide_server application, and use gdb to step-into route_guide_client:

221     int main(int argc, char** argv) {
(gdb) n
223       std::string db = routeguide::GetDbFileContent(argc, argv);
(gdb)
226                               grpc::InsecureChannelCredentials()),
(gdb)
[New Thread 0x7ffff502f700 (LWP 25947)]
[New Thread 0x7ffff482e700 (LWP 25948)]
225           grpc::CreateChannel("localhost:50051",
(gdb)
227           db);
(gdb)
DB parsed, loaded 100 features.

Check current network status:

$ netstat -an | grep 50051
tcp6       0      0 :::50051                :::*                    LISTEN

From this, we can know grpc::CreateChannel() function doesn’t create connection to server, and it just does some initialization work.

(2) Since simple RPC is one-request/one-response mode, and it’s really easy, I won’t discuss it here.

(3) Server-side streaming RPC is one-request/multiple-response mode. The client code skeleton is like this:

std::unique_ptr<ClientReader<Feature> > reader(
        stub_->ListFeatures(&context, rect));
while (reader->Read(&feature)) {
  ......
}
Status status = reader->Finish();
if (status.ok()) {
  ......
} else {
  ......
}

After client sends request (stub_->ListFeatures(&context, rect)), it will read all the responses from server until reader->Read() returns false. Then using reader->Finish() to check the status. The server code is simple, just send all responses:

for (......) {
    ......
    writer->Write(f);
 } 

But how about client not want to continue to process the responses in the middle? It likes this:

while (reader->Read(&feature)) {
  ......
  if (some error) {
    break;
  } 
}
Status status = reader->Finish();
if (status.ok()) {
  ......
} else {
  ......
}

After experimenting, the client will block at reader->Finish() function. You can refer Be careful of using grpc::ClientStreamingInterface::Finish() function. There are 2 methods I can figure out to handle this case:
a) Don’t call reader->Finish():

while (reader->Read(&feature)) {
  ......
  if (some error) {
    break;
  } 
}

b) Or although there is error, still consume other messages:

while (reader->Read(&feature)) {
  ......
  if (some error) {
    break;
  } 
}
while (reader->Read(&feature)) {
    // do nothing, just comsume other messages.
}
Status status = reader->Finish();
if (status.ok()) {
  ......
} else {
  ......
}

Honestly, I don’t dive into the gRPC code. But from testing, the above 2 methods seems all work correctly.

(4) Client-side streaming RPC is multiple-request/one-response mode, the client code framework is like this:

std::unique_ptr<ClientWriter<Point> > writer(
    stub_->RecordRoute(&context, &stats));
for (......) {
  ......
  if (!writer->Write(f.location())) {
    // Broken stream.
    break;
  }
  ......
}
writer->WritesDone();
Status status = writer->Finish();
if (status.ok()) {
  // Check stats from server
} else {
  ......
}

The client code is simple, just tries best to send all requests. Once finish transmitting, calls writer->WritesDone() and blocks writer->Finish() to wait server’s response. For server code, it’s also brief:

while (reader->Read(&point)) {
  ......
}

If the program breaks the while loop half way:

while (reader->Read(&point)) {
  ......
  if (some error) {
    break;
  } 
}

That is OK too.

(5) The last flow is bidirectional streaming RPC, which is multiple-request/multiple-response mode. For the server code framework, it is neat, just sends response according request:

while (stream->Read(&note)) {
  ......
  stream->Write(n);
  ......
}

But for client side, it is a little complicated:

stream->Write(note);
stream->Read(&server_note);
stream->Write(note);
stream->Read(&server_note);
......
stream->WritesDone();
Status status = stream->Finish();
if (!status.ok()) {
  ......
} else {
  ......
}

You must pay attention to the same problem as Server-side streaming RPC, that is, if there is pending server’s responses before stream->Finish(), it will cause client block forever:

while (reader->Read(&feature)) {
  ......
  if (some error) {
    break;
  } 
}
Status status = reader->Finish();

Epilogue:
In a summary, gRPC is a brilliant communication framework which hides so many dirty details and let you focus on message definitions. If you are considering doing some network programming, you can try it, and I think it can’t make you disappointed.

Use more instructions in building CMake ExternalProject

When use CMake to build ExternalProject:

ExternalProject_Add(
    ......
)

It seems you can’t add multiple same command, for example, PATCH_COMMAND:

ExternalProject_Add(
    ......
    PATCH_COMMAND ......
    PATCH_COMMAND ......
    ......
)

You may get errors like following:

......
File /usr/bin/patch is read-only; trying to patch anyway
/usr/bin/patch: **** Can't create temporary file /usr/bin/patch.oLnbOsk : Permission denied
......

Instead you can COMMAND to add more instructions:

ExternalProject_Add(
    ......
    PATCH_COMMAND ......
    COMMAND ......
    ......
)

References:
CMake ExternalProject_Add: How to build multiple msbuild targets?;
ExternalProject.

Install googletest on OpenBSD

To use googletest on OpenBSD, I need to install CMake firstly:

# pkg_add cmake

Then Download googletest and create build directory:

# git clone https://github.com/google/googletest.git
# cd googletest
# mkdir build
# cd build

If use default gcc compiler, the compilation will generate following errors:

# make
[  2%] Building CXX object googlemock/CMakeFiles/gmock.dir/__/googletest/src/gtest-all.cc.o
cc1plus: warnings being treated as errors
In file included from /usr/include/g++/tr1/tuple:159,
                 from /root/Project/googletest/googletest/include/gtest/internal/gtest-port.h:746,
                 from /root/Project/googletest/googletest/include/gtest/internal/gtest-internal.h:40,
                 from /root/Project/googletest/googletest/include/gtest/gtest.h:58,
                 from /root/Project/googletest/googletest/src/gtest-all.cc:39:
/usr/include/g++/tr1/functional: In constructor 'std::tr1::_Mem_fn<_Res _Class::*>::_Mem_fn(_Res _Class::*)':
/usr/include/g++/tr1/functional:470: warning: declaration of '__pm' shadows a member of 'this'
In file included from /usr/include/g++/tr1/tuple:159,
                 from /root/Project/googletest/googletest/include/gtest/internal/gtest-port.h:746,
                 from /root/Project/googletest/googletest/include/gtest/internal/gtest-internal.h:40,
                 from /root/Project/googletest/googletest/include/gtest/gtest.h:58,
                 from /root/Project/googletest/googletest/src/gtest-all.cc:39:
/usr/include/g++/tr1/functional: In constructor 'std::tr1::_Simple_type_wrapper<_Tp>::_Simple_type_wrapper(_Tp)':
/usr/include/g++/tr1/functional:804: warning: declaration of '__value' shadows a member of 'this'
*** Error 1 in . (googlemock/CMakeFiles/gmock.dir/build.make:63 'googlemock/CMakeFiles/gmock.dir/__/googletest/src/gtest-all.cc.o': cd /root...)
*** Error 1 in . (CMakeFiles/Makefile2:90 'googlemock/CMakeFiles/gmock.dir/all')
*** Error 1 in /root/Project/googletest/build (Makefile:139 'all')

So I switch to clang, and all build is OK:

# CC=clang cmake ..
# make
# make install