Tag: gRPC

a server program which uses gRPC crashed when started:

$ ./server 60001
......
E1123 15:37:54.133480971   14408 server_chttp2.c:38]         {"created":"@1511422674.133408109","description":"No address added out of total 1 resolved","file":"src/core/ext/transport/chttp2/server/chttp2_server.c","file_line":245,"referenced_errors":[{"created":"@1511422674.133405147","description":"Failed to add any wildcard listeners","file":"src/core/lib/iomgr/tcp_server_posix.c","file_line":338,"referenced_errors":[{"created":"@1511422674.133394827","description":"Unable to configure socket","fd":4,"file":"src/core/lib/iomgr/tcp_server_utils_posix_common.c","file_line":200,"referenced_errors":[{"created":"@1511422674.133385167","description":"OS Error","errno":98,"file":"src/core/lib/iomgr/tcp_server_utils_posix_common.c","file_line":173,"os_error":"Address already in use","syscall":"bind"}]},{"created":"@1511422674.133404647","description":"Unable to configure socket","fd":4,"file":"src/core/lib/iomgr/tcp_server_utils_posix_common.c","file_line":200,"referenced_errors":[{"created":"@1511422674.133401558","description":"OS Error","errno":98,"file":"src/core/lib/iomgr/tcp_server_utils_posix_common.c","file_line":173,"os_error":"Address already in use","syscall":"bind"}]}]}]}
Segmentation fault (core dumped)

It is weird because it runs well yesterday. Check the core dump file:

......
Core was generated by `./server 60001'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x00007f30edbbf9b0 in pthread_mutex_lock () from /usr/lib/libpthread.so.0
(gdb) bt
#0  0x00007f30edbbf9b0 in pthread_mutex_lock () from /usr/lib/libpthread.so.0
#1  0x000055c312c87077 in grpc::Server::Wait() ()
......

No clue. So format the above json log:

{  
   "created":"@1511422674.133408109",
   "description":"No address added out of total 1 resolved",
   "file":"src/core/ext/transport/chttp2/server/chttp2_server.c",
   "file_line":245,
   "referenced_errors":[  
      {  
         "created":"@1511422674.133405147",
         "description":"Failed to add any wildcard listeners",
         "file":"src/core/lib/iomgr/tcp_server_posix.c",
         "file_line":338,
         "referenced_errors":[  
            {  
               "created":"@1511422674.133394827",
               "description":"Unable to configure socket",
               "fd":4,
               "file":"src/core/lib/iomgr/tcp_server_utils_posix_common.c",
               "file_line":200,
               "referenced_errors":[  
                  {  
                     "created":"@1511422674.133385167",
                     "description":"OS Error",
                     "errno":98,
                     "file":"src/core/lib/iomgr/tcp_server_utils_posix_common.c",
                     "file_line":173,
                     "os_error":"Address already in use",
                     "syscall":"bind"
                  }
               ]
            },
            {  
               "created":"@1511422674.133404647",
               "description":"Unable to configure socket",
               "fd":4,
               "file":"src/core/lib/iomgr/tcp_server_utils_posix_common.c",
               "file_line":200,
               "referenced_errors":[  
                  {  
                     "created":"@1511422674.133401558",
                     "description":"OS Error",
                     "errno":98,
                     "file":"src/core/lib/iomgr/tcp_server_utils_posix_common.c",
                     "file_line":173,
                     "os_error":"Address already in use",
                     "syscall":"bind"
                  }
               ]
            }
         ]
      }

Oh, I see. “Address already in use” means the 60001 port is occupied already. I switch to another port, and it works.

The route_guide program is a very good tutorial to learn gRPC, I will use it as an example to analyze the gRPC programming.

Prologue:
Compile the program with debug information, so it is easy to use gdb to manipulate them. Modify Makefile:

CXXFLAGS += -std=c++11 -g

(1) Start route_guide_server application, and use gdb to step-into route_guide_client:

221     int main(int argc, char** argv) {
(gdb) n
223       std::string db = routeguide::GetDbFileContent(argc, argv);
(gdb)
226                               grpc::InsecureChannelCredentials()),
(gdb)
[New Thread 0x7ffff502f700 (LWP 25947)]
[New Thread 0x7ffff482e700 (LWP 25948)]
225           grpc::CreateChannel("localhost:50051",
(gdb)
227           db);
(gdb)
DB parsed, loaded 100 features.

Check current network status:

$ netstat -an | grep 50051
tcp6       0      0 :::50051                :::*                    LISTEN

From this, we can know grpc::CreateChannel() function doesn’t create connection to server, and it just does some initialization work.

(2) Since simple RPC is one-request/one-response mode, and it’s really easy, I won’t discuss it here.

(3) Server-side streaming RPC is one-request/multiple-response mode. The client code skeleton is like this:

std::unique_ptr<ClientReader<Feature> > reader(
        stub_->ListFeatures(&context, rect));
while (reader->Read(&feature)) {
  ......
}
Status status = reader->Finish();
if (status.ok()) {
  ......
} else {
  ......
}

After client sends request (stub_->ListFeatures(&context, rect)), it will read all the responses from server until reader->Read() returns false. Then using reader->Finish() to check the status. The server code is simple, just send all responses:

for (......) {
    ......
    writer->Write(f);
 } 

But how about client not want to continue to process the responses in the middle? It likes this:

while (reader->Read(&feature)) {
  ......
  if (some error) {
    break;
  } 
}
Status status = reader->Finish();
if (status.ok()) {
  ......
} else {
  ......
}

After experimenting, the client will block at reader->Finish() function. You can refer Be careful of using grpc::ClientStreamingInterface::Finish() function. There are 2 methods I can figure out to handle this case:
a) Don’t call reader->Finish():

while (reader->Read(&feature)) {
  ......
  if (some error) {
    break;
  } 
}

b) Or although there is error, still consume other messages:

while (reader->Read(&feature)) {
  ......
  if (some error) {
    break;
  } 
}
while (reader->Read(&feature)) {
    // do nothing, just comsume other messages.
}
Status status = reader->Finish();
if (status.ok()) {
  ......
} else {
  ......
}

Honestly, I don’t dive into the gRPC code. But from testing, the above 2 methods seems all work correctly.

(4) Client-side streaming RPC is multiple-request/one-response mode, the client code framework is like this:

std::unique_ptr<ClientWriter<Point> > writer(
    stub_->RecordRoute(&context, &stats));
for (......) {
  ......
  if (!writer->Write(f.location())) {
    // Broken stream.
    break;
  }
  ......
}
writer->WritesDone();
Status status = writer->Finish();
if (status.ok()) {
  // Check stats from server
} else {
  ......
}

The client code is simple, just tries best to send all requests. Once finish transmitting, calls writer->WritesDone() and blocks writer->Finish() to wait server’s response. For server code, it’s also brief:

while (reader->Read(&point)) {
  ......
}

If the program breaks the while loop half way:

while (reader->Read(&point)) {
  ......
  if (some error) {
    break;
  } 
}

That is OK too.

(5) The last flow is bidirectional streaming RPC, which is multiple-request/multiple-response mode. For the server code framework, it is neat, just sends response according request:

while (stream->Read(&note)) {
  ......
  stream->Write(n);
  ......
}

But for client side, it is a little complicated:

stream->Write(note);
stream->Read(&server_note);
stream->Write(note);
stream->Read(&server_note);
......
stream->WritesDone();
Status status = stream->Finish();
if (!status.ok()) {
  ......
} else {
  ......
}

You must pay attention to the same problem as Server-side streaming RPC, that is, if there is pending server’s responses before stream->Finish(), it will cause client block forever:

while (reader->Read(&feature)) {
  ......
  if (some error) {
    break;
  } 
}
Status status = reader->Finish();

Epilogue:
In a summary, gRPC is a brilliant communication framework which hides so many dirty details and let you focus on message definitions. If you are considering doing some network programming, you can try it, and I think it can’t make you disappointed.