Let’s see a Go web program:
package main
import (
"fmt"
"io"
"io/ioutil"
"net/http"
"os"
"time"
)
func main() {
for {
resp, err := http.Get("https://www.google.com/")
if err != nil {
fmt.Fprintf(os.Stderr, "fetch: %v\n", err)
os.Exit(1)
}
_, err = io.Copy(ioutil.Discard, resp.Body)
resp.Body.Close()
if err != nil {
fmt.Fprintf(os.Stderr, "fetch: reading: %v\n", err)
os.Exit(1)
}
time.Sleep(45 * time.Second)
}
}
The logic of above application is not hard, just retrieve information from the specified Website URL. I am a layman of web development, and think the HTTP communication should be “short-lived”, which means when HTTPclient issues a request, it will start a TCP connection with HTTP server, once the client receives the response, this session should end, and the TCP connection should be demolished. But is the fact really like this? I use lsof command to check my guess:
# lsof -P -n -p 907
......
fetch 907 root 3u IPv4 0xfffff80013677810 0t0 TCP 192.168.80.129:32618->xxx.xxx.xxx.xxx:8080 (ESTABLISHED)
......
Oh! My assumption is wrong, and there is a “long-lived” TCP connection. Why does this happen? When I come across the network related troubles, I will always seek help from tcpdump and wireshark and try to capture packets for analysis. This time should not be exception, and the communication flow is as the following picture:
(1) The start number of packet is 4, that’s because the first 3 packets are TCP handshake, and it is safe to ignore them;
(2) Packet 4 ~ 43 is the first HTTP GET flow, and this process lasts about 2 seconds, and ends at 19:20:37;
(3) Then after half a minute, at 19:21:07, there occurs a TCP keep-alive packet on the wire. Oh dear! The root cause has been found! Although the HTTP session is over, the TCP connection still exists and uses keep-alive mechanism to make the TCP passway alive, so this TCP route can be reused by following HTTP messages;
(4) As expected, 15 seconds later, a new HTTP session begins at packet 46, which is exactly 45 seconds pass after the first HTTPconversation.
That’s the effect of TCP keep-alive, which keeps the TCP connection alive and can be reused by following HTTP sessions. Another thing you should pay attention of reusing connection is the Response.Body must be read to completion and closed (Please refer here):
type Response struct {
......
// Body represents the response body.
//
// The http Client and Transport guarantee that Body is always
// non-nil, even on responses without a body or responses with
// a zero-length body. It is the caller's responsibility to
// close Body. The default HTTP client's Transport does not
// attempt to reuse HTTP/1.0 or HTTP/1.1 TCP connections
// ("keep-alive") unless the Body is read to completion and is
// closed.
//
// The Body is automatically dechunked if the server replied
// with a "chunked" Transfer-Encoding.
Body io.ReadCloser
......
}
As an example, modify the above program as follows:
package main
import (
"fmt"
"io"
"io/ioutil"
"net/http"
"os"
"time"
)
func closeResp(resp *http.Response) {
_, err := io.Copy(ioutil.Discard, resp.Body)
resp.Body.Close()
if err != nil {
fmt.Fprintf(os.Stderr, "fetch: reading: %v\n", err)
os.Exit(1)
}
}
func main() {
for {
resp1, err := http.Get("https://www.google.com/")
if err != nil {
fmt.Fprintf(os.Stderr, "fetch: %v\n", err)
os.Exit(1)
}
resp2, err := http.Get("https://www.facebook.com/")
if err != nil {
fmt.Fprintf(os.Stderr, "fetch: %v\n", err)
os.Exit(1)
}
time.Sleep(45 * time.Second)
for _, v := range []*http.Response{resp1, resp2} {
closeResp(v)
}
}
}
During running it, You will see 2 TCP connections, not 1:
# lsof -P -n -p 1982
......
fetch 1982 root 3u IPv4 0xfffff80013677810 0t0 TCP 192.168.80.129:43793->xxx.xxx.xxx.xxx:8080 (ESTABLISHED)
......
fetch 1982 root 6u IPv4 0xfffff80013677000 0t0 TCP 192.168.80.129:12105->xxx.xxx.xxx.xxx:8080 (ESTABLISHED)
If you call closeResp function before issuing new HTTP request, the TCP connection can be reused:
package main
import (
"fmt"
"io"
"io/ioutil"
"net/http"
"os"
"time"
)
func closeResp(resp *http.Response) {
_, err := io.Copy(ioutil.Discard, resp.Body)
resp.Body.Close()
if err != nil {
fmt.Fprintf(os.Stderr, "fetch: reading: %v\n", err)
os.Exit(1)
}
}
func main() {
for {
resp1, err := http.Get("https://www.google.com/")
if err != nil {
fmt.Fprintf(os.Stderr, "fetch: %v\n", err)
os.Exit(1)
}
closeResp(resp1)
time.Sleep(45 * time.Second)
resp2, err := http.Get("https://www.facebook.com/")
if err != nil {
fmt.Fprintf(os.Stderr, "fetch: %v\n", err)
os.Exit(1)
}
closeResp(resp2)
}
}
P.S., the full code is here.
References:
Package http;
Reusing http connections in Golang;
Is HTTP Shortlived?.
And what if “Connection” “close” header is sent , there is still a need to read and close response body to actually close the connection?
@Milan, As I have said in post: “I am a layman of web development”. But per my understanding, if “http.Get()” method returns nil error, you should always close the response body to avoid resource leak. Thx!
the example you give in the code about re-use the TCP connection is mis-leading, if you want to demonstrate the re-use socket, you need to create two http.get with same target, the way you put there has 1 connection to google and 1 connection to facebook, the underlaying TCP socket are going to two different DstIP, you will never re-use that socket for sure, since you won’t send Google’s traffic to FB.
The general idea is correct, and you can go more fancy if you want, to have go routine to do the actual work, and you will see big different on how the re-use work.
agree
Thanks for this ijformative read, I hve shared it on Facebook.
Hello! Quick question that’s totally off topic. Do you
know how to make your site mobile friendly? My website looks weird
when browsing from my iphone4. I’m trying to find a theme or
plugin that might be able to fix this problem. If you have any suggestions, please share.
Cheers!
Very sorry. I am not a expert in wordpress, so no good advice for you.
hi nanxiao,
thanks for your sharing.
But the result I got in my machine with your demo code is different from what you shown here.