Nan Xiao's Blog

A system software / performance engineer's home

Month: July 2016

Fix “please format Go code with ‘gofmt -s'” issue in building Swarmkit

When you do modifications in Swarmkit source code, e.g., doc.go, sometimes executing “make” command will show the following errors:

# make
 fmt
doc.go
 please format Go code with 'gofmt -s'

But executing “gofmt -s” command on doc.go seems not take effect:

# gofmt -s doc.go
// Package swarmkit implements a framework for task orchestration.
package swarmkit
# make
 fmt
doc.go
 please format Go code with 'gofmt -s'
make: *** [fmt] Error 1

The solution should be running “go fmt” command on this file, then the build process is OK:

# go fmt doc.go
doc.go
# make
 fmt
 bin/swarmd
......

Wget only recognizes http_proxy, not https_proxy

My Ubuntu 16.04 LTS works behind proxy. I have set HTTP_PROXY and HTTPS_PROXY environmental variables:

HTTP_PROXY=http://web-proxy.corp.xxxxxx.com:8080/
HTTPS_PROXY=https://web-proxy.corp.xxxxx.com:8080/

But wget can’t work:

# wget -P /tmp https://github.com/NVIDIA/nvidia-docker/releases/download/v1.0.0-rc.3/nvidia-docker_1.0.0.rc.3-1_amd64.deb
--2016-07-14 22:51:12--  https://github.com/NVIDIA/nvidia-docker/releases/download/v1.0.0-rc.3/nvidia-docker_1.0.0.rc.3-1_amd64.deb
Resolving github.com (github.com)... 192.30.253.112
Connecting to github.com (github.com)|192.30.253.112|:443... connected.
ERROR: cannot verify github.com's certificate, issued by ‘O=Fortinet Ltd.,CN=FG3K6C3A15800021’:
  Self-signed certificate encountered.
    ERROR: certificate common name ‘FG3K6C3A15800021’ doesn't match requested host name ‘github.com’.
To connect to github.com insecurely, use `--no-check-certificate'.
root@ubuntu:~# wget --no-check-certificate -P /tmp https://github.com/NVIDIA/nvidia-docker/releases/download/v1.0.0-rc.3/nvidia-docker_1.0.0.rc.3-1_amd64.deb

After setting http_proxy and https_proxy:

http_proxy=http://web-proxy.corp.xxxxxx.com:8080/
https_proxy=https://web-proxy.corp.xxxxx.com:8080/
HTTP_PROXY=http://web-proxy.corp.xxxxxx.com:8080/
HTTPS_PROXY=https://web-proxy.corp.xxxxx.com:8080/

Now wget works:

# wget -P /tmp https://github.com/NVIDIA/nvidia-docker/releases/download/v1.0.0-rc.3/nvidia-docker_1.0.0.rc.3-1_amd64.deb     --2016-07-14 22:57:30--  https://github.com/NVIDIA/nvidia-docker/releases/download/v1.0.0-rc.3/nvidia-docker_1.0.0.rc.3-1_amd64.deb
Resolving web-proxy.xxxxxx.hp.com (web-proxy.xxxxxx.hp.com)... xxx.xxx.xxx.xxx
Connecting to web-proxy.xxxxxx.hp.com (web-proxy.xxxxxx.hp.com)|xxx.xxx.xxx.xxx|:8080... connected.
Proxy request sent, awaiting response... 302 Found
......

So we can conclude that wget is picky about uppercase and lowercase words.

 

Build docker from source behind proxy

If you want to build Docker from source like this:

# git clone https://github.com/docker/docker.git
# cd docker
# make

But your working server is actually behind a proxy, I think you may run into errors as the follows:

# make
mkdir bundles
docker build  -t "docker-dev:master" -f "Dockerfile" .
Sending build context to Docker daemon 145.8 MB
Step 1 : FROM debian:jessie
 ---> f854eed3f31f
Step 2 : RUN apt-key adv --keyserver hkp://p80.pool.sks-keyservers.net:80 --recv-keys E871F18B51E0147C77796AC81196BA81F6B0FC61  || apt-key adv --keyserver hkp://pgp.mit.edu:80 --recv-keys E871F18B51E0147C77796AC81196BA81F6B0FC61
 ---> Running in fede03b56767
Executing: gpg --ignore-time-conflict --no-options --no-default-keyring --homedir /tmp/tmp.MjO7kIEOm8 --no-auto-check-trustdb --trust-model always --primary-keyring /etc/apt/trusted.gpg --keyring /etc/apt/trusted.gpg.d/debian-archive-jessie-automatic.gpg --keyring /etc/apt/trusted.gpg.d/debian-archive-jessie-security-automatic.gpg --keyring /etc/apt/trusted.gpg.d/debian-archive-jessie-stable.gpg --keyring /etc/apt/trusted.gpg.d/debian-archive-squeeze-automatic.gpg --keyring /etc/apt/trusted.gpg.d/debian-archive-squeeze-stable.gpg --keyring /etc/apt/trusted.gpg.d/debian-archive-wheezy-automatic.gpg --keyring /etc/apt/trusted.gpg.d/debian-archive-wheezy-stable.gpg --keyserver hkp://p80.pool.sks-keyservers.net:80 --recv-keys E871F18B51E0147C77796AC81196BA81F6B0FC61
gpg: requesting key F6B0FC61 from hkp server p80.pool.sks-keyservers.net
gpgkeys: key E871F18B51E0147C77796AC81196BA81F6B0FC61 can't be retrieved
gpg: no valid OpenPGP data found.
gpg: Total number processed: 0
Executing: gpg --ignore-time-conflict --no-options --no-default-keyring --homedir /tmp/tmp.6clUfj4AwL --no-auto-check-trustdb --trust-model always --keyring /etc/apt/trusted.gpg --primary-keyring /etc/apt/trusted.gpg --keyring /etc/apt/trusted.gpg.d/debian-archive-jessie-automatic.gpg --keyring /etc/apt/trusted.gpg.d/debian-archive-jessie-security-automatic.gpg --keyring /etc/apt/trusted.gpg.d/debian-archive-jessie-stable.gpg --keyring /etc/apt/trusted.gpg.d/debian-archive-squeeze-automatic.gpg --keyring /etc/apt/trusted.gpg.d/debian-archive-squeeze-stable.gpg --keyring /etc/apt/trusted.gpg.d/debian-archive-wheezy-automatic.gpg --keyring /etc/apt/trusted.gpg.d/debian-archive-wheezy-stable.gpg --keyserver hkp://pgp.mit.edu:80 --recv-keys E871F18B51E0147C77796AC81196BA81F6B0FC61
gpg: requesting key F6B0FC61 from hkp server pgp.mit.edu
gpgkeys: key E871F18B51E0147C77796AC81196BA81F6B0FC61 can't be retrieved
gpg: no valid OpenPGP data found.
gpg: Total number processed: 0
The command '/bin/sh -c apt-key adv --keyserver hkp://p80.pool.sks-keyservers.net:80 --recv-keys E871F18B51E0147C77796AC81196BA81F6B0FC61 || apt-key adv --keyserver hkp://pgp.mit.edu:80 --recv-keys E871F18B51E0147C77796AC81196BA81F6B0FC61' returned a non-zero code: 2
Makefile:70: recipe for target 'build' failed
make: *** [build] Error 1

Or:

......
RUN apt-get update && apt-get install -y       apparmor        apt-utils       aufs-tools      automake        bash-completion   binutils-mingw-w64       bsdmainutils    btrfs-tools     build-essential         clang   createrepo      curl    dpkg-sig        gcc-mingw-w64      git     iptables        jq      libapparmor-dev         libcap-dev      libltdl-dev     libsqlite3-dev  libsystemd-journal-dev  libtool    mercurial       net-tools       pkg-config      python-dev      python-mock     python-pip      python-websocket        ubuntu-zfs xfsprogs        libzfs-dev      tar     zip     --no-install-recommends         && pip install awscli==1.10.15
 ---> Running in 37080c364862
Get:1 http://ppa.launchpad.net trusty InRelease [8127 B]
Get:2 http://httpredir.debian.org jessie InRelease [8127 B]
Get:3 http://security.debian.org jessie/updates InRelease [8127 B]
Splitting up /var/lib/apt/lists/partial/ppa.launchpad.net_zfs-native_stable_ubuntu_dists_trusty_InRelease into data and signature failedIgn http://ppa.launchpad.net trusty InRelease
E: GPG error: http://ppa.launchpad.net trusty InRelease: Clearsigned file isn't valid, got 'NODATA' (does the network require authentication?)
The command '/bin/sh -c apt-get update && apt-get install -y    apparmor        apt-utils       aufs-tools      automake        bash-completion    binutils-mingw-w64      bsdmainutils    btrfs-tools     build-essential         clang   createrepo      curl    dpkg-sig        gcc-mingw-w64      git     iptables        jq      libapparmor-dev         libcap-dev      libltdl-dev     libsqlite3-dev  libsystemd-journal-dev     libtool         mercurial       net-tools       pkg-config      python-dev      python-mock     python-pip      python-websocket  ubuntu-zfs       xfsprogs        libzfs-dev      tar     zip     --no-install-recommends         && pip install awscli==1.10.15' returned a non-zero code: 100
make: *** [build] Error 1

These reports can make you crazy!

The solution is adding proxy into Dockerfile which resides in the root directory of Docker folder:

......
FROM debian:jessie
ENV http_proxy http://web-proxy.corp.xxxxxx.com:8080/
ENV https_proxy https://web-proxy.corp.xxxxxx.com:8080/
......

Then the make progress will be smooth!

P.S., after discussing in reddit, the correct and idiomatic method should be this:

make DOCKER_BUILD_ARGS="--build-arg http_proxy=http://web-proxy.corp.xxxxxx.com:8080/ --build-arg https_proxy=https://web-proxy.corp.xxxxxx.com:8080/"

Powered by WordPress & Theme by Anders Norén