Nan Xiao's Blog

A system software / performance engineer's home

Tag: proxy

Wget only recognizes http_proxy, not https_proxy

My Ubuntu 16.04 LTS works behind proxy. I have set HTTP_PROXY and HTTPS_PROXY environmental variables:

HTTP_PROXY=http://web-proxy.corp.xxxxxx.com:8080/
HTTPS_PROXY=https://web-proxy.corp.xxxxx.com:8080/

But wget can’t work:

# wget -P /tmp https://github.com/NVIDIA/nvidia-docker/releases/download/v1.0.0-rc.3/nvidia-docker_1.0.0.rc.3-1_amd64.deb
--2016-07-14 22:51:12--  https://github.com/NVIDIA/nvidia-docker/releases/download/v1.0.0-rc.3/nvidia-docker_1.0.0.rc.3-1_amd64.deb
Resolving github.com (github.com)... 192.30.253.112
Connecting to github.com (github.com)|192.30.253.112|:443... connected.
ERROR: cannot verify github.com's certificate, issued by ‘O=Fortinet Ltd.,CN=FG3K6C3A15800021’:
  Self-signed certificate encountered.
    ERROR: certificate common name ‘FG3K6C3A15800021’ doesn't match requested host name ‘github.com’.
To connect to github.com insecurely, use `--no-check-certificate'.
root@ubuntu:~# wget --no-check-certificate -P /tmp https://github.com/NVIDIA/nvidia-docker/releases/download/v1.0.0-rc.3/nvidia-docker_1.0.0.rc.3-1_amd64.deb

After setting http_proxy and https_proxy:

http_proxy=http://web-proxy.corp.xxxxxx.com:8080/
https_proxy=https://web-proxy.corp.xxxxx.com:8080/
HTTP_PROXY=http://web-proxy.corp.xxxxxx.com:8080/
HTTPS_PROXY=https://web-proxy.corp.xxxxx.com:8080/

Now wget works:

# wget -P /tmp https://github.com/NVIDIA/nvidia-docker/releases/download/v1.0.0-rc.3/nvidia-docker_1.0.0.rc.3-1_amd64.deb     --2016-07-14 22:57:30--  https://github.com/NVIDIA/nvidia-docker/releases/download/v1.0.0-rc.3/nvidia-docker_1.0.0.rc.3-1_amd64.deb
Resolving web-proxy.xxxxxx.hp.com (web-proxy.xxxxxx.hp.com)... xxx.xxx.xxx.xxx
Connecting to web-proxy.xxxxxx.hp.com (web-proxy.xxxxxx.hp.com)|xxx.xxx.xxx.xxx|:8080... connected.
Proxy request sent, awaiting response... 302 Found
......

So we can conclude that wget is picky about uppercase and lowercase words.

 

Build docker from source behind proxy

If you want to build Docker from source like this:

# git clone https://github.com/docker/docker.git
# cd docker
# make

But your working server is actually behind a proxy, I think you may run into errors as the follows:

# make
mkdir bundles
docker build  -t "docker-dev:master" -f "Dockerfile" .
Sending build context to Docker daemon 145.8 MB
Step 1 : FROM debian:jessie
 ---> f854eed3f31f
Step 2 : RUN apt-key adv --keyserver hkp://p80.pool.sks-keyservers.net:80 --recv-keys E871F18B51E0147C77796AC81196BA81F6B0FC61  || apt-key adv --keyserver hkp://pgp.mit.edu:80 --recv-keys E871F18B51E0147C77796AC81196BA81F6B0FC61
 ---> Running in fede03b56767
Executing: gpg --ignore-time-conflict --no-options --no-default-keyring --homedir /tmp/tmp.MjO7kIEOm8 --no-auto-check-trustdb --trust-model always --primary-keyring /etc/apt/trusted.gpg --keyring /etc/apt/trusted.gpg.d/debian-archive-jessie-automatic.gpg --keyring /etc/apt/trusted.gpg.d/debian-archive-jessie-security-automatic.gpg --keyring /etc/apt/trusted.gpg.d/debian-archive-jessie-stable.gpg --keyring /etc/apt/trusted.gpg.d/debian-archive-squeeze-automatic.gpg --keyring /etc/apt/trusted.gpg.d/debian-archive-squeeze-stable.gpg --keyring /etc/apt/trusted.gpg.d/debian-archive-wheezy-automatic.gpg --keyring /etc/apt/trusted.gpg.d/debian-archive-wheezy-stable.gpg --keyserver hkp://p80.pool.sks-keyservers.net:80 --recv-keys E871F18B51E0147C77796AC81196BA81F6B0FC61
gpg: requesting key F6B0FC61 from hkp server p80.pool.sks-keyservers.net
gpgkeys: key E871F18B51E0147C77796AC81196BA81F6B0FC61 can't be retrieved
gpg: no valid OpenPGP data found.
gpg: Total number processed: 0
Executing: gpg --ignore-time-conflict --no-options --no-default-keyring --homedir /tmp/tmp.6clUfj4AwL --no-auto-check-trustdb --trust-model always --keyring /etc/apt/trusted.gpg --primary-keyring /etc/apt/trusted.gpg --keyring /etc/apt/trusted.gpg.d/debian-archive-jessie-automatic.gpg --keyring /etc/apt/trusted.gpg.d/debian-archive-jessie-security-automatic.gpg --keyring /etc/apt/trusted.gpg.d/debian-archive-jessie-stable.gpg --keyring /etc/apt/trusted.gpg.d/debian-archive-squeeze-automatic.gpg --keyring /etc/apt/trusted.gpg.d/debian-archive-squeeze-stable.gpg --keyring /etc/apt/trusted.gpg.d/debian-archive-wheezy-automatic.gpg --keyring /etc/apt/trusted.gpg.d/debian-archive-wheezy-stable.gpg --keyserver hkp://pgp.mit.edu:80 --recv-keys E871F18B51E0147C77796AC81196BA81F6B0FC61
gpg: requesting key F6B0FC61 from hkp server pgp.mit.edu
gpgkeys: key E871F18B51E0147C77796AC81196BA81F6B0FC61 can't be retrieved
gpg: no valid OpenPGP data found.
gpg: Total number processed: 0
The command '/bin/sh -c apt-key adv --keyserver hkp://p80.pool.sks-keyservers.net:80 --recv-keys E871F18B51E0147C77796AC81196BA81F6B0FC61 || apt-key adv --keyserver hkp://pgp.mit.edu:80 --recv-keys E871F18B51E0147C77796AC81196BA81F6B0FC61' returned a non-zero code: 2
Makefile:70: recipe for target 'build' failed
make: *** [build] Error 1

Or:

......
RUN apt-get update && apt-get install -y       apparmor        apt-utils       aufs-tools      automake        bash-completion   binutils-mingw-w64       bsdmainutils    btrfs-tools     build-essential         clang   createrepo      curl    dpkg-sig        gcc-mingw-w64      git     iptables        jq      libapparmor-dev         libcap-dev      libltdl-dev     libsqlite3-dev  libsystemd-journal-dev  libtool    mercurial       net-tools       pkg-config      python-dev      python-mock     python-pip      python-websocket        ubuntu-zfs xfsprogs        libzfs-dev      tar     zip     --no-install-recommends         && pip install awscli==1.10.15
 ---> Running in 37080c364862
Get:1 http://ppa.launchpad.net trusty InRelease [8127 B]
Get:2 http://httpredir.debian.org jessie InRelease [8127 B]
Get:3 http://security.debian.org jessie/updates InRelease [8127 B]
Splitting up /var/lib/apt/lists/partial/ppa.launchpad.net_zfs-native_stable_ubuntu_dists_trusty_InRelease into data and signature failedIgn http://ppa.launchpad.net trusty InRelease
E: GPG error: http://ppa.launchpad.net trusty InRelease: Clearsigned file isn't valid, got 'NODATA' (does the network require authentication?)
The command '/bin/sh -c apt-get update && apt-get install -y    apparmor        apt-utils       aufs-tools      automake        bash-completion    binutils-mingw-w64      bsdmainutils    btrfs-tools     build-essential         clang   createrepo      curl    dpkg-sig        gcc-mingw-w64      git     iptables        jq      libapparmor-dev         libcap-dev      libltdl-dev     libsqlite3-dev  libsystemd-journal-dev     libtool         mercurial       net-tools       pkg-config      python-dev      python-mock     python-pip      python-websocket  ubuntu-zfs       xfsprogs        libzfs-dev      tar     zip     --no-install-recommends         && pip install awscli==1.10.15' returned a non-zero code: 100
make: *** [build] Error 1

These reports can make you crazy!

The solution is adding proxy into Dockerfile which resides in the root directory of Docker folder:

......
FROM debian:jessie
ENV http_proxy http://web-proxy.corp.xxxxxx.com:8080/
ENV https_proxy https://web-proxy.corp.xxxxxx.com:8080/
......

Then the make progress will be smooth!

P.S., after discussing in reddit, the correct and idiomatic method should be this:

make DOCKER_BUILD_ARGS="--build-arg http_proxy=http://web-proxy.corp.xxxxxx.com:8080/ --build-arg https_proxy=https://web-proxy.corp.xxxxxx.com:8080/"

Install docker on Ubuntu 14.04

If you want to play docker on Ubuntu 14.04, please pay attention to the installation instruction: it is “apt-get install docker.io“, not “apt-get install docker“. You can find the difference between them by following command:

# apt-cache search docker
......
docker - System tray for KDE3/GNOME2 docklet applications
......
docker.io - Linux container runtime
......

OK! Since you have set up docker successfully, you can check its process now:

# ps -ef | grep docker
root       4715      1  0 13:22 ?        00:00:00 /usr/bin/docker -d
root       4857   4691  0 13:50 pts/0    00:00:00 grep --color=auto docker
# pstree -ps 4715
init(1)───docker(4715)─┬─{docker}(4717)
                       ├─{docker}(4722)
                       ├─{docker}(4723)
                       ├─{docker}(4724)
                       ├─{docker}(4734)
                       ├─{docker}(4754)
                       ├─{docker}(4762)
                       ├─{docker}(4769)
                       └─{docker}(4793)

You can use “service start docker” and “service stop docker” to start and stop docker daemon.

If your host runs behind proxy, you may meet problems when pulling image:

# docker run hell-world
Unable to find image 'hell-world:latest' locally
Pulling repository hell-world
FATA[0005] Get https://index.docker.io/v1/repositories/library/hell-world/images: x509: certificate is valid for FG3K6C3A15800021, not index.docker.io

The solution is add proxy configurations in /etc/default/docker:

......
# If you need Docker to use an HTTP proxy, it can also be specified here.
export http_proxy="http://web-proxy.corp.xxxxxx.com:8080/"
export https_proxy="https://web-proxy.corp.xxxxxx.com:8080/"
......

Then you can download images successfully:

# docker run hello-world
Unable to find image 'hello-world:latest' locally
latest: Pulling from hello-world
d59cd4c39e50: Pull complete
f1d956dc5945: Pull complete
Digest: sha256:4f32210e234b4ad5cac92efacc0a3d602b02476c754f13d517e1ada048e5a8ba
Status: Downloaded newer image for hello-world:latest

Hello from Docker.
This message shows that your installation appears to be working correctly.
......

Now that all the preparations are ready, please enjoy playing docker!

 

How to pull docker image behind proxy on RHEL7?

My host OS is RHEL7, and running behind proxy. The output of executing docker run hello-world is like this:

# docker run hello-world
Unable to find image 'hello-world:latest' locally
Trying to pull repository registry.access.redhat.com/hello-world ... failed
Trying to pull repository docker.io/library/hello-world ... failed
Error while pulling image: Get https://index.docker.io/v1/repositories/library/hello-world/images: x509: certificate is valid for FG3K6C3A15800021, not index.docker.io

It prompts pull image failed, so I need to configure proxy to make docker work correctly:

(1) Add proxy info in /etc/sysconfig/docker file:

HTTP_PROXY="http://web-proxy.corp.xxxxxx.com:8080"
HTTPS_PROXY="http://web-proxy.corp.xxxxxx.com:8080"
http_proxy="${HTTP_PROXY}"
https_proxy="${HTTPS_PROXY}"

(2) Restart docker service:

# service docker restart

Then docker works OK now:

# docker run hello-world
Unable to find image 'hello-world:latest' locally
Trying to pull repository registry.access.redhat.com/hello-world ... not found
Trying to pull repository docker.io/library/hello-world ... latest: Pulling from library/hello-world
3f12c794407e: Pull complete
975b84d108f1: Pull complete
......

References:
Cannot download Docker images behind a proxy

Set proxy for Maven

Sometimes, your Maven needs proxy to download object:

[INFO] Scanning for projects...
Downloading: http://repo.maven.apache.org/maven2/org/apache/apache/11/apache-11.pom
[ERROR] The build could not read 1 project -> [Help 1]
[ERROR]
[ERROR]   The project org.apache.mesos:mesos:0.25.0 (/home/nan/mesos-0.25.0/build/src/java/mesos.pom) has 1 error
[ERROR]     Non-resolvable parent POM: Could not transfer artifact org.apache:apache:pom:11 from/to central (http://repo.maven.apache.org/maven2): Connection to http://repo.maven.apache.org refused and 'parent.relativePath' points at wrong local POM @ line 18, column 11: Connection timed out -> [Help 2]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/ProjectBuildingException
[ERROR] [Help 2] http://cwiki.apache.org/confluence/display/MAVEN/UnresolvableModelException

The method is configuring proxy setting in ${HOME}/.m2/settings.xml:

$ echo ${HOME}
/home/nan
$ cat ${HOME}/.m2/settings.xml
<settings>
        <proxies>
                <proxy>
                        <active>true</active>
                        <protocol>http</protocol>
                        <host>web-proxy.xxxxxx.com</host>
                        <port>8080</port>
                </proxy>
        </proxies>
</settings>

Then it can work!

P.S. My Maven version is 3.0.5.

 

Powered by WordPress & Theme by Anders Norén