Deploy Docker Swarm cluster on one host

Sometimes, you just want to learn the internal mechanics of Docker Swarm, but unfortunately there is only one Linux box at hand, and you don’t want to bother to install Virtual Machines on it. In this scenario, you certainly can build a Docker Swarm cluster on one host, and this tutorial will provide a detailed guide:

(1) Make sure the Go environment has been ready on your system, if not, please follow this document to setup it. Also remember add$GOPATH/bin into $PATH environment variable.

(2) Install Docker Swarm:

# go get -u github.com/docker/swarm

Execute swarm command to check whether Docker Swarm is well equipped:

# swarm
Usage: swarm [OPTIONS] COMMAND [arg...]

A Docker-native clustering system

Version: 1.2.3 (HEAD)

Options:
  --debug                       debug mode [$DEBUG]
  --log-level, -l "info"        Log level (options: debug, info, warn, error, fatal, panic)
  --experimental                enable experimental features
  --help, -h                    show help
  --version, -v                 print the version
......

(3) Modify the Docker configuration file. E.g., on my RHEL 7, the file is /etc/sysconfig/docker:

# systemctl show docker
......
EnvironmentFile=/etc/sysconfig/docker (ignore_errors=yes)
......

Add “-H tcp://127.0.0.1:2375” in OPTIONS field:

# cat /etc/sysconfig/docker
# /etc/sysconfig/docker

# Modify these options if you want to change the way the docker daemon runs
OPTIONS='--selinux-enabled -H tcp://127.0.0.1:2375 -H unix:///var/run/docker.sock'

Restart Docker, and check whether the new OPTIONS takes effect:

# systemctl restart docker
# systemctl status docker
● docker.service - Docker Application Container Engine
   Loaded: loaded (/usr/lib/systemd/system/docker.service; disabled; vendor preset: disabled)
   Active: active (running) since Wed 2016-06-08 12:32:19 CST; 10s ago
     Docs: http://docs.docker.com
 Main PID: 14429 (sh)
   CGroup: /system.slice/docker.service
           ├─14429 /bin/sh -c /usr/bin/docker-current daemon $OPTIONS            $DOCKER_STORAGE_OPTIONS            $DOCKER_NETWORK_OPTI...
           ├─14430 /usr/bin/docker-current daemon --selinux-enabled -H tcp://127.0.0.1:2375 -H unix:///var/run/docker.sock --add-registr...
           └─14431 /usr/bin/forward-journald -tag docker
......

(4) Run “swarm create” command to create token for the cluster:

# swarm create
d10eacbda9763b0740548a2a4c2f1a59

(5) Execute swarm join to create a Docker Swarm node:

# swarm join --addr 127.0.0.1:2375 token://d10eacbda9763b0740548a2a4c2f1a59
INFO[0000] Registering on the discovery service every 1m0s...  addr=127.0.0.1:2375 discovery=token://d10eacbda9763b0740548a2a4c2f1a59
......

You should notice that the argument of --addr option is the IP and port of the Docker engine on this host. Since we have set theOPTIONS in Docker configuration file in step 3, the IP should be 127.0.0.1 whilst port is 2375.

(6) Open a new terminal, and create the manager of the cluster. Because port 2375 is occupied by Docker engine, we use another available port:

# swarm manage -H 127.0.0.1:3375 token://d10eacbda9763b0740548a2a4c2f1a59
INFO[0000] Listening for HTTP                            addr=127.0.0.1:3375 proto=tcp
INFO[0001] Registered Engine localhost.localdomain at 127.0.0.1:2375

Through the log, you can see the node and manager have communicated successfully.

Now, you can think a Docker engine is listening on tcp://127.0.0.1:3375, but actually, there is one Docker cluster behindtcp://127.0.0.1:3375, even though the cluster has only one node. You can play docker client commands now, such as get the cluster info:

# docker -H tcp://127.0.0.1:3375 info
Containers: 0
Images: 5
Server Version: swarm/1.2.3
Role: primary
Strategy: spread
Filters: health, port, containerslots, dependency, affinity, constraint
Nodes: 1
 localhost.localdomain: 127.0.0.1:2375
  └ ID: ZUIV:BMPV:3B5R:2WBC:JXEI:2S6H:XM3H:66W5:UZQI:NJON:JY4T:HIFB
  └ Status: Healthy
  └ Containers: 0 (0 Running, 0 Paused, 0 Stopped)
  └ Reserved CPUs: 0 / 8
  └ Reserved Memory: 0 B / 12.1 GiB
  └ Labels: executiondriver=native-0.2, kernelversion=3.10.0-327.el7.x86_64, operatingsystem=Red Hat Network, storagedriver=devicemapper
  └ UpdatedAt: 2016-06-08T04:58:05Z
  └ ServerVersion: 1.9.1
Kernel Version: 3.10.0-327.el7.x86_64
......

Or run a container:

# docker -H tcp://127.0.0.1:3375 run hello-world

Hello from Docker.
This message shows that your installation appears to be working correctly.

To generate this message, Docker took the following steps:
 1. The Docker client contacted the Docker daemon.
......

Enjoy Docker Swarm now!

Reference:
Swarm docs;
Docker Swarm Tutorial and Examples.

 

The tips of learning Linux kernel

As the Linux kernel has become one of the most gigantic and complex software project in the world, its complication scare many novices away. In this post, I will give some personal experience on how to learn Linux kernel, and hope these tips can offer some help to newcomers.

(1) Download vanilla kernel and install it.

Yes, I suggest you can find a physical machine, or if you really don’t have one at hand, virtual machine is also OK. Download the newest vanilla kernel from kernel.org, then build and install it. This process isn’t too hard and makes you conquer the fear of Linux kernel. After your first successful setting up of Linux kernel, and read the release version number from uname -r output:

# uname -r
4.6.0

I think this will enable you gain more confidence.

(2) Study the elementary skills of Linux kernel programming.

Looking back when you begin user-space C programming on *nix platform, you need to know allocating memory through malloc; opening file through fopen/open; using pthread library to construct concurrent program, and so on. Linux is nothing more than a platform, and you also need to study the rules of playing with it. For example, you should be familiar with how to tweak list (list.h); giving out memory should use kmalloc, etc. There are many classical books and tutorials elaborate these knowledge. Although some posts seem outdated (the version of kernel is still 2.6.x.), but they are also applicable to current.

(3) Dive into one module.

Once you get the basic expertise of Linux kernel programming, you should focus on one aspect of the kernel. If you are a full-time kernel programmer, congratulations! You should concentrate on your work area and try to be the expert of this domain. If kernel is just your hobby, you should select one module which you have great interest on. I.e., if you are curious about debugging, kdump should be your taste; if you pay close attention to dynamic tracing, BPF will be the right stuff which you want to find. After picking out the part you want to contribute, you should dig into the code and attempt to master every detail of it. You should also subscribe the related mailing list to acquaint the newest progress. The final goal is to check in meaningful patches for kernel, from a trivial typo to an enhanced feature. Think your code will run on millions of thousands of devices, it is really amazing!

(4) Others

When you meet an issue, you can try to get help from mailing list or forums. You can also try to take part in local community to recognize people in the same camp. Anyway, Endeavor to utilize all the resource you can find.

Happy hacking!

 

Install docker on Ubuntu 14.04

If you want to play docker on Ubuntu 14.04, please pay attention to the installation instruction: it is “apt-get install docker.io“, not “apt-get install docker“. You can find the difference between them by following command:

# apt-cache search docker
......
docker - System tray for KDE3/GNOME2 docklet applications
......
docker.io - Linux container runtime
......

OK! Since you have set up docker successfully, you can check its process now:

# ps -ef | grep docker
root       4715      1  0 13:22 ?        00:00:00 /usr/bin/docker -d
root       4857   4691  0 13:50 pts/0    00:00:00 grep --color=auto docker
# pstree -ps 4715
init(1)───docker(4715)─┬─{docker}(4717)
                       ├─{docker}(4722)
                       ├─{docker}(4723)
                       ├─{docker}(4724)
                       ├─{docker}(4734)
                       ├─{docker}(4754)
                       ├─{docker}(4762)
                       ├─{docker}(4769)
                       └─{docker}(4793)

You can use “service start docker” and “service stop docker” to start and stop docker daemon.

If your host runs behind proxy, you may meet problems when pulling image:

# docker run hell-world
Unable to find image 'hell-world:latest' locally
Pulling repository hell-world
FATA[0005] Get https://index.docker.io/v1/repositories/library/hell-world/images: x509: certificate is valid for FG3K6C3A15800021, not index.docker.io

The solution is add proxy configurations in /etc/default/docker:

......
# If you need Docker to use an HTTP proxy, it can also be specified here.
export http_proxy="http://web-proxy.corp.xxxxxx.com:8080/"
export https_proxy="https://web-proxy.corp.xxxxxx.com:8080/"
......

Then you can download images successfully:

# docker run hello-world
Unable to find image 'hello-world:latest' locally
latest: Pulling from hello-world
d59cd4c39e50: Pull complete
f1d956dc5945: Pull complete
Digest: sha256:4f32210e234b4ad5cac92efacc0a3d602b02476c754f13d517e1ada048e5a8ba
Status: Downloaded newer image for hello-world:latest

Hello from Docker.
This message shows that your installation appears to be working correctly.
......

Now that all the preparations are ready, please enjoy playing docker!

 

A brief intro of delve

delve is a debugger developed in Golang and also dedicated to help trouble-shooting Golang programs (Home page is here). Though it is still in pre-1.0 release, I think it is stable enough for daily use. BTW, if you find some bugs, you can report it to developers and help to make delve more stronger! P.S., Albeit fmt.Print buddies are useful in most cases, I strongly recommend you try to usedelve to inspect the internal mechanism of your code, because it can help you know Golang deeper, not just superficial stuff.

Installing delve is very simple: taking Linux platform as an example, it is no different from setting up other Golang projects, just “go get” is enough:

go get github.com/derekparker/delve/cmd/dlv 

Now, in $GOPATH/bin, there is an extra dlv executable binary file (Notice: the project is named delve, while the executable file is calleddlv. I even made a foolish mistake when began to use it).Run dlv command, and it will show you a detailed manual of delve:

# dlv
Delve is a source level debugger for Go programs.

Delve enables you to interact with your program by controlling the execution of the process,
evaluating variables, and providing information of thread / goroutine state, CPU register state and more.

The goal of this tool is to provide a simple yet powerful interface for debugging Go programs.

Usage:
  dlv [command]

Available Commands:
  version     Prints version.
  run         Deprecated command. Use 'debug' instead.
  debug       Compile and begin debugging program.
......

Let’s check this artificial Hello.go program:

package main

import "fmt"

func main() {
        var s []byte
        s = append(s, []byte("Hello, Debugging!")...)
        fmt.Println(string(s))
}

Use delve to debug it:

# dlv debug Hello.go
Type 'help' for list of commands.
(dlv) help
The following commands are available:
    help (alias: h) ------------- Prints the help message.
    break (alias: b) ------------ Sets a breakpoint.
    trace (alias: t) ------------ Set tracepoint.
    restart (alias: r) ---------- Restart process.
    continue (alias: c) --------- Run until breakpoint or program termination.
    step (alias: s) ------------- Single step through program.
    step-instruction (alias: si)  Single step a single cpu instruction.
    next (alias: n) ------------- Step over to next source line.
    threads --------------------- Print out info for every traced thread.
    thread (alias: tr) ---------- Switch to the specified thread.
......

If you are familiar with gdb, you will find the commands are very similar, and I promise you can master delve soon.

An interesting thing is that the delve doesn’t provide start command which gdb offers, so you should try to set breakpoints first, then run continue command:

(dlv) b Hello.go:8
Breakpoint 1 set at 0x4011ea for main.main() ./Hello.go:8
(dlv) c
> main.main() ./Hello.go:8 (hits goroutine(1):1 total:1) (PC: 0x4011ea)
     3: import "fmt"
     4:
     5: func main() {
     6:         var s []byte
     7:         s = append(s, []byte("Hello, Debugging!")...)
=>   8:         fmt.Println(string(s))
     9: }
(dlv) p s
[]uint8 len: 17, cap: 32, [72,101,108,108,111,44,32,68,101,98,117,103,103,105,110,103,33]
(dlv) goroutines
[4 goroutines]
* Goroutine 1 - User: ./Hello.go:8 main.main (0x4011ea)
  Goroutine 2 - User: /usr/local/go/src/runtime/proc.go:263 runtime.gopark (0x42a153)
  Goroutine 3 - User: /usr/local/go/src/runtime/proc.go:263 runtime.gopark (0x42a153)
  Goroutine 4 - User: /usr/local/go/src/runtime/mfinal.go:144 runtime.runfinq (0x413f80)

Cool! Isn’t it? Now You can observe almost everything you want to know about your program.

Happy Debugging! Happy delving!

 

Watch out for the permission of “tmp” directory during installation of SAP HANA

My colleague creates a SLES11SP3 docker image, and wants to install SAP HANA database in the SLES11SP3 container. But the error occurs during installation:

# ./hdbinst
......
Creating instance...
  hdbparam: Working configuration directory:  "/hana/shared/H00/global/hdb/custom/config"
  hdbnsutil: creating persistence ...
  hdbnsutil: writing initial topology...
  hdbnsutil: writing initial license: status check = 2
Installation failed
  error installing
    Cannot create Instance
      Cannot start sapstartsrv
        Waiting for sapstartsrv failed: timeout reached (120)
        Waiting for sapstartsrv failed: timeout reached (120)

Log file written to '/var/tmp/hdb_H00_install_2016-05-04_19.18.27/hdbinst.log' on host 'fe769d9f6bae'.

Check the /var/tmp/hdb_H00_install_2016-05-04_19.18.27/hdbinst.log:

19:22:42.406 - INFO:     Starting service
19:22:42.406 - INFO:       Starting external program /usr/sap/H00/HDB00/exe/sapstartsrv
19:22:42.406 - INFO:         Command line is: /usr/sap/H00/HDB00/exe/sapstartsrv pf=/hana/shared/H00/profile/H00_HDB00_fe769d9f6bae -D -u h00adm
19:22:42.438 - INFO:         Output line 1: Impromptu CCC initialization by 'rscpCInit'.
19:22:42.438 - INFO:         Output line 2:   See SAP note 1266393.
19:22:42.726 - INFO:         Output line 3: Impromptu CCC initialization by 'rscpCInit'.
19:22:42.726 - INFO:         Output line 4:   See SAP note 1266393.
19:22:42.857 - INFO:         Program terminated with exit code 0
19:22:42.857 - INFO:     Waiting for sapstartsv...
19:22:42.862 - INFO:       sapstartsrv is not running: Net::HTTPS: connect: Connection refused
19:22:43.864 - INFO:       sapstartsrv is not running: Net::HTTPS: connect: Connection refused
......
19:24:43.091 - ERR :       Waiting for sapstartsrv failed: timeout reached (120)
19:24:43.092 - INFO:     Checking unix domain socket
19:24:43.093 - ERR :       Cannot establish http connection to unix domain socket '/tmp/.sapstream50013' (No such file or directory)
19:24:43.093 - INFO:       sapstartsrv is not running: connect: No such file or directory
19:24:44.094 - ERR :       Cannot establish http connection to unix domain socket '/tmp/.sapstream50013' (No such file or directory)
19:24:44.094 - INFO:       sapstartsrv is not running: connect: No such file or directory
......

After tough debugging, the reason is the tmp folder doesn’t grant write permissions to users except root:

drwxr-xr-x    2 root root  4096 May  4 19:17 tmp

Change the permission of /tmp:

# chmod a+w /tmp

Then the installation is successful!