*NIX & Hacking —— 第2期

做一本我感兴趣的杂志,就这么简单!

Debug

Introduction to Debuggers

Docker

DOCKER MACHINE 0.3.0 DEEP DIVE

Git

First aid git

Golang

go-lang-cheat-sheet

Kernel

Kernel and userspace tracing: with LTTng and his friends

Regular Expression

Five Invaluable Techniques to Improve Regex Performance

Rust

First Rust Program Pain (So you can avoid it…)

Scala

Various ways to run Scala code
Why does the same scala code work OK in command line while not in Intellij?

Unix

A little collection of cool unix terminal/console/curses tools
How big is the pipe buffer?
Stack Smashing On A Modern Linux System
Unix history repository

Easter egg

DMA (Direct Memory Access)
Get Unix Jobs
Here are 13 tech jobs that pay at least $130,000 across the country

搭建Spark开发环境

本文使用docker搭建Spark环境,使用的image文件是sequenceiq提供的1.3.0版本

首先pull Spark image文件:

docker pull sequenceiq/spark:1.3.0

pull成功后,运行Spark

docker run -i -t -h sandbox sequenceiq/spark:1.3.0 bash

测试Spark是否工作正常:

bash-4.1# spark-shell --master yarn-client --driver-memory 1g --executor-memory 1g --executor-cores 1
......
scala> sc.parallelize(1 to 1000).count()
......
res0: Long = 1000

输出1000,OK!

(1)启动spark-shell,输出log很多,解决方法如下:
a)把/usr/local/spark/conf文件夹下的log4j.properties.template文件复制生成一份log4j.properties文件:

bash-4.1# cd /usr/local/spark/conf
bash-4.1# cp log4j.properties.template log4j.properties

b)把log4j.properties文件里的“log4j.rootCategory=INFO, console”改成“log4j.rootCategory=WARN, console”即可。

(2)启动spark-shell会有以下warning

15/05/25 04:49:28 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

提示找不到hadoop的库文件,解决办法如下:

export LD_LIBRARY_PATH=/usr/local/hadoop/lib/native/:$LD_LIBRARY_PATH

请参考stackoverflow的相关讨论:
a)Hadoop “Unable to load native-hadoop library for your platform” error on CentOS
b)Hadoop “Unable to load native-hadoop library for your platform” error on docker-spark?

(3)在Quick Start中提到如下例子:

scala> val textFile = sc.textFile("README.md")
......
scala> textFile.count() // Number of items in this RDD

执行会有错误:

scala> textFile.count()
org.apache.hadoop.mapred.InvalidInputException: Input path does not exist: hdfs://sandbox:9000/user/root/README.md
        at org.apache.hadoop.mapred.FileInputFormat.singleThreadedListStatus(FileInputFormat.java:285)
        at org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:228)
        at org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:304)
        at org.apache.spark.rdd.HadoopRDD.getPartitions(HadoopRDD.scala:203)
        at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:219)
        at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:217)
        at scala.Option.getOrElse(Option.scala:120)
        at org.apache.spark.rdd.RDD.partitions(RDD.scala:217)

可以看到程序尝试从hdfs中寻找文件,所以报错。

解决方法有两种:
a) 指定本地文件系统:

scala> val textFile = sc.textFile("file:///usr/local/spark/README.md")
textFile: org.apache.spark.rdd.RDD[String] = file:///usr/local/spark/README.md MapPartitionsRDD[3] at textFile at <console>:21

scala> textFile.count()
res1: Long = 98

b)上传文件到hdfs上:

bash-4.1# hadoop fs -put /usr/local/spark/README.md README.md

接着运行spark-shell:

bash-4.1# spark-shell --master yarn-client --driver-memory 1g --executor-memory 1g --executor-cores 1
Spark assembly has been built with Hive, including Datanucleus jars on classpath
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 1.3.0
      /_/

Using Scala version 2.10.4 (Java HotSpot(TM) 64-Bit Server VM, Java 1.7.0_51)
Type in expressions to have them evaluated.
Type :help for more information.
15/05/25 05:22:15 WARN Client: SPARK_JAR detected in the system environment. This variable has been deprecated in favor of the spark.yarn.jar configuration variable.
15/05/25 05:22:15 WARN Client: SPARK_JAR detected in the system environment. This variable has been deprecated in favor of the spark.yarn.jar configuration variable.
Spark context available as sc.
SQL context available as sqlContext.

scala> val textFile = sc.textFile("README.md")
textFile: org.apache.spark.rdd.RDD[String] = README.md MapPartitionsRDD[1] at textFile at <console>:21

scala> textFile.count()
res0: Long = 98

参考邮件:
Spark Quick Start – call to open README.md needs explicit fs prefix

P.S.在主机(非docker环境)下载sparkhttps://spark.apache.org/downloads.html)运行时,会有以下warning

log4j:WARN No appenders could be found for logger (org.apache.hadoop.metrics2.lib.MutableMetricsFactory).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.

解决办法是把/path/to/spark/conf文件夹下的log4j.properties.template文件复制生成一份log4j.properties文件即可。

参考stackoverflow的讨论:
log4j:WARN No appenders could be found for logger (running jar file, not web app)

docker笔记(4)—— 如何进入一个正在运行的“docker container”?

docker attach [container-id]”命令有时会hang住,执行Ctrl+C命令也不起作用:

[root@localhost ~]# docker attach a972e69ab444
^C^C^C^C^C^C^C^C^C^C

使用pstack命令查看函数调用栈:

[root@localhost ~]# pstack 29744
Thread 5 (Thread 0x7f9079bd8700 (LWP 29745)):
#0  runtime.futex () at /usr/lib/golang/src/pkg/runtime/sys_linux_amd64.s:269
#1  0x0000000000417717 in runtime.futexsleep () at /usr/lib/golang/src/pkg/runtime/os_linux.c:49
#2  0x0000000001161c58 in runtime.sched ()
#3  0x0000000000000000 in ?? ()
Thread 4 (Thread 0x7f90792d7700 (LWP 29746)):
#0  runtime.futex () at /usr/lib/golang/src/pkg/runtime/sys_linux_amd64.s:269
#1  0x0000000000417782 in runtime.futexsleep () at /usr/lib/golang/src/pkg/runtime/os_linux.c:55
#2  0x00007f907b830f60 in ?? ()
#3  0x0000000000000000 in ?? ()
Thread 3 (Thread 0x7f9078ad6700 (LWP 29747)):
#0  runtime.futex () at /usr/lib/golang/src/pkg/runtime/sys_linux_amd64.s:269
#1  0x0000000000417717 in runtime.futexsleep () at /usr/lib/golang/src/pkg/runtime/os_linux.c:49
#2  0x00000000011618a0 in text/template.zero ()
#3  0x0000000000000000 in ?? ()
Thread 2 (Thread 0x7f9073fff700 (LWP 29748)):
#0  runtime.futex () at /usr/lib/golang/src/pkg/runtime/sys_linux_amd64.s:269
#1  0x0000000000417717 in runtime.futexsleep () at /usr/lib/golang/src/pkg/runtime/os_linux.c:49
#2  0x000000c2080952f0 in ?? ()
#3  0x0000000000000000 in ?? ()
Thread 1 (Thread 0x7f907b9e1800 (LWP 29744)):
#0  runtime.epollwait () at /usr/lib/golang/src/pkg/runtime/sys_linux_amd64.s:385
#1  0x00000000004175dd in runtime.netpoll () at /usr/lib/golang/src/pkg/runtime/netpoll_epoll.c:78
#2  0x00007fff00000004 in ?? ()
#3  0x00007fff58720fd0 in ?? ()
#4  0xffffffff00000080 in ?? ()
#5  0x0000000000000000 in ?? ()

可以使用“docker exec -it [container-id] bash”命令进入正在运行的container

[root@localhost ~]# docker exec -it a972e69ab444 bash
bash-4.1# ls
bin   dev  home  lib64  mnt  pam-1.1.1-17.el6.src.rpm  root      sbin     srv  tmp  var
boot  etc  lib   media  opt  proc                      rpmbuild  selinux  sys  usr
bash-4.1#

docker attach相当于复用了container当前使用的tty,因此在docker attach内执行exit,会导致正在运行的container退出。而docker exec会新建立一个tty,在docker exec中执行exit不会导致container退出。

 

参考资料:
(1)Docker – Enter Running Container with new TTY

docker笔记(3)—— selinux导致docker工作不正常

最近几天在研究docker备份文件(操作系统是RHEL7docker版本是1.5.0)。仿照docker文档,执行如下命令:

[[email protected] data]#docker create -v /dbdata --name dbdata training/postgres /bin/true
[[email protected] data]#docker run -d --volumes-from dbdata --name db1 training/postgres
[[email protected] data]# docker run --volumes-from dbdata -v $(pwd):/backup ubuntu tar cvf /backup/backup.tar /dbdata
tar: /backup/backup.tar: Cannot open: Permission denied
tar: Error is not recoverable: exiting now

看到Permission denied这个提示,自然首先怀疑用户没有写权限的问题。检查一下当前目录的权限:

[[email protected] data]# ls -alt
total 4
drwxrwxrwx.  2 root root    6 May  7 21:33 .
drwxrwx-w-. 15 root root 4096 May  7 21:33 ..

应该是没问题的。经过在stackoverflow上的一番讨论,得到的建议是有可能是selinux捣的鬼。查看了一下selinux状态:

[root@localhost root]# sestatus
SELinux status:                 enabled
SELinuxfs mount:                /sys/fs/selinux
SELinux root directory:         /etc/selinux
Loaded policy name:             targeted
Current mode:                   enforcing
Mode from config file:          enforcing
Policy MLS status:              enabled
Policy deny_unknown status:     allowed
Max kernel policy version:      28

果断把模式改为permissive:

[root@localhost data]# setenforce 0
[root@localhost data]# sestatus
SELinux status:                 enabled
SELinuxfs mount:                /sys/fs/selinux
SELinux root directory:         /etc/selinux
Loaded policy name:             targeted
Current mode:                   permissive
Mode from config file:          enforcing
Policy MLS status:              enabled
Policy deny_unknown status:     allowed
Max kernel policy version:      28

马上工作正常:

[root@localhost data]# docker run --volumes-from dbdata -v $(pwd):/backup ubuntu tar cvf /backup/backup.tar /dbdata
tar: Removing leading `/' from member names
/dbdata/

因为时间原因,没有往下深究。总之,在使用docker时,要留意一下selinux,有可能会引起很奇怪的问题。

更新:

最近又碰到这个问题,可以参考这篇总结

参考资料:
(1)Why does docker prompt “Permission denied” when backing up the data volume?
(2)How to disable SELinux without restart?
(3)Quick-Tip: Turning off or disabling SELinux

使用gitlab docker image搭建git server

这两天折腾了一下gitlab,遇到一些问题,记录一下,以便日后查阅。

(1)gitlab使用的是docker image(下载地址:https://registry.hub.docker.com/u/genezys/gitlab/),这个比较顺利,按README操作即可。

(2)接下来,就是用git client(版本:1.8.3.1)访问git server,这个过程相当痛苦。把结论总结在这里:

a)由于gitlab container22端口会映射到宿主机的2222端口,所以需要在“~/.ssh/config”文件中加上“Port 2222”这一句;

b)使用“git clone”命令clone项目。其中repository地址可以从gitlab web页面查到。例如:“[email protected]:root/test.git”,但要注意的是需要把IP改成宿主机的IP。举例,如果宿主机IP10.137.20.113,则改为“[email protected]:root/test.git”;

c)最后需要指出的是,如果git clone命令不指定目的文件夹,则默认目的文件夹为xxx.git中的xxx。以上面命令为例,则目的文件夹为test。一定要确保目的文件夹不存在,或者里面没有内容。否则会提示“fatal: destination path 'xxx' already exists and is not an empty directory.”。

P.S. 关于git clone命令访问非标准SSH端口,也可参考我的这篇文章

参考资料:
1)How can I use git client to access gitlab docker?
2)How to get Git to clone into current directory

docker笔记(2)——“docker daemon”,“image”和“container”

使用“service docker start”命令实际上是启动docker daemon程序,可以用“ps”命令查看一下:

[root@localhost ~]# ps -ef | grep docker
root     24791     1  0 Mar18 ?        00:00:03 /usr/bin/docker -d --selinux-enabled
root     24969 24950  0 03:25 pts/0    00:00:00 grep --color=auto docker

-d”选项表明开启daemon模式。

Docker中最重要的概念是imagescontainersimages可以比做虚拟机软件(VirtualBox或者VMware)的虚拟磁盘镜像文件(VirtualBox的格式是*.vdiVMware*.vmdk)。从images可以创建一个或多个containers进程,这个可以比做从虚拟磁盘镜像文件创建出虚拟机程序。

使用docker run命令可以创建一个containers,并且在containers中运行命令,如下所示:

[root@localhost ~]# docker run --rm -ti ubuntu /bin/bash
root@88129ebf9b61:/# ls

新的containers运行的是一个ubuntu镜像,并且进入bash交互模式。

docker笔记(1)—— RHEL 7.0安装docker

官方RHEL 7.0安装docker的文档在这里。由于这个需要用户注册,所以在这里我介绍另一种方法:使用CentOSdocker rpm包。

(1)CentOS的软件包在这里:http://cbs.centos.org/repos/virt7-testing/x86_64/os/,你可以配置到yum源(软件仓库)的配置文件里,类似这样:

[centos-extra]
name=centos extra
baseurl=http://cbs.centos.org/repos/virt7-testing/x86_64/os/
enabled=1
gpgcheck=0

(2)运行“yum install docker”命令。

(3)安装成功后,运行“docker version”命令:

[[email protected] yum.repos.d]# docker version
Client version: 1.5.0
Client API version: 1.17
Go version (client): go1.3.3
Git commit (client): a8a31ef/1.5.0
OS/Arch (client): linux/amd64
FATA[0000] Get http:///var/run/docker.sock/v1.17/version: dial unix /var/run/docker.sock: no such file or directory. Are you trying to connect to a TLS-enabled daemon without TLS?

可以看到有“FATA[0000]......”提示,原因是没有启动docker daemon程序,使用“service docker start”可以启动docker程序。再次执行“docker version”命令:

[[email protected] bin]# docker version
Client version: 1.5.0
Client API version: 1.17
Go version (client): go1.3.3
Git commit (client): a8a31ef/1.5.0
OS/Arch (client): linux/amd64
Server version: 1.5.0
Server API version: 1.17
Go version (server): go1.3.3
Git commit (server): a8a31ef/1.5.0

可以看到“FATA[0000]......”提示没有了。

(4)接下来的步骤可参考这里