Category: Life

How I improved myself in 2014

I graduated from school in 2008, and my first job is an embedded software engineer. After working for more than 2 years, I joined in another traditional telecommunication company (let me use A to stand for it). Fast forward to 2014, the business of A company seemed stagnated for a while: more and more people began to use OTT applications from the second decade of 21th century, and it was a big hurt to telecommunication companies like A. Though I still had salary increase every year, the amount can be ignored. But the good thing is A company has good work-life-balance culture. Besides that, my daily work is:
a) Add new trivial features for one system which is written in C and run on Solaris 10.
b) Answer technical questions which support engineers don’t know about.
Since I had been very familiar with the stuff I was in charge of, I knew I could not absorb new knowledge from A company, so I did following things to improve myself in 2014:

(1) Read English technical articles.
Yes, before 2014, I had mostly used Chinese key words to search related technical posts and read them. After switching to English, I am exposed to another world which has plentiful resource for me to explore.

(2) Write English blog.
I realised that if I just write Chinese articles, only the people who know Chinese can understand them. But if I record my thoughts in English, the people around the world can read them. Believe it or not, my current employer even referred one of my blog post before I became its employee.

(3) Learn git.
A company still used svn in 2014, and it was enough for our development procedure. But I knew git is the trend. I still remember I read Pro git and practised the commands from the book in a server; Pro git is the best tutorial of git until now.

(4) Study Go.
Go became popular in 2014, and the market for C developer began to shrink. The reason for me to learn Go is simple: hope to secure my future career. I even spent lot of time in trying to run Go on Solaris, but no success. The irony is until this year (2022), I am still a full-time C developer and never got a Go job.

On top of the above things, I also did others: wrote some DTrace tutorials in Chinese, joined some IRC chat rooms to learn English, etc. All in all, I learnt a lot of new things in 2014.

Not every software engineer can work in leading companies which have advanced technologies and did cool things, but everyone can improve himself/herself by self-learning. Hope my story can give others inspiration.

Illumos makes me nostalgic

During 2010 – 2014, I worked as a software engineer in a telecommunication company which uses Solaris&SPARC as its server, and the working experience is a really pleasant memory for me. So even I have already left the company for more than 5 years, when I meet illumos, it always lets me reminisce about some good old days.

Before I joined in that company, I had only used some Linux distributions, e.g, Fedora 4. Honestly, I didn’t know what is the difference between Linux and Solaris at that time. I still remember the first Solaris tool I learned is pstack, which my colleague showed me how to analyze application’s core dump in a brief way. Although Solaris ships its own debugger, i.e., mdb, it was a shame I was still used to use gdb (installed from Sunfreeware). Since only gdb 6.x was provided, I always tried to build cutting-edge gdb from source code, and it worked smoothly on Solaris.

When talking about our commercial server, IIRC, it had only 2GiB memory. Yes, in the second decade of 21st century, when notebook already has 8GiB memory, our 32-bit application run on 2GiB machine.  Both sever and application could date back to 2005. Our company didn’t have a big name, and throughput of our application was ~30 transactions per second. The server was surprisingly stable. I can’t remember we ever restarted it once during 2010 – 2014, and it matched what this picture has said (a Solaris server run for more than 10 years).

Last week, I came across this tweet, so I wanted to relive Solaris once again (even though it is called illumos now). Download and install OmniOS, quite straightforward. Build illumos code, meet some problems, but fix them with the help of enthusiastic people. Find a typo in manual, go through the whole flow of contributing: create issue, send patch to review, send RTI, done! The flow is stricter than “fire-and-forget” mode, but I think it is necessary for code quality. BTW,  I find some Solaris‘s peculiar still exist on illumos. E.g., even in 64-bit environment, gcc will generate 32-bit executable by default, and you should use “-m64” compile option to claim you want 64-bit one.

Thanks for your time and patience to read this ramble rants. If there is only one take-away, that is besides Linux and BSD families, there is another less-popular Unix: illumos. Though it is a small community, it doesn’t have outdated tools. You can give it a shot if you are interested, and maybe you will like it.

Tuning performance is harder than debugging bugs

In the past week, I focused on resolving an application performance issue, i.e., try to pinpoint why the code didn’t run as fast as I expected. Once upon a time, I am convinced that tuning performance is indeed harder than debugging bugs.

I have more than 10 years experience in software programming, and in recent 4 years, I spend ~20% working time in performance tuning related work: mostly application, sometimes the whole system. Regarding to debugging software bug, if the bug can be always reproduced, it should not be hard to find the root cause. Please notice, I never say it should be easy to fix: e.g., some huge technical debt. If for some reasons, the bug is not 100% reproducible, e.g., the notorious multi-thread bug, you can resort to methods to increase reproduce ratio and help you to pinpoint the culprit: add more logs, change execution time sequence, and so on. However, when talking about performance issue, the thing is “you don’t know something you don’t know“.

In most cases, as a software engineer, you don’t need to keep a watchful eye on hardware, Operating System, compiler, etc. You just need to concentrate on your own code. But to make your program performant, it is not enough to only analyze your code, you need to find answers to questions like this: why does the program run slower in this more powerful platform? Why does profiling make program run even faster? Why can’t multi-thread give a big performance rise? The more you dive into, the more you find you don’t know: architectures of CPU, the mechanism behind Operating System, tons of compiler’s options, and so forth. Even small catch can make your program hiccup! Furthermore, the stackoverflow is not the good place to call for help for performance issue, so the only guy you can rely on is yourself at most of time.

Nonetheless, the fun of performance tuning is also here: after days even weeks of endeavor, I finally find the bottleneck. It is not only exciting experience but every time I learn something I totally don’t know before amid this process. Performance tuning forces you to get a whole picture of the computer system, not only the code you write. This can broaden your view and let you know the essence of computer science.

performance tuning is harder than debugging bugs, but it also pays off! Enjoy it!

What I learn from Practical Binary Analysis as a non-reverse-engineering engineer?

I spent the past two weeks in reading Practical Binary Analysis. Since I am not a professional reverse engineer, I glossed over the “Part III: Advanced Binary Analysis”, so I only read half the book. Even though, I still get a big gain:

(1) Know better of ELF file. On *nix Operating system, ELF file is everywhere: executable file, object file, shared library and coredump. “Chapter 2: The ELF format” gives me a clear explanation of the composition of ELF. E.g., I know why some functions have “@plt” suffix when using gdb to debug it.

(2) Master many tricks about GNU Binutils. GNU Binutils is a toolbox which provides versatile command line programs to analyze ELF files. Literally it relies heavily on BFD library. I also get some sense about how to use BFD library to tweak ELF files.

(3) “Appendix A: A crash course on X86 assembly” is a good tutorial for refreshing X86 architecture and assembly language.

(4) Others: E.g., I understand how to use LD_PRELOAD environmental variable and dynamic linking functions to manipulate shared library.

All in all, if you are working on *nix (although this book is based on Linux, I think most knowledge are also applicable to other *nix), you should try to read this book. I promise it is not a waste of time and you can always learn something, believe me!

What you need may be “pipeline +Unix commands” only

I came across Taco Bell Programming recently, and think this article is worthy to read for every software engineer. The post mentions a scenario which you may consider to use Hadoop to solve but actually xargs may be a simpler and better choice. This reminds me a similar experience: last year a client wanted me to process a data file which has 5 million records. After some investigations, no novel technologies, a concise awk script (less than 10 lines) worked like a charm! What surprised me more is that awk is just a single-thread program, no nifty concurrency involved.

The IT field never lacks “new” technologies: cloud computing, big data, high concurrency, etc. However, the thinkings behind these “fancy” words may date back to the era when Unix arose. Unix command line tools are invaluable treasure. In many cases, picking the right components and using pipeline to glue them can satisfy your requirement perfectly. So spending some time in reviewing Unixcommand line manual instead of chasing state-of-the-art techniques exhaustedly, you may gain more.

BTW, if your data set can be disposed by an awk script, it should not be called “big data”.