The byproducts of reading OpenBSD netcat code

When I took part in a training last year, I heard about netcat for the first time. During that class, the tutor showed some hacks and tricks of using netcat which appealed to me and motivated me to learn the guts of it. Fortunately, in the past 2 months, I was not so busy that I can spend my spare time to dive into OpenBSD‘s netcat source code, and got abundant byproducts during this process.

(1) Brush up socket programming. I wrote my first network application more than 10 years ago, and always think the socket APIs are marvelous. Just ~10 functions (socket, bind, listen, accept…) with some IO multiplexing buddies (select, poll, epoll…) connect the whole world, wonderful! From that time, I developed a habit that is when touching a new programming language, network programming is an essential exercise. Even though I don’t write socket related code now, reading netcat socket code indeed refresh my knowledge and teach me new stuff.

(2) Write a tutorial about netcat. I am mediocre programmer and will forget things when I don’t use it for a long time. So I just take notes of what I think is useful. IMHO, this “tutorial” doesn’t really mean teach others something, but just a journal which I can refer when I need in the future.

(3) Submit patches to netcat. During reading code, I also found bugs and some enhancements. Though trivial contributions toOpenBSD, I am still happy and enjoy it.

(4) Implement a C++ encapsulation of libtls. OpenBSD‘s netcat supports tls/ssl connection, but it needs you take full care of resource management (memory, socket, etc), otherwise a small mistake can lead to resource leak which is fatal for long-live applications (In fact, the two bugs I reported to OpenBSD are all related resource leak). Therefore I develop a simple C++ library which wraps the libtls and hope it can free developer from this troublesome problem and put more energy in application logic part.

Long story to short, reading classical source code is a rewarding process, and you can consider to try it yourself.

The Dilemma brought by “outdated technology”

This week, I came across an interesting post: Do You Know Cobol? If So, There Might Be a Job for You. The general idea is many big finance companies still use the systems that are developed in Cobol, which is an ancient programming language. Currently, there are few people who master Cobol, and even worse, most of them will or already retire. IMHO, this article gives an good example about companies and personal engineers’ dilemma brought by “outdated technology”.

Let me tell two stories first:
(1) I once worked in HPE for nearly two years, and HPE has its own Unix Operating System: HP-UX. Actually, my team did HP-UX related work before I joined in, but at that time, all team’s work was already switched to Linux. Why? Because Linux had dominated the servers market then. To earn money, more and more resource should be invested in Linux area, and for HP-UX, basic maintenance and development is enough. As fat as I know, HP-UX is still the backbone of many critical services, such as banks, telecommunication Operators, etc. But even for HPE itself, HP-UX‘s priority becomes very lower now.

(2) There is a service which was launched in mid-1990s. It is a 32-bit program, written in C programming language, and stable enough to serve the people all over the world every day. About ~20 years later, one engineer noticed Year_2038_problem because the program definitely use time_t which is 32-bit long. He began to discuss with leader to transform the program to 64-bit, but both of them knew it was not as simple as adding only -m64 compile option. After ~20 years, the code had become “mature”, what I mean is the code repository was very large; about ~40 engineers had ever committed code, and some modules had changed into a total “black box”, what I mean is no one knew the logic behind it, but it really worked as a charm! To transform it into 64-bit program, maybe the compilation can pass, but no one know whether it indeed work! It needs careful code review and sufficient testing, but seems not worth cause it is a problem which will occur ~20 years later. Therefore, this task lies in “Todo” list year by year. Every one pray the service should be shut down by 2038.

These examples all narrate one fact, most “outdated technology” are not too “bad”, such as Cobol, HP-UX or 32-bit program, but for some reasons, they are not main-steam now. For most companies, the overhaul of services which are constructed by these “outdated technology” is not accepted: besides the notable time & person cost, one glitch can bring catastrophic result, even can let company close. But at the other side, the amount of engineers who master these “outdated technology” also become smaller and smaller, so the companies can only explore the potentials from their internal staffs mostly.

For engineers, there is also a dilemma. You can pick “outdated technology” as a hobby, but there is a huge risk to adopt it as a full-time job. Maybe one day, you need to find job again, and many biased companies will reject you just because they deem you don’t know some “hyped technology”, and can only tame dinosaurs. Ridiculous! Isn’t it? But it is the reality we must adapt to.

If a project helps you, please give it a star!

I used to fork projects from github when I thought they are useful: even not currently but maybe I need it in the future, so I did a backup of it. However, I didn’t always give the project a star. Graudally, I found this is not a good habit. Currenlty, When I fork a project, I acknowledge it gives me valuable information, I always give this project a star.

The reason behind it is simple: for a repository in github, no matter it belongs to a company or an independent developer, is an outcome of hardworking. Especially for an individual, who mostly sacrifices his own spare time to foster the project and share it freely. You don’t need to pay even one penny to obatin knowledge from others’ IP, and only one click of mouse can express your gratitude and respect for original author. It is not too hard, correct?

Actually, the author can often get motivated by this simple action because it means his endeavour gets a recognization and he earns potential reputation in his career; this may drive he to polish project continuously. The users, including you, will get benefit from a high-qualified project with doubt, so there will be a “win-win” result finally.

Based on above exposition, maybe everyone should give a star for the projects which have helped you. This is the cheapest method to support Free& Open Source Projects, no matter it is software, hardware, and whatever.

A performance engineering team’s story

It is a story about a performance engineer team: most is genuine, and some is made-up.

Server performance related work requires a broad knowledge and experience: software, hardware, Operating System kernel, and so on. The team members’ background is also a rich variety: embedded system developer, PCB designer, QA, DevOps, compiler researcher, etc.

The team’s daily work is like this:

(1) Since company’s main products are servers, run SPEC benchmark programs and cooperate with other team to get the best result is the most important task. Since only your servers exceed other competitors, the customers are willing to pay money for them.

(2) It is no controversial that Linux has dominated the server market, so being proficient in Linux profiling and tracing tools (from perf,ftrace to BPF) is a compulsory class. Meanwhile, the company also has its own proprietary Unix which is still serving some critical business: bank, telecommunication, etc. The members also need to use some esoteric tools to help diagnosing issues on this proprietary Unix. At the same time, coding is another important task: develop company’s own profiling tools, and contribute to FOSS projects.

(3) Support customers and other teams. E.g., one customer wants to know the Oracle‘s performance on some server models; the other finds that Docker doesn’t run as expected. Another colleague comes to your desk: “When you are free, could you help to check how to optimize this program? Boss isn’t satisfied with it”.

(4) The team encourages sharing. During every weekly meeting, you can introduces a Unix command trick, a debugging skill, like this. Members can have 5% ~ 10% time to do hobby projects, but the prerequisite is work first. This benefit is not free lunch, you should report it in the meeting too.

So what is the point of this article? Nothing, just telling a story, that’s it.

Learn new technology through writing a tutorial about it

I like to get my feet wet on new technologies, but find if I don’t use it for some time, e.g., several months, I will forget a lot of details, not sure whether other people have the same feeling :-). To let me get a quick refreshment of the technology after a while, I resort to the old school method: writing notes. But one day, I came out a idea: why not try to write a tutorial during studying instead of only recording? So in the past 2 years, Golang 101 hacks and OpenMP Little Book are born. The whole process is really rewarding :

(1) Sometimes you think you have grasped the knowledge, but when you begin to write an article to explain it, you will find there are some points you can’t understand thoroughly. To make your article more clearly, you need to write code to verify it, look for help in the internet, etc. Among this process, you will get a deeper understanding.

(2) Your tutorial can be reviewed by other brilliant engineers who can point out your mistakes, in the meantime, the tutorial can also help others. E.g., I find my tutorial is quoted in stackoverflow’s answer occasionally, and it really encourages me!

(3) Since I am not a native English speaker, creating an English tutorial can also help to improve and practice my English skills. I highly recommend you use English to compose, because that can make your idea shared among the people all over the world!

Based on above points, writing technological tutorial is definitely a win-win process. Why not give a shot? I think you can!