A performance engineering team’s story

It is a story about a performance engineer team: most is genuine, and some is made-up.

Server performance related work requires a broad knowledge and experience: software, hardware, Operating System kernel, and so on. The team members’ background is also a rich variety: embedded system developer, PCB designer, QA, DevOps, compiler researcher, etc.

The team’s daily work is like this:

(1) Since company’s main products are servers, run SPEC benchmark programs and cooperate with other team to get the best result is the most important task. Since only your servers exceed other competitors, the customers are willing to pay money for them.

(2) It is no controversial that Linux has dominated the server market, so being proficient in Linux profiling and tracing tools (from perf,ftrace to BPF) is a compulsory class. Meanwhile, the company also has its own proprietary Unix which is still serving some critical business: bank, telecommunication, etc. The members also need to use some esoteric tools to help diagnosing issues on this proprietary Unix. At the same time, coding is another important task: develop company’s own profiling tools, and contribute to FOSS projects.

(3) Support customers and other teams. E.g., one customer wants to know the Oracle‘s performance on some server models; the other finds that Docker doesn’t run as expected. Another colleague comes to your desk: “When you are free, could you help to check how to optimize this program? Boss isn’t satisfied with it”.

(4) The team encourages sharing. During every weekly meeting, you can introduces a Unix command trick, a debugging skill, like this. Members can have 5% ~ 10% time to do hobby projects, but the prerequisite is work first. This benefit is not free lunch, you should report it in the meeting too.

So what is the point of this article? Nothing, just telling a story, that’s it.

Learn new technology through writing a tutorial about it

I like to get my feet wet on new technologies, but find if I don’t use it for some time, e.g., several months, I will forget a lot of details, not sure whether other people have the same feeling :-). To let me get a quick refreshment of the technology after a while, I resort to the old school method: writing notes. But one day, I came out a idea: why not try to write a tutorial during studying instead of only recording? So in the past 2 years, Golang 101 hacks and OpenMP Little Book are born. The whole process is really rewarding :

(1) Sometimes you think you have grasped the knowledge, but when you begin to write an article to explain it, you will find there are some points you can’t understand thoroughly. To make your article more clearly, you need to write code to verify it, look for help in the internet, etc. Among this process, you will get a deeper understanding.

(2) Your tutorial can be reviewed by other brilliant engineers who can point out your mistakes, in the meantime, the tutorial can also help others. E.g., I find my tutorial is quoted in stackoverflow’s answer occasionally, and it really encourages me!

(3) Since I am not a native English speaker, creating an English tutorial can also help to improve and practice my English skills. I highly recommend you use English to compose, because that can make your idea shared among the people all over the world!

Based on above points, writing technological tutorial is definitely a win-win process. Why not give a shot? I think you can!

What is a good software engineer job?

After working as a professional software engineer for 10 years, I want to share what I think is a good software job here.

(1) The opportunity of trying different stuff.
You have the chance to touch the different technology, not necessarily the newest. E.g., after working as an application engineer for one or two years, you can make your hands dirty on kernel. Be bored of using C++, it is time to have a taste of rust or Go. All in all, keep freshment is an important factor to motivate you.

(2) Contribute to FOSS.
No one will deny that both the tech giants and start-ups depend on FOSS (Free/Open Source Software) greatly now (Even Microsoft “loves” Linux). If your employer encourage you to contribute back to the project rather than only exploit it, it is really beneficial for community, yourself and company. “A stone kill three birds!”

(3) Sponsor you attend tech conferences and training.
Attending tech conference can expand your horizon, know more new people, and be aware of what other companies are doing and concentrating on in your area. If you can give a talk on some famous conferences, it will earn reputation for both you and your company. Besides this, life-long studying is important for software engineer, so it will be fantastic if your boss is willing to pay money for your enrichment.

(4) Encourage innovation and knowledge sharing.
It will be a great favor if you have ~10% work time for your own project. Actually, the company can select and cultivate seed from these projects, and it may bring potential success. The classical example is gmail. On top of that, Knowledge sharing is a good section in team meeting in which you can learn skills from your colleagues.

Besides the aspects mentioned above, I think there are some other factors. For example, salary, welfare, promotion, etc. Since these are not software job exclusive, and a little complicated among different industries, companies and even countries, I won’t elaborate it here.

The time spent on practising white board test may not be worthy

Once upon a time, I spent my spare time in practising white board test, i.e., try to solve data structure / algorithm issues from leetcodehackerrank, etc. But after nearly one or two months, I decided to stop this activity since I find the time spent on it is not worthy, at least for me.

I am not a clever software engineer, even a little dumb. Except very basic problems, I find a lot of puzzles which has “medium” difficulty level will cause me spend hours to fix it. I mean although I can resolve part of them, but I am not sure whether this is the optimal solution, so I would read others’ answers which claims consume very little time carrefully to understand them. For some extreame cases which I could’t get a clue after two hours, it usually costs me to take much longer time to get the point of the puzzles.

Gradually, I found though this practice can have some effect in improving my coding ability, whereas compared to the time consumed, it is not cost-effective, and honestly not very helpful for my daily work. Take C++ as an example, the STL and boost have provided almost all the data structures and algorithms which you need. You just need to call std::sort, and don’t care it is “bubble sort” or “quick sort” under the hood. So in reality, I seldom have the opportunity to implement the sort algorithm from scratch. Furthermore, most problems just need single-thread to resolve, but I find multi-thread and synchronization are real needs in daily life. Unfortunately, concurreny programming ability can’t be polished through resloving these puzzles. Last but not least, researching these problems really consume too much time. If I really need the realted knowledge of one specified problem at work, studying it then is not too late.

After stopping these practice, I leverage the spare time for following tasks: read C++ classical books and learn more about standard libraries (There is another example, hash table is a commonly used data structure, and STL provides a lot of member functions for it, but I find many puzzles related to hash table only need very few methods, so you can’t get a comprehensive understanding of hash table class in STL by resolving these problems only); watch techncical conference videos (this can expand horizon and improve my English skills) and study basic knowledges for computet and maths (Definitely, the data structures and algorithms are important, I think we should know the internals well, but no need implementing them ourselvels).

To wrap up, currently I feel the time is fully utilized and more worthy compared to practise white board tests before. What’s your opinions? Feedback welcome!

Ability must be ageless

Yesterday I came across this impressive video: Ability is ageless, and want to share some thoughts here.

I don’t know about other countries, whereas in China, ageism does exist more or less in software companies. I have read some stories and news about elder engineers who were laid off without convinced reasons. Employers think the elder employees have families and want more work-life balance, so they won’t work over-time without complaint like fresh graduates. Furthermore, the elder engineers are harder to manage than freshmen. I even read a job description which was like this: we don’t welcome the applicants who are older than 30, since they will lack innovation.

Regarding myself, I am 35 years old. 10 years ago, in January, 2008, I left school and got my first full-time job. In fact, currently I don’t reduce my working time in a week compared with 10 years ago. The experience accumulated in past 10 years is literally very precious, and I can ulitize it to help other younger colleagues. Besides this, I don’t come to a standstill, and keep to make my hands dirty on new fields in computer science which I didn’t touch before, wrtie blogs and tutorials, and take part in technology meetups and conferences actively. At least in my opinion, I become more mature and valuable accompanied with older age.

Ability must be ageless, and it is what I want to say.