Reading Notes - Designing Distributed Systems

Recently I read a book Designing Distributed Systems, which is written by Brendan Burns, and published by O’Reilly earlier this year. This book introduces the patterns and components used in the development of distributed systems.

Before I started to read this book, I had three questions in my mind, and try to find the answers from the book. Those three questions are:

1. What’s the most important difference between designing distributed systems and single machine systems?
2. Why container technology, such as docker, kubernates, is so popular? How could it be helpful?
3. What are the common patterns used in distributed systems design, and when shall I use them?

This book does give me the answers, at least partial ones. I put my reading notes into a Google Slides, and you can find it here to read the details. A PDF version in light background color is available here.

The short answers to my questions are as in the following:

What’s the most important difference between designing distributed systems and single machine systems?

• Designing distributed systems can be significantly more complicated to design, build, and debug correctly.
• Designing distributed systems need much more efforts in designing for scalability and reliability.
• In a distributed system, tasks/data are spreaded to multiple workers. It requires techniques like containers and load balancing to utilize parallelisation
• Containers are not only useful for applications which have components running on multiple machines, but also for single machine applications.
• The goal of containerization is to establish boundaries around specific resources, team ownership, separation of concerns.
• The benefits include resource isolation, scaling teams, reuse components and modules, break big problems into smaller ones (Small, focused applications are easier to understand, be tested, updated and deployed)

What are the common patterns used in distributed systems design, and when shall I use them?

The book describes three types of patterns.

• Serving patterns: sharded services, scatter/gather, FaaS, etc.
• Batch computational patterns: Work queue, Event-driven batch processing, coordinated batch processsing, etc.

You can find more detailed description of each design pattern in my reading notes.

Performance Profiling Tools on Windows

Last year, I wrote a blog post about CPU Profiling and the tools on Linux. Today I’m going to write about a few Performance profiling tools on Windows platform. Last week I was working on profiling and analyzing the build process of a project on Windows, and during the process I experimented with a few different tools.

Performance Monitor (perfmon)

Performance Monitor is a small utility provided by Windows OS, you can start it by running command perfmon. With perfmon, you can monitor real-time system performance, and record performance to files for post analysis. This tool provides some extremely useful interfaces in its GUI.

Real-time Performance

To view current performance activity, you just need click on the Performance Monitor button in the left panel:

Effective Jenkins Plugins (1)

(Update 2018-05-13: Here is a translation of this post in Hindi, provided by Nikol @ DealsDaddy.)

Jenkins, originally founded in 2006 as “Hudson”, is one of the leading automation applications which support building, deploying and automating software projects. One great advantage of Jenkins is there are hundreds of plugins available which enable various kinds of extended features needed in the Continous Integration and Continuous Delivery process. As I just checked on the Jenkins Plugins page, there are 873 plugins that fall into five categories: Platforms, User interface, Administration, Source code management and Build management.

Effectively using Jenkins plugins makes your experience with Jenkins more productive. I’m going to occasionally write about Jenkins plugins that I used or learned about. The first post will start from some of the plugins I used when I worked on building a Continuous Delivery system last year (from 2015 to 2016).

Job Configuration History

JobConfigHistory Plugin

This plugin saves every change made to a job. It allows you to see the history of job configurations, compare configuration differences, and restore a particular version of config. You can also see which user makes a particular change if you configured a security policy.

Python: Why Decorators Are Useful

In Python, by definition decorators are functions that accept a function as an argument, and return a new function as their return value. The reason why decorators exist in Python, but not in other similar language such as Ruby, is that functions are objects in Python. Functions can be assigned to variables and passed around the same as other object in Python. For example, a list can have functions as its elements, and functions can take other functions as arguments. Functions can be defined inside another function, and this forms closures.

When to use decorators?

It’s easy to understand what decorators are, while the real question you may have is: Why decorators are useful? When shall I use decorators in my Python program?

In some way, I see decorator functions are useful whenever you need process or extend the inputs or outputs of a given function (or more often, multiple functions) in some way you want. Here I list three usages of decorators that I can think of: