euccas.github.io

why a developer writes

Python: Why Decorators Are Useful

| Comments

In Python, by definition decorators are functions that accept a function as an argument, and return a new function as their return value. The reason why decorators exist in Python, but not in other similar language such as Ruby, is that functions are objects in Python. Functions can be assigned to variables and passed around the same as other object in Python. For example, a list can have functions as its elements, and functions can take other functions as arguments. Functions can be defined inside another function, and this forms closures.

When to use decorators?

It’s easy to understand what decorators are, while the real question you may have is: Why decorators are useful? When shall I use decorators in my Python program?

In some way, I see decorator functions are useful whenever you need process or extend the inputs or outputs of a given function (or more often, multiple functions) in some way you want. Here I list three usages of decorators that I can think of:

CPU Profiling Tools on Linux

| Comments

Profiling is an effective method to provide measurements for the performance of software applications. With profiling, you get fine grained information for the components of an application, such as how often a function is called, how long a routine takes to execute and how much time are spent of different spots in the code. With these information, you could identify the performance bottlenecks and the poorly implemented parts in a software application, and find effective methods to improve them.

In this post I’ll write a brief summary of two profiling methods: Instrumentation and Sampling, and four CPU profiling tools on Linux: perf, gprof, Valgrind and Google’s gperftools.

Profiling Methods

Different profiling methods use different ways to measure the performance of an application when it is executed. Instrumentation and Sampling are the two categories that profiling methods fall into.

Facebook Infrastructure: Streaming Video Engine (SVE)

| Comments

In last year’s Facebook F8 conference, Sachin Kulkarni, who worked on Facebook’s Video Infrastructure, gave a talk (watch it here) to introduce the design of Facebook’s Streaming Video Engine System (SVE). I found this talk particularly interesting because it revealed, in a very well structured, concise yet informative way, how Facebook infrastructure team came up with a solution to build a video system solving user frustrations by reviewing the end-to-end process, and how such a design meet the goal of being fast, flexible, scalable, and efficient. After watching the presentation video for a few times, I thought it would be helpful to write down some notes here, for my own reviewing in the future, and for people who might be interested in Facebook’s media infrastructure.

Sharing on Facebook started from largely text, and quickly changed to be largely photos. Since 2014, more videos started to be posted and shared among users. The challenge was, building a video processing system is much harder than building a text or image processing system. Videos are greedy, they will consume all your resources: CPU, memory, disk, network, and anything else.

Before building the Streaming Video Engine system, the team started by reviewing Facebook’s existing video uploading and processing process, which was slow and not scalable. They found several problems need change or improvement:

How Instagram Moved to Python 3

| Comments

Instagram, the famous brunch sharing app, presented in PyCon 2017 and gave a talk in the keynote session on “How Instagram moves to Python 3”. If you have 15 minutes, read the interview with the speakers, Hui Ding and Lisa Guo from Instagram Infrastructure team, here. If you have 45 minutes, watch their PyCon talk video, here. If you have only 5 minutes, continue reading, right here.

Instagram’s backend, which serves over 400 million active users every day, is built on Python/Django stack. The decision on whether moving from Python 2 to Python 3, was really a decision on whether investing in a version of the language that was mature, but wasn’t going anywhere (Python 2 is expected to retire in 2020) – or the language that was the next version and had great and growing community support. The major motivations behind Instagram’s migration to Python 3 are:

  • Typing support for dev velocity
  • Better performance than Python 2
  • Community continues to make Python 3 better and faster

The whole migration process took about 10 months, in roughly 3 stages.

CMake: Use the Correct Options to Solve Linker Errors

| Comments

A few months ago when I worked on a project using zlib to compress and decompress files, I once met linker errors complaining about unable to resolve symbols of zlib functions:

1
cannot resolve symbols _gzbuffer, _gzopen, ...

In the end I fixed these linker errors by using TARGET_LINK_LIBRARIES command in the project’s CMakefile to specify the linker package dependency, as the following:

1
TARGET_LINK_LIBRARIES(myProject zlibstatic)

When I was looking for solutions to fix those linker errors, I found several related CMake commands which look quite similar and could be confusing in terms of their functions and when to use them. Here is a quick summary of these commands.

Related CMake commands:

  • add_dependencies
  • link_directories
  • link_libraries
  • target_link_libraries