Linux System and Application Observability (book)

BPF (eBPF) tracing is a superpower that can analyze everything, and I'll show you how in my upcoming book BPF Performance Tools: Linux System and Application Observability, coming soon from Addison Wesley. The book includes over 150 BPF observability tools that you can run to find performance wins and troubleshoot software, and also shows you how to write your own. Over one hundred of these BPF tools are newly-developed for this book; you can see many of them in this diagram:

I've found this diagram to be a useful reminder and checklist when debugging production issues at Netflix, and I have it printed on a wall so I can glance at it. You may find it helps you, too, which is why it is on the cover.

BPF is a new in-demand technology, and I expect that, over time, a dozen or more books about it may be published – as happens for other hot new technologies. Guess how many books there are about Docker? I lost count at 30! Some other future BPF books may be good, and I'll recommend them as appropriate. To answer the question why you should buy my book:

I was previously a professional technical trainer for Sun Educational Services, where my classes included DTrace, another tracing tool similar to BPF tracing. I learned firsthand how to bring beginners from different backgrounds up to speed. Early courses covered only the DTrace language, and expected students to figure out what to do with it. I found it was more effective to teach by example: explain the goals and targets of performance analysis, and then share real tools to meet those goals, and how to develop new ones. I've followed this proven approach in my book.

I've taken a break from training, but I have an opportunity to start again at Netflix, and I'll be using this book as materials for an internal BPF course. This course will be internal only: you'll need to join Netflix to attend my class! (Although I'll submit a short version for USENIX LISA/SREcon.)

We (the BPF community) began planning this book in 2017, and posted an announcement about it so that anyone looking could find it. At the time, BCC was great for complex tools, but it was still in development and changing. Also, no higher-level front end had been completed, but we knew we needed one beyond BCC. Since then, BCC has largely stabilized, and bpftrace has become a mature high-level front-end: one that is built from the ground up for BPF. bpftrace is already in use for production analysis at companies including Netflix, Facebook, and Shopify. It allows tool source to be so short and concise it can be included in the book in its entirety. It's like including pseudocode that you can run.

The last nine months have been hectic as I and others have added features to bpftrace, fixed bugs, and stabilized the API. The analysis of so many things unearthed many bugs and missing parts; fixing these was good not only for the book, but also for bpftrace. A special thanks to Matheus Marchini, Willian Gasper, Alastair Robertson, Dale Hamel, Dan Xu, Augusto Mecking Caringi, and others, for all their work in these months. I'm also grateful that my fiancée, Deirdré Straughan, could find the time outside of her AWS job to do technical copy-edit. Thanks also to Alastair Robertson for creating bpftrace, Alexei Starovoitov and Daniel Borkmann for creating eBPF, and over forty other people involved in reviewing and contributing to this book.

As you'll see, this book is the start of a new era for Linux observability: the power to easily see anything and everything, in production. A time where you can pose arbitrary questions of the system, and it can answer them. A time where you are free to think about performance in new ways, and you can fetch hard data to support or disprove your new ideas.

It is, but I had a static goal for the book: to get all of the included 150+ tracing tools to work. After many months of development, we've reached that milestone. I also involved the main BPF developers so I could document their planned work, to make this book as future-proof as possible.

BPF Type Format (BTF) is the only large change expected for tracing in the nearish future. It will provide kernel struct information beyond the linux-headers package, and is summarized in the book. A few of the tools declare missing structs as a workaround, and some years from now it will be possible to remove those workarounds.

What I expect to break sooner is the kprobe-based source code in the book. kprobes is considered an unstable API as it instruments kernel functions, and a kernel engineer may rename a function in a later kernel, breaking the tool. This is why the tools use the stable tracepoint interface whenever possible. But it's not always possible, and this is the nature of tracing. This is not, however, the hardest problem you'll face. The hardest problem is knowing what to do with BPF tracing, and even broken tools are a source of useful ideas. I will create a GitHub repository for the book tools as-is plus updated versions, and allow everyone to submit updates as they are found to be needed.

Systems Performance has sold very well (thanks, everyone!). It's recommended reading for new hires at several companies, including some teams at Facebook and SUSE. It is the encyclopedia of performance, covering many topics with a balance of theory and practice. BPF Performance Tools focuses on one topic only – observability – and goes deeper. The BPF book is about doing analysis right now, with these tools and capabilities. Now that I've finished it, I can do a second edition of Systems Performance (not right away…).

BPF is the hot new technology, and it is an extreme privilege to have written the book on it. I was determined to give you the best book imaginable, and not waste this opportunity. I hope you enjoy it, and it helps you find valuable performance wins. Good luck!

For URLs where to find the book and updates, please see my website BPF Performance Tools (book).

Source: Brendangregg.com

Powered by NewsAPI.org

Keywords:

Linux • System • Observability • Berkeley Packet Filter • Berkeley Packet Filter • Superpower • Berkeley Packet Filter • Linux • System • Observability • Addison-Wesley • Berkeley Packet Filter • Observability • Software • Berkeley Packet Filter • Programming tool • Software development • Debugging • Netflix • BPF Party • Docker (software) • BPF Party • Technical trainer • Sun Microsystems • DTrace • Berkeley Packet Filter • DTrace • Profiling (computer programming) • Netflix • British Pacific Fleet • Only You (1994 film) • Netflix • BPF Party • Cubic crystal system • Compiler • Cubic crystal system • BPF Party • Netflix • Facebook • Shopify • Pseudocode • Application programming interface • Banu Aws • Copy editing • Berkeley Packet Filter • Linux • Observability • System • You Are Free • Software development • Berkeley Packet Filter • Future proof • Berkeley Packet Filter • Data type • File format • Bidirectional texture function • Kernel (operating system) • Record (computer science) • Linux • Include directive • Package manager • Programming tool • Record (computer science) • Source code • Application programming interface • Kernel (operating system) • Subroutine • Kernel (operating system) • Engineering • Subroutine • Kernel (operating system) • Tool • Debian • Ftrace • Interface (computing) • Berkeley Packet Filter • Source code • GitHub • Software repository • Programming tool • Patch (computing) • Computer • Facebook • SUSE Linux distributions • BPF Party • Observability • BPF Party • Systems engineering • Berkeley Packet Filter • URL • Berkeley Packet Filter •