Learning from Giants #3
API Pagination, eBPF, Linear's Product building method, multiplayer sync problem at Dropbox, and Spotify's Golden Paths.
👋 Hi, this is Mathias with your weekly drop of the 1% best, most actionable, and timeless resources to grow as an engineering or product leader. Handpicked from the best authors and companies.
Did a friend send this to you? Subscribe to get these weekly drops directly in your inbox. Read the archive for even more great content. Also: I share these articles daily on LinkedIn.
Popular Solutions to Paginate API Endpoints
API Pagination is so simple every company does it differently.
At first, the problem sounds simple: collections tend to grow over time and reach size and query duration limits. They must be split into pages.
Engineers end up making many trade-offs between user requirements and database capabilities:
"Can the list be sorted? Is there any default order?"
"Can the list be filtered? If so, which filter params should it accept?"
📗 Ignacio Chiazzo's Paginating Requests in APIs goes over the three most popular implementation strategies to split API responses into pages. The Shopify engineer explains why there is no silver-bullet pagination solution and details the pros and cons of offsets, keysets, and cursors. He then concludes with a quick survey of how Facebook, Stripe, Slack, Shopify, and Github solve that problem. Spoiler alert: all roads lead to cursors.
PS: Ignacio worked on Shopify's recent move from offset-based to cursor-based pagination, so he knows a thing or two about hitting the limits of offset-based pagination!
An Introduction to eBPF from Teleport
If you're not following the Linux observability and networking world closely, you may have missed eBPF.
Let's start with the problem. Say I want to :
Apply custom logic globally to a Linux system safely.
Record all system calls globally to a Linux system safely.
These are difficult because of the strict separation between user and kernel spaces. Only kernel space has global visibility and control. However, letting custom logic into the kernel is highly unsafe.
"So what does eBPF do? It lets programmers safely execute custom byte code within the Linux kernel without modifying or adding to kernel source code."
📗 Teleport's What is eBPF and how does it work? is the missing step-by-step introduction to eBPF. Virag Mody describes why eBPF exists, the risks of touching the kernel space, and where it succeeded where others have failed. He then explains how Teleport uses it to audit ssh sessions.
eBPF opens a new world of possibilities!
"Already these programs have been used for networking, debugging, tracing, firewalls, and more."
Fun fact: eBPF must surely be an acronym, right? Not anymore! The official website says eBPF "is no longer an acronym for anything."
Linear’s Product Building Method
Successful products have "on-point" operating systems.
And when they share them, it's an incredible learning opportunity.
As an individual, there are plently of ideas to bring to your team.
As a founder, you're able to compare your processes against the best!
📗 Linear's Linear Method describes the foundational principles the Linear team used to create such a successful product. Although this may feel like an ad for the company, I found every sentence truly enlightening and helpful as a product leader. It takes a deep understanding to clearly layout these principles with no jargon and short sentences.
The Architecture of Dropbox Replay: a multiplayer synchronization problem
On Thursdays, I share real-world problem-solving examples from top-tier teams. Reading them is the best way to:
Stand on the shoulders of giants and learn from them.
Fight your internal impostor syndrome: you probably could have designed it too!
Today's problem came up while building Dropbox Replay Live Review:
"Keeping a virtual (video) screening room with multiple collaborators in sync is a harder problem to solve than you might think. [...] Anyone in a Live Review session can pause, adjust the playback speed, or scrub to a different frame at any time."
In short: they have an event ordering problem with significant UX constraints.
📗 Dropbox's How Dropbox Replay keeps everyone in sync is remarkable because it's a detailed walkthrough with clear examples of how the team solved this problem. Alan Rogers, Daniel Wagner, and Siya Yang explain their "tracer bullet" (iterative) process towards the right user experience. And if you're a Golang speaker, there even are code samples!
Golden Paths: Spotify’s Solution to Tooling Fragmentation and Unclear Best Practices
Tooling fragmentation and inconsistency kill software productivity.
How long does onboarding to a new codebase or project take for an existing engineer? If the answer is more than one day, you probably have a fragmentation problem.
"[...] the only way to find out how to do something was to ask your colleague. 'Rumour-driven development', we endearingly called it".
Is it a price to pay to keep autonomous teams? Spotify's Platform Developer Experience teams said no, and introduced the idea of a Golden Path.
"The Golden Path is the opinionated and supported path to build your system, and the Golden Path tutorial walks you through this path."
📗 Spotify's How We Use Golden Paths to Solve Fragmentation in Our Software Ecosystem is one of these simple but truly inspiring reads. Gary Niemen explains the Golden Path and Golden Path tutorials in the Spotify engineering org and how they helped the company solve its fragmentation problem.
"There was a time when engineers at Spotify couldn't imagine life with Golden Paths; now we can't imagine life without them."
Fun fact: the "Golden Path" name comes from Frank Herbert's "Children of Dune":
"His spice-induced visions show him a myriad of possible futures where humanity has become extinct and only one where humanity survives. He names this future 'The Golden Path' and resolves to bring it to fruition.
Reply to this email to react, recommend us articles, or just give feedback!
See you next week!