Learning from Giants #52
The most opinionated database design doc in the world, Reddit's image serving and optimization stack, and defining Design system components collaboratively.
👋 Hi, this is Mathias with your weekly drop of the 1% best, most actionable, and timeless resources to grow as an engineering or product leader. Handpicked from the best authors and companies. Guaranteed 100% GPT-free content.
Did a friend send this to you? Subscribe to get these weekly drops directly in your inbox. Read the archive for even more great content. Also: I share these articles daily on LinkedIn.
TigerBeetle’s database design document
TigerBeetle shows how specialization can enable building a database an order of magnitude more performant and resilient.
TigerBeetle is a distributed financial accounting database and possibly the most opinionated database in the world.
"TigerBeetle implements the latest research and technology to deliver unprecedented safety, durability and performance while reducing operational costs by orders of magnitude and providing a fantastic developer experience."
The team goes to extreme lengths to reach their expected performance and resilience. And one important axis is simplifying the business domain as much as possible.
"In theory, TigerBeetle is a replicated state machine that takes an initial starting state (account opening balances), and applies a set of input events (transfers) in deterministic order, after first replicating these input events safely, to arrive at a final state (account closing balances)."
That's it. Everything else stems or is made possible by these simple foundations.
📗 Joran Greef's Design Document of TigerBeetle is a fascinating read because it describes the lengths he goes to build the fastest and safest accounting database. Every detail is rigorously thought-through, from every fixed-size data structure to the replication protocol and timestamp management. And as a bonus, TigerBeetle is open source so you can see the resulting codebase.
These principles can apply to all software: simplifying the business domain and closing (or opening) future space for evolution is an important decision that can dramatically impact whatever you're looking to optimize. In Tigerbeetle's case, it's all for performance and resilience, but in most other cases, it's OK to trade some performance for optionality.
Just-in-time image optimization at Reddit’s scale
Reddit serves billions of images per day.
Building a great browsing experience in a media-heavy platform requires lightning-fast image loading. Reddit does that through image optimization (reducing the size, picking the correct format) and caching the transformed images.
"Reddit has been using a just-in-time image optimizer relying on third-party vendors since 2015. [...] it made sense to move this functionality in-house due to cost and control over the end-to-end user experience"
The cost aspect led to the decision to focus on images. So the team set out to replace this central piece of the Reddit platform.
"Our task was to change almost everything about how billions of images are served daily without the user ever noticing and without breaking any of the upstream company functions."
The solution chains the following components:
A Content Delivery Network (CDN). Answers all image-related requests and is in charge of caching responses. Image optimization is costly and lengthy, so the Cache hit ratio is the important CDN metric.
An image optimization service based on the C library libvips. Only called when the transformed image is not in the cache.
"As a result of moving image optimization in-house, we were able to:
Reduce our costs for animated GIFs to a mere 0.9% of the original cost
Reduce p99 cache-miss latency for encoding animated GIFs from 20s to 4s
Reduce bytes served for static images by ~20%"
📗 Walter Michelin's Just In Time Image Optimization at Reddit scale describes the mission-critical project the Reddit team undertook to move image processing and caching in-house. A good example of pragmatic engineering: start from the problem, identify the metrics, and build from there.
A collaborative approach to defining design systems components
Design systems have very different users with different agendas. Engineers optimize for clean code, and designers for a clean look & feel. This misalignment leads to components being misused into stretched use cases, making the overall system more brittle and costly to evolve.
For example:
"Modal, prompt, and popover are components that may look alike, but are intended to solve very different problems."
"So, how can we come up with component definitions that everyone agrees on, and that can scale with the system as it continues to evolve?"
Start with a clear, specific purpose.
"When everyone agreed on the component’s purpose, naming and behaviors suddenly became clear."
Write the component's design rationale.
"We start with just the concept (modal), followed by a short, detailed description (e.g. a modal displays content in a layer above the page, requiring the user’s interaction to proceed), and then usage documentation, along with “do” and “don’t” examples."
That design rationale will pave the way for consistency across your design system.
📗 Wart Burggraaf's A collaborative approach to defining components describes how thinking about purpose can ease collaboration on design systems. If we take a step back, a design system is a product, and like all products, the first step to clarity and alignment is a clear vision. The purpose is that vision.