Learning from Giants #66
How Meta gives trusted time in their datacenters, why structure and systems are the essence of design work, ClickHouse's six join algorithms, and a fantastic re-post: Spotify's Golden Paths.
Hi! This is Mathias with your weekly drop of the 1% best, most actionable, and timeless resources to grow as an engineering or product leader. Handpicked from the best authors and companies. Guaranteed 100% GPT-free content.
Did a friend send this to you? Subscribe to get these weekly drops directly in your inbox. Read the archive for even more great content. Also, I share these articles daily on LinkedIn.
How Precision Time Protocol is being deployed at Meta
How Meta spent millions to get time accuracy in its data centers from microseconds to nanoseconds.
Synchronization is the main problem of distributed systems, but what makes it the toughest is that you can't just use time. That's because using time means agreeing on what time it is, which is itself a synchronization problem!
That's why most distributed systems like databases are built with the assumption that you can't trust machine time, which complicates many things and ultimately affects these systems' latency.
But time isn't just a virtual concept; the physics world defines it. If only machines could get access to a trustable physics time! It turns out that's what hardware engineers have iterated on for years, and as data centers and distributed systems grow even larger, that problem is as important as ever.
So, Meta research scientists and engineers built a full-stack solution from the GNSS antenna on the roof to the hardware appliance, the time server, the client, and their monitoring.
What's fascinating is that many tiny details become large challenges when you want to get to nanoseconds time accuracy.
The time appliance needs to be configured on the fly to adapt its configuration to the state of all its components.
The travel time of the time information from the appliance to the client, located anywhere in the datacenter, needs to be accounted for as a "time offset".
Time isn't a strict value anymore, but a "window of uncertainty"
"As you may see, the API doesn't return the current time (aka time.Now()). Instead, it returns a window of time which contains the actual time with a very high degree of probability."
📗 Ahmad Byagowi's How Precision Time Protocol is being deployed at Meta is a fascinating long read about the entire stack the Meta teams designed to get nanosecond time accuracy. If anything, it's a good reminder not to trust time.Now()!
How to Turn Good Design Direction into a Good System
"Systematization is the rigor of design."
"There's this myth in creative spaces that systems are where good ideas go to die. [...] But structure and systems are not just a designer's most important tools — they are, ultimately, what design is."
And it's true in everything! We always give more credit to the initial idea than it deserves and less to execution. That leads people to spend too much time on ideation compared to the execution of structure and systems.
"When this happens, I'll find myself in reviews of initial layouts or compositions asking why there are so many new ideas present."
So Christopher Butler came up with the "Idea to Structure Ratio".
Idea Generation — 10%
"Good ideas probably account for about 10% of the effort in a design project. That's right. Just ten percent."
"The idea is to show how those elements can work together in a variety of different ways and secure your constituents' comfort with the visual language without asking them to approve specific layouts."
Structure — 20%
"Creating a structure to support them — creating the pages, screens, or scenes that provide a consistent landscape for attention and use — is about 20%."
This is the wireframing stage. The idea is to go from a very rough structure to "just enough detail to demonstrate the information architecture of the space as well as visually indicate the priority of information and possible actions."
Sub-Structure — 40%
"Integrating the sub-structure in design — considering taxonomies of information, connections between elements (both functional and conceptual), and the admin layer of user experience — is actually 40%."
"This is the comprehensive systematization of a thing. It's the largest portion of effort and value because of how detailed it is, how much collaboration it requires, and how much it impacts the strength, scalability, and overall elegance of the final product. [...] This is where designed components become system modules."
Visual System — 20%
"Creating the final visual system — the complete visual language and pattern library that is the result of joining the initial idea to the outer and inner structures — is the next 20%."
This is the high-fidelity design stage.
"This is the stage where, if the previous two aren't handled well, I start to see a lot of unnecessary new creation that distracts from progress being made or obstructs a system from cohering or remaining together."
Documentation — 10%
"Finally, the documentation of the visual system is the last 10%."
📗 Christopher Butler's How to Turn Good Design Direction into a Good System is a very dense explanation of how Chris runs his design projects and of the steps where he spends most of his time. Twenty years of iteration and experience that you should definitely read.
Choosing the right join algorithm (ClickHouse)
Database joins don't have a one-size-fits-all solution. Like most database operations, finding the optimal algorithm to execute a join statement is a latency vs. memory tradeoff that heavily depends on the pre-existing alignment of the data layout and the join sort order.
By thinking in reverse, we can design the best table layout by considering the most common query and join patterns the table will have to support. But you never request a specific JOIN algorithm, your database guesses for you! Let's learn how it does that so we can design the highest-performance tables.
ClickHouse is an analytical database that can support massive datasets. For the reasons above, it supports six different JOIN algorithms that it'll pick depending on the situation.
The Direct join applies when the right-hand side table supports fast key-value lookups. Thanks to this, it streams through the left table, requesting the right table for each row.
"The Direct join is ClickHouse's fastest join algorithm."
If it cannot do key-value lookups on the table, ClickHouse will build one using a hash table, preferably in memory if the dataset size allows. That's how the three Hash Joins work.
"The algorithm can be constrained by high memory usage. Plus, the creation of the in-memory hash table from the right-hand side table of the join is single-threaded and can become a bottleneck in terms of join execution time if the right-hand side table is very large."
When the tables' existing sort order matches and is along the join key, ClickHouse can use a third class of algorithms: merge joins. Merge joins can stream both tables in parallel, which brings significant memory improvements.
"In such cases, the join performance can be competitive with some of the hash join algorithms from the chart above while generally requiring significantly less main memory."
📗 ClickHouse's Choosing the Right Join Algorithm describes the six JOIN algorithms the database engine supports and how and when they perform best. The author also describes how ClickHouse chooses the best-fitting algorithm based on simple heuristics.
Top 0.1% re-post: How Spotify uses “Golden Paths” to solve fragmentation in their software ecosystem
Tooling fragmentation and inconsistency kill software productivity. How long does onboarding to a new codebase or project take for an existing engineer? If the answer is more than one day, you probably have a fragmentation problem.
"[...] the only way to find out how to do something was to ask your colleague. 'Rumour-driven development', we endearingly called it".
Is it a price to pay to keep autonomous teams? Spotify's Platform Developer Experience teams said no, and introduced the idea of a Golden Path.
"The Golden Path is the opinionated and supported path to build your system, and the Golden Path tutorial walks you through this path."
📗 Spotify's How We Use Golden Paths to Solve Fragmentation in Our Software Ecosystem is one of these simple but truly inspiring reads. Gary Niemen explains the Golden Path and Golden Path tutorials in the Spotify engineering org and how they helped the company solve its fragmentation problem.
"There was a time when engineers at Spotify couldn't imagine life with Golden Paths; now we can't imagine life without them."
Fun fact: the Golden Path name comes from Frank Herbert's "Children of Dune":
"His spice-induced visions show him a myriad of possible futures where humanity has become extinct and only one where humanity survives. He names this future 'The Golden Path' and resolves to bring it to fruition.”