Learning from Giants #68
How Slack takes bigger product bets, Pinterest's wide-column database on RocksDB, Pattern parties to keep codebases consistent, and a fantastic re-post: a survey of API token solutions.
Hi! This is Mathias with your weekly drop of the 1% best, most actionable, and timeless resources to grow as an engineering or product leader. Handpicked from the best authors and companies. Guaranteed 100% GPT-free content.
Did a friend send this to you? Subscribe to get these weekly drops directly in your inbox. Read the archive for even more great content. Also, I share these articles daily on LinkedIn.
Slack's CPO’s principles to take Bigger, Bolder Product Bets
As start-ups scale, small teams become organizations. With them, product development can almost halt as A/B-tested optimization replaces taking big bets.
Slack's Chief Product Officer, Noah Weiss, saw this happen at Slack and shares what he learned from trying to make bigger bets possible again.
"Data can help solve easy problems, but it doesn't actually solve the hard problems."
Noah draws his playbook from observing what small start-ups do that large organizations cannot and trying to set up processes and principles to offset these differences.
1. Share context
"Does everyone making the decision have the same knowledge and information? Is everyone using the same backdrop to weigh the pros and cons fairly?"
"The two fundamental practices to start cracking this code are transparency and repetition."
Transparency makes the right data widely available to everyone, and repetition ensures that context stays at the center of the decision-making process.
2. Build trust
"If there's no trust, it's really hard for there to be conflict. And healthy conflict is necessary to get to any great resolution."
Consistency is what builds trust. Leaders must be consistent in their commitments, feedback, and actions.
"You need to have a culture where you're not putting individual blame when outcomes go wrong. On any individual decision, optimize for the pace of learning and the impact will follow."
3. Factor in risk
"A key skill for any seasoned product leader is knowing which choices warrant careful scrutiny and which don't require sweating the small stuff."
Noah recommends using Amazon's famous "one-way/two-way doors" decision-making framework.
"We have this saying at Slack that you should walk confidently through two-way door decisions. [...] The more two-way door decisions product leaders can identify, the faster they can move."
📗 Noah Weiss' How to take Bigger, Bolder Product Bets is an essential reminder that if leaders don't intentionally set up their organization with operating principles focused on fast and good decision-making, it will ultimately plateau to local optima. Noah also recommends using the 70:20:10 product roadmap rule to manage your organization's global risk portfolio.
Building Pinterest’s wide column database using RocksDB
"In the past two years, we have onboarded more than 300 production use cases, and the service handles millions of requests per second, storing petabytes of data."
That's Rockstorewidecolumn, Pinterest's wide-column distributed database built on top of RocksDB.
1. What's a wide-column database?
"While a simple key value database can be viewed as a persistent hash map, a wide column database can be interpreted as a two dimensional key-value store with a flexible columnar structure."
That means the database doesn't have a fixed schema. It has datasets that are collections of rows, each with as many columns with as many values as wanted.
2. How is it built?
The database relies on a distributed deployment of RocksDB, a key-value store built and maintained by Meta.
"RocksDB is a single node key value store. In order to build a distributed and replicated service using RocksDB, we built a real time replicator library: Rocksplicator."
The wide-column database is implemented on that simple key-value store interface:
"Every RocksDB key is a concatenation of row key, column name, and timestamp of the update operation."
"The value of the column is the RocksDB value."
"While data is sorted in ascending order by default in RocksDB, in our case, we use a custom comparator to sort the keys in descending timestamp order so that the latest versions of a value are accessed first."
That's how the database can handle not having a schema: it stores individual values separately, each in a different key and value. And because the key contains the write timestamp, it's "append only", which also enables value versioning and TTL.
3. A use case: storing large user sequences
Pinterest's core engine saves every content users consume to personalize their feed. That requires storing all user content views and having fast access to them to build feeds in real time.
"The main requirement for the datastore supporting large-scale user sequences at Pinterest was to provide a persistent, append-only log for storing user events."
📗 Rajath Prasad's Building Pinterest's new wide column database using RocksDB details how the team built that datastore. Rockstorewidecolumn is another example of how simple distributed key/value stores can be the foundations of all kinds of databases. And if you want to build your own, Rocksplicator is open source.
Keeping the codebase consistent with pattern parties
Consistency across a large codebase is the most underrated principle in software. And the upside is significant! With high consistency, teams ship much faster, common patterns are known to everyone, and engineers can debug and troubleshoot the entire codebase.
But it's also harmful when used to block all innovation or justify bad practices (e.g. it's not my fault; I copied it from an existing file!). And it is only a helpful principle when your codebase is already consistent.
The largest risk is slowly and silently diverging away from that consistent state.
"At incident.io, we've introduced Pattern Parties as a way to keep our codebase consistent, our design pattern documentation up to date, and our engineers on top of the patterns."
"For a number of days, the engineering team is dedicated to the Pattern Party [...]. We choose a handful of related patterns that we think would be most impactful to implement across the entire codebase, and as a team we apply them as widely as possible."
Each pattern party focuses on a specific pattern that is first agreed upon and then documented and explained to the team. Then, the entire team takes two days to apply the pattern to the whole codebase.
Pattern Parties are not just a way to keep the codebase healthy:
They make patterns explicit and documented.
They onboard all engineers onto the patterns by having them apply them at least once.
They show everyone how important consistency is.
📗 Kelsey Mills' Keeping the codebase consistent with Pattern Parties is another super insightful write-up by the incident.io team about their culture and processes. I love how it makes so many implicit things explicit.
Top 0.1% re-post: API tokens: a tedious survey
As a software engineer, you probably know what JWTs are. But have you ever asked yourself if they were the right fit for your problem? How much do you know about their alternatives?
JWTs have become the go-to API token implementation for a lot of companies because:
They're standardized: there's an IETF working group and multiple RFCs
BigCos like Google & Apple widely use them.
They have an excellent library ecosystem due to 1. and 2.
Yet apart from JWTs, there is a multitude of really good API token schemes:
The unreasonable: username & password
The boring (which doesn't mean bad): random tokens in a database
Somewhat complex: platform tokens, i.e. encrypted user data
Hipster: PASETO (Platform-Agnostic SEcurity TOkens) & Protobuf Tokens
Token-less: Cryptographic signature of requests with a secret key
(Too?) smart: Facebook's CATs, Macaroons and Biscuits
📗 Thomas Ptacek's API Tokens: A Tedious Survey enumerates all of these token schemes to solve the API token problem: "How can I authenticate and identify users calling a set of API endpoints?". It's a visibly passionate, opinionated, eye-opening article.
After starting with the usual boring options, Thomas rants against JWTs before writing a detailed introduction to Macaroons and their cousins. He then ends with a scorecard of solutions, rating their safety, hipsterism (yes! :), complexity, flexibility, and scalability.
Lots of wisdom in the conclusion too:
"I continue to believe that boring, trustworthy random tokens are underrated and that people burn a lot of complexity chasing statelessness they can't achieve and won’t need because token databases for most systems outside of Facebook aren’t hard to scale.
"A couple months ago, I’d have said that Macaroons are underrated in a different way, the way Big Star’s “#1 Record” is. Now I think there's merely underrated like the first Sex Pistols show; everyone who read about them created their own token format. [...] I’d hesitate to recommend them for a typical CRUD "application."
"But, don’t use JWT."