I got feedback that not everyone reads work-related articles on Saturdays (fair enough). So I’ll try sending them on Tuesdays for a few weeks!

Did a friend send this to you? Subscribe to get these weekly drops directly in your inbox. Read the archive for even more great content. Also, I share these articles daily on LinkedIn.

Subscribe now

Ledger: Stripe’s system for tracking and validating money movement

Payments are a complex industry to scale because you're functionally (and legally) required to care about every single penny, regardless of your scale. As they scale to billions of payments daily and hundreds of third parties, the full explainability and reconciliation of theory with the actual bank accounts can become a significant issue.

👉 That's what the Data Quality Platform and Ledger team at Stripe set out to solve.

"Stripe supports more than 135 currencies and payment methods through partnerships with local banks and financial networks in 185 countries."

"Each day, Ledger sees five billion events and 99.99% of our dollar volume is fully ingested and verified within four days"

1. The Data Quality (DQ) Platform

The data quality platform standardizes and centralizes all money movements at Stripe in a database they call Ledger. The platform then offers observability workflows on top of Ledger.

The centralization aspect is super important. Dozens of teams work separately to integrate with local banks and payment providers. They each make design decisions to optimize for their providers and end up with different systems. These systems are nearly impossible to track globally.

2. Ledger: the underlying database

"We want to represent all activity on the Stripe platform in a common data structure that can be analyzed by a single system."

"Ledger encodes a state machine representation of producer systems, and models its behavior as a logical fund flow—the movement of balances (events) between accounts (states)."

So Ledger tracks and analyses each transaction passing through dozens of distributed systems as if it were part of one unified system. Because it does double-entry bookkeeping with these events and accounts, if money gets repeatedly stuck in a system or state, the balance of that state will immediately grow and trigger alerts.

3. Data quality metrics, alerts, and remediation

"We analyze these [transaction-level data flows] with a set of trustworthy DQ metrics that measure the health of a fund flow. Clearing (did the fund flow complete correctly), Timeliness (did the data arrive on time), and Completeness."

"These measurements roll up to create a unified DQ score [...] turning a complex distributed analysis problem into a straightforward tabulation exercise"

The team built an entire observability stack on top of Ledger. Because they know who owns what, teams are alerted in real-time of metric drops and can easily see the impact of deployment and bug fixes.

"Combined, we have the ability to measure and analyze data quality, identify root-cause problems, and flexibly interact with the underlying data constructs to manage our problem load over time."

📗 Ilya Ganelin's Ledger: Stripe's system for tracking and validating money movements is a fascinating peek at the system, keeping the payment behemoth's systems sane, explainable, and reliable. It shows how much value can be created by solving problems with the right system design and abstractions and how that can scale to solve a global problem thanks to the software's near-infinite scale.

Read the full article on Stripe's blog

Measuring Happy GMV, and why I’m over Net Promoter Score.

1. NPS may not be the right north star metric.

"How NPS is calculated makes it an easily gamed, statistically noisy measure of intent, not of behavior."

It also is largely insignificant in early-stage companies because of the smaller sample size.

"What’s critical for any company to keep in their purview is [..] how much happier a person is when they use your product versus any substitute."

"If you make your customers happy enough, you become indispensable. That’s the key to achieving net revenue retention (and organic growth). A noisy algorithm based on how likely someone says they are to refer a friend or colleague is not."

2. Tracking Happy GMV

recommends looking at another metric she calls Happy GMV.

"Ask yourself “what is my best guess at the buyer and seller experience that will lead to retention?” and then measure the percentage of your potential buyers and sellers that get that experience."

"The GMV that qualifies is your Happy GMV."

She gives the example of a ride-sharing marketplace. An initial guess for rider-happy GMV could be the percentage of riders who find a driver quickly and rate them 4 or 5 stars.

Happy GMV is a leading metric and is much more actionable than NPS! You can use it to make daily operational decisions and prioritization.

"Over time, as your understanding of your customers improves and you start to have statistical significance in your cohorts, you’ll be able to tune your “Happy GMV” thresholds to more accurately reflect the experience that leads to retention and keep you focused on the most important levers."

📗 's Measuring Happy GMV and Why I’m Over Net Promoter Score is a super actionable article for most companies and, in particular, marketplaces. Sarah has been a long-time partner at Benchmark, a Tier 1 super opinionated VC fund with deep expertise in marketplaces.

Read the full article on Sarah's blog

Understanding LSM Trees: What Powers Write-Heavy Databases

Cassandra, ScyllaDB, RocksDB, Google BigTable, and many other databases share one thing. They all use the same core data structure to read and write data, albeit with different implementations, because it allows for write-heavy workloads.

That data structure is the log-structured merge tree (LSM tree). Here's a super gentle introduction:

1. The Write path

"The write path is optimized by only performing sequential writes"

That's the main point. LSM trees only write large sequential (i.e. sorted by key) blobs of data to disk. Since they're sorted, they're a lot easier to query when looking for a specific key. These blobs are called segments or runs and are part of the Sorted String Table (SSTable).

SSTable doesn't work independently because writes do not come as large, pre-sorted blobs of data. They come as individual, unorganized key+value instructions.

"This is solved by using an in-memory tree structure. This is frequently referred to as a memtable."

"As writes come in, the data is added to this red-black tree. Our writes get stored in this red-black tree until the tree reaches a predefined size."

When the Memtable is large enough, it's written to disk, adding a new segment to our SSTable.

2. The Read path.

Finding a specific value in the SSTable requires scanning all segments from the most recent and stopping once the given key is found. In practice, databases use optimizations like indexes to help with scanning.

"What about looking up records that do not exist? We will still end up looping over all segment files and fail to find the key in each segment. This is something that a bloom filter can help us out with."

The Bloom filter can tell you it's 100% sure that some keys aren't in the dataset, with high compression, at the cost of being wrong in rare instances.

3. Optimization

You may think the performance must grow terribly with the number of segments. That is correct.

"These segment files need to be cleaned up and maintained in order to prevent the number of segment files from getting out of hand. This is the responsibility of a process called compaction. Compaction continuously combines old segments together into newer segments."

📗 Braden Groom's Understanding LSM Trees: What Powers Write-Heavy Databases is the best quick introduction to the data structure you'll find online. It may seem simple in the article, but each concept varies widely in implementation from one database to the other, with some small tricks everywhere to push performance. Although it's a 2020 article, it's not even on the Internet anymore, so thanks to the Internet Archive for keeping it up!

Read the full article on Braden's blog

Top 0.1% re-post: Engineering Levels: Avoiding the Scope Trap

All tech companies end up building an engineering ladder. It makes sense for engineers to have a clear path forward. When they do, they often are heavily inspired by the dozens of publicly shared ladders.

That’s what Honeycomb did at first, but soon they started seeing gaps and small unsettling details. The biggest one? The importance that they gave to growing scope compared to other criteria.

“A ladder that conflates advancement with scope is a ladder that only rewards engineers who work on the largest projects. This doesn’t seem fair.”

“More often than not, scope is the enemy. We would rather reward engineers who find clever ways to limit scope by decomposing problems in both time and size.”

📗 Honeycomb’s Engineering Levels: Avoid the Scope Trap is an eye-opening post on how the company defined engineer career growth. It’s great because Ben Darfler explains the problems with their current solution and how they fixed them. It's super actionable and relatable.

The main result is a two-dimensional ladder, adding ownership to balance the impact of scope.

“Scope progresses from focusing on tasks to features to projects, products, and the company as a whole.”

“Ownership progresses from focusing on the execution of work to the process of delivering work, to the discovery of solutions to deliver, to the discovery of problems to solve.”

Read the full article

I got feedback that not everyone reads work-related articles on Saturdays (fair enough). So I’ll try sending them on Tuesdays for a few weeks!

Did a friend send this to you? Subscribe to get these weekly drops directly in your inbox. Read the archive for even more great content. Also, I share these articles daily on LinkedIn.

Subscribe now

From data to data viz

Every chart tells a story, but not all stories are best told by a pie chart.

Using the appropriate graph can turn a vague illustration into a central part of your communication. There are dozens of graphs to pick from, including some complex ones like Chord diagrams, Sankeys, and sunbursts, so how can you make the right choice?

Use a decision tree made by experts!

"Pick the main [data] type using the buttons below. Then let the decision tree guide you toward your graphic possibilities".

Once you've picked a chart, you still have a few layout and density decisions to make, and here again, you could make mistakes.

"Avoid the rainbow color palette when you map a numeric variable."

"Placing individual plots horizontally or vertically is an important choice to make"

Experts can help us here too!

📗 Yan Holtz's From data to Viz is a precious resource to help you improve your data visualization skills with very simple heuristics. Use their graph decision tree and read the gallery of typical caveats to get started now!

Read the full guide

How to pass a coding interview with Robert Heaton

A large part of all coding interviews fail for dumb reasons. You can't claim you're the best engineer there is and take 10 minutes to get your IDE to run code.

So here are a few must-haves for any coding interview:

1. "Set up a basic hello-world program beforehand and make sure you can run it."

2. "Start by drafting your program using prose of pseudocode. Don't start by writing actual code."

3. "Once you've started writing code, run it as frequently as you can."

Ideally you should be doing that using unit tests. It's cleaner and safer than running by hand.

"Don't use a testing framework unless you're required to or you're very confident with it."

4. Keep your code reasonably simple.

"Don't abstract too soon is good advice for the real world, but it's particularly important in an interview. The costs of choosing an incorrect abstraction are higher than normal."

And don't try to cover all cases in your first iteration. Leave TODOs for edge cases and clean them up once the base case works.

Additionally, remember you're there to complete a coding challenge in a given amount of time, so everything you do should be optimized around time efficiency.

5. "Talk as much as you reasonably can". Use it to surface misunderstandings as early as possible.

6. "Consider asking for help." Succeeding after getting some help is often better than failing by staying stuck.

📗 Robert Heaton's How to pass a coding interview with me is a collection of 22 very sound tips the author has learned by interviewing hundreds of candidates in their careers. If you're interviewing or considering interviewing you should definitely have a read.

Read the full article on Robert's blog

Deep dive into Neon’s storage engine

Separation of Compute and Storage is the hottest thing in databases. Combined with cloud storage services like AWS S3 and serverless technology like Kubernetes, this pattern allows for virtually infinite-sized databases.

How does this separate storage work in practice?

Neon defines itself as a Serverless Postgres database with separate storage and compute. The compute part is the actual Postgres engine, but the team swapped Postgres' on-disk storage with a remote solution: separate storage servers backed by cloud storage.

Storage engine: under the hood

Neon's storage engine manages Postgres' storage primitives: pages and write-ahead-log (WAL) entries.

"In Neon, [the write-ahead-log of changes] is streamed to three safekeeper nodes, which provide durability and synchronous replication using Paxos-like consensus algorithm. The safekeeper nodes then stream the logs to the pageservers, where the logs are processed, reorganized, and written to immutable files. Finally, these files are uploaded to cloud storage for long-term storage."

Write path

All durable data is append-only: no data is ever modified.

"Neon eliminates the distinction between active data and backups, treating all data the same in the storage system. Because old data is not modified, whenever new data is digested into the system, all the history stays accessible up to a configurable limit."

Every update is given a monotonically integer number (the log sequence number, LSN) which acts as a cluster-wide clock.

"When a read request comes in, the pageserver finds the last image of the page by the LSN, and any write-ahead-log (WAL) records on top of it, applies the WAL records if needed to reconstruct the page, and returns the materialized page to the caller."

Every page is thus simply an ordered list of changes. Since all reads specify a sequence number, the storage server replays that change history up to the given number and returns the built page value.

But that solution slows down as the number of changes grows.

Neon fixes that by compacting the changes into Image layers which are complete images of the page at a given sequence number. This way, the pageserver only needs to apply the changes after the most recent image.

📗 Heikki Linnakangas' Deep dive into Neon storage engine is a simple introduction to storage and compute separation in data stores with a real-world example: Neon. Even though it states the separation simplifies the overall system, the article also shows how it opens the database to a new class of distributed systems challenges.

Read the full article on Neon's blog

Top 0.1% re-post: Stripe: Thinking Like a Civilization

/dev/payments

That's the initial name of one of our time's biggest and most inspiring private companies. Do you have any idea which one?

Hint: its founders are siblings and probably two of the most brilliant business people on the planet.

"In Patrick and John Collison, the payments company has a founding pair unusual in their intellect, rarer still in their ego-less vision. From that inspiration, the brothers have manifested an entire culture, characterized by long-term thinking, rapid execution, and the desire to build something meaningful. They've done so while constructing a remarkable business capable of dominating the internet's financial infrastructure even as it empowers millions of other merchants."

"All of which is to say that Stripe thinks like a civilization. It is well on its way to becoming one."

📗 The Generalist's Stripe: Thinking Like a Civilization may be a slight stretch from what I usually share, but I believe as a product or engineering leader, there are some stories you must know. Moreover, the essay is packed with anecdotes and details about Stripe's culture and how the Collison brothers see the world.

Read the full article on the Generalist

Did a friend send this to you? Subscribe to get these weekly drops directly in your inbox. Read the archive for even more great content. Also, I share these articles daily on LinkedIn.

Subscribe now

Examples of great URL design

URLs don't have to be boring or, worse, opaque data blobs.

Even though browsers tend to hide them more and more, URLs are an essential element of web applications. When done right, they can help your user navigate faster by typing in specific URLs, understand links without clicking, or even participate in your brand's storytelling. Let's look at cool examples:

StackOverflow appends an optional, human-readable slug to all their question URLs

Example: stackoverflow.com/questions/16245767/creating-a-blob-from-a-base64-string-in-javascript/

"The beauty is :slug is an optional parameter in the URL." It's only added for the human reading it! The computer only uses the preceding ID.

Slack uses clever URL design to promote their marketing campaign "Slack is ...".

"I've always found delight in URLs that try to form natural language sentences."

“The URL is the Product” in some technical products.

The URL is the Product in technical products like Github and NPM, which can be powerful when it neatly maps domain semantics.

Example: github.com/django/django/compare/4.2.7...main compares two refs like it would in git.

📗 Jim Nielsen's Examples of Great URL Design is a great reminder that URLs aren't just technical details. When crafted with care, they can make the user's experience delightful and sometimes even efficient!

Read the full article on Jim's blog

When and How to Build Second Products

Building a Second Product can be a business-defining moment or a total miss.

Wild successes include Snapchat Stories and Uber Eats.

"Most companies struggle to understand when they need to start investing in adding new product value vs. just continuing to grow off the traction of their initial product/market fit."

When to build a second product?

"Companies can rarely ride one product into the IPO sunset anymore. [...] it is becoming necessary earlier and earlier in a company's lifecycle due to a confluence of factors."

There are multiple factors to explain it. Tech products are a lot easier to build nowadays, making every space more crowded and overall competition fiercer.

To answer the "When" question, you must analyze your growth and market dynamics. Can you grow 10x on your current product? Do your market size, retention, and acquisition metrics allow it?

For companies like Figma, the answer is Yes, and they can ride on a single product for a while. Others, like Canvas, had to keep moving to stay relevant and launch many new products. Adjacent spaces but widely different personas and market dynamics.

How to build a second product?

"The main vectors on which product expansion should be evaluated is whether the expansion changes the product, changes the target market, or changes the core competencies required to deliver the product's value."

But what matters is "how" success is defined for this new product and how it impacts the product-building model. And it's not the Product/Market fit!

"New products may only need to influence one of the three (retention, monetization, growth) to be successful. But the key is, they need to influence it for the overall company, not just the product itself."

That's why the internal startup or innovation department model rarely creates massively successful second products. They’re designed in silos and assess projects separately. Instead, new products must be large bets into which a company invests significant energy.

📗 Casey Winters' When and How to Build Second Products is a must-read for product leaders. It'll force you to reflect on your growth model and think analytically about new products. I did my best to summarize the main takeaways, but that’s really only 5% of the article. Go read it!

Read the full article on Casey's blog

Hiding credentials from Rails at Fly.io

"At the stage we're at, all secrets are hazmat. [...] And Rails itself is the portion of our attack surface we're least confident about."

So the team at Fly.io developed a hardened solution to store and use secrets that never expose anything plaintext to Rails while still using Rails as their main app framework.

They call it Tokenizer.

"Tokenizer is a stateless HTTP proxy that holds the private key of a Curve25519 keypair. When we get a new 3rd party API secret, we encrypt it to Tokenizer's public key."

Tokenizer is written in Go, a strongly typed memory-safe language that makes it much safer than Rails.

Anytime the Rails app needs to make an HTTP request using these secrets, it does so through Tokenizer. It sends the original request with additional Proxy-Tokenizer headers containing the encrypted secrets and instructions to use them once decrypted.

"Each Proxy-Tokenizer header carries an encrypted secret and instructions for Tokenizer to rewrite the request in some way, usually by injecting the decrypted plaintext into a header."

Of course, these secrets can be locked down to specific hosts to prevent the most basic forms of extraction.

And there is more! While Tokenizer handles most forms of Bearer tokens and request signatures, the Fly team created its sister system, SSOkenizer, to secure multi-step OAuth2 flows without exposing any step to Rails.

📗 Ben Toews' Tokenized Tokens is an excellent write-up on securing secrets without impacting the developer experience. It's a fantastic system design and, even better, it’s fully open-source!

Read the full article on Fly's blog

Top 0.1% re-post: The Importance of Writing

"Writing is a normalizing medium. No matter what you look like, how old you are, how you speak, or how confident you are, you can sit on your own and formulate your thoughts [...] An impactful message is an impactful message."

Writing helps in many ways, yet it takes effort to learn and improve.

Firstly, the best way to get better at writing is just to write as much as you possibly can."

📗 James Stanier's The importance of writing is a post about the impact of writing at work. It will convince you of its usefulness if you're not already, but the article's value is in the actionable advice it contains. If you only have 2 minutes, skip to the five ways to practice being more impactful at work through your writing.

The one that resonated with me most is "Think through problems by writing them out." This practice has worked remarkably well for me over the past years.

"Writing is so powerful. Wield that power for your benefit."

Read the full article on James' blog

Did a friend send this to you? Subscribe to get these weekly drops directly in your inbox. Read the archive for even more great content. Also, I share these articles daily on LinkedIn.

Subscribe now

Slack's CPO’s principles to take Bigger, Bolder Product Bets

As start-ups scale, small teams become organizations. With them, product development can almost halt as A/B-tested optimization replaces taking big bets.

Slack's Chief Product Officer, Noah Weiss, saw this happen at Slack and shares what he learned from trying to make bigger bets possible again.

"Data can help solve easy problems, but it doesn't actually solve the hard problems."

Noah draws his playbook from observing what small start-ups do that large organizations cannot and trying to set up processes and principles to offset these differences.

1. Share context

"Does everyone making the decision have the same knowledge and information? Is everyone using the same backdrop to weigh the pros and cons fairly?"

"The two fundamental practices to start cracking this code are transparency and repetition."

Transparency makes the right data widely available to everyone, and repetition ensures that context stays at the center of the decision-making process.

2. Build trust

"If there's no trust, it's really hard for there to be conflict. And healthy conflict is necessary to get to any great resolution."

Consistency is what builds trust. Leaders must be consistent in their commitments, feedback, and actions.

"You need to have a culture where you're not putting individual blame when outcomes go wrong. On any individual decision, optimize for the pace of learning and the impact will follow."

3. Factor in risk

"A key skill for any seasoned product leader is knowing which choices warrant careful scrutiny and which don't require sweating the small stuff."

Noah recommends using Amazon's famous "one-way/two-way doors" decision-making framework.

"We have this saying at Slack that you should walk confidently through two-way door decisions. [...] The more two-way door decisions product leaders can identify, the faster they can move."

📗 Noah Weiss' How to take Bigger, Bolder Product Bets is an essential reminder that if leaders don't intentionally set up their organization with operating principles focused on fast and good decision-making, it will ultimately plateau to local optima. Noah also recommends using the 70:20:10 product roadmap rule to manage your organization's global risk portfolio.

Read the full article

Building Pinterest’s wide column database using RocksDB

"In the past two years, we have onboarded more than 300 production use cases, and the service handles millions of requests per second, storing petabytes of data."

That's Rockstorewidecolumn, Pinterest's wide-column distributed database built on top of RocksDB.

1. What's a wide-column database?

"While a simple key value database can be viewed as a persistent hash map, a wide column database can be interpreted as a two dimensional key-value store with a flexible columnar structure."

That means the database doesn't have a fixed schema. It has datasets that are collections of rows, each with as many columns with as many values as wanted.

2. How is it built?

The database relies on a distributed deployment of RocksDB, a key-value store built and maintained by Meta.

"RocksDB is a single node key value store. In order to build a distributed and replicated service using RocksDB, we built a real time replicator library: Rocksplicator."

The wide-column database is implemented on that simple key-value store interface:

1. "Every RocksDB key is a concatenation of row key, column name, and timestamp of the update operation."

2. "The value of the column is the RocksDB value."

3. "While data is sorted in ascending order by default in RocksDB, in our case, we use a custom comparator to sort the keys in descending timestamp order so that the latest versions of a value are accessed first."

That's how the database can handle not having a schema: it stores individual values separately, each in a different key and value. And because the key contains the write timestamp, it's "append only", which also enables value versioning and TTL.

3. A use case: storing large user sequences

Pinterest's core engine saves every content users consume to personalize their feed. That requires storing all user content views and having fast access to them to build feeds in real time.

"The main requirement for the datastore supporting large-scale user sequences at Pinterest was to provide a persistent, append-only log for storing user events."

📗 Rajath Prasad's Building Pinterest's new wide column database using RocksDB details how the team built that datastore. Rockstorewidecolumn is another example of how simple distributed key/value stores can be the foundations of all kinds of databases. And if you want to build your own, Rocksplicator is open source.

Read the full article

Keeping the codebase consistent with pattern parties

Consistency across a large codebase is the most underrated principle in software. And the upside is significant! With high consistency, teams ship much faster, common patterns are known to everyone, and engineers can debug and troubleshoot the entire codebase.

But it's also harmful when used to block all innovation or justify bad practices (e.g. it's not my fault; I copied it from an existing file!). And it is only a helpful principle when your codebase is already consistent.

The largest risk is slowly and silently diverging away from that consistent state.

"At incident.io, we've introduced Pattern Parties as a way to keep our codebase consistent, our design pattern documentation up to date, and our engineers on top of the patterns."

"For a number of days, the engineering team is dedicated to the Pattern Party [...]. We choose a handful of related patterns that we think would be most impactful to implement across the entire codebase, and as a team we apply them as widely as possible."

Each pattern party focuses on a specific pattern that is first agreed upon and then documented and explained to the team. Then, the entire team takes two days to apply the pattern to the whole codebase.

Pattern Parties are not just a way to keep the codebase healthy:

• They make patterns explicit and documented.

• They onboard all engineers onto the patterns by having them apply them at least once.

• They show everyone how important consistency is.

📗 Kelsey Mills' Keeping the codebase consistent with Pattern Parties is another super insightful write-up by the incident.io team about their culture and processes. I love how it makes so many implicit things explicit.

Read the full article

Top 0.1% re-post: API tokens: a tedious survey

As a software engineer, you probably know what JWTs are. But have you ever asked yourself if they were the right fit for your problem? How much do you know about their alternatives?

JWTs have become the go-to API token implementation for a lot of companies because:

1. They're standardized: there's an IETF working group and multiple RFCs

2. BigCos like Google & Apple widely use them.

3. They have an excellent library ecosystem due to 1. and 2.

Yet apart from JWTs, there is a multitude of really good API token schemes:

• The unreasonable: username & password

• The boring (which doesn't mean bad): random tokens in a database

• Somewhat complex: platform tokens, i.e. encrypted user data

• Hipster: PASETO (Platform-Agnostic SEcurity TOkens) & Protobuf Tokens

• Token-less: Cryptographic signature of requests with a secret key

• (Too?) smart: Facebook's CATs, Macaroons and Biscuits

📗 Thomas Ptacek's API Tokens: A Tedious Survey enumerates all of these token schemes to solve the API token problem: "How can I authenticate and identify users calling a set of API endpoints?". It's a visibly passionate, opinionated, eye-opening article.

After starting with the usual boring options, Thomas rants against JWTs before writing a detailed introduction to Macaroons and their cousins. He then ends with a scorecard of solutions, rating their safety, hipsterism (yes! :), complexity, flexibility, and scalability.

Lots of wisdom in the conclusion too:

"I continue to believe that boring, trustworthy random tokens are underrated and that people burn a lot of complexity chasing statelessness they can't achieve and won’t need because token databases for most systems outside of Facebook aren’t hard to scale.

"A couple months ago, I’d have said that Macaroons are underrated in a different way, the way Big Star’s “#1 Record” is. Now I think there's merely underrated like the first Sex Pistols show; everyone who read about them created their own token format. [...] I’d hesitate to recommend them for a typical CRUD "application."

"But, don’t use JWT."

Read the full article

Did a friend send this to you? Subscribe to get these weekly drops directly in your inbox. Read the archive for even more great content. Also, I share these articles daily on LinkedIn.

Subscribe now

Small Change-Lists make Google fast

Google engineers' secret trick to ship 100x faster than the industry average.

Small change lists, really small ones, make a massive difference on a team's velocity. They're quicker, easier to visualize, better reviewed, and ultimately contain fewer bugs.

Why does everyone know about this rule, but most don't follow it?

They don't truly know what is "small"

"The right size for a CL is one self-contained change. [...] The CL makes a minimal change that addresses just one thing."

"There are no hard and fast rules about how large is "too large." 100 lines is usually a reasonable size for a CL, and 1000 lines is usually too large, but it's up to the judgment of your reviewer."

Yes, Google recommends 100-line changes. If you're experienced, you know that's truly small. Still, that limited size shouldn't make the changes harder to understand or split away tests.

The team isn't prioritizing reviews enough, and small CLs pile up

"You want to find some way to work that won't block you while you're waiting for review. This could involve having multiple projects to work on simultaneously, finding reviewers who agree to be immediately available, doing in-person reviews, pair programming, or splitting your CLs in a way that allows you to continue working immediately."

High-velocity teams usually use all the above: efficient tooling to stack CLs, strict review SLAs, and a lot of sync review and work.

They don't know how to split

"It's often useful to think about how to split and organize those CLs at a high level before diving into coding."

There are many ways to split work: by file, feature, or layer. Sometimes, it's harder to do, but there are some general rules you must always follow:

• Separate refactorings. They're the hardest to review, so they should never be mixed up with new functionality.

• CLs should include related test code.

• Keep the CI green for every chunk.

📗 Google's Small CLs article is part of the company's published Code Review guidelines. It's small, super actionable, and probably the most impactful practice one can have on a team's velocity.

Read the full guide

Building and operating a pretty big storage system called S3

AWS's S3 is a lot more than disk storage as a service. Its massive scale leads to emergent properties that no disk can match.

Andrew Warfield worked on S3 for many years – in a recent presentation, he explains how he understands and reasons about the system:

"S3 is an object storage service with an HTTP REST API. There is a frontend fleet with a REST API, a namespace service, a storage fleet that's full of hard disks, and a fleet that does background operations."

The main feat of S3 is managing a very spiky workload on cheap hard drives, all while enabling extreme I/O performance. The 20TB hard drives are incredibly cheap storage, but their I/O performance doesn't scale with size.

"This tension between HDDs growing in capacity but staying flat for performance is a central influence in S3's design. We need to scale the number of bytes we store by moving to the largest drives we can as aggressively as we can."

That's solved by splitting data into chunks and replicating it across hundreds of drives, which enables the distribution of load so widely that spiky workloads are averaged out in the massive number of requests.

"As I really started to understand the system I realized that it was the scale of customers and workloads using the system in aggregate that really allow it to be built differently, and building at this scale means that any one of those individual workloads is able to burst to a level of performance that just wouldn't be practical to build if they were building without this scale."

📗 Andy Warfield's Building and operating a pretty big storage system called S3 is a fascinating read about the unique scale of S3. While Andy makes it look simple, the data distribution is probably only one half of the problem, the write path. I imagine choosing which disk to read from and orchestrating these reads is at least as interesting!

Read the full article on W. Vogels' blog

An interactive guide to CRDTs

Every app has real-time collaboration now. Build yours with CRDTs!

And while you'll likely use an off-the-shelf solution (sponsors are welcome 😉), a little theory never hurts!

"CRDT stands for Conflict-free Replicated Data Type."

It's a kind of data structure that can be stored on different computers (peers). Peers may have different states at different points in time, but are guaranteed to eventually converge on a single agreed-upon state."

There's your real-time collaboration feature. Let's get a little more specific: there are two approaches to CRDTs: state-based or operation-based.

"State-based CRDTs transmit their full state between peers, and a new state is obtained by merging all the states together. Operation-based CRDTs transmit only the actions that users take, which can be used to calculate a new state."

As often happens in software, the best-looking solution is a lot more complex to get right. So, let's study the simplest first: state-based replication.

The theory behind state-based CRDTs is indeed simple. The peers exchange states to agree on a common one and rely on a merge function to apply others' changes to their local state. The CRDT works if this merge function satisfies the following mathematical properties: commutativity, associativity, and idempotency.

The magic is that because these properties are composable, you can build complex CRDTs by combining the simplest valid CRDT constructs.

One of these simple constructs is the Last Write Wins (LWW) register.

"Last Write Wins Registers, as the name suggests, simply overwrite their current value with the last value written. They determine which write occurred last using timestamps, represented here by integers that increment whenever the value is updated."

📗 Jake Lazaroff's An interactive guide to CRDTs then builds an LWW register using Typescript, and combines multiple ones into an LWW Map. All of this is illustrated with brilliant interactive examples. The smoothest introduction to CRDTs!

Read the full interactive guide

Top 0.1% re-post: The power of Product Thinking

If you're building a product, your job is to make hypotheses and guesses. Iterate and experiment, right?

Yet each of these iterations has a cost, so how do you avoid betting on guesswork?

💡 Product thinking.

"Product thinking is the skill of knowing what makes a product useful — and loved — by people. [...] Product thinking is a habit, an eye, a mindset."

In a fast-changing environment, product thinking can make the difference between 0 and 1. And lucky us, it can be developed!

"The two most important habits are observation and inquiry."

"Observation is about paying attention to people's reactions when they encounter products or services in their day-to-day lives."

"Inquiry comes from genuine curiosity about people and their behaviors, and can take different forms depending on how you learn best. The key is understanding the "why" behind the reactions."

📗 's The Power of Product Thinking is a reference article. After explaining that product thinking will make you and your products stand out, the author describes how to develop it with simple habits. With practice and curiosity, observation and inquiry become a defining mindset.

Read the full article

Did a friend send this to you? Subscribe to get these weekly drops directly in your inbox. Read the archive for even more great content. Also, I share these articles daily on LinkedIn.

Subscribe now

How Precision Time Protocol is being deployed at Meta

How Meta spent millions to get time accuracy in its data centers from microseconds to nanoseconds.

Synchronization is the main problem of distributed systems, but what makes it the toughest is that you can't just use time. That's because using time means agreeing on what time it is, which is itself a synchronization problem!

That's why most distributed systems like databases are built with the assumption that you can't trust machine time, which complicates many things and ultimately affects these systems' latency.

But time isn't just a virtual concept; the physics world defines it. If only machines could get access to a trustable physics time! It turns out that's what hardware engineers have iterated on for years, and as data centers and distributed systems grow even larger, that problem is as important as ever.

So, Meta research scientists and engineers built a full-stack solution from the GNSS antenna on the roof to the hardware appliance, the time server, the client, and their monitoring.

What's fascinating is that many tiny details become large challenges when you want to get to nanoseconds time accuracy.

1. The time appliance needs to be configured on the fly to adapt its configuration to the state of all its components.

2. The travel time of the time information from the appliance to the client, located anywhere in the datacenter, needs to be accounted for as a "time offset".

3. Time isn't a strict value anymore, but a "window of uncertainty"

"As you may see, the API doesn't return the current time (aka time.Now()). Instead, it returns a window of time which contains the actual time with a very high degree of probability."

📗 Ahmad Byagowi's How Precision Time Protocol is being deployed at Meta is a fascinating long read about the entire stack the Meta teams designed to get nanosecond time accuracy. If anything, it's a good reminder not to trust time.Now()!

Read the full article on Meta's blog

How to Turn Good Design Direction into a Good System

"There's this myth in creative spaces that systems are where good ideas go to die. [...] But structure and systems are not just a designer's most important tools — they are, ultimately, what design is."

And it's true in everything! We always give more credit to the initial idea than it deserves and less to execution. That leads people to spend too much time on ideation compared to the execution of structure and systems.

"When this happens, I'll find myself in reviews of initial layouts or compositions asking why there are so many new ideas present."

So Christopher Butler came up with the "Idea to Structure Ratio".

1. Idea Generation — 10%

"Good ideas probably account for about 10% of the effort in a design project. That's right. Just ten percent."

"The idea is to show how those elements can work together in a variety of different ways and secure your constituents' comfort with the visual language without asking them to approve specific layouts."

2. Structure — 20%

"Creating a structure to support them — creating the pages, screens, or scenes that provide a consistent landscape for attention and use — is about 20%."

This is the wireframing stage. The idea is to go from a very rough structure to "just enough detail to demonstrate the information architecture of the space as well as visually indicate the priority of information and possible actions."

3. Sub-Structure — 40%

"Integrating the sub-structure in design — considering taxonomies of information, connections between elements (both functional and conceptual), and the admin layer of user experience — is actually 40%."

"This is the comprehensive systematization of a thing. It's the largest portion of effort and value because of how detailed it is, how much collaboration it requires, and how much it impacts the strength, scalability, and overall elegance of the final product. [...] This is where designed components become system modules."

4. Visual System — 20%

"Creating the final visual system — the complete visual language and pattern library that is the result of joining the initial idea to the outer and inner structures — is the next 20%."

This is the high-fidelity design stage.

"This is the stage where, if the previous two aren't handled well, I start to see a lot of unnecessary new creation that distracts from progress being made or obstructs a system from cohering or remaining together."

5. Documentation — 10%

"Finally, the documentation of the visual system is the last 10%."

📗 Christopher Butler's How to Turn Good Design Direction into a Good System is a very dense explanation of how Chris runs his design projects and of the steps where he spends most of his time. Twenty years of iteration and experience that you should definitely read.

Read the full article on Chris's blog

Choosing the right join algorithm (ClickHouse)

Database joins don't have a one-size-fits-all solution. Like most database operations, finding the optimal algorithm to execute a join statement is a latency vs. memory tradeoff that heavily depends on the pre-existing alignment of the data layout and the join sort order.

By thinking in reverse, we can design the best table layout by considering the most common query and join patterns the table will have to support. But you never request a specific JOIN algorithm, your database guesses for you! Let's learn how it does that so we can design the highest-performance tables.

ClickHouse is an analytical database that can support massive datasets. For the reasons above, it supports six different JOIN algorithms that it'll pick depending on the situation.

1. The Direct join applies when the right-hand side table supports fast key-value lookups. Thanks to this, it streams through the left table, requesting the right table for each row.

"The Direct join is ClickHouse's fastest join algorithm."

2. If it cannot do key-value lookups on the table, ClickHouse will build one using a hash table, preferably in memory if the dataset size allows. That's how the three Hash Joins work.

"The algorithm can be constrained by high memory usage. Plus, the creation of the in-memory hash table from the right-hand side table of the join is single-threaded and can become a bottleneck in terms of join execution time if the right-hand side table is very large."

3. When the tables' existing sort order matches and is along the join key, ClickHouse can use a third class of algorithms: merge joins. Merge joins can stream both tables in parallel, which brings significant memory improvements.

"In such cases, the join performance can be competitive with some of the hash join algorithms from the chart above while generally requiring significantly less main memory."

📗 ClickHouse's Choosing the Right Join Algorithm describes the six JOIN algorithms the database engine supports and how and when they perform best. The author also describes how ClickHouse chooses the best-fitting algorithm based on simple heuristics.

Read the full article

Top 0.1% re-post: How Spotify uses “Golden Paths” to solve fragmentation in their software ecosystem

Tooling fragmentation and inconsistency kill software productivity. How long does onboarding to a new codebase or project take for an existing engineer? If the answer is more than one day, you probably have a fragmentation problem.

"[...] the only way to find out how to do something was to ask your colleague. 'Rumour-driven development', we endearingly called it".

Is it a price to pay to keep autonomous teams? Spotify's Platform Developer Experience teams said no, and introduced the idea of a Golden Path.

"The Golden Path is the opinionated and supported path to build your system, and the Golden Path tutorial walks you through this path."

📗 Spotify's How We Use Golden Paths to Solve Fragmentation in Our Software Ecosystem is one of these simple but truly inspiring reads. Gary Niemen explains the Golden Path and Golden Path tutorials in the Spotify engineering org and how they helped the company solve its fragmentation problem.

"There was a time when engineers at Spotify couldn't imagine life with Golden Paths; now we can't imagine life without them."

Fun fact: the Golden Path name comes from Frank Herbert's "Children of Dune":

"His spice-induced visions show him a myriad of possible futures where humanity has become extinct and only one where humanity survives. He names this future 'The Golden Path' and resolves to bring it to fruition.”

Read the full article on Spotify's blog

Did a friend send this to you? Subscribe to get these weekly drops directly in your inbox. Read the archive for even more great content. Also, I share these articles daily on LinkedIn.

Subscribe now

Making sense of React Server Components

React Server Components are the most confusing React update in many years. Thankfully, you will likely use them through a framework like Next.js that abstracts away complexity and will never have to deal with them directly. Yet RSCs are such a paradigm shift that learning about them is helpful.

The main problem

Server-side rendering is everywhere now. Its initial idea was to simulate a client environment on the server to render a first HTML page and display it to the user while the large Javascript bundle is loading.

As the SSR frameworks expanded, developers started asking to customize that initial HTML page and fill it with user data to make it more relevant.

"In order for this to work, we'd need to be able to give React a chunk of code that it runs exclusively on the server, to do the database query. But that hasn't been an option with React… even with Server Side Rendering, all of our components render on both the server and the client."

The solution: React Server Components

React Server Components solve that problem: they're guaranteed to only run on the server and won't be part of the browser Javascript bundle. The other components are called Client Components.

"Server Components never re-render. They run once on the server to generate the UI. The rendered value is sent to the client and locked in place."

This paradigm has significant consequences on what RSCs can and cannot do:

• RSCs cannot hold mutable state.

• RSCs cannot receive props from Client Components. Thus, they cannot be imported by Client Components. If they are, React automatically turns them into Client Components.

• All components are considered React Server Components unless specified.

"One of the biggest "ah-ha" moments I had with React Server Components was the realization that this new paradigm is all about creating client boundaries."

Indeed, because of the propagation of the "Client-ness" in the import tree, boundaries and structure define what will be interactive and what will render only once on the server.

📗 Josh Comeau's Making Sense of React Server Components is an excellent introduction to RSCs. Josh uses his creativity and React and animation skills to describe how they work in detail. While you will probably only use them through frameworks, the article details weird but logical design decisions that, once understood, will save you hours of madness.

Read the full article on Josh's blog

Golden Rules for Organizational Structure

Your 200-person org doesn't need VPs, Directors AND managers. Here's how to fix it.

From ~100 FTEs up to thousands, it's a good practice for every tech organization to reflect on its structure. A periodic audit of the overall organization design can help uncover and fix flaws. What exactly should you look for?

Joshua Burgin has spent most of his 30-year career as a tech executive and gives Golden rules and numbers for organizational design.

"The guidance principles are divided into the following areas: Span of control, Layers, PM to Engineering Ratios, Engineering Seniority Proportions"

Span of control

"This refers to the number of direct reports that a manager oversees and/or manages within an organization.

• Managers should oversee 6 to 10 reports. [...] Directors, VPs & above are capable of managing 8-12 reports

• No one with a Management title (including Directors and VPs) with less than 5 direct reports."

Senior ICs can also report to VPs or Director, which the author recommends, through this principle:

"Managers, Directors, and VPs should ideally be at the same or higher level as the ICs they oversee."

Layers

"In organizational design, "layers" refer to the hierarchical levels or depth within a company's structure."

It's no secret that organizations should be kept as flat as possible.

• "The median gap between the lowest level ICs and VPs should be 2 layers of management (IC → Manager/Director → VP should be the norm)

• The maximum gap between the lowest level ICs and VPs should not exceed 3 managers."

Engineering ratios

These are the rules that apply the earliest in a company's lifespan.

First, you should keep the feature teams vs. common services (sometimes called platform) teams ratio between 70/30 (usual) and 90/10 (high-performing orgs).

Second, you should balance seniority inside the organization and its teams.

• "Aim for 30% of your org junior/entry level. You can have a little less or more, but not more than 40%.

• 40-50% of your org should be mid-career level."

• 20% of your org should be the senior-most ICs in technical leadership (typically "Senior Staff" and "Principal Engineers")."

📗 's Golden Rules for Organizational Structure gives reference numbers to design healthy organizations. Beyond people ratios, he also mentions that every organization is different and may warrant breaking these rules as long as the organization's design is intentional and continuously questioned.

Joshua Burgin’s Substack

Golden Rules for Organizational Structure

In the realm of organizational structure inside a technology (or R&D) org, I’ve observed many different structures over the last 25 years. Through that, I’ve developed a set of guiding principles and best practices I think provide broadly optimal results. While these aren't rigid mandates, they serve as valuable starting points for fostering productive …

Read more

3 years ago · 224 likes · 14 comments · Joshua Burgin

A shallow survey of OLAP and HTAP query engines

Snowflake, ClickHouse, DuckDB, Velox... Are the hundreds of OLAP query engines really different?

As it turns out, many converge to similar designs over time. Yet studying these differences is a fantastic learning opportunity. Let's look at the main attributes:

Data layout

All systems seem to agree on the PAX layout (Partition Attributes Across) to store data. The idea is that rows are grouped together, and then the values are grouped by column within each row group. The main differences lie in the size of these partitions, usually in the hundreds of Mb range.

"For hot data we still see PAX-like layouts, but with much smaller row groups because we want entire query pipelines (see below) to fit in cpu cache."

Compression

The main question is generic vs. specialized compression. The issue with generic compression is that it usually requires unpacking an entire compressed block when reading random values. However, since compression significantly cuts storage and bandwidth costs, it's an acceptable trade-off.

"For cold data most systems apply both one or more specialized algorithms and then some generic algorithm afterwards."

Vectorized vs pipelined queries

• Vectorized query execution processes multiple data elements (as a vector) in a single instruction, improving computational efficiency by reducing instruction overhead.

• Pipelined query execution splits a query into stages, allowing different stages to work concurrently on different parts of the data.

"The pure OLAP systems generally use entirely vectorized style."

📗 Jamie Brandon's A shallow survey of OLAP and HTAP query engines is an exploratory article into the design and trade-offs that characterize these systems. As it opens more doors than it closes, the article is an excellent entry point into the deep domain of query engine optimization.

Read the full article on Jamie's blog

Top 0.1% re-post: The Linear Method

Standing on the Shoulders of Giants is nearing two years, and 300 articles have been shared. Some articles are one in a thousand and deserve more than one post. Every week, I’ll re-post one of these fantastic pieces of knowledge. Starting with the Linear method, as my thoughts go to the Linear team that has had a bad database outage last week and has lived through what I imagine wasn’t their best week.

Successful products have "on-point" operating systems. And when they share them, it's an incredible learning opportunity. As an individual, there are plenty of ideas to bring to your team. As a founder, you can compare your processes against the best!

The Linear Method collects the aspirations and operating cadence of the team and how they continuously roadmap and build. It’s down to earth, opinionated, but also built on the principle that rules are guidelines, not laws.

Overflowing inboxes are a good sign unless they're bug reports. Don’t worry too much about organizing all the [user] feedback. Collect it and use it as a research library when developing new features.

At Linear, we don’t write user stories and think they’re an anti-pattern in product development. We write short and simple issues that describe the task in plain language instead.

📗 Linear's The Linear Method describes the foundational principles the Linear team used to create such a successful product. I found every section truly enlightening and helpful as a product leader. It takes a deep understanding to clearly lay out these principles with no jargon and short sentences.

Read the full method

Did a friend send this to you? Subscribe to get these weekly drops directly in your inbox. Read the archive for even more great content. Also: I share these articles daily on LinkedIn.

Subscribe now

How Rust and Wasm power Cloudflare's 1.1.1.1

1.1.1.1, Cloudflare's DNS resolver, has grown exponentially since its 2018 release.

Unsurprisingly, Cloudflare started with an off-the-shelf solution called Knot Resolver but quickly outgrew it and needed more and safer customization.

"We forged a platform that we are happy with, and we call it BigPineapple."

BigPinneapple is Cloudflare's DNS resolution system, hosted in hundreds of edge data centers. It was built with the following goals in mind:

• High performance and throughput. Total latency should be in the milliseconds.

• Correctness. DNS resolution is old and has many edge cases and non-RFC-compliant servers that must be supported.

• Extensibility. Cloudflare's value-add beyond a simple DNS resolver is in the customization and new use cases it can enable.

The article describes BigPinneapple's system architecture and design choices. It's built using a Rust async framework called Tokio, and composed of separate components deployed together. Here are the most interesting ones:

1. The server speaks many protocols because DNS is queried over many of them and turns these heterogeneous requests into common DNS query Cloudflare calls "Frames." It offloads query resolution to a set of workers.

2. The cache is central to the overall server performance. Because DNS resolution involves making multiple network requests, it can be slow, hence the caching needs. Because of how large the DNS space is, it can't fit in one cache. Cloudflare uses consistent hashing to distribute queries to different nodes in the same data center to split caches between nodes.

3. Conductor manages outbound connections smartly. Every non-cached DNS query leads to network calls to upstream DNS nameservers with different performance, capacity, and availability. "The conductor is able to make these decisions by tracking the upstream server's metrics, such as RTT, QoS, etc."

4. The Sandbox is the extensibility component. It runs WebAssembly modules as "callbacks" that are registered at different points of the BigPinneaple request lifecycle.

"BigPineapple's Wasm runtime is currently powered by Wasmer. The runtime allows each module to run in its own instance, with an isolated memory and a signal trap, which naturally solved the module isolation problem we described before."

📗 Ambang Wen's How Rust and Wasm power Cloudflare's 1.1.1.1 describes the architecture of one of the largest DNS resolvers on the planet. It goes into sufficient detail to be insightful, calling out protocols and techniques the team used to build and operate the system at such a scale.

Read the full article on Cloudflare

How to Beat Larger Competitors

"It doesn't matter how big your company is, there is always someone bigger."

Start-ups often shrug off larger companies when talking about competition. We build our two-dimensional competitive charts without them and fill them with smaller companies that look just like us. That's wrong in many ways. Even if large companies aren't your main risk or focus, their size can threaten your business.

"Big companies can outlast you, so they don't need to win, they just need to wait for you to fail."

So, you need a strategy to beat and protect yourself against them. You need to take the initiative. came up with a framework to think about your fight against these larger companies:

Deposition

"Strategy 1: Deposition. You can [...] frame their product as "last generation" or "table stakes". This is called depositioning."

Depositioning is a powerful marketing message, but it is even stronger when you build your entire company and product around that message. One example is all the new ERP-like companies whose central message is "Old ERPs are old and unusable, ERPs are dead, we're the future".

Commoditize

"Strategy 2: Commoditize. Once a business is big, it cannot easily adjust to a new business model [...] If you can offer a similar service for 1/10th of the price it's unlikely they can find a way to compete."

Driving price down is a risky strategy for your image and not always the best way to build a healthy business. Still, if it's powered by a fundamental change in how the business is done, it can be the best competitive advantage.

Disrupt

"Strategy 3: Disrupt. If you do something that is radically different then you have a chance to disrupt their business."

Delivering similar value in a better way is also a recipe for explosive growth because, depending on switching costs, the existing market can switch to your solution in a landslide. Think Uber vs. Taxis. While the word is overused, disruption is probably the most common way start-ups win. It's also the most common way they lose to smaller competitors once they're giants. A never-ending cycle!

"Leaders in any given category are rarely leaders for more than a decade or two, as smaller competitors eventually find ways to chip away at their advantages."

📗 ' How to Beat Larger Competitors gives an actionable framework to think about big companies in your space. In some markets, beating larger competitors can also give you access to large existing markets without needing to create a new one, which can tremendously accelerate growth.

The Breaking Point

How to Beat Larger Competitors

If you liked reading this, please click the ❤️ button on this post so more people can discover it on Substack. Thanks! It is inevitable that you will face competition from bigger companies. Any business that is growing and has happy customers becomes an attractive opportunity for a huge company looking for growth, or a threat they need to negate. It does…

Read more

3 years ago · 17 likes · Sean Byrnes

Graceful behavior at Capacity

🚒 "The database is stuck at 100% ; all requests are timing out!"

vs.

😎 "We're currently experiencing issues; clients may see elevated error rates."

This shows the difference between graceful degradation and total outage from the outside. Yet from the inside, the same thing may be happening: high contention for some shared resource, like a database. Without proper protection in place, high contention can spiral into congestion collapse.

"Congestion collapse is when a system is receiving a high request rate but achieving throughput much lower than its capacity."

Once you've identified that bottleneck system (all systems are, at some point, but the first is the limiting one), there are different ways to protect it that all achieve the same goal:

"The basic strategy to resolve contention is to deliberately limit concurrency to levels which don't cause undue contention."

Adding a request queue can only solve traffic spikes, but under heavy load, queues will fill until either latency becomes too much or they're full.

"There's only one solution: somehow have less work to do. We refer to either or both of these strategies as backpressure."

There are two general ways to apply backpressure:

1. Flow control is a mechanism through which the server asks the client to slow down incoming requests only to receive what it can process. But that requires controlling most clients.

2. Load shedding is simple: drop requests entering your system to accept what the system can process. Multiple strategies exist to select such requests, from random to request tiering.

📗 's Graceful behavior at capacity explains that because all systems have limits, it's essential to plan for when such limits are reached. The author then describes the main concepts and techniques commonly employed to protect systems under heavy load and ensure they don't degrade beyond their own capacity.

Read the full article on Nelson's blog

Did a friend send this to you? Subscribe to get these weekly drops directly in your inbox. Read the archive for even more great content. Also: I share these articles daily on LinkedIn.

Subscribe now

The Log: What every software engineer should know about real-time data’s unifying abstraction

Spoiler alert: This 2013 article leads to the origin story of Kafka, the now famous log-based system.

But first, let's start with a refresher on the Log.

"Sometimes called write-ahead logs or commit logs or transaction logs, logs have been around almost as long as computers and are at the heart of many distributed data systems and real-time application architectures."

“A log is perhaps the simplest possible storage abstraction. It is an append-only, totally-ordered sequence of records ordered by time."

It shines as a tool to coordinate software components and systems. If all systems are deterministic, you can reduce the coordination and data replication problems to agreeing on a common log of records.

"The purpose of the log here is to squeeze all the non-determinism out of the input stream to ensure that each replica processing this input stays in sync."

In practice, we can use logs in multiple ways: log all requests (e.g., for a database, the SQL queries) or log the underlying data changes.

Most systems use a Log as an internal coordination component. However, the article shows it can also be used as a first-class citizen to solve data integration problems between systems. That's Kafka.

The problem Kafka solves is simple: the amount of data we process is growing exponentially, and specialized systems are outscaling the "one big generalist database" model. Think ElasticSearch for search, BigQuery and Snowflake for analytics. The more systems we operate, the more data needs to be moved from one to the other. Kafka makes that data movement simpler by separating concerns.

"This experience led me to focus on building Kafka to combine what we had seen in messaging systems with the log concept popular in databases and distributed system internals. We wanted something to act as a central pipeline first for all activity data, and eventually for many other uses, including data deployment out of Hadoop, monitoring data, etc."

📗 Jay Kreps' The Log: What every software engineer should know about real-time data's unifying abstraction is a foundational article describing the story that led him to build Kafka at LinkedIn in the 2010s. After the general introduction about the Log, Jay describes the challenges the team faced building and scaling the Kafka Unified Log.

Read the full article on LinkedIn's blog

The Concept that explains everything about Marketplaces

"Marketplaces are simply businesses that don't sell goods or services, but instead sell the reduction of transaction costs."

There are many theories about these transaction costs. With the help of decades of theory, the author defines four categories:

1. Search and information costs: discovering what is available to purchase and comparing alternatives.

2. Bargaining and decision costs: coming to an agreement between buyer and seller, including establishing the final price and terms.

3. Enforcement and policing costs: ensuring that both sides holds up their end of the deal.

4. Distribution costs: actually getting the good or service to the end consumer

All marketplaces help reduce one, many, or all of these costs. Consequently, studying a market's transaction costs is a great way to identify marketplace opportunities. The bigger the cost savings, the more the market will expand.

For example, Uber reduces the bargaining and decision costs:

"Price transparency used to be a key problem when hiring a taxi. Your only option was to get in the car and watch the meter run up. Now Uber calculates the price ahead of time, making it easier to decide whether or not to take a ride."

"Kevin Kwok wrote a great piece about how Uber and Lyft grew the TAM for private cars. Not only that, but they grew TAM precisely in areas where transaction costs were highest."

Looking further, a marketplace's take rate can increase as it reduces costs for buyers and sellers. So, the logical evolution for them is to take on more responsibilities and work on the heaviest costs, which are Enforcement costs and Distribution costs. That's also how competitors can disrupt them. For instance, Amazon disrupted eBay with its asset-heavy, fully-managed model, taking on a lot more costs that eBay did.

📗 's The Concept That Explains Everything About Marketplaces is an excellent introduction to the transaction cost theory of marketplaces. It's an angle that seems obvious initially but has a lot of depth when analyzed further.

Dan Hock's Essays

The Concept That Explains Everything About Marketplaces

While most undergrads were out drinking whatever they drank in the 30s, Ronald Coase was working on a theory that would help him win the Nobel Prize more than 50 years later. In “The Nature of the Firm”, Coase demonstrated why businesses must exist at all, instead of individuals just contracting directly with each other. At the core of his rationale was …

Read more

3 years ago · 94 likes · 5 comments · Dan Hockenmaier

How to become insanely Well-Connected

Networking and relationships are games you can't cheat at. It's also a lifelong one that, when done correctly, has a compounding impact. Your end score is the sum of your life's work to create meaningful connections.

“The best way to be highly influential is to be human to everyone you meet."

Beyond that mantra, here are Chris' Seven Rules for Making Memorable Connections.

1. Convey genuine appreciation. Actively project warmth and high energy. It's been observed people like you when they feel liked by you."

2. Listen with intent. Being a great listener is an art. You must strive to actively listen, react, and follow up on every sentence to show you're there, not already thinking about your next meeting.

3. Use humility markers. Acknowledging your own fallibility and human imperfection can go a long way toward making yourself relatable."

4. Offer unvarnished honesty. Honesty creates trust, and trust is core to all relationships. It's also probably the easiest on the list to stand out.

5. Blue-sky brainstorm. [...] Give before thinking about what you get. Always offer something of value before expecting or asking for something in return." Brainstorming with your interlocutor is the simplest form of giving.

6. End every meeting or conversation with the feeling and optimism you'd like to have at the start of your next conversation with the person." Creating a lasting relationship will require many interactions. Whatever you say last will stay, and make the next one easy or hard.

7. Don't fake it till you make it. And beyond not lying, always come extra prepared to meetings with a specific goal in mind and knowledge about how they can help you reach it. That's particularly true when reaching out to higher-profile people.

📗 's How to Become Insanely Well-Connected is a great article to prepare for 2024. Networking and relationships should be at the top of your list of things to improve professionally, and Chris' advice on the matter is both actionable and obviously powerful.

Read the full article on First Round

Did a friend send this to you? Subscribe to get these weekly drops directly in your inbox. Read the archive for even more great content. Also: I share these articles daily on LinkedIn.

Subscribe now

Solving the Engineering Strategy Crisis

Your Engineering team deserves a Strategy, too.

While absent from most roadmaps, engineering work is core to a tech company's team and product health. And if it's not explicit, your company probably has an unspoken engineering strategy.

What's Engineering Strategy? Taking inspiration from Richard Rumelt's Good Strategy, Bad Strategy, Will Larson says this:

"I believe that Engineering strategy comes down to two core components:

1. Honest diagnosis that engages with the reality your organization's current needs and challenges"

2. Practical approach to move forward while addressing the circumstances raised in the diagnosis".

Far from the fancy missions and visions, strategy must be rooted in the current situation and truth. The more honest it is about the present, the more it can be used to guide the future.

Why is it important?

Explicit strategy aligns people around a common approach, creates consistency within the company, focuses resources on improvements to that common approach that benefit everyone, and facilitates hard decisions.

I know what you're thinking: these aren't specific to engineering. Think about scaling your org, onboarding new people, and decentralizing decision-making. Having a written strategy will make these processes smoother. So write that strategy down!

"You can solve half the engineering strategy crisis by just writing stuff down."

📗 Will Larson's Solving the Engineering strategy crisis then explains how to define (or find) your strategy. It's often already there, but you need to put the right words and align people with it. If you don't know where to start, Will gives examples of engineering strategies of his previous companies: Stripe, Uber, Calm. You'll see how radically different they can be!

Read the full article on Will's blog

Scaling data processing at Mux with an embedded key-value store

How Mux, a video infrastructure company, moved their view management system to a "Shared Nothing" architecture.

The Mux Data team owns a system that does viewer analytics and stores finished views to allow resuming views. Because the requirements of analytics (large-scale aggregations) and view resumption (row-level queries) are opposite, they split it into two sub-systems:

1. Analytics are powered by an OLAP system, ClickHouse, ingests all view events from Kafka.

2. Row-level access was powered by Riak, a distributed key-value store used as a 6-hour cache, also ingesting events from Kafka.

"Riak became increasingly difficult to operate as Mux Data usage grew, compounded by a growing number of live video events. Unlike on-demand video consumption, live events have fundamentally different viewership patterns."

Traffic became much spikier, with requests in the hundred thousand per second. Because Riak is a distributed database, its capacity isn't elastic. Scaling it up and down requires human intervention. So, the team had to provision it for peak usage, which is very inefficient overall.

Even then, Riak would stall when provisioned to its maximum capacity, and data ingestion would fall behind.

"Riak often became a write-throughput bottleneck when viewers left a stream en masse. Degraded Riak write performance could cause stream-processing lag, possibly leading to delayed data in our Mux Data Monitoring dashboard."

At this point, the team knew they couldn't continue scaling Riak. They went back to the drawing board. And they realized something. They didn't need a unified store because Kafka already partitioned their events into independent streams.

"Rather than running a networked, distributed KV store (Riak, Redis, etc.), we could instead have the event processors manage many independent partition-scoped embedded KV stores on local disk."

"We leveraged the keyed, partitioned nature of the data in Kafka to scope each cache to a single Kafka topic partition, leading to a cache architecture following the Shared Nothing pattern [from Martin Kleppmann's Designing Data-Intensive Applications book]."

The team removed Riak's network requests and coordination bottlenecks by writing directly into an embedded (understand local) key-value store inside the processing Pod. The trade-off is having to manually manage the persistence of that data during rebalancing and upgrades.

📗 Scott Kidder's Scaling data processing with an embedded key-value store," tells the story of a team that started with a pragmatic choice and successfully re-evaluated their options once it began to show scale issues. Scott also clearly explains that there is no silver bullet; this complex solution works for Mux because of its architecture and spiky, write-heavy workload. And last, it's not simple to set up! Even the world-class Mux team struggled to make the persistence work at all times, so think twice before building your own distributed system!

Read the full article on Mux's blog

Your Codebase deserves Better Names

Poor naming is probably the fastest and dumbest way to accumulate tech debt. But it also is the most common. In particular when English isn't your team's primary language. Luckily, it's 1) easy to improve at, 2) trivial to refactor, and 3) can even improve the code itself!

Beyond these apparent reasons, putting effort into naming has a valuable side-effect: it helps detect spaghetti code:

"When it feels impossible to name something, step back and figure out what you really want your code to do before you write any more of it."

So, what's a great name? It should be:

1. Descriptive. [...] Names should describe what the code does, in detail."

2. Specific. [...] making sure your words are as precise and specific as possible is essential for a great name."

Get rid of the doXYZ and getXYZ methods and the foo and bar variables. Using specific and consistent nouns and verbs to describe concepts makes a massive difference to the readability of your code.

To find these specific words, first scan the codebase for similar occurrences, and if you can't find any, use a thesaurus (I use Power Thesaurus) to generate ideas. Start with a concept and explore synonyms until you discover the perfect one.

3. In line with codebase conventions". Consistency is underrated.

4. Grammatical."

For non-English speaking teams, grammar is usually the biggest hurdle. That's OK. Accept it and use Grammarly or translation tools to ensure your names are correct; it's the only way to learn.

Grammar-wise, there are a few general rules that you can always apply:

1. The most important noun should be last, and adjectives before it: ConnectedOrderBuilder, NetworkConnectivityInspector.

2. Use verbs for actions and place them first: renderBlueLogo, openDatabaseConnection.

📗 Emma Ferguson's Your Codebase Deserves Better Names is a rare actionable article on naming. While the famous saying is that "cache invalidation and naming things" are the most complex problems in computer science, there are many more articles on the former. So, thank you, Emma, for clarifying this simple but impactful topic!

Read the full article on Samsara's blog

Did a friend send this to you? Subscribe to get these weekly drops directly in your inbox. Read the archive for even more great content. Also: I share these articles daily on LinkedIn.

Subscribe now

The Value Engineering method

Product people, this is for you! Switching from an output to an outcome-driven product culture isn't enough. You must fundamentally change the way you work as a company to make a difference. Otherwise, your shiny new outcomes just map to your old outputs.

"In an outcome-driven environment, organizations have shifted focus to funding teams and outcomes over work. [...] The reality is that many initiatives continue to run through 'completion' and are judged on their results."

To be effective, one must match an outcome-driven culture with an incremental approach to value creation–with checkpoints and milestones. That's called Value Engineering.

"The Value Engineering approach recognizes these initiatives as a portfolio of bets and uses an evidence-based approach to make decisions as new information is revealed. [...] It is an evidence-based decision making model for identifying when to pivot, persevere, or stop investments."

That decision-making model is based on incremental bets to create learnings and near-instant usage of these learnings. Here's a template for each initiative:

1. Hypothesis: We believe [this capability] will result in [this outcome]. We will have confidence to proceed when we see a [measurable signal].

2. Desired outcome: Explanation of a future state, ideally measurable. This might be an OKR, a target for a KPI, or just an articulation of the future."

3. Max bet: If you could buy the outcome off the shelf, how much would you pay? This can be in terms of dollars, but it's often easier to define this as people over time (e.g. 2 teams over 16 weeks)"

After each incremental bet, we ask the same question:

"Do we pivot, persevere, or quit?"

"That decision may be relative to other investments. Because we can plot other initiatives across these axes, we can visualize whether or not we have longer-running initiatives worth doubling down on or earlier initiatives showing more promise."

That last point is what makes the Value Engineering method so strong. It gives you a framework to visualize initiatives relative to each other and compare them to make the necessary trade-offs.

📗 's Value Engineering article is a must-read for product people in outcome-driven cultures. As Barry puts it, "With one checkpoint (at the end), the only potential scenarios are luck or overcommitment.". Building that test, learn, and iterate culture is a necessary first step, but it's the plotting and visualization that will make your decisions more intentional and strategic.

Read the full article

Access Path Selection in a Relational Database Management System

SQL and relational databases are 50 years old!

Most modern systems still use the language and query optimization concepts from the 70s. The papers of that era from the IBM Research Division are a treasure trove of canonical papers. In particular, one paper is the father of many systems.

"Access Path Selection in a Relational Database Management System" from Patricia Griffiths Selinger.

The IBM team built System R, one of the first-ever relational databases.

"System R is an experimental database management system developed to carry out research on the relational model of data."

The paper describes how System R does Access Path Selection, meaning turning a SQL query into a plan to retrieve the data from disk, filter, and join it to satisfy the query.

"The four phases of [SQL] statement processing are:

1. Parsing,

2. Optimization,

3. Code generation,

4. Execution."

Parsing is quite simple to understand. Code generation and execution are intelligent ways to execute the defined plan. The focus of the article is on the second step: the OPTIMIZER.

▶︎ "The OPTIMIZER accumulates the names of tables and columns referenced in the query and looks them up in the System R catalogs to verify their existence and to retrieve information about them. [...] Finally the OPTIMIZER performs access path selection. It first determines the evaluation order among the query blocks in the statement. Then for each query block, the relations in the FROM list are processed. If there is more than one relation in a block, permutations of the join order and of the method of joining are evaluated. The access paths that minimize total cost for the block are chosen from a tree of alternate path choices."

That's the high-level approach System R and many following databases took for the next 50 years. Compute the cost of possible access paths, compare them, and take the smallest. But how does the costing work?

▶︎ "During catalog lookup, the OPTIMIZER retrieves statistics on the relations in the query and on the access paths available on each relation. [...] Using these statistics, the OPTIMIZER assigns a selectivity factor 'F' for each boolean factor in the predicate list. This selectivity factor very roughly corresponds to the expected fraction of tuples which will satisfy the predicate."

For example, the selectivity factor of a single field = ‘value’ clause is computed as 1 / field cardinality.

By merging that selectivity factor with the costs of accessing each path, the optimizer can make trade-offs between sequential and index reads and, more generally, choose the lowest cost path.

📗 Patricia Griffiths Selinger's Access Path Selection in a Relational Database Management System paper from 1979 then covers the cost formula for single path access and joins, as well as the selection algorithms involved—a nice read into the world of databases.

Read the full paper

Absolutely positioning things relatively at Canva

How Canva automatically turns designs into responsive websites.

For their new Websites product, a Canva team needed to go beyond the limits of the Canva editor.

"Canva is a design platform where users can create designs by freely dragging and dropping elements using our fixed dimensions editor."

"Our challenge is whether we can do this automatically at scale. Can we take any design and render it responsively? And just for a challenge, can we do all this without JavaScript?"

Their initial thought was to stick with absolute positioning and scale everything according to the device's dimensions. Users place elements inside the drag-and-drop editor, giving fixed coordinates for each.

"There are a few optimizations we can make, but there's one big problem: the text becomes impossible to read [on smaller devices]."

And because absolute is, well, absolute, text will overflow the elements if you don't scale it down with the rest of the design. So, absolute positioning was out of the picture.

"We need a simple absolute positioning system, but the reflowing text should influence the position of other elements. That is, the best of both absolute positioning and the normal relative flow."

The team achieved this using CSS Grids. In the world of CSS Grids, you define as many rows and columns as needed and give each element a start, end row, and column on the grid.

Because rows and columns can't overlap, they can stretch to accommodate content and thus avoid the pitfalls of absolute positioning.

"To calculate the grid-template-columns, we calculate the offset as a percentage of the design/screen width from the previous column."

The trick is not to use percentages only but to combine them with max-content:

min-max (12vw, max-content)

Max-content will ensure the columns and rows can grow beyond the fixed percentage to always fit their content.

📗 Benjamin Morris's CSS: Absolutely positioning things relatively describes the team's journey to figure out the right software solution to the complex problem that is solving responsiveness auto-magically. Beyond performance, the team put much effort into creating the best user experience, making it a super interesting read.

Read the full article on Canva's blog

Did a friend send this to you? Subscribe to get these weekly drops directly in your inbox. Read the archive for even more great content. Also: I share these articles daily on LinkedIn.

Subscribe now

CSS Selectors: a visual guide and reference

Every style you apply with Javascript has a more elegant CSS-only alternative. And that's a great way to tell apart senior frontend engineers from React-only tinkerers. They have a deep knowledge of CSS selectors.

“CSS selectors are patterns used in CSS to select and style HTML elements on a page."

And with every new browser version comes a new set of CSS language extensions, including many new selectors. So they're something you must learn and re-learn.

Along with traditional CSS selectors, CSS has pseudo-classes and pseudo-elements. They're called pseudo because they're not user-applied classes nor actual HTML elements but can be selected precisely like classes and elements.

“Pseudo-classes allow us to define styles based on an element's state or its relation to other elements. [...] Pseudo-classes start with a colon character (:)."

“Pseudo-elements give you the power to target and style specific parts of an element, such as its first line or before and after its content. Pseudo-elements start with two colons (::)."

📗 Sébastien Noël's CSS Selectors: A Visual Guide & Reference describes the most popular and useful CSS selectors. It's an effective way to grow your frontend engineering skills and replace your 100-line Javascript style functions with an elegant CSS solution.

Read the full article

The taming of the B-trees

B-trees, binary trees, what’s the difference? Let's learn from engineers who have had to make this critical decision to build a database: Scylla DB.

"One of the core components that greatly affects ScyllaDB's performance is the in-memory cache of the user data (we call it the row cache). And one of the key factors to achieving the performance goal is a good selection of collections".

Let's start with definitions.

1. A binary tree is a tree data structure in which each node has at most two children.

2. The tree is called a B-tree when the number of children per node is higher than two. Fun fact: the "B" doesn't stand for anything officially; it's just "B".

"The common belief about binary vs B- trees is that the former ones should be used when the data is stored in the RAM, whilst the latter trees should live in the disk. [...] However, there are many reasons why B-trees are often a good choice for in-memory collections."

With newer hardware and all datasets getting larger, B-trees and their variants can significantly improve performance over binary trees.

"On a B-tree, this search will consist of two phases — intra-node search and descending the tree — executed one after another."

“The two-phased B-tree search can be made better with respect to branch predictions. The trick is in making the intra-node search linear, i.e. walking the array of keys forward key-by-key. In this case, there will be only a "should we move forward" condition that's much more predictable."

And with SIMD instructions doing linear search on up to 64 values simultaneously, that intra-node search can be further optimized to give B-trees a big edge.

📗 Pavel Emelyanov's The Taming of the B-Trees goes further than describing how great B-trees can be. The rest of the article describes B+-trees, a variant of the B-trees that only have values on leaf nodes and the solutions the Scylla engineers had to develop to use B+-trees in production. Academic theory often needs real-world adaptation that companies rarely share, making the post even more valuable.

Read the full article on ScyllaDB's blog

Why write?

"Why write an essay when you can type a few words and have AI generate one for you?"

Writing is a lot more than coming up with words and sentences. The process can create a higher value than output. Let's explore why:

1. “Writing is not just a vehicle to share ideas with others but also a way to understand them better yourself."

2. Writing requires the compression of an idea." That compression requires deep understanding of the essence of the topic.

Why write in the ocean of mediocre AI-generated content? Why now?

Think about it: if no one understands anything to the level of being able to write about it anymore, this can be your superpower.

📗 Farnam Street's Why Write is a quick reminder that the value of writing goes beyond sharing content with the world. The profound personal impact is almost more valuable; it's like journaling. Write about what you did and what you'll do. Write about your strategy.

Read the full essay on FS

Did a friend send this to you? Subscribe to get these weekly drops directly in your inbox. Read the archive for even more great content. Also: I share these articles daily on LinkedIn.

Subscribe now

Putting the "You" in CPU: a rabbit hole into how your computer runs programs

As modern software engineers, our day job happens a thousand abstraction layers above the CPU. Unless you're from a pure CS background, you've never bothered to learn how it works. Why should you?

Because it's cool, and it's beautifully simple!

"The one thing that surprised me over and over again while writing this article was how simple computers are."

That simple? Let's dive into it:

"The central processing unit (CPU) of a computer is in charge of computation. [...] The "instructions" that CPUs execute are just binary data: a byte or two to represent what instruction is being run (the opcode), followed by whatever data is needed to run the instruction."

"What we call machine code is nothing but a series of these binary instructions in a row. Assembly is a helpful syntax for reading and writing machine code that's easier for humans to read and write than raw bits; it is always compiled to the binary that your CPU knows how to read."

"The CPU always reads machine code directly from RAM, and code can't be run if it isn't loaded into RAM."

The CPU fetches instructions from RAM, executes them, and then moves on to fetch and execute more instructions. While it does that, it is in one of two modes: User or Kernel.

"In kernel mode, anything goes: the CPU is allowed to execute any supported instruction and access any memory. In user mode, only a subset of instructions is allowed, I/O and memory access is limited, and many CPU settings are locked."

Processors start in kernel mode and switch to user mode when executing non-OS code. User code can only access memory and secure instructions by securely switching back to kernel mode for a defined set of instructions: system calls.

"Programs typically use these syscalls by calling shared library functions. These wrap machine code [...] that transfers control to the OS kernel and switches rings. The kernel does its business and switches back to user mode and returns to the program code."

📗 Lexi Mattick's Putting the You in CPU is a long-form read on how computers run code, starting from the lowest system abstraction. What's unique about it is how clearly the author describes the different layers, and how, step-by-step, they build your understanding of computers' internals.

Read the full series

Slack’s resilient networking infrastructure

Maintaining high-availability SLAs for a real-time product with millions of customers globally is a complex networking challenge. Slack is an excellent example with relatively low latency requirements. The team has designed a redundant networking architecture that can sustain quite a lot of failure. And lucky us, they have written about it!

Slack hosts their central application services and databases in the us-east-1 AWS region. They're resilient to failure thanks to a multi-availability zone setup.

That means all requests, like new messages, flow through the US, wherever you're in the world. Not only does this create latency problems, but it can also harm availability if your network can't route requests to the US. So Slack has edge points-of-presence (PoPs) close to the users to ensure the requests enter Slack's network as soon as possible and cross the world in that faster, more resilient network.

"These edge PoPs sit closer to our users to reduce latency, improve performance, and connect them back to Slack's main region in the AWS us-east-1."

Even though they don't host the primary Slack services, edge PoPs each have a collection of caches, image proxies, and services to ensure all requests are handled as close to the user as possible.

How is traffic routed to and through these edge PoPs?

Everything happens at the DNS level through a collection of primary and backup domains with different rules to select the IP (so the data center) the DNS records point to.

Primary domains route to the geographically closest edge PoP that is up. Backup domains distribute the load between the three nearest PoPs to ensure clients can skip an unhealthy PoP after a few retries.

📗 Rafael Elvira's Traffic 101: Packets Mostly Flow describes Slack's distributed networking architecture that enables it to handle traffic gracefully during partial outage events. It's an interesting insight into how such a large company plans for the unexpected!

Read the full article on Slack's blog

Design communication is a critical skill

The best designers excel at communication. And not being able to communicate "why" clearly is probably designers' most common issue.

It makes sense in your head; your solution is the best. How can you communicate that effectively to stakeholders?

"One of the most important skills for designers is the ability to communicate design intent. We need to help stakeholders understand why a design decision provides the best possible solution to a problem."

So, as with all crucial skills, it's good to go back to basics periodically. Here are the four communication foundations:

1. Product over process. Focus your communication on the product; show how you're changing it.

2. Explain the problems.

"Set the stage for what you're sharing by starting with the problems. Never assume that stakeholders have as much context as you do."

3. Link solutions to problems.

4. Explain why solutions matter.

"Stakeholders often need to understand the why so that they understand the value of what you're proposing. [...] With every decision you make, ask yourself — "how will this affect the user?""

Beyond these general principles, the golden rule is to tailor your communication to your stakeholders. People have very different interests and value different angles based on their roles and positions in the company.

📗 Kazdem Cattapan's Design communication is a critical skill is a very actionable read around communicating design decisions. Communication is essential to build trust with all stakeholders, and understanding how to do it effectively can 10x your impact.

Read the full article on Shopify's blog

Did a friend send this to you? Subscribe to get these weekly drops directly in your inbox. Read the archive for even more great content. Also: I share these articles daily on LinkedIn.

Subscribe now

The Architecture challenges behind Cloudflare Waiting rooms

Waiting Rooms protects web applications against traffic spikes by serving excess users a temporary branded waiting room, dynamically admitting users as spots become available on their sites.

The hard part about building Waiting Rooms is that to ensure good protection, the decision to queue a user or let them through has to be evaluated synchronously on every request. And worse, because it's spike protection, the decision is to let the user through most of the time, so the added latency must be minimal.

"Waiting Room is built on Workers that runs across a global network of Cloudflare data centers. To optimize for minimal latency and enhanced performance, these requests are routed to the data center with the most geographical proximity."

So here's the problem: make a millisecond-latency decision about every user while keeping the queue in sync for all users worldwide.

Considering a cross-continent request can take tens of milliseconds, all solutions based on a central state machine are off the table. The team had to build an eventually consistent solution, which brought a new set of problems:

• Protect too aggressively, and you'll end up queuing users for no reason.

• Protect reactively, and you'll miss the first spike of users.

The Waiting Room engineers designed a solution: distribute the remaining utilization slots between data centers based on the active users of the last minute. The centralized process only distributes tickets to the workers, which they can then freely allocate to incoming users.

After a few iterations, engineers employed datacenter-local counters to ensure an even distribution between Workers of the same DC.

📗 George Thomas's How Waiting Room makes queueing decisions on Cloudflare's highly distributed network is an in-depth walkthrough of the thought process and iterations the Waiting Room engineers went through to solve that complex distributed systems problem.

Read the full article from Cloudflare

Fresh Work and the 80/15/5 Rule

Are you feeling bored at your day job? Or that you need to be challenged or learn more?

You can do something about it! It doesn't have to come from your manager. Start by freeing up time to explore new things. They can be closely related to your current task or have a distant link. It's all about the balance.

"Doing things that are new keeps you energized and interested. [...] But doing things you're already good at ensures you accomplish goals other people care about."

designed a simple rule: 80/15/5:

• 80% of your time goes to low-risk/reasonable-reward work

• 15% of your time goes to related high-risk/high-reward work

• 5% of your time goes to satisfying your own curiosity with no thought of reward"

The 80/15/5 rule is not just about carving out time for novelty. It's about continuous movement towards what energizes you, slowing moving out of the current 80% and using the 15 and 5 to create your next new 80%.

"While you're doing your 80% task you're gradually teaching the next generation of your team to take that task over. [...] About the time you've found a 15% task to become your new 80%, your successor is ready to take over your previous 80% task."

📗 's Fresh Work 80/15/5 rule is simple but powerful. Use it to take control of your work and career or as a manager to help unblock people and enable them to create their path.

How Exit Traps can make your Bash scripts more robust and reliable

"A simple, useful idiom to make your bash scripts more robust."

Bash scripts are often cobbled together, bypassing the high standards we uphold ourselves to when writing code. Yet their impact is enormous compared to their relatively few lines of code, as is their ability to blow up in your face.

Not writing unit tests is one thing. But the most dangerous part is the undefined behavior of the script failing halfway.

This can lead to funny situations:

• A database maintenance bash script failed and didn't reach the "restart database" line.

• A large data file operation crashed and didn't reach the "cleanup" line.

See where I'm going?

"The secret sauce is a pseudo-signal provided by bash, called EXIT, that you can trap; commands or functions trapped on it will execute when the script exits for any reason."

That simple! Now you know what to do with your teardown and restart methods!

"You place any code that you want to be certain to run in this "finish" function."

📗 Aaron Maxwell's How Exit Traps can make your bash scripts way more robust and reliable" is a short read introducing that trap method. I'm often reluctant to share code articles to ensure they're actionable by all readers, but bash is not just code!

Read the full article on Aaron's blog

Did a friend send this to you? Subscribe to get these weekly drops directly in your inbox. Read the archive for even more great content. Also: I share these articles daily on LinkedIn.

Subscribe now

What is a Vector Database

The infrastructure behind the LLM revolution.

"All of these new applications rely on vector embeddings, a type of data representation that carries within it semantic information that's critical for the AI to gain understanding and maintain a long-term memory they can draw upon when executing complex tasks."

What are embeddings exactly? They're large vectors representing a model's understanding of a data point.

"Embeddings are generated by AI models (such as Large Language Models) and have a large number of attributes or features."

LLMs associate a vector representation: embeddings for every piece of data of our world. Adding long-term memory to a model is just that: storing a database of past conversations and allowing the model to query that database freely. But traditional relational databases don't shine at managing and indexing embeddings. That's because the default operation is not an exact query; it's a similarity search.

"In vector databases, we apply a similarity metric to find a vector that is the most similar to our query. A vector database uses a combination of different algorithms that all participate in Approximate Nearest Neighbor (ANN) search."

So vector databases are just this: databases specialized in doing that similarity search query on a lot of data very efficiently. They can also query both the embeddings and the actual data in case filtering should happen on both. Approximate Nearest Neighbour search on large datasets is not simple and usually combines many techniques and tricks. Here are some examples:

• Random Projection

"The basic idea behind random projection is to project the high-dimensional vectors to a lower-dimensional space using a random projection matrix."

• Locality-sensitive hashing

"LSH maps similar vectors into "buckets" using a set of hashing functions. The query vector is hashed to a particular table and then compared with the other vectors in that same table to find the closest matches."

📗 Roie Schwaber-Cohen's What is a Vector Database introduces this new breed of databases, why they're needed, and how they work. Considering they're everywhere and used in most LLM applications, it's an excellent time to jump on the train. Beyond marketing and shiny new databases, I also hear Postgres has a pgvector extension that gives everyone's favorite database vector search superpowers! Once dropped, operators will stow the pallet to a fixed pallet location.

Read the full article on Pinecone's blog

In Defense of Strategy

"There's a belief in tech that execution is all that matters, that strategy is for wimpy MBAs."

And it's easy to pride ourselves on having the best execution! Everything feels so fast!

"The better you are at execution, the faster you can run in any direction. A good strategy helps you run fast in the right direction."

Early-stage startups are often protected in the first months because they're so small and unsuccessful that no one will attempt to compete with them. That time is precious, and founders must focus on digging their moat.

"If startups need to dig moats before their uncertainty runs out, strategy is how they do it. If moats are the what, strategy is the how."

If the above is so obvious, why are startups struggling with strategy? I would say that most fall into two buckets:

1. Startups that get too busy with execution and think it'll be enough.

2. Startups that have Bad Strategy.

Packy's first advice is to read Good Strategy, Bad Strategy by Richard Rumelt.

"It's my favorite strategy book because it was so easy to put into practice, and because I recognized so many of the characteristics of bad strategy."

Then, the author explains what "Good Strategy" is.

"A strategy is a high-level plan to achieve one or more goals under conditions of uncertainty, designed through recognizing the challenge (diagnosis), setting a direction to overcome it (guiding policy), and detailing steps to implement the policy (coherent actions)."

"The best time to craft a strategy is before you start the company; the second best time is now."

📗 Packy McCormick's In Defense of Strategy then goes deeper into the three core elements of a strategy: diagnosis, guiding policy, and coherent actions, and ends with the practical example of a past startup. Beyond being a reminder that strategy is not just a big empty word, the article is also a guide to thinking about your strategy and a call to action to keep refining it.

Not Boring by Packy McCormick

In Defense of Strategy

Welcome to the 579 newly Not Boring people who have joined us since last Tuesday! If you haven’t subscribed, join 208,497 smart, curious folks by subscribing here…

Read more

3 years ago · 191 likes · 23 comments · Packy McCormick

How to get useful answers to your questions

"Often when I ask a vague or underspecified question, what happens is one of:

1. the person starts by explaining a bunch of stuff I already know

2. the person explains some things which I don't know, but which I don't think are relevant to my problem

3. the person starts giving a relevant explanation, but using terminology that I don't understand, so I still end up being confused"

And it takes a lot of confidence to stand up to the person, interrupt them, and ask a clarifying question, especially in a junior-to-senior conversation.

So here's a framework to solve that:

1. You can get more useful answers if you optimize your first question.

"My favourite tactic is to ask a yes/no question. What I love about this is that there's a much lower chance that the person answering will go off on an irrelevant tangent."

"My second favourite tactic is to state my understanding of how the system works." This way, your coworker will be able to identify where your gaps are and specify their answer.

2. Do not be afraid of following-up.

While it takes some confidence, never accept responses that don't answer your question or that you don't understand. Interrupt, clarify your initial question, and reflect on whether you understand the answers before closing the conversation.

📗 Julia Evans' How to get useful answers to your questions is an impacting read to recommend to anyone, especially in their early or late careers. I found that junior people tend not to be confident enough to ask questions, and very senior ones do not because they don't want to show gaps and weaknesses.

While some articles will teach you something new, others' power is to put concepts and ideas you already know in a novel way. The latter is sometimes even more powerful because human brains love patterns and structure. This article is an excellent example.

Read the full article on Julia's blog

Did a friend send this to you? Subscribe to get these weekly drops directly in your inbox. Read the archive for even more great content. Also: I share these articles daily on LinkedIn.

Subscribe now

What it means to design a platform

"Platform design isn’t just another flavor of UX or product design. There are challenges, mental models, and requisite skills that set platform design apart."

As a product leader, designer, or engineer, understanding the differences between platform and application design can help create better strategies and powerful self-reinforcing loops.

What exactly sets platform design apart?

1. Interfaces. APIs, components, services. How they're available to the platform user can guide them into building amazing, or terrible experiences. Think both simple and flexible. Because of the nature of platforms, interfaces are often one-way door decisions that are costly to reverse and thus require extra care.

2. "Incentives drive the motivation of both platform- and end- users."

3. "Emergence is the open-ended feedback loop that platforms can create and maintain. By designing for Emergence, not against it, we enable users to discover applications we never imagined."

The most obvious dimension of platform design, Emergence, must be enabled, managed, and monitored purposefully.

4. Second-order thinking requires thinking two steps ahead and anticipating how the platform can drive positive loops and mitigate bad ones. The first step is for user usage, and the second is for users' use of platform-born tools.

📗 Matthew Strom is a platform design lead at one of the most impressive platform companies, Stripe. If you're building a platform, thinking about it from these angles may help you find the flywheel we're all after.

Read the full article on Matthew's blog

The architecture of the perfect UX for LLM chatbots

Shopify's Sidekick.

We've all experienced it: LLM chatbots are slow. It's an excellent challenge for product engineers aiming for a great experience because the classic request/response scheme doesn't work.

LLM chatbot answers are streams of tokens, that are turned into streams of characters. So, by leveraging streaming, we can cut that response time. Streaming will render responses token by token, turning it into the ChatGPT experience we know.

But LLM chatbots aren't helpful as is; they become useful when they can search for additional content and, more generally, use tools.

"The typical tool integration goes:

1. Receive user input.

2. Ask the LLM to consult one or more tools that perform operations.

3. Receive tool responses.

4. Ask the LLM to assemble the tool responses into a final answer."

This worsens our slowness problem because data can only be rendered after step 4. That can mean dozens of seconds of wait time. The Sidekick team's solution is interesting: de-couple to parallelize invocations.

"We've made a tweak to break the tool invocation and output generation out of the main LLM response, to let the initial LLM roundtrip directly respond to the user, with placeholders that get asynchronously populated."

That's a game changer because this architecture allows streaming the response without waiting for intermediate invocations. The UI orchestrates multiple multiplexed LLM streams to build the final response while rendering instantly to show the user maximum progress.

📗 Ates Goral's Sidekick's Improved Streaming Experience shows how much the unstructured aspect of LLMs allows engineers to be creative with architecture. Asynchronous tool resolution and the UI-driven process make a lot of sense in that context.

Read the full article on Shopify's blog

Everything you need to know about Message Queues

"Message Queues are a way to transfer information between two systems. This information—a message—can be data, metadata, signals, or a combination of all three."

While there are many different ways to transfer information, most lead to high coupling, 1-to-1 transfers only, and failure correlation.

"[With a message queue], one system is decoupled from the other in terms of responsibility, time, bandwidth, internal workings, load and geography."

Queues' killer feature is this decoupling. But as with everything software, there is no silver bullet, and decoupling comes with its trade-offs. Even though it's often system-specific, these usually are latency, consistency, and complexity.

When evaluating queues, you must look at a few dimensions:

1. Delivery guarantees can be At-Least Once, At-Most Once, or Exactly Once.

While Exactly Once is the best-sounding one, it's also the trickiest and, in most cases, a big smoke screen for complexity. If you don't know what to pick, go for At-Least Once.

"[At-Least Once] is the most common delivery mechanism, and it's the simplest to reason about and implement. If I have a message for you, I will read it to you, and keep doing so again and again until you acknowledge it."

By architecting your code for failure and retries, you'll build a significantly more resilient system than praying for at least once delivery guarantees to do the job for you.

2. Strict ordering vs. parallelism.

Again, the best-looking guarantees are also the most limiting. Opting for messages to be strictly processed in the order they were published makes parallelized processing forbidden and puts a speed limit on the overall process.

📗 's The Big Little Guide to Message Queues is an excellent introduction to queues and their main attributes. If you're reading this article and still think that you need Exactly Once and Strict Ordering, here's a last piece of wisdom from Sudhir:

Most problems that seem like they have a hard FIFO requirement can often lend themselves to parallelism and out-of-order delivery with a little bit of creativity."

"Designing our messages to allow for out-of-order delivery and idempotency makes the system more resilient in general, while also letting use more parallel messaging systems—often saving time, money and a lot of operational work."

Read the full article on Sudhir's blog

Did a friend send this to you? Subscribe to get these weekly drops directly in your inbox. Read the archive for even more great content. Also: I share these articles daily on LinkedIn.

Subscribe now

How to do Great Work by Paul Graham

A life-changing and driving read for ambitious people.

📗 How to do Great Work by Paul Graham (Viaweb, Y Combinator, and many more) is an essay and a recipe for ambitious people on how to get the most out of work. Great work is impactful towards your ambitions and ideally also impactful to the entire world or a subset of it.

"If you collected lists of techniques for doing great work in a lot of different fields, what would the intersection look like? I decided to find out by making it."

It's a fascinating piece filled with deep thoughts and insight from someone who has given much thought to the topic and advised many people.

I usually try to summarize the articles I share, but this one cannot be summarized. Reading it is a journey towards your ambitions and personal fulfillment. Getting to the end already means something about you. I assume few people do.

Still, here are a few thought-provoking quotes:

"What should your projects be? Whatever seems to you excitingly ambitious. [...] What are you excessively curious about — curious to a degree that would bore most other people? That's what you're looking for."

"The three most powerful motives are curiosity, delight, and the desire to do something impressive. Sometimes they converge, and that combination is the most powerful of all."

"Writing a page a day doesn't sound like much, but if you do it every day you'll write a book a year. That's the key: consistency. [...] If you do work that compounds, you'll get exponential growth. [...] Learning, for example, is an instance of this phenomenon: the more you learn about something, the easier it is to learn more."

"Consciously cultivate your taste in the work done in your field. Until you know which is the best and what makes it so, you don't know what you're aiming for. And that is what you're aiming for, because if you don't try to be the best, you won't even be good."

Read the full essay on PG's blog

What is a Merkle Tree?

"It is useful to think of computing the Merkle tree as computing a collision-resistant hash function, which takes n inputs (e.g., files) and outputs the Merkle root hash."

A Merkle tree is a recursive tree of hashes from a set of leaf elements we want to guarantee integrity of. Each element's content is hashed and used to build that tree from the leaves to the root hash. That root hash is effectively a collision-resistant representation of the entire collection of elements.

"The beauty of Merkle trees is that a prover, who has a large set of data can convince a verifier, who has access to the set's Merkle root hash, that a piece of data is in this large set by giving the verifier a Merkle proof."

What's the Merkle proof? Simply put, it's the set of complementary node hashes needed to rebuild the root hash, starting for the element being verified. In the example below, given h1,8 (the Merkle root hash), verifying h4 would only require three hashes: h3, h1,2, and h5,8. It seems trivial on eight elements but scales very well to millions.

📗 Alin Tomescu's What is a Merkle Tree defines that data structure that is foundational to many distributed systems by giving three examples: integrity of a set of files, Bitcoin transaction verification, and corruption detection when transmitting data. The perfect combination of applied theory!

Read the full article

The difference between root certificate authorities, intermediates, and resellers

As we live in a post-Lets Encrypt world, SSL certificates aren't the currency they once were. Yet they still are invaluable to secure the Internet.

That security revolves around the essential concept of trust with some cryptography on top. Browsers only accept certificates for HTTPS webpages if they were issued by a publicly-trusted SSL certificate authority. But how does it work? Who are these Certificate Authorities, and why are so many?

"There are two types of CA certificates: root and intermediate."

"Root CA certificates, also known as "trust anchors" or just "roots", are shipped with your browser."

"Intermediate CA certificates [...] are certificates with the "CA" boolean set to true in the Basic Constraints extension, and which were issued by a root or another intermediate."

Intermediate CA certificates recursively point to other certificates. If the recursion ends with a trusted root CA certificate, browsers know they're trusted. Simple. So why are there so many different issuers but so few trusted root CAs (Firefox trusts around 50 organizations)?

Because SSL certificates have long been a lucrative business, with many different models that are complex to tell apart:

1. Root CAs often have their own business selling certificates.

2. But they can also issue "external" intermediate certificates to other organizations that can issue certificates, too.

3. Or open a reseller API for anyone to sell certificates issued by the CA.

And the entire thing is recursive!

📗 Andrew Ayer's The Difference Between Root Certificate Authorities, Intermediates, and Resellers describes the ecosystem around certificate issuance. Even in a world where Lets Encrypt enables any organization to generate trusted certificates for free, these intermediates and resellers still have immense businesses.=

Read the full article on Andrew's blog

Did a friend send this to you? Subscribe to get these weekly drops directly in your inbox. Read the archive for even more great content. Also: I share these articles daily on LinkedIn.

Subscribe now

Shift left on go-to-market to build better products

GTM is part of the product and part of the PM's job. And like other considerations, it shouldn't happen in a vacuum. Go-to-market must influence and be influenced by users, problems, opportunities, solutions, and execution.

But what's GTM, exactly? Many things.

Go to Market is a catch all term for the process of bringing a product to market and making it available to customers."

Distribution. How will I get the customer to benefit from the value? How will I communicate that value? How do I want customers to think about the product?

"Great PMs shift left on go to market, considering and developing their distribution strategy while they're still prioritizing problems to solve, long before a PRD is written, let alone a line of code. And they build better products because of it."

How do they build better products? By closing the feedback loop and using early learnings and GTM reflections to inform their product strategy.

You can do it too! Try the famous Working backwards Amazon method of writing your product's press release first. Write a FAQ. Or even simpler, the author suggests writing the website copy for that new product. These will force you to step into your customer's shoes and think about GTM.

"Once you begin acting as though GTM is your job (it is), you'll find that you have better ideas not only about how, where, and when to distribute your product, but actually about what to build."

📗 Frank Tisellano's Shift left on go-to-market to build better products goes deeper into why GTM is essential to the Product job. The author also gives the brilliant example of Airbnb to illustrate how embedding a distribution strategy into your product can make a huge difference.

Read the full article on Frank's blog

Mid-career stuckness

Read about it before (or while) it strikes!

After a few years of career, it's easy to be caught in an existential crisis. Whatever you were hired to do as an individual, you now do ten times faster, and better. You feel as good technically, if not better than some Senior colleagues, yet they get paid twice as much as you and have a more prestigious title.

Relax–you're experiencing your first career stuckness, a first wall to hit and crawl over.

"The most common cause [...] is the lack of an understanding of their power, the power of others, and then not developing their sources of power and using those to influence others to get things done."

Still unclear? Let me develop the idea:

1. Your manager evaluates your impact. The impact is all that matters.

2. There is a smaller chance than when you started that you can do your job ten times faster now to have more impact.

3. The answer is others. One of the main traits of a Senior person, beyond autonomy, is their ability to have impact through and with others. Influence. Leadership.

“Leadership is a process of influencing a group of people to produce some common goals."

How do you build that influence? By exercising some power. While it can have a negative connotation when used against someone's will, power is broader and kinder than this! Learn to recognize your strengths and how to use them effectively.

Your powers can be:

1. Technical expertise. Not just ability but something greater that can benefit your organization and peers.

2. How you feel about others and others feel about you.

"When people see you as a role model, like working with you, want to be associated with you, or are drawn towards you for your charisma, you can influence them without formal authority."

3. Your role, title, and relationships in the company.

4. And, of course, your ability to give, reward, and punish.

But they can be many other things relative to your peers' needs.

📗 Subbu Allamaraju's Mid Career Stuckness is a must-read for all professionals. Beyond the "leadership is power" dogma, the author gives pragmatic advice on how to grow your understanding of leadership, influence, and power.

Read the full article on Subbu's blog

SQL queries don’t start with SELECT

When building queries involving multiple operators, we often ask the same questions. One is "Can you filter the result of a window function in a WHERE or HAVING or something?"

"This led me to a bigger question – what order do SQL queries actually run in?"

Julia Evans means in what order is the SQL expression evaluated, which will drive what you can and cannot do with SQL. The order is as follows:

That mental model can help you evaluate faster whether you can apply WHERE to data transformed by window functions.

🚨 Beware! This is just the SQL evaluation, not data processing.

"Database engines in practice don't actually run queries by joining, and then filtering, and then grouping, because they implement a bunch of optimizations reorder things to make the query run faster as long as reordering things won't change the results of the query."

📗 Julia Evans's SQL queries don't start with SELECT is an excellent diagram to keep in mind when writing SQL queries.

Read the full article on Julia's blog

Did a friend send this to you? Subscribe to get these weekly drops directly in your inbox. Read the archive for even more great content. Also: I share these articles daily on LinkedIn.

Subscribe now

M. Hashimoto’s approach to building Large Technical Projects

How Mitchell Hashimoto (Terraform, Vagrant, and many more) builds and completes large projects.

"Whether it's building a new project from scratch, implementing a big feature, or beginning a large refactor, it can be difficult to stay motivated and complete large technical projects."

Who best than Mitchell, founder of Hashicorp and founding engineer of many of Hashicorp's impressive suite of tools, to take advice from on that topic?

He advises to cadence the visible results to keep your excitement high and use continuous demo-ability and that excitement factor to order your work. How does he stay excited about the project?

"The goal is to always give yourself a good demo. [...] Initially, I try to think what a realistic project is where I can see results as soon as possible."

Break down the large projects into small sub-projects with visible results, and start building to assemble them into a first demo.

Once you reach a runnable demo, you can return to small sub-problems and fix the most critical issues. A usable demo helps prioritize these problems: you can now feel firsthand what needs improvement.

📗 Mitchell Hashimoto's My Approach to Building Large Technical Project explains the method through the example of a terminal emulator personal project Mitchell has been building. While the advice may seem generic, it's essential to remember that motivation is deeply personal.

"Everyone I think needs to find some process to reinforce their motivation in a healthy way. I realized seeing results motivates me really strongly, I've built my work style around that, and it has worked well for me thus far."

Read the full article on Mitchell's blog

Emoji under the hood

Did you know that complex emojis are just combinations of existing ones?

👨 + 💻 → 👨‍💻

🐻 + ❄️ → 🐻‍❄️

The Unicode implementation of emojis is quite fascinating.

"At their simplest, they are just that: another symbol in a Unicode table. That's why emoji behave like any other letter: they can be typed in a text field, copied, pasted, rendered in a plain text document,..."

The simple ones have a code of their own.

🐙 is just U+1F419.

The rest are the source of many engineer nightmares because one character maps not to one but to a sequence of codepoints.

"Meet Grapheme Clusters. Grapheme cluster is a sequence of codepoints that is considered a single human-perceived glyph."

"Grapheme clusters create many complications for programmers. You can't just do substring(0, 10) to take the first 10 characters—you might split an emoji in half (or an acute, so don't do it anyway)!"

⚠️ You can't cut strings, can't measure an accurate length, or reverse them without a grapheme-cluster-aware library. So complex emojis are combinations of multiple codes, some of which are valid standalone emojis (but not all).

"Instead of adding a new codepoint for each emoji and skin tone combination, only five new codepoints were added. [...] Together they form a ligature: 👋 (U+1F44B WAVING HAND SIGN) directly followed by 🏽 (U+1F3FD MEDIUM SKIN TONE MODIFIER) becomes 👋🏽."

"U+200D is called ZERO-WIDTH JOINER, or ZWJ for short. It works similarly to what we saw with skin tone, but this time you can join two self-sufficient emoji into one. Not all combinations work, but many do, sometimes in surprising ways!"

📗 Nikita Prokopov's Emojis under the hood details all the hidden secrets of emojis' unicode representation. It's a lot of fun but also quite actionable knowledge. For instance, you do not need a large mapping table to turn an ISO country code into its flag, but a little emoji magic!

Read the full article on Nikita's blog

Scaling Stateful Objects

"It is widely (MIS)believed that stateless Server-Side Apps are The Only Way™ to scale Server-Side."

Every serious project starts with a stateless handler and a database. And for most, the best practice is to persist every action to the database. At scale, there are many solutions to scaling reads, from caches to replicas.

But writes are a lot more complex to scale without losing ACID guarantees.

For your application, the more you write, the more this becomes a problem. Reads will almost scale infinitely but writes undoubtedly won't.

“keeping our request handlers stateless, does NOT really solve the scalability problem; instead – it merely pushes it to the database."

"In a pretty much any serious real-world interactive system, it is database which is The Bottleneck™."

But write-heavy applications exist, the most famous being online games and all other interactive, real-time applications like Figma. These applications cannot guarantee scalability and low latency if they push every action to durable storage.

"Server-Side Apps with an In-Memory State can easily save us 10x-1000x of database load."

"Of course, these performance benefits of Stateful Server-Side Apps don't come for free (nothing does). The currency we'll be paying with for this drastically improved performance, is Lack of Durability. In other words – if our Stateful Server-Side App crashes, we'll lose all the state which haven't been saved yet to the DB."

📗 IT Hare's Scaling Stateful Objects is a timeless read (except for a few numbers) about the complexity of scaling write-heavy applications. It raises a valid point of durability vs. performance and offers many patterns to solve this. We engineers have a love-hate relationship with stateful systems, but there is still innovation in the space, like Cloudflare's Durable Objects.

Read the full article on IT Hare

Did a friend send this to you? Subscribe to get these weekly drops directly in your inbox. Read the archive for even more great content. Also: I share these articles daily on LinkedIn.

Subscribe now

TigerBeetle’s database design document

TigerBeetle shows how specialization can enable building a database an order of magnitude more performant and resilient.

TigerBeetle is a distributed financial accounting database and possibly the most opinionated database in the world.

"TigerBeetle implements the latest research and technology to deliver unprecedented safety, durability and performance while reducing operational costs by orders of magnitude and providing a fantastic developer experience."

The team goes to extreme lengths to reach their expected performance and resilience. And one important axis is simplifying the business domain as much as possible.

"In theory, TigerBeetle is a replicated state machine that takes an initial starting state (account opening balances), and applies a set of input events (transfers) in deterministic order, after first replicating these input events safely, to arrive at a final state (account closing balances)."

That's it. Everything else stems or is made possible by these simple foundations.

📗 Joran Greef's Design Document of TigerBeetle is a fascinating read because it describes the lengths he goes to build the fastest and safest accounting database. Every detail is rigorously thought-through, from every fixed-size data structure to the replication protocol and timestamp management. And as a bonus, TigerBeetle is open source so you can see the resulting codebase.

These principles can apply to all software: simplifying the business domain and closing (or opening) future space for evolution is an important decision that can dramatically impact whatever you're looking to optimize. In Tigerbeetle's case, it's all for performance and resilience, but in most other cases, it's OK to trade some performance for optionality.

Read the full doc on Github

Just-in-time image optimization at Reddit’s scale

Reddit serves billions of images per day.

Building a great browsing experience in a media-heavy platform requires lightning-fast image loading. Reddit does that through image optimization (reducing the size, picking the correct format) and caching the transformed images.

"Reddit has been using a just-in-time image optimizer relying on third-party vendors since 2015. [...] it made sense to move this functionality in-house due to cost and control over the end-to-end user experience"

The cost aspect led to the decision to focus on images. So the team set out to replace this central piece of the Reddit platform.

"Our task was to change almost everything about how billions of images are served daily without the user ever noticing and without breaking any of the upstream company functions."

The solution chains the following components:

1. A Content Delivery Network (CDN). Answers all image-related requests and is in charge of caching responses. Image optimization is costly and lengthy, so the Cache hit ratio is the important CDN metric.

2. An image optimization service based on the C library libvips. Only called when the transformed image is not in the cache.

"As a result of moving image optimization in-house, we were able to:

• Reduce our costs for animated GIFs to a mere 0.9% of the original cost

• Reduce p99 cache-miss latency for encoding animated GIFs from 20s to 4s

• Reduce bytes served for static images by ~20%"

📗 Walter Michelin's Just In Time Image Optimization at Reddit scale describes the mission-critical project the Reddit team undertook to move image processing and caching in-house. A good example of pragmatic engineering: start from the problem, identify the metrics, and build from there.

Read the full article on Reddit's blog

A collaborative approach to defining design systems components

Design systems have very different users with different agendas. Engineers optimize for clean code, and designers for a clean look & feel. This misalignment leads to components being misused into stretched use cases, making the overall system more brittle and costly to evolve.

For example:

"Modal, prompt, and popover are components that may look alike, but are intended to solve very different problems."

"So, how can we come up with component definitions that everyone agrees on, and that can scale with the system as it continues to evolve?"

1. Start with a clear, specific purpose.

"When everyone agreed on the component’s purpose, naming and behaviors suddenly became clear."

2. Write the component's design rationale.

"We start with just the concept (modal), followed by a short, detailed description (e.g. a modal displays content in a layer above the page, requiring the user’s interaction to proceed), and then usage documentation, along with “do” and “don’t” examples."

That design rationale will pave the way for consistency across your design system.

📗 Wart Burggraaf's A collaborative approach to defining components describes how thinking about purpose can ease collaboration on design systems. If we take a step back, a design system is a product, and like all products, the first step to clarity and alignment is a clear vision. The purpose is that vision.

Read it on designsystems.com