Learning from Giants #63
The foundational article about the Log that gave birth to Kafka at LinkedIn, seing Marketplaces through the lens of transaction cost reduction, and how to become insanely well-connected.
👋 Hi, this is Mathias with your weekly drop of the 1% best, most actionable, and timeless resources to grow as an engineering or product leader. Handpicked from the best authors and companies. Guaranteed 100% GPT-free content.
Did a friend send this to you? Subscribe to get these weekly drops directly in your inbox. Read the archive for even more great content. Also: I share these articles daily on LinkedIn.
The Log: What every software engineer should know about real-time data’s unifying abstraction
Spoiler alert: This 2013 article leads to the origin story of Kafka, the now famous log-based system.
But first, let's start with a refresher on the Log.
"Sometimes called write-ahead logs or commit logs or transaction logs, logs have been around almost as long as computers and are at the heart of many distributed data systems and real-time application architectures."
“A log is perhaps the simplest possible storage abstraction. It is an append-only, totally-ordered sequence of records ordered by time."
It shines as a tool to coordinate software components and systems. If all systems are deterministic, you can reduce the coordination and data replication problems to agreeing on a common log of records.
"The purpose of the log here is to squeeze all the non-determinism out of the input stream to ensure that each replica processing this input stays in sync."
In practice, we can use logs in multiple ways: log all requests (e.g., for a database, the SQL queries) or log the underlying data changes.
Most systems use a Log as an internal coordination component. However, the article shows it can also be used as a first-class citizen to solve data integration problems between systems. That's Kafka.
The problem Kafka solves is simple: the amount of data we process is growing exponentially, and specialized systems are outscaling the "one big generalist database" model. Think ElasticSearch for search, BigQuery and Snowflake for analytics. The more systems we operate, the more data needs to be moved from one to the other. Kafka makes that data movement simpler by separating concerns.
"This experience led me to focus on building Kafka to combine what we had seen in messaging systems with the log concept popular in databases and distributed system internals. We wanted something to act as a central pipeline first for all activity data, and eventually for many other uses, including data deployment out of Hadoop, monitoring data, etc."
📗 Jay Kreps' The Log: What every software engineer should know about real-time data's unifying abstraction is a foundational article describing the story that led him to build Kafka at LinkedIn in the 2010s. After the general introduction about the Log, Jay describes the challenges the team faced building and scaling the Kafka Unified Log.
The Concept that explains everything about Marketplaces
"Marketplaces are simply businesses that don't sell goods or services, but instead sell the reduction of transaction costs."
There are many theories about these transaction costs. With the help of decades of theory, the author defines four categories:
Search and information costs: discovering what is available to purchase and comparing alternatives.
Bargaining and decision costs: coming to an agreement between buyer and seller, including establishing the final price and terms.
Enforcement and policing costs: ensuring that both sides holds up their end of the deal.
Distribution costs: actually getting the good or service to the end consumer
All marketplaces help reduce one, many, or all of these costs. Consequently, studying a market's transaction costs is a great way to identify marketplace opportunities. The bigger the cost savings, the more the market will expand.
For example, Uber reduces the bargaining and decision costs:
"Price transparency used to be a key problem when hiring a taxi. Your only option was to get in the car and watch the meter run up. Now Uber calculates the price ahead of time, making it easier to decide whether or not to take a ride."
"Kevin Kwok wrote a great piece about how Uber and Lyft grew the TAM for private cars. Not only that, but they grew TAM precisely in areas where transaction costs were highest."
Looking further, a marketplace's take rate can increase as it reduces costs for buyers and sellers. So, the logical evolution for them is to take on more responsibilities and work on the heaviest costs, which are Enforcement costs and Distribution costs. That's also how competitors can disrupt them. For instance, Amazon disrupted eBay with its asset-heavy, fully-managed model, taking on a lot more costs that eBay did.
📗
's The Concept That Explains Everything About Marketplaces is an excellent introduction to the transaction cost theory of marketplaces. It's an angle that seems obvious initially but has a lot of depth when analyzed further.
How to become insanely Well-Connected
Networking and relationships are games you can't cheat at. It's also a lifelong one that, when done correctly, has a compounding impact. Your end score is the sum of your life's work to create meaningful connections.
“The best way to be highly influential is to be human to everyone you meet."
Beyond that mantra, here are Chris' Seven Rules for Making Memorable Connections.
Convey genuine appreciation. Actively project warmth and high energy. It's been observed people like you when they feel liked by you."
Listen with intent. Being a great listener is an art. You must strive to actively listen, react, and follow up on every sentence to show you're there, not already thinking about your next meeting.
Use humility markers. Acknowledging your own fallibility and human imperfection can go a long way toward making yourself relatable."
Offer unvarnished honesty. Honesty creates trust, and trust is core to all relationships. It's also probably the easiest on the list to stand out.
Blue-sky brainstorm. [...] Give before thinking about what you get. Always offer something of value before expecting or asking for something in return." Brainstorming with your interlocutor is the simplest form of giving.
End every meeting or conversation with the feeling and optimism you'd like to have at the start of your next conversation with the person." Creating a lasting relationship will require many interactions. Whatever you say last will stay, and make the next one easy or hard.
Don't fake it till you make it. And beyond not lying, always come extra prepared to meetings with a specific goal in mind and knowledge about how they can help you reach it. That's particularly true when reaching out to higher-profile people.
📗
's How to Become Insanely Well-Connected is a great article to prepare for 2024. Networking and relationships should be at the top of your list of things to improve professionally, and Chris' advice on the matter is both actionable and obviously powerful.