Learning from Giants #70
10x your data viz skills and impact, 22 tips to pass (and grade) a coding interview, a deep dive into Neon DB's storage engine, and a fantastic re-post on Stripe's history and culture.
Hi! This is Mathias with your weekly drop of the 1% best, most actionable, and timeless resources to grow as an engineering or product leader. Handpicked from the best authors and companies. Guaranteed 100% GPT-free content.
I got feedback that not everyone reads work-related articles on Saturdays (fair enough). So I’ll try sending them on Tuesdays for a few weeks!
Did a friend send this to you? Subscribe to get these weekly drops directly in your inbox. Read the archive for even more great content. Also, I share these articles daily on LinkedIn.
From data to data viz
Every chart tells a story, but not all stories are best told by a pie chart.
Using the appropriate graph can turn a vague illustration into a central part of your communication. There are dozens of graphs to pick from, including some complex ones like Chord diagrams, Sankeys, and sunbursts, so how can you make the right choice?
Use a decision tree made by experts!
"Pick the main [data] type using the buttons below. Then let the decision tree guide you toward your graphic possibilities".
Once you've picked a chart, you still have a few layout and density decisions to make, and here again, you could make mistakes.
"Avoid the rainbow color palette when you map a numeric variable."
"Placing individual plots horizontally or vertically is an important choice to make"
Experts can help us here too!
📗 Yan Holtz's From data to Viz is a precious resource to help you improve your data visualization skills with very simple heuristics. Use their graph decision tree and read the gallery of typical caveats to get started now!
How to pass a coding interview with Robert Heaton
A large part of all coding interviews fail for dumb reasons. You can't claim you're the best engineer there is and take 10 minutes to get your IDE to run code.
So here are a few must-haves for any coding interview:
"Set up a basic hello-world program beforehand and make sure you can run it."
"Start by drafting your program using prose of pseudocode. Don't start by writing actual code."
"Once you've started writing code, run it as frequently as you can."
Ideally you should be doing that using unit tests. It's cleaner and safer than running by hand.
"Don't use a testing framework unless you're required to or you're very confident with it."
Keep your code reasonably simple.
"Don't abstract too soon is good advice for the real world, but it's particularly important in an interview. The costs of choosing an incorrect abstraction are higher than normal."
And don't try to cover all cases in your first iteration. Leave TODOs for edge cases and clean them up once the base case works.
Additionally, remember you're there to complete a coding challenge in a given amount of time, so everything you do should be optimized around time efficiency.
"Talk as much as you reasonably can". Use it to surface misunderstandings as early as possible.
"Consider asking for help." Succeeding after getting some help is often better than failing by staying stuck.
📗 Robert Heaton's How to pass a coding interview with me is a collection of 22 very sound tips the author has learned by interviewing hundreds of candidates in their careers. If you're interviewing or considering interviewing you should definitely have a read.
Deep dive into Neon’s storage engine
Separation of Compute and Storage is the hottest thing in databases. Combined with cloud storage services like AWS S3 and serverless technology like Kubernetes, this pattern allows for virtually infinite-sized databases.
How does this separate storage work in practice?
Neon defines itself as a Serverless Postgres database with separate storage and compute. The compute part is the actual Postgres engine, but the team swapped Postgres' on-disk storage with a remote solution: separate storage servers backed by cloud storage.
Storage engine: under the hood
Neon's storage engine manages Postgres' storage primitives: pages and write-ahead-log (WAL) entries.
"In Neon, [the write-ahead-log of changes] is streamed to three safekeeper nodes, which provide durability and synchronous replication using Paxos-like consensus algorithm. The safekeeper nodes then stream the logs to the pageservers, where the logs are processed, reorganized, and written to immutable files. Finally, these files are uploaded to cloud storage for long-term storage."
Write path
All durable data is append-only: no data is ever modified.
"Neon eliminates the distinction between active data and backups, treating all data the same in the storage system. Because old data is not modified, whenever new data is digested into the system, all the history stays accessible up to a configurable limit."
Every update is given a monotonically integer number (the log sequence number, LSN) which acts as a cluster-wide clock.
"When a read request comes in, the pageserver finds the last image of the page by the LSN, and any write-ahead-log (WAL) records on top of it, applies the WAL records if needed to reconstruct the page, and returns the materialized page to the caller."
Every page is thus simply an ordered list of changes. Since all reads specify a sequence number, the storage server replays that change history up to the given number and returns the built page value.
But that solution slows down as the number of changes grows.
Neon fixes that by compacting the changes into Image layers which are complete images of the page at a given sequence number. This way, the pageserver only needs to apply the changes after the most recent image.
📗 Heikki Linnakangas' Deep dive into Neon storage engine is a simple introduction to storage and compute separation in data stores with a real-world example: Neon. Even though it states the separation simplifies the overall system, the article also shows how it opens the database to a new class of distributed systems challenges.
Top 0.1% re-post: Stripe: Thinking Like a Civilization
/dev/payments
That's the initial name of one of our time's biggest and most inspiring private companies. Do you have any idea which one?
Hint: its founders are siblings and probably two of the most brilliant business people on the planet.
"In Patrick and John Collison, the payments company has a founding pair unusual in their intellect, rarer still in their ego-less vision. From that inspiration, the brothers have manifested an entire culture, characterized by long-term thinking, rapid execution, and the desire to build something meaningful. They've done so while constructing a remarkable business capable of dominating the internet's financial infrastructure even as it empowers millions of other merchants."
"All of which is to say that Stripe thinks like a civilization. It is well on its way to becoming one."
📗 The Generalist's Stripe: Thinking Like a Civilization may be a slight stretch from what I usually share, but I believe as a product or engineering leader, there are some stories you must know. Moreover, the essay is packed with anecdotes and details about Stripe's culture and how the Collison brothers see the world.