Learning from Giants #25
Learning from Giants #25
JavaScript: the unexpected parts, Amazon's 1998 Distributed computing manifesto, Staging environment best practices, and a chapter book summary on Replication in distributed systems.
👋 Hi, this is Mathias with your weekly drop of the 1% best, most actionable, and timeless resources to grow as an engineering or product leader. Handpicked from the best authors and companies.
Did a friend send this to you? Subscribe to get these weekly drops directly in your inbox. Read the archive for even more great content. Also: I share these articles daily on LinkedIn.
JavaScript: the crazy, unexpected parts
I usually avoid programming language-specific content because it ignores parts of my readers. But JavaScript is not a language; it's synonymous with the Web now. It's a must-know.
Although JavaScript is everywhere, and actually because it is everywhere, it's a very complex language with dark corners you do not want to end up in.
"Everything in JavaScript acts like an object, with the only two exceptions being null and undefined."
"The typeof operator (together with instanceof) is probably the biggest design flaw of JavaScript, as it is almost completely broken."
And this is just a random selection of two very telling examples. It's a complete mess.
📗 Ivo Wetzel et. al's JavaScript Garden is a collection of documentation about the weirdest parts of the language. It contains many examples and advice on bypassing most of them correctly, with acceptable performance.
The Distributed Computing Manifesto (Amazon, 1998)
Some areas in software are harder to Stand on the Shoulders of Giants than others. You can build a complex full-stack web reactive application with a simple `npx create-remix` and instantly stand on decades of iteration and invention. On the other hand, you can't design great software architecture with a one-liner.
System design is where many companies and engineers thus fail exactly where thousands of engineers have already failed.
How can we avoid these mistakes? How can we do micro-services without breaking every principle they exist to solve?
By learning from giants and understanding that all the current architectural trends weren't born out of thin air. They result from thousands of iteration cycles from brilliant engineers and millions of mistakes and aborted bad ideas.
What better example than Amazon, which has pushed the boundaries of high-performance, planet-scale computing, and workflows for almost three decades?
"It was more than just a high-performance website, we are talking about everything from high-volume transaction processing to machine learning, security, robotics, binning millions of products – anything that you could find in a distributed systems textbook was happening at Amazon, and it was happening at unbelievable scale."
"Over the next two decades, Amazon would move from a monolith to a service-oriented architecture, to microservices, then to microservices running over a shared infrastructure platform. All of this was being done before terms like service-oriented architecture existed."
Every time the business grew 10x, everything had to be simplified and re-invented by brilliant engineers. This time in 1998, a document first proposed the idea of service-oriented architecture.
"It is clear that we need to create and implement a new architecture if Amazon's processing is to scale to the point where it can support ten times our current order volume. The question is, what form should the new architecture take and how do we move towards realizing it?"
📗 In The Distributed Computing Manifesto, Werner Vogels shares this internal Amazon document from 1998. It pitches the move from a two-tier architecture (application and central database) to a service-oriented architecture where data and workflow processing are split into different services, and business logic layers are decoupled from data storage. This a move many companies still have to do in 2022, and it may help you get it right too!
Best practices for staging environments
"Let's talk about staging."
Staging is a controversial topic for start-ups, which I previously covered with a contrarian article. This article needed balancing, so here's one favoring staging environments.
Staging is where you validate the known-unknowns of your systems.
The staging question is, in fact, a confidence and risk tolerance question. There is a level of confidence in your deployment and code that you can only get with a staging environment.
However many tests you will write, unit, integration tests, you will always miss some cases. How well do you know your production environment? What's the impact of your load balancer?
"This isn't to say you should stop writing tests. Tests catch known-knowns, the step before the known-unknowns of staging. But tests are not a replacement for staging."
But like everything, staging environments have dos and don’ts and some great articles to start!
📗 Alice Goldfuss's Best practices for staging environments is a rare objective take on staging environments. It doesn't try to enter staging debate but instead states why you'll need a staging environment at some point, how you can set it up, and common mistakes to avoid.
Designing Data-Intensive Applications: Replication
Designing Data-Intensive Applications by Martin Kleppman. The one book every engineer touching distributed systems must read. But it's many hundreds of pages, so definitely not today's topic. Instead, I found some great engineers summaries.
Let's focus on one chapter: Replication. A fundamental distributed computing topic.
"Why do we replicate?
Reduce latency.
Increase availability.
Increase throughput.”
Replicating data in itself is not particularly hard. scp target destination, right? But it's keeping the guarantees that non-replicated data offered that is hard. Liveness, transactions, etc. And gracefully handling failure scenarii.
📗 Adam Conrad's summary of the Replication chapter is a peek at the knowledge goldmine the Designing data-Intensive Applications book is. It describes replication strategies, whether with or without leader election, and the real-world use cases behind.