Learning from Giants #26

SPACE engineering productivity metrics, Writing tests, Measuring product health, and the Amazon Redshift reinvented paper.

👋 Hi, this is Mathias with your weekly drop of the 1% best, most actionable, and timeless resources to grow as an engineering or product leader. Handpicked from the best authors and companies.

Did a friend send this to you? Subscribe to get these weekly drops directly in your inbox. Read the archive for even more great content. Also: I share these articles daily on LinkedIn.

---

Measuring Engineering Productivity with SPACE Metrics

As a manager / CTO / VPE, measuring your engineering team's productivity is crucial for driving success and innovation within your organization.

And there were many attempts at this, starting with the famously bad "lines of code" metric. And these wrong metrics have been used to manage teams and drive them into the wall for decades.

"SPACE was written to provide a better understanding of what developer productivity is, as well as share ideas for how teams can start measuring it."

First, a significant difference with most approaches:

"Productivity can't be reduced to just activity."

So here's the SPACE framework:

"S – Satisfaction & Well Being

P – Performance

A – Activity

C – Collaboration & Communication

E – Efficiency & Flow"

Each category can be measured using many different metrics (see picture below). And you don't need to track all of them.

Some example category metrics. Source: SPACE paper.

So, where to start?

The author suggests starting with a Why. What is your goal? Why do you want to measure your team? And then use SPACE as a discovery framework to figure out the metrics.

📗

Abi Noda

's summary of The SPACE of Developer Productivity paper describes this novel way of analyzing engineering efficiency. It opens the question beyond activity and explores how satisfaction, collaboration, or flow can be as crucial as lines of code.

Measuring Productivity with SPACE

---

How I Write Tests

Testing is the foundation of good software engineering, but it can be challenging to get right. If we spend more than half our coding time on tests, we should invest as much time into learning how to write tests as we spend learning how to code.

"I fully subscribe to the definition of legacy code as code without an automated test suite. I'm convinced that the best thing you can do to encourage fast progress in a test suite is to design for testing and have a fast, reliable, comprehensive test suite."

📗 In How I Write Tests,

Nelson Elhage

shares his test-driven-development philosophy, which focuses on working on modules one at a time and writing tests before moving on to the next module. He also discusses the importance of avoiding manual testing and investing in testing tools.

Other useful techniques covered in the article include using fakes or mocks, writing regression tests, and using real-world examples in tests.

"If testing your application in code is hard or frustrating, it's worth the effort to invest in your own testing tooling to make it easier! Tests are an important part of your codebase, and you wouldn't skip important features just because they were hard!"

---

Metrics to measure Product Health

"Good products grow well as they penetrate their market, retain and engage their users, and bring users back regularly with a healthy content loop."

📈 GROWTH METRICS: growth can be analyzed through active users (DAU, WAU, and MAU), changes in active users (D/D, W/W, M/M, Y/Y), the Quick Ratio (the ratio of the sum of new and resurrected users over the churned users), new users/MAU, and net growth/MAU.

📊 RETENTION METRICS: These include retention, resurrection, and churn rates. Retention rate is the percentage of users who return to the product after a specified time period (e.g. 30 days). Resurrection rate is the percentage of users who were inactive at a specific point in time but returned to the product later. Churn rate is the percentage of users who were active at a specific point in time but are no longer active.

📌 STICKINESS: Stickiness measures how much users engage with the product. Stickiness can be calculated by dividing the average daily active users by the average monthly active users. A product is considered sticky if it has a stickiness of more than 0.1.

👀 ENGAGEMENT: Engagement can be measured in various ways, depending on the product type. It will often be a business metric. For example, social media products can use daily likes per user, e-commerce products daily orders per user, and gaming products daily game sessions per user.

📗 Sequoia's Measuring Product Health is a must-read for anyone in product management or entrepreneurship. By tracking growth, retention, stickiness, and engagement metrics, companies can identify areas for improvement and make informed decisions about their product's future. The article introduces all these different metric families, in which you'll have to find the important ones for your business.

---

Amazon Redshift Reinvented

"Redshift is a column-oriented massively parallel processing system. [...] Tens of thousands of customers use Redshift to process Exabytes of data daily."

You probably know Redshift already, so let's skip to the interesting parts:

1. Like many modern data systems, Redshift's architecture separates compute from storage.

Storage is managed by the Redshift Managed Storage layer (RMS), based on heavy caching of S3 data on SSD-backed nodes.

"RMS is designed for a durability of 99.999999999% and 99.99% availability over a given year, across multiple availability zones (AZs)"

2. To optimize Compute, it is heavily distributed. It considers the cluster and data topology to push predicates as down as possible to the physical storage. AWS engineers did much optimization work at the query planning layer.

"To reduce the number of blocks that need to be scanned, Redshift evaluates query predicates over zone maps i.e., small hash tables that contain the min/max values per block and leverages late materialization."

Redshift architecture. Source: Amazon Web Services.

3. Further compute optimization is done using C++ code generation, which then runs on hardware specifically designed for the workload.

"Redshift generates C++ code specific to the query plan and the schema being executed. The generated code is then compiled and the binary is shipped to the compute nodes for execution."

📗The Amazon Redshift Re-invented paper, published in SIGMOD'22, discusses the evolution of Amazon Redshift since its launch in 2015. As always with large papers, I'm sharing a summary by Murat Demirbas so you can get the most out of it.