Thoughts on B. Rich's "Skunk Works"

Published: 2016-09-19

Tagged: readings

I just put down Ben Rich's and Leo Janos' "Skunk Works" and stared at the ceiling while I waited for everything to sink in. It's a great book about about the history of some spectacular pieces of technology and how they were made. Apart from day dreaming about becoming a fighter pilot when I was a kid, I'm not much into aviation. I picked up this book because a post on HN mentioned that the part about how the sausage is made is supremely interesting to anyone in CS/IT.

In short, the book tells the story of Skunk Works - a division of Lockheed Martin that designs and builds the most advanced, coolest, and secret planes. I don't want to spoil the book for potential readers so here's two things that really stood out to me as a developer:

Prototypes and Testing

All of the projects undertaken by Skunk Works were revolutionary in some way and had to tackle challenges that no one before encountered. In order to successfully compete on time and budget they used off the shelf components to build incremental prototypes. This allowed them to ignore the stuff that was already figured out and focus on the real problems. When building prototypes of the F-117 they used engines, computers, hydraulic systems, and other parts from other planes. This allowed them tofocus on the thing that didn't exist, which was the whole point of the project: the stealth technology.

We use a similar process in software land. As often as we're able to use ready-made components for the generic parts we're still left trying to wrap our heads around on how to shape the thought-stuff of code to meet some specific requirement. We have to iterate over a couple of prototypes to get us there quickly and then throw them away. It's really tempting to continue building on that prototype but unless you're going to refactor it top to bottom, you'll quickly get stranded in a bog of technical debt.

Apart from prototyping, they also used techniques eerily similar to the Toyota Production System - they had the white-collar designers work on the shop floor with machinists and technicians; anyone with an idea could talk to the chief; they established good relationships with trusted suppliers; they put a lot of energy into refining their production process by building custom tools or producing their own resources. Where am I going with this?

They exhibited a maniacal focus on quality assurance. The book describes how QA people worked alongside technicians. Whenever a defect was found it was brought straight to the technician responsible for them to fix and learn from. This short feedback loop allowed the techs to rapidly increase their skill level and keep defects in the final product to a minimum. We already have a lot of tools to handle this. My favorite is TDD - it lets me sleep at night knowing that something isn't going to break when that one user has a funny letter in their name. My own experience tells me that quality assurance in the form of that 30% of "extra" time is often left out of projects to get a project out the door faster. This is a mistake as that 30% is often turned into 30+% of unplanned work when users eventually find bugs in your product. If you include QA in your work, you gain two benefits:

1. This work is planned and scheduled. No sudden fires to put out.
2. You gain trust of your users for delivering a quality product.

Leadership

The directors of Skunk Works that are described in this book - Kelly Johnson, and Ben Rich - were both good and experienced engineers. They could identify promising ideas, provide great feedback, or troubleshoot project issues just as well as anyone working under them.

Now, the author (Ben Rich) reminds us that things weren't always sunshine and rainbows: people got annoyed, fought, some quit. But one thing that comes up again and again, although not directly, is that both of these guys were dead honest. If Skunk Works got a contract to research and develop some never before seen thingamajig and their research showed them the project was headed nowhere, Skunk Works would just give the money back. When Kelly Johnson had in mind to build a record breaking plane like the U-2, he straight up told his people that they'd have to work at minimum six day weeks.

The last part resonated very strongly with me because, in my experience, overtime is rarely planned. It just comes up when something horrible happens management-wise. Features get added in during a sprint, new requirements are introduced without moving deadlines to accomodate that fact, flying saucers drop out-of-scope work on you, and the list goes on. Usually it's the developers or designers that have go all-in at this point and forget about daylight for a while. After a while you learn to feel this coming but everybody still tells you that it'll be different this time. Unplanned work is a huge red flag and just like unplanned outages should be avoided at all costs.