"Kill It with Fire" Review

Published: 2023-09-10

Tagged: software legacy-code essay

In my career, I've seen only a handful of teams with a structured way to deal with legacy software. Most approached the problem without a clear sense of direction or priority. Makes the outcome of such work a coin toss.

It's surprising how common this is given how many of us work on such code. And I'm not talking about wrangling COBOL or running ancient mainframes.

There's no single point when a shiny piece of software turns into a smelly pile of junk that nobody likes. It happens gradually, often beginning before the official release date. Early decisions ossify into technical debt. Stability & performance issues expand to consume more and more of the team's time. The process and the problems it creates seem as inevitable as growing old.

In "Kill It with Fire" (KIwF), Marianne Bellotti offers a pragmatic approach to dealing with these problems. Given that managers, tech leads, and senior ICs are usually the best positioned to handle these issues--or at least that's where these issues naturally accumulate within an organization--they stand to benefit from giving this book a read.

The book's main claim is that problems with legacy software originate from deep (ie. legacy) human psychology: we promise we'll fix our shortcuts later; we underestimate how much effort rewriting a system from scratch will take; we overestimate our knowledge of the existing system and its constraints; we mistake consistency--keeping up with trends, for example--with creating value; we focus on the wrong goals, or, alternatively, we define them so loosely that our projects fizzle out.

All these patterns are reflected in the workings of the organization around us. They're magnified, in fact, by the awesome power of collaboration, which leads to all sorts of failures, from projects being late and over budget to projects failing to finish at all.

In other words, it's not technical issues that lead to failure, but organizational ones.

Bellotti spends only a little while on explaining why this happens. If you're familiar with common biases, nothing here will surprise you much. Still, it's interesting to learn how such thinking patterns influence the enterprise of producing software specifically. Her primary focus, and the rest of the book, is more practical: how to increase the chance we'll succeed when we have to work on legacy code?

The approach offered breaks the whole problem into two halves: one about the people above you and one about those below you.

Up the chain of command, you have directors, product managers, and other business-oriented folks. To them you need to prove that your work on legacy software has value. You can only do that by first figuring out, then by talking about risks, timelines, and costs (incl. opportunity costs).

KIwF offers a toolbox of ideas and techniques to help you with that. And while many of them sounded obvious at first, my memory tells me that most engineers do the exact opposite of what Bellotti suggests. Most will argue for the necessity of their project by saying that X is old, or that everybody uses Y now, or that building Z would make for a good blog post. Better arguments promise improvements in velocity or efficiency, but are so vague as to be meaningless--think: migrating a dozen or two cloud instances to Kubernetes without a clear idea of how many hours that will save or how it will increase the number of daily deployments.

What Bellotti's advice leads to, as I understand it, is adversarial collaboration. Your role in this is to come up with ways to make the software better--ultimately--for end users. Leadership's role is to test your ideas to leave only the best ones. And while this may not always result in the best possible decision, it should at least minimize the chances of your team flat out wasting your time.

Let's now turn to the other half of the problem, the one about the colleagues you lead. Here the emphasis is on focus and motivation.

It's hard to motivate engineers to do good work on legacy software. Such software is simply not cool. It involves languages and frameworks that everyone already knows. There are few exciting questions or developments in the area. And since the code has changed hands a few times already, it's likely that maintenance makes up for the vast majority of effort expanded on it. That means engineers are fixing bugs and troubleshooting arcane issues. Unlike feature launches, nobody celebrates solving a particularly ugly problem. None of this looks good on a resume either.

Alan Kay was right when he said that software is pop culture. It's why engineers like working on cool things. It earns them respect from their peers. It ensures a steady supply of interesting challenges that can be turned into blog posts, tech talks, or lunch and learns. Finally, cool work ensures career growth by capturing the attention of recruiters and interviewers.

So how do you tackle this? Bellotti points out that the problem contains its solution: find ways to make engineers feel seen. She remarks that this type of reward is often more powerful than a monetary one--something that Jerry Weinberg wrote about in "The Psychology of Computer Programming" over 50 years ago!

The counterpart to motivation is focus. KIwF reminds us that we're all just sophisticated apes so we'll often do what incentives tell us to do. And if those are vague, we'll naturally find ways to spend our time on things that have little to do with our objectives.

Many engineers will seek to inject some cool into their work by advocating for using a new language, framework, or library--never mind that doing so would add engineer-months to the timeline for essentially no benefit. Others, lacking a way to measure progress, will become sidelined by abstract questions about programming style or the superiority of one code editor over another.

This struck close to home. My own repeating (thankfully less often now) mistake here is focusing too much on improving the system for future engineers. That entails taking the longer, harder way to build something so that it lasts. It's is fun and earns peer respect. But it has nothing to do with business objectives--after all, many software projects are bets about an uncertain future, so something good now is many times better than something better later.

Taking a step back, we can see that KIwF is a shot at improving an industry-wide efficiency wicked problem: getting value out of older code. And since engineer time is so expensive, that seems like a noble endeavor. On a more personal level, however, it just bugs me that I've seen the same problems pop up repeatedly only to be handled in the same awkward and unsatisfying ways.

It's as if the whole industry decided to pretend that this is something we just can't get right because we haven't found a single elegant solution which appeals to our appetite for cool. In this context, Bellotti with her pragmatic approach offers us a chance to manage the problem on a whole new level.