Heroic Efforts Block Automatic Deploys

In software development, and especially when deploying a software, there is now and then heroic efforts:

• There is only one person who can configure some part,
• There is a super developer who only knows how to deploy software to some environment,
• There is only one developer who understands integrations to other systems,
• There is only one database guru who knows is it safe to make a change to some table,
• Calling to “Steve” to save ongoing deployment at midnight,
• etc.

We all have been in that situation, either being that hero or seeing heroic efforts. Those aren’t always even difficult, but they rely on some certain person. I have also done some heroic efforts. At the beginning of my career, I was sometimes even proud of them. But the more I gained experience I shamed more of those times (“Why did I make this so difficult or haven’t taught to someone else also?”). In recent years I have noticed that I should try to automate them (“There has to be a way to do this boring repeated manual step automatically and make life easier”).

I recently read a great book, The DevOps Handbook by Gene Kim et. al. (must read for every developer!), and the title of this blog was stuck in my mind. It was something I hadn’t thought clearly even if I had faced it many times. The more you have heroic efforts when deploying, the more difficult it is to automate deployment.

When heroic efforts are necessary to succeed, then a team is in a dark place. (Mike Kavis in forbes.com)

Heroic Efforts are Bottleneck

Heroic efforts are a bottleneck and risk for your deployment process. If you need a certain person to do some part of the process, it will be stuck until he has done it. You might think that a few minutes of work from “John” per deploy is nothing. But when you every time depend on him, others won’t know how to do it. Unfortunately, there will inevitably be a day when this (manual) heroic effort will go wrong and it will be an expensive deploy.

Why there are heroic efforts so often? Is it something that human has to do? Is the logic too complex? Or there hasn’t been time or resources to teach it to others? There isn’t a single answer and it depends on a situation. Maybe there has been a good reason to do them that way at that time. You’ve taken “heroic debt” (just like technical debt) but that will cost you more little by little if you don’t pay it back. On each heroic effort, you pay interest on heroic debt.

Start Paying Off Heroic Debt

If you keep taking heroic debt and don’t pay it off, you can’t automate your deploy. Because you don’t do deploys completely same way every time (they have manual heroic steps), you won’t have reliable deploys that does always same things. You should work to pay off heroic debt and automate these steps to make your deploys reliable.

Don’t think that you are too busy, start working today (read more from my blog post Too Busy to Improve?). Or don’t fall into common “that can’t be automated” or “doesn’t work in our organization”; try to automate them! You can start with the easiest heroic efforts step by step (though the definition of the heroic effort itself says they aren’t easy). In some issues, you might have to re-architecture your software to be able to decrease heroic efforts and automate them. The more you can pay off heroic debt, more productive you will be and your deploys will be better.

Ps. It can be also opposite, you will have to take heroic debt because you don’t have automation:

Without automation, you may start to rely on heroic efforts by individuals—so that every time you are doing a deployment, someone is having to do a figurative tightrope-walk to get everything accomplished. (Sam Fell in devops.com blog)