Chad Perrin: SOB

31 May 2007

10 tips for developing deployment procedures (or: Deployment Is Development)

Filed under: Geek,Popular — apotheon @ 07:56

People* have a tendency to think of deployment as something the admins do — if developers put any thought into it at all, it’s usually an afterthought, and not one that is considered very important. That is, of course, a huge mistake: deployment is a development concern, too.

  1. Have at least two (types of) boxes on which you test deployments at every stage of development. One should be an example of a standard machine in the production environment, and the other should be whatever close approximation you have as part of the development environment. That means that during development of a desktop application, for instance, you should at every major development stage test it on both an end user desktop system and on your own development workstation (or close equivalent). Make sure the thing deploys under wide-ranging circumstances, and ensure that if it doesn’t deploy properly during testing it hurts enough to make you consider deployment matters very carefully. For independent consultants, this may mean requesting a test box from the admins at whatever organization is paying you for development.
  2. Make sure you actually test deployment at every stage. It is becoming ever more fashionable to develop in stages so that at every interim goal you have something that actually runs when it comes time for testing and project specification assessment. When those interim goals are met, make sure you deploy and test as though it were the final deployment to the production environment, and make sure you don’t skip either the production environment test box or your own box in the deployment test (though of course you might want to try the test box first in case it blows up). Don’t consider the milestone reached until it deploys smoothly to both.
  3. Make sure deployment/installation is a largely automated process. Use scripts where necessary, employ the system’s software management applications for deployment where at all possible to integrate it with the environment and reduce confusion, and ensure no steps of deployment are missed. Have a complete, functional, automated installation process working and tested before you declare your interim goal met.
  4. Use automated testing scripts to verify that everything is where it needs to be after a test deployment, and make sure that it’s an automated, idiot-proof verifier so you can incorporate it in the final product. Ensure that, at minimum, it logs any discrepancies it finds so that you have a record of what went wrong during deployment, providing you with a neat and orderly to-do list of things to fix. Make sure that the end-user or sysadmin that will be doing deployment can use the verification system without your help, since somewhere down the line someone you’ve never met will be doing exactly that — guaranteed, assuming your software is good enough that it’s used more than once. Your verification should check locations, version numbers, and file permissions. Integrate your testing scripts into the automated deployment process. Actually use it yourself for test deployments, even on your own machine, and design it so that’s possible without a bunch of special-case handling.
  5. Make sure real human beings test the software after every test deployment to make sure it operates as expected at this stage of development.
  6. Keep necessary configuration to a bare minimum. Best of all would be to have no necessary configuration options at all, with the possible exception of asking whether custom configuration options are desired. Provide opportunity for custom configuration, though, in case something nonstandard is desired.
  7. Provide easy rollback in case of a failure. Test the rollback every time. You should keep track of original system state, and include an automated “undeployment” process for returning the system to that state. This should, just like all other operations, work on both the test machine and your own machine, without any special case handling necessary. Provide opportunities for custom configuration of the rollback procedure, and for some sysadmin you’ve never met before to use only parts of the automated process in combination with manual changes if necessary, but make sure that no user interference is actually required for the rollback procedure.
  8. Follow these guidelines every time. You may think a given project is not big and important enough to bother, but if it’s small and unimportant that just makes it easier to follow these guidelines. I’m not perfect — I don’t always follow these guidelines myself. I should, though, and I intend to do so a lot more in the future. Practice makes perfect, and you definitely want things to be as close to perfect as possible — right?
  9. There’s no excuse for deployment failure. If deployment fails, it means you forgot to account for something. If someone tries (and fails) to deploy it on the wrong type of system, the problem is that you did not account for that type of system — not that the choice of system on which to deploy it was wrong. If some other piece of software changed things so that your deployment didn’t work, the problem is that you did not account for the potentially varied circumstances in deployment. You may not want to tell them that it’s your fault, but you should think of it as your fault anyway: even if it’s because the guy deploying it is an idiot, the problem is that you didn’t make deployment idiot-proof enough.
  10. Extra Credit: Voluntarily test-deploy other people’s software on your system all the time, and help them solve their deployment issues as much as possible — but without actually doing development for them unless it’s specifically appropriate for you to do so. Being on the other end of the developer/deployer divide can help you hone your sense for how to develop high quality deployment procedures for your software. The more you know about what your software is going to be doing, and the conditions under which it’s running — and the more you know about the problems of others’ software deployments — the better you’ll be at developing quality deployment procedures for your own software.

That all adds up to being a better developer with happier clients. While you may consider deployment an “admin problem”, I guarantee most admins and end users think of software that’s difficult to deploy as a “developer problem”, and whether you think that’s fair or not, it affects your success and reputation as a developer. After all, if it doesn’t deploy, it doesn’t work.

* We’re assuming developers qualify as “people”, here.

(Credit where it’s due: This was in part inspired by Justin James’ Time to improve application deployment and a comment to it by Wayne M. titled Deployment Philosophy.)


  1. Wow, this was a bad time for me to read this. (Not your fault, obviously!)

    Actually I’m pleased to note that we follow many (but not all) of these tips in our own deployment schemes. And the rules are slightly different when you’re deploying a Web application that you host, since you don’t have to worry about multiple platforms. You’d think that having a homogeneous enterprise (Windows Server 2003/SQL Server 2005/ASP.NET 2.0) would greatly simplify deployment.

    Except there are enough significant differences between Windows Server and Windows XP Pro (which our development computers run) to make local deployment and testing challenging, to say the least. Our application is three-tiered, so it’s difficult for me to run anything other than just the Web interface locally; if I make a change to the app or database layers I have to test it elsewhere. So we probably have more test and staging and sandbox environments than you’d normally see at a shop our size.

    Which brings me to my last point: never underestimate the value of a good SCM. (Which is not to say that you have here; I realize you’re writing from a developer’s perspective.) Good SCMs are developers – even if they’re not software engineers, they still have to be expert script programmers. They have to be familiar not only with the deployment environments, but with the architecture of your application. We recently lost our full-time SCM and his absence is keenly felt: currently one of the QA engineers is pulling double duty as SCM, and I know it’s been a struggle for him to stay on top of multiple environments and learn the tools and scripts used to do the build. Plus, you know, it’s all Microsoft, all the time. TFS doesn’t actually stand for Totally F*cked Software, but it could.

    Comment by Brian Martinez — 31 May 2007 @ 11:00

  2. And the rules are slightly different when you’re deploying a Web application that you host, since you don’t have to worry about multiple platforms.

    Actually, you might have to worry about it eventually — especially working with MS Windows as the deployment platform, since there’s no such thing as a smooth upgrade path when you start needing the capabilities (or at least patch support) of a newer system. One of the reasons to keep an assumption in mind about potential deployment to unexpected system configurations is that system upgrades, replacements, or expansions in the future might suddenly crop up, making you the unfamiliar deployment environment.

    . . . but in the short term, that can certainly seem like a fortuitous shortcut to take.

    I’m curious about your issues with TFS, if only because I don’t have any experience with it at all and I like to, err, know stuff. Do you have some specifics worth sharing?

    Comment by apotheon — 1 June 2007 @ 12:20

  3. […] Finally, the last thing on the first page of the programming reddit when I looked was 10 tips for developing deployment procedures (or: Deployment Is Development). Yes, I wrote it. In fact, it’s the SOB post just before this one. I wasn’t even aware it was on reddit until I saw it there tonight. I’ve also noticed, in the process of following links around, that both OOP and the death of modularity and The One Important Factor of programming languages are still getting some action, through links various sources. Include Elegance (which I’m almost embarrassed to admit has been called a “seminal work”), and I might have the beginning of a “Joel On Software” or “Hackers And Painters” style of book in my future. Then again, maybe I’m just feeling too full of myself. […]

    Pingback by Chad Perrin: SOB » the goings-on in coderspace — 2 June 2007 @ 01:06

  4. […] I’ve already talked about ensuring you achieve good results with software deployment by ceasing to consider deployment procedures as separate from development. Since packaging/installation, aka “deployment procedure”, actually involves thinking about the structure of the software and writing code, it’s an easy thing to incorporate deployment development in your application development process. The two are a natural fit, which might help people to realize that they are, in fact, one — not two — after all. The deployment procedure for a piece of software is part of its interface. Learn that simple truth, and you should be able to put it all together. Voila, you’re set. […]

    Pingback by Chad Perrin: SOB » The DRY Principle and Documentation — 11 June 2007 @ 04:33

RSS feed for comments on this post.

Sorry, the comment form is closed at this time.

All original content Copyright Chad Perrin: Distributed under the terms of the Open Works License