In my early days of IT, I logged many an hour contending with poor application deployment practices. I often found myself working until the wee hours while chasing down some issue that needed to be fixed before the rest of the office came to work the next day. To my credit, this would happen only once per category of deployment problems because the next day I would document what went wrong, how I found the cause, and how I fixed it. Because most of my work for the first decade of my career was for very small companies, these marathon engagements with last-minute issues were common place, mostly because the budgets were so small and the time frames so short that the last minute was the only time left to deal with deployment issues.
When I began working for large companies I breathed a sigh of relief as the days of working late were gone (except sometimes by choice for extra income). Then I started working for huge companies and I suddenly found myself working late again to fix some app deployment issue before the users arrived the following morning expecting everything to be better than when they last used the application. The only difference this time around was that I was fixing issues caused by others, as the team was now larger than just me with the occasional assistant or two.
Over the years I’ve learned and developed deployment processes that have gotten me home by dinner. In this article, I share my application deployment process tips and explain how you can reap the benefits in your work. While I generally prefer to approach things from the positive, these process tips all came as a direct correction to real-life negatives. Understanding the specific issues that inspired some of these approaches should be helpful for those who have experienced the same issues.
Application Deployment Tip #1: Start Early, Start Small
“A good plan … executed now is better than a perfect plan executed next week.” — George S. Patton
One of the most common causes for failure at any point in the release process is that steps are defined (and often times attempted for the first time) far too late in the project cycle. This is especially true of build processes.
It is very common for development to continue for a very long time before there is a build process. One developer uses the IDE build, another compiles from the command line, a third uses a Maven script he developed … pointing to local paths on this machine. Strangely, it often becomes obvious that there is either no defined build process — or the process is not followed — long before it is decided how the builds should be done.
Team members run across problems with integrating their components or find that bug fixes they had done have been un-done by someone else fixing a different bug. When these warning signs crop up, it is easy to assume that the urgency of the work that caused the issue is more important than the issue itself. As a result, some individual or ad-hoc group implements a quick fix and the issue continues to grow.
Just as it is much easier for a single developer to maintain their own processes because the context is small, it is very easy to define a process before or at compile time.
Many people have very strong and differing opinions on what the build process should be. These closely-held processes often are incompatible at best (and mutually-destructive at worse). IDE builds, ANT builds, Maven builds, manual builds and manually initiated builds or automated builds at check in or at set time intervals are all valid approaches. What is important is that any given project (or, depending again on context, any enterprise) use one application deployment process and use that one process every time. I have worked with the build master approach (manual builds done by a single person with a back-up person), the continuous automation approach and many in-between. I have found that every one of them worked when everyone followed them. The trick is getting people to follow them, which I will leave to your experience because that topic is beyond the scope of this article.
Application Deployment Tip #2: Thou Shall Control Source with Source Control
Just as a good set of architecture principles should include some things that are obvious, so should the processes for deployment. I’ve never been anywhere where source control isn’t used, but I’ve been on many projects where it was not introduced until it was difficult to do so. It doesn’t matter if the source control application is commercial or open source, what matters is having source control in place and using it regularly. (You’ll find an in-depth article about good source control habits at Developer.com. While the article is focused on specifics to support Eclipse, it applies to all development environments.)
Application Deployment Tip #3: Work as a Team, Even Individually
“Red light stop, green light go, yellow light go very fast” — Jeff Bridges as Starman
Some folks out there still code with pure text editors. If they are still employed, it is generally because they are very good and extremely diligent in their habits. Since building is outside of coding, they’ll adapt whatever build process you ask them to, and make great build masters if you go that route (which I personally recommend only for small teams). The vast majority of us, though, use an IDE. Even if you know your APIs very well and only like the syntax coloring, an IDE is a tool for speed. But speed can be a double-edged sword: you can build components faster, but also propagate issues faster.
If a warning appears at save or compile time, most developers I know (but have not worked with much) will ignore it. That is like ignoring the yellow light at the intersection. If you are paying attention to everything else, you’ll survive. But if your attention wanders at the wrong moment, your day (and probably evening, too) is going to be totaled.
IDE warnings alert you that something in the code could be problematic. Sometimes this is because the IDE does not understand perfectly valid alternative approaches. In such cases, most IDEs these days can be set to list only issues at a certain level of severity and configured to ignore certain issues. Most of these customizations can be stored and distributed throughout the team so that everyone can handle them consistently.
If you find that the IDE you use is not entirely adequate for your code standards, look into the several commercial and open source offerings. Not all integrate with the IDE (which is ideal, the idea being the earlier an issue is detected the less impact it has on the application, project, and/or others), but the key to any tool’s productivity is ensuring that it is used consistently.
Application Deployment Tip #4: If it Doesn’t Work, it Wasn’t Tested
“The operation was a success, but the patient died.” — Unknown
Maybe it is just me who confuses unit testing with functional testing. I see functional testing as running the entire application, moving from one task to another. When I say unit testing, I refer to testing a process from start to finish and including as many variations as reasonably possible. Sometimes that process goes beyond the boundaries of the code I am writing. In those cases, I follow it through with either the other components involved in the process or surrogate methods for those components. The surrogate methods mimic as much as possible the functionality of the final components (since a surrogate should be used only when the real thing does not yet exist or is not yet accessible). Many people think of unit testing as testing a single method with a single value, while some consider unit testing as successful compilation. I have even worked with a few who consider their work done if their IDE doesn’t raises any errors before compilation.
For the purpose of getting home at a reasonable hour (or at least being able to leave without keeping others late), let’s use my definition. This means that every piece of code needs to be tested as thoroughly as practical before checking it in to source control. Failing to do so will result in the code breaking the system at some point — worst of all, just about the time it becomes critical, when it will be very hard to find.
Application Deployment Tip #6: Perform Dress Rehearsals
“No battle plan survives contact with the enemy.” — Colin Powell
The best learned lessons are those hardest won, and those that are won twice are learned perfectly or never at all. I have written out perfect, step-by-step instructions for deployments that went off without a hitch. I didn’t have to fully validate those plans except in my own mind once or twice. Unfortunately, one of those instances occurred early in my career and it took a few dinners (followed by late night snacks) in front of the monitor to figure out what had gone wrong. I finally decided that for the rest of my career I was going to walk through my plan from start to finish before the go-live — even if I was 100% confident in it.
The dress-rehearsal is great because you do all of the things you will do for the “real” release only without any of the stress of users and bosses and bonuses waiting for you to finish. And if you find that you missed a step in your plans or a step is not detailed enough, it is much easier to address that earlier than late at night.
Application Deployment Tip #7: Context-Specific Is OK
During those earlier years I developed habits that got me home on time. When I found that simply having those habits myself was no longer enough, I worked on codifying them into documented processes. Since the habits themselves had formed organically rather than through a conscious process, the documented processes tended to be very context-specific. While I have covered the broad strokes of what is applicable to most (if not all) application projects here, I strongly recommend enhancing and elaborating the application deployment process to specific applications, projects, enterprises or teams (as best fits your delivery culture).
In addition to all the reasons already stated, there is one other reason to have a plan for deployment and follow it: if you have to work late, tired hands and a frustrated mind are much more likely to create more mistakes, and sometimes bigger ones, too. All of the lessons I referenced are real and personal. They kept me away from friends, fun and family before I discovered the application deployment processes to control how late I had to work to deliver what users expected and deserved — so that they could go home on time, too.