It was about a year or so back—I was in the middle of figuring out how to pass an authorization token between trust boundaries when I got a call from our CIO. He had just come back from some conference sponsored by *** (Vendor's name withheld to protect the clueless) and was brimming with new acronyms.

"Udi", he says, "I just heard that for us to realize the potential of our SOA, we should be using an ESB."

I'm sure he said a lot more than that, but that first sentence was enough for me to tune out. I managed to get through the conversation without catching a case of acronym-itis, but my train of thought was broken. I wasted one Google search to find out that ESB meant "Enterprise Service Bus," got fed up, and went to Starbucks. Of all the overused terms in software, "Enterprise" is by far the most annoying—although "Service" seems to be moving up in the world.

I've been developing loosely-coupled systems for awhile now, using all kinds of technologies, and never thought that I was doing anything particularly ground-breaking. So when the CIO came down and introduced me to the army of high-priced consultants that were going to help us redesign our software to become more "service-oriented" I really was interested in seeing what would be different. Soon after, I realized that the vendor-driven SOA meant nothing more than Web Services, XML, and loose coupling—with the mindset of loose coupling being the most important. I'd been "service-oriented" all this time and didn't even know it. Oh, and it turns out that we didn't have to redesign anything.

Imagine my utter joy in hearing that the reason our project was in trouble was that we didn't have an ESB. And all this time I thought it was because the requirements were changing every two weeks.

It was about a month after that that our project manager got promoted for doing such a great SOA implementation and was now in charge of making the whole company, oops, sorry—enterprise, service oriented. I was made "acting project manager" and managed to do one really smart thing pretty quick—get our software through testing and deployed within the month, before too many new requests came in (I think it was possible because most of the stakeholders were on holiday). The system wasn't that complex; we had the standard HR, accounting, inventory, etc., functionality split up into the same high-level components. Each top-level component had its own server-cluster and database. We pulled the data from each of the database to the data warehouse using classic ETL. Data flowed between the components primarily using publish-subscribe semantics while the client-side software just request-responded what it needed from each component.

Trouble in Paradise
We kept on working, pushing out features as fast as we could, until one morning I found the sysadmin at my door.

"We had a problem with some of the disks on the accounting cluster, so we installed it on a different cluster and brought the first one down. We tried a simple test to make sure everything was OK, and it wasn't. A bunch of things don't seem to be working."

After looking around for a bit I found out that the sysadmins had forgotten to update the config files on the servers and the clients. We restarted the server components and they worked fine, but we couldn't really go around restarting and changing config files on all the clients. Luckily, the sysadmins had it set up so that every client on our domain that logged in to the network could be sent a script to run, so pushing out the new config files was easy. As for the restarting part, we called up the help desk and told them that if (when) someone called about why their software wasn't working properly, just to tell them to close it and run it again—which apparently is their standard first suggestion anyway.

Bigger Trouble in Paradise
Well, things lurched along like that for a while as we put out more and more functionality—put up a Web front-end, tied in some business partners, etc. I thought I had everything under control until our COO charged in, with the CIO in tow.

"Our business partners haven't been able to send us orders for almost a week," he fumed. "What did you do?!"

When money talks, you'd better believe everybody listens. After some seriously hectic hours of peering through diffs between the deployed source and the previously deployed source, we were getting nowhere. Somebody, I don't remember who, had the common sense to get the sysadmins in there too. It turned out to have been the same problem as before, but this time with the inventory cluster. So we used the same solution. The problem was that our business partners' software didn't get the updated config files. While I was pondering how we could get the same system to work with external partners, my boss called me in for an urgent meeting.

"I just got a call from Jim (the CIO) and he wants you and me to help him explain what happened to the CEO."

I started to get that sinking feeling, the kind where you know things are going from bad to worse. That afternoon, we all filed in to the chief's office, bracing ourselves for the worst. He got directly to the point.

"If anything like this happens again, you three are fired. Now get the hell out of my office."

How's that for motivation?

And to top it all off, before he hurried off to another meeting, Jim asks us "You guys know about our first audit for Sarbanes-Oxley in three months, right? I don't want to see any more screw-ups, and this SOX stuff is getting people anxious. We need full audit trails on everything."

It looks like Moore's law will continue indefinitely: You will need to handle twice as much crap today as you did 18 months ago.

It's All in the Message
It was at about this point where I realized that I needed help. I called up one of my old partners in crime, Clem, who had been doing large-scale distributed systems development for awhile. I told him my sorry tale and asked if he could give me a hand. Unfortunately, he was in the middle of some serious crunch time, but he left me with these pearls of wisdom:

"Udi, it's all in the message. Forget about remote method invocations and pub-subbing events—down on the wire it's all just messages. The trick is to think of your system as passing messages at the application level as well.

Asynchronous message passing over queues. It's really quite simple.

Once you've packaged everything into the message, that message can be dynamically routed anywhere, and so can its responses. The application doesn't need to bind against any specific endpoint—it just drops a message addressed to some logical location. Infrastructure can make sure that messages get to the logical recipient, even if they change physical locations.

That infrastructure is what brings about the "Bus" architectural style between your distributed components."

Luckily I was writing down what he said, because I had to re-read it at least a dozen times for it to sink in. Flashback to that original conversation with the CIO—so that's what ESBs are for! Well, you wouldn't have guessed it with all the hype going on—IT/Business Alignment, like that's going to happen any time soon.

After talking with some ESB vendors, I understood some nuances in what Clem told me. The message passing at the application level is really passing logical messages—a message is an object just like any other. The transformation that logical message undergoes in order to be sent across the wire is something else entirely. We can transform our message to and from XML, binary, text based key-value pairs—anything we need. Finally, the transport used to pass that wire-representation between machines is an infrastructure detail that is also independent of the logical message.

Once my mind warped itself around asynchronous messaging, the whole SOA thing became clear. The top-level components we were developing were providing top-level services—requests would queue up like people would at the teller in a bank. A component could send out the exact same message either as a broadcast or a unicast, where the recipients would be able to use the same semantics either way. Exposing a method, subscribing to an event, and handling a message were all the same, both internally and externally.

I can't explain how much this simplified my view of the distributed world. It kind of felt like dominos—as one thing fell into place, it knocked down something else. I was finally beginning to understand what needed to be changed in our system—and all it took was a multi-million dollar mistake and nearly getting fired.

Needless to say, the whole SOX thing caused all hell to break loose. Our team wasn't compliant, but then neither was any other team. The same goes for most of the company's software. But, the reassuring thing for me, was knowing where I was going with our system. It took some time—we redesigned most of the communication paths, found a vendor whose product met our needs (at the right price), and several months later, we rolled out the new version. I wouldn't say that the rollout was flawless, but I will tell you this—when the sysadmins moved a service from one cluster to another, no config files needed to be pushed out in order for things to work, and, more importantly, no orders were lost. I even got promoted from "Acting Project Manager" to "Project Manager."