s is often the case in title fights, it didn’t really get interesting until the last round.
On an otherwise uninspiring Monday night, in the grand ballroom of a Redwood City, CA hotel, I was listening, against my will, to every bad song that ever made it into a “Rocky” movie soundtrack. The event was a boxing-themed “Smackdown” between J2EE and Microsoft .NET, organized and presented with admirable amiability and nonpartisanship by the Software Development Forum. It was ostensibly intended to focus on Web services but only isolated portions of the 2+ hour match were Web services-specific.
Three team members each from Sun Microsystems and Microsoft were sent to opposite “corners.” During the event a series of questions would be read. One member from each team would have five minutes to address each question. Each question and its two five-minute answers composed a round. Though it was a little heavy-handed, the boxing metaphor was actually sort of cute and helped take the sharp edges off of deep-seated animosities in the room. Unfortunately, they really could have saved us all some time by getting right to the last question, which, of course, was about Pet Shop.
I was eager to hear both sides debate the highly controversial results of Microsoft’s “benchmark” of .NET based on the Java Pet Storeapplication (a rather infamous quasi-standard traditionally used for testing and evangelizing best practices for multi-tier Java applications), but in the end I was left with only with a strong feeling of futility and deja vu.
Microsoft’s argument: In the absence of any industry-standard benchmark that can be used to do a fair side-by-side comparison of Java and .NET applications, Java Pet Store seemed the best chance for Microsoft to perform a comparative test using an industry-recognized standard. After all, as Dino Chiesa, Strategist for .NET Developer Solutions Group pointed out, Oracle recently used Pet Store as a de facto benchmark and Sun itself calls Java Pet Store a “blueprint application.” While the results have been controversial, the data, Chiesa contended, shows a 10x performance advantage for .NET over J2EE and dramatic cost/performance advantages for .NET. In short, Chiesa inferred, the data speaks for itself: .NET is faster and provides a better return on investment.
Sun’s argument: There are so many flaws in the way the .NET version of Pet Store was handled that the results are devoid of credibility. First, Sun argues, Microsoft funded the report by sponsoring the costs of The Middleware Company, which performed the testing. “Funny thing about benchmarks that you fund,” said Tom Daly, Staff Engineer for Sun, “they tend to come out in your favor.”
Among the other problems with Microsoft’s benchmark, according to Daly: the Java implementations were based on JDK 1.3 instead of the faster version 1.4, no run rules, no peer reviews, and very little disclosure. Sun emphasized that the Java Pet Store is not a benchmark, and Daly said emphatically the “lines of code comparison is just plain wrong.” The .NET code, he said, was optimized in several ways that stacked the results.
My roll up is that Microsoft (and The Middleware Company) made some errors (of arguable severity) in judgment and whether those errors are material or superficial does not change the fact that the credibility of this .NET Pet Shop benchmark is damaged beyond repair.
Rematch!
At the “Smackdown” many attendees and at least one member of the panel of judges volubly advocated a rematch. In other words, an independent, mutually agreeable benchmark carried out by one or more third parties?one that, win or lose, the vendors agree to ratify as fair and impartial.
I think almost everyone likes that idea, including me. The unfortunate reality is that the cost and logistics of performing such a benchmark makes it highly unlikely. Even if both companies were willing to split the considerable cost of doing so (and only Microsoft would likely want to?after all, unlike with Java, Microsoft is virtually the sole beneficiary of .NET’s financial success), the chances that they could ever agree upon a testbed are a million to one.
But in the end, it doesn’t matter. Customers and implementers get no direct benefit from a successfully implemented benchmark. If we could all sit down and design the perfect, fair, infallible benchmark?invulnerable to any accusation of bias?what would happen? Well, one platform or the other would be negligibly faster than the other one. Not incredibly faster, not orders of magnitude faster?a little faster, and even then only in that specific configuration. Give the product to a customer and let them optimize it for their specific environment and you will see a negligible performance gap become a significant performance advantage.
What is it that we think we might get out of a benchmark? Both platforms are highly competent and high performing. The great news for the business world is that you can’t really mess up this decision too badly. If one platform makes more sense for you, based on other business issues, you already know what your choice is. If you’re one of the relatively few organizations that’s on the fence, well your decision boils down to, not the choice of two evils, but the choice of two solutions. A benchmark would certainly be interesting to someone who’s making “the big decision” but in the end it’s what youdo with it that matters, not what they do with it in some lab. Finally, Web services interoperability ensures that no matter which decision you make now, with a little planning you’ll still be able to use the fruits of that decision in the future, even if your company makes dramatic technological changes down the line.
Will the results of a benchmark matter to those who have already staked their flag in one turf or another? If their platform loses will they run screaming to their ISVs, demanding an explanation? Will they frantically rip applications out by their figurative roots? Nope. It is the very rare enterprise that makes a technology investment based on transactions per second alone. And a very myopic IT manager who’d second-guess his choice based on a single benchmark.
There are only two groups of people that have a vested interest in a completed benchmark: 1) Employees of Microsoft, Sun, and other major vendors who have staked their businesses on Java (Oracle, BEA, IBM), and 2) each platform’s fanatically partisan proponents. As partisanship increases among the mainstream proponents of each platform, it becomes less likely that even the fiction of an infallible benchmark would have any notable effect. At worst, the response will be: We’ll get ’em next time.
So, in the end, the Pet Shop controversy boils down to a lot of childish arguing about who’s got the bigger bat. Oops, I mean ball. No, I mean … oh, never mind.
Same Old Song and Dance
I might feel a little differently if it weren’t for the fact that every “round” in SDForum’s Smackdown prior to the PetShop round was really just the same old philosophical disagreements that these two vendors (and their respective communities) have been arguing over for months and sometimes years.
A question on security prompted Sun’s team to (yawn) take aim at IIS. They even referenced a report from Gartner that suggested people move to ChiliSoft ASP instead of IIS. It’s an argument that we took issue witha long time ago.
Sun criticized Microsoft both for taking too long to become interested in security and then for making rapid changes to the security model that required customers to fix existing implementations. Fix it or don’t, Microsoft: Either way you lose.
Another question had the two sides arguing about the importance of APIs and application portability, comments that ultimately had the crowd booing Microsoft’s Chisea. “We think much more valuable than API standards is protocol standards,” said Chisea, “protocol standards that enable interoperability.” Chisea downplayed the importance of portability saying that once deployed, applications rarely get moved.
It’s interesting that protocol interoperability inspired boos from an audience that had come to hear a debate on Web services, aka protocol interoperability. Web services and SOAP interoperability make the platform and language used to create those services completely immaterial. On the other hand, perhaps the reaction was incited only by the idea of moving away from APIs.
A few minutes later, Microsoft’s David Weller criticized the Java method and its reliance on multifarious APIs for creating “classpath hell.” Weller cited code efficiency and built in source code control as clear advantages of .NET.
Sun lauded the Java Community Process?650 members strong?and noted proudly that 55 percent of the current JSRs aren’t led by Sun. It criticized Microsoft’s relative inattention to open standards. “The CLI and C# may be in ECMA,” said Bill Day, “but there’s a long list of things that Microsoft controls.”
Microsoft put a big checkmark next to cost in its column, saying that .NET could be implemented at 10 percent to 20 percent of the cost of Java. “Windows isthe application server,” said Weller.
To rebut, Sun’s Daly cast shadows on .NET’s maturity and stability as well as Microsoft’s business model. “Risk is a big cost,” he said. He asked the crowd to consider how Microsoft continues to be so successful if its costs are so low. “What are the hidden costs of [using] .NET?” Daly asked.
What I was seeing in this boxing ring wasn’t a bout that will result in the emergence of a single champion, but rather the sad spectacle of two aging champions who haven’t yet realized that their fleeting moments in the spotlight are almost over, whose importance will be eclipsed by the younger blood of XML Web services even as they pound each other over old grievances.
Crowd Gives Sun a Hollow Victory
Had a winner been declared, I think Sun would have won by decision. They were more aggressive, more direct, and had the better portion of the crowd on their side.
But none of the arguments presented by either side are new to developers. Each platform has its philosophy. Microsoft’s that interoperability and developer productivity are the worthwhile rewards of tight integration and platform optimization. Java’s that choice, flexibility, and portability should not be relinquished for any other goal. Java’s that better software requires community agreement. Microsoft’s that better software is better software, regardless of how it’s invented. Java’s that your code should run on every platform. Microsoft’s that every language should run on your platform.
Neither philosophy is inherently wrong. They only become right and wrong if you decide that one of those tenets is essential to you. And once you’ve done that, a little thing like a Pet Shop benchmark isn’t going to make one bit of difference.
