At a conference in New York City, Todd Papaioannou, former chief cloud architect for Yahoo, talked about his experiences setting up 45,000 Hadoop servers in Yahoo’s 400,000 node private cloud. His group ended up creating “a bunch of redneck architecture” cobbled together from bits and pieces of various libraries, and the result wasn’t very stable, explained Papaioannou.
“Hadoop is hard – let’s make no bones about it,” he said. “It’s damn hard to use. It’s low-level infrastructure software, and most people out there are not used to using low-level infrastructure software.”
To make Hadoop easier to use, Papaioannou has founded a new company called Continuity, which creates an abstraction layer on top of Hadoop. He says Continuity’s AppFabric will help other companies to avoid the application architecture difficulties they experienced at Yahoo.
