Learning from Yahoo’s Hadoop Mistakes

Learning from Yahoo’s Hadoop Mistakes

At a conference in New York City, Todd Papaioannou, former chief cloud architect for Yahoo, talked about his experiences setting up 45,000 Hadoop servers in Yahoo’s 400,000 node private cloud. His group ended up creating “a bunch of redneck architecture” cobbled together from bits and pieces of various libraries, and the result wasn’t very stable, explained Papaioannou.

“Hadoop is hard – let’s make no bones about it,” he said. “It’s damn hard to use. It’s low-level infrastructure software, and most people out there are not used to using low-level infrastructure software.”

To make Hadoop easier to use, Papaioannou has founded a new company called Continuity, which creates an abstraction layer on top of Hadoop. He says Continuity’s AppFabric will help other companies to avoid the application architecture difficulties they experienced at Yahoo.

View article


About Our Editorial Process

At DevX, we’re dedicated to tech entrepreneurship. Our team closely follows industry shifts, new products, AI breakthroughs, technology trends, and funding announcements. Articles undergo thorough editing to ensure accuracy and clarity, reflecting DevX’s style and supporting entrepreneurs in the tech sphere.

See our full editorial policy.

About Our Journalist