Online-traffic analysis vendor Quantcast has released an open source big-data management engine. Called Quantcast File System (QFS) 1.0, the tool is designed to offer faster performance than Hadoop, which has become the de facto standard for big data processing. InformationWeek reports, “According to Quantcast, QFS outperforms Hadoop because its client software is written in C++ rather than the slower Java, and its core services are compiled in C++ rather than C and Java, as is the case for Hadoop. QFS also encodes data using the same Reed-Solomon algorithm used to compact data onto DVDs, which lays data out in nine stripes, each of which is painted on a different physical disk and could be painted on entirely separate storage racks in the cluster.”
Quantcast developed QFS for its own internal purposes. Based on Google’s Kosmos Distributed File System (KFS), QFS processes 500 billion data records per month–more than 20 petabytes per day–for Quantcast and its clients.
