I’ve spoken about the slippery definition of Big Data in two recent blog posts: Big Data Today, Not So Tomorrow? and The Big Data Long Tail. Since we define Big Data in relative terms, the challenges that constitute Big Data problems will continue to shift as our tools mature.
Fair enough. But the same argument goes backward. We have always had Big Data problems, because at any point in time, some size of data set always ended up pushing the boundaries of the best tools of the day. So whether those tools were abaci, adding machines, or UNIVACs, Big Data — and Big Data problems — have always been with us.
Here’s the rub: if Big Data have been with us since the dawn of time, then why is there such a hullabaloo about them now? The answer: marketing hype. Sure, Hadoop is exciting (if you’re into that kind of thing), and the whole NoSQL movement has some impressive synergies with the Cloud, but every new technology has its day in the sun. Hell, even relational databases were new and sexy once!
But today, the hype is around Big Data, and not just the products, either. Yes, people are jumping on this bandwagon as well, centered on the mysterious Data Scientist, with all sorts of pundits and gurus hanging on for the ride. Beware. There’s really no such thing as a Big Data Expert. Because if someone really knew today’s Big Data problems and solutions well enough to be an expert, then those problems would be the Big Data of the past, not the present.
Of course, there are experts in analytics, data warehousing, and even emerging technologies like the aforementioned Hadoop and NoSQL. Such true experts are worth their weight in gold, and you’ll be lucky to hire one. But beware the snake oil salesman who bills himself (or herself) as a Big Data Expert.