Whats the Big Data

Chuck Hollis calls Data Scientists “rock stars” and argues that they are “a fundamentally different profession with a different profile than the BI analysts that came before [them].  They’re more likely to have advanced degrees, frequently have a background in the sciences (vs. business) and they interact with data in more ways — and using different tools.”

Over at Vator they call them “rocket scientists” and “data junkies.” And an article in the November/December issue of the IEEE Intelligent Systems explores a “what if” scenario in which data scientists are criminals. No quotation marks.

Asking good questions as the key to unleashing the potential of big data got significant blog time this past week. Continue reading →

The Wikipedia article on Big Data says it “requires exceptional technologies to efficiently process large quantities of data within tolerable elapsed times.” The examples given (Hadoop, MapReduce, Cloud Computing, etc.) do not include one very exceptional technology, the human brain, and a new way to harness its power, “crowdsourcing.” In the 2006 Wired article in which he coined the term, Jeff Howe wrote: “Just as distributed computing projects like UC Berkeley’s SETI@home have tapped the unused processing power of millions of individual computers, so distributed labor networks are using the Internet to exploit the spare processing power of millions of human brains.” Isn’t crowdsourcing one of the “exceptional technologies” required by Big Data?

To find out more about crowdsourcing and its role in the service of Big Data,  I attended yesterday a Crowdsortium Meetup. Karim Lakhani from the Harvard Business School opened with a brief keynote, reminding us of (Bill) Joy’s Law: “No matter where you are, most smart people work for someone else.” Following him was a panel with the aforementioned Howe, Dwayne Spradlin (CEO of Innocentive), Doron Reuveni (CEO of uTest), Dan Sullivan (CEO of Appswell), moderated expertly by Jim Savage, partner and co-founder of Longworth Venture Partners. Continue reading →

“The United States alone faces a shortage of 140,000 to 190,000 people with analytical expertise and 1.5 million managers and analysts with the skills to understand and make decisions based on the analysis of big data.” –McKinsey Continue reading →