3 Scenarios for the Future of Data Science

The last couple of weeks were great for the future of data science. First Wikibon, and then IDC, promised a big data market in 2015 of between $16.9 billion (IDC) to $32.1 billion (Wikibon) (more on these reports in Chuck Hollis’ Big Data: From Meme to Marketplace). And the Strata conference showcased the promising startups and data scientists that are going to make the big big data market a reality (see Daniel Tunkelang’s excellent summary here).  Tim O’Reilly aptly summarized all of this excitement by declaring that “data science is the new black.”

So where do we go from here? How will data scientists’ careers shape up over the next decade?   Continue reading

Posted in Data Science, Predictions | Leave a comment

Big Data and the Demise of Analog Retail

News today that the CEO of Best Buy has abruptly stepped down. According to the Wall Street Journal he did do apparently because of his “personal conduct.” But the recently announced $1.7 billion quartely loss is still the news that matters most to the future of Best Buy and other “Big Box” retailers. And while the cost of operating brick-and-mortar stores as opposed to selling online is what seems to most as the culprit, I would argue that missing the potential of big data is–or will be–the great undoing of traditional retailers.

The Journal article quotes Craig Johnson of retail consultancy Customer Growth Partners: “Best Buy is a very dated store experience, rooted in the 1990s, and they need someone visionary.” Question is, what exactly is dated about the “dated store experience”? Johnson provides the numbers that most commentators focus on: Best Buy’s operating income per square foot was  $18.52 last year (down from $50.61 in 2006). By contrast, “Apple’s retail stores reaped an astronomical $4,700 per square foot last year.”

Indeed, Best Buy finds itself “stuck in the middle,” to use Michael Porter’s terms, between Apple’s product differentiation (both the design of the actual products sold and the design of its stores) and Amazon’s cost leadership. But maybe Porter’s terms are also somewhat dated. Maybe we are witnessing the rise of a completely new big data “generic strategy” which leaves Best Buy and other traditional retailers “stuck outside.” They are left outside of the big data analytics mainstream, stuck on the bank of the river of data that is generated by online sales, watching their online competitors generating not only less-costly sales transactions but also data–on transactions, locations, logistics, customers, potential customers–and knowledge that is used in a virtuous circle to generate more sales and increase customer loyalty. Continue reading

Posted in Misc | Leave a comment

Domain Expertise vs. Machine Learning: The Debate Continues

“The startup’s three co-founders have backgrounds in engineering and data science, but not weather, and there are no meteorological models involved. By keeping weather predictions within a two-hour window, they believe statistics are sufficient.”–Mashable in “Can Statistics Predict Weather Without Meteorologists? This App Thinks So” on Ourcast, a  new app that uses real-time radar data and crowdsourcing to predict how weather at a given location will change within the next two hours.  Continue reading

Posted in Data Science, Machine Learning, Statistics | Leave a comment

Kirk Borne on Data Science: Start Small, Think Big

Posted in Data Science | Leave a comment

Big Data Startups News

Wikibon’s Jeff Kelly bravely put a stake in the ground recently, first among IT market observers, by estimating the big data market at $5 billion, growing to $50 billion in five years. Kelly’s 5/50/5 plan is a great guide to the initial jostling for market position in this very promising and very emerging market. it shows that most–if not all–of the innovation in big data came from startups, and some have already been acquired by established IT firms.

The big data market, as defined by Wikibon, includes the hardware, software, and services designed to address the shortcomings of traditional data base technologies in handling large data sets. This means that the $5 billion estimate is a conservative one as it represents a narrow market, the market comprised of what we could call the hardware, software, and services platforms for big data. Continue reading

Posted in Misc | Leave a comment

Management Education in the Age of Big Data

What if business schools based their entire curriculum on the fundamentals of business analytics?

McKinsey estimates that the demand for “deep analytical positions” in the U.S. will exceed supply by 140,000 to 190,000 positions and that there will be a need for 1.5 million additional ”managers and analysts who can ask the right questions and consume the results of the analysis of big data effectively.” Continue reading

Posted in Misc | Leave a comment

Big Data Bytes: “Information technology has entered a big-data era”

“From social media to medical revolutions anchored in metadata analyses, wherein astronomical feats of data crunching enable heretofore unimaginable services and businesses, we are on the cusp of unimaginable new markets.”–Mark Mills and Julio Ottino, “The Coming Tech-Led Boom,”  The Wall Street Journal, January 30, 2012

“The data fabric is the next middleware”–Todd Papaioannou of http://continuuity.com/ quoted in Derrick Harris, “5 low-profile startups that could change the face of big data,” GigaOm, January 28, 2012

“You can’t have a conversation in today’s business technology world without touching on the topic of Big Data….companies such as Yahoo, Amazon, comScore and AOL have turned to Hadoop to both scale. According to some recent research from Infineta Systems, a WAN optimisation startup, traditional data storage runs $5 per gigabyte, but storing the same data costs about 25 cents per gigabyte using Hadoop.”–Michael Friedenberg, “Why Big Data Means a Big Year for Hadoop,” techworld.com, January 29, 2012

Posted in Misc | Leave a comment

Big Data Bytes of the Week: What’s a Data Scientist?

What’s a Data Scientist? Joshua Konkle, Vice President at DCIG, quoted (scroll down) a few definitions earlier this week: Continue reading

Posted in Misc | Leave a comment

Big Data News Roundup

IBM’s Watson visited a few conferences last week. Watson’s lead developer, David Ferrucci delivered a keynote at the ACM’s 2011 Federated Computing Research Conference in San Jose, CA. Continue reading

Posted in Artificial Intelligence | Leave a comment

Hunch.com: Training the Web to be Your Friend

Would you like the Web to understand your inner GPS?

Then go to Hunch.com and start training the Web. After answering a few questions about your tastes, preferences, and opinions, you will get a set of recommendations from other Hunch members for movies, books, restaurants, recipes, music, vacation spots, shops, gadgets and other goodies.  Hunch will also predict how well you’ll like each recommendation. The more you interact with the site (e.g., answering more questions, rating Hunch’s predictions, adding a descriptive tag, making a recommendation), the more accurate and relevant its recommendations become. Continue reading

Posted in Misc | Leave a comment