WhatsTheBigData

Louis Columbus at Forbes.com surveys key big data forecasts and market size estimates, including Gartner’s recent Hype Cycle for Big Data. The winning technologies in the immediate future? “Column-Store DBMS, Cloud Computing, In-Memory Database Management Systems will be the three most transformational technologies in the next five years.  Gartner goes on to predict that Complex Event Processing, Content Analytics, Context-Enriched Services, Hybrid Cloud Computing, Information Capabilities Framework and Telematics round out the technologies the research firm considers transformational.”

More on the report from Beth Schultz at AllAnalytics:

Gartner’s Hype Cycle is extremely crowded, with nearly 50 technologies represented on it. Many of them are clustered at what the firm calls the peak of inflated expectations, which it says indicates the high level of interest and experimentation in this area. As experimentation increases, many technologies will slide into the “trough of disillusionment,” as MapReduce, text analytics, and in-memory data grids have already done, the report says. This reflects the fact that, even though these technologies have been around for a while, their use as big-data technologies is a newer development.

Interestingly, Gartner says it doesn’t believe big-data will be a hyped term for too long. “Unlike other Hype Cycles, which are published year after year, we believe it is possible that within two to three years, the ability to address new sources and types, and increasing volumes of information will be ‘table stakes’ — part of the cost of entry of playing in the global economy,” the report says. “When the hype goes, so will the Hype Cycle.”

Bruce Reading, CEO of VoltDB, has an interesting and original take on the big data landscape.

Last year, Dave Feinleib published the Big Data Landscape, “to organize this rapidly growing technology sector.” One prominent data scientist told me “it’s just a bunch of logos on a slide,” but it has become a popular reference point for categorizing the different players in this bustling market. Sqrrl, a big data start-up, published recently its own version of Feinleib’s chart, its “take on the big data ecosystem.” Sqrrl’s eleven big data “buckets” are somewhat different from Feinleib’s, demonstrating a lack of agreement, understandable at this stage, on what exactly are the different segments of the big data market and what to call them. Furthermore, Sqrrl positions itself “at the intersection of four of these boxes” which raises questions about the accuracy of its positioning  of other big data companies inside just one or two boxes.

Another interesting recent attempt to make sense of the big data landscape comes from The 451’s Matt Aslett in the form of a “Database Landscape Map.” Taking its inspiration from the map of the London Underground and a content technology map from the Real Story Group, it charts the links between an ever-expanding database market and the data storing/organizing/mining technologies and tools (Hadoop, NoSQL, NewSQL…) that now form the core of the big data market.

Which brings me to Bruce Reading, VoltDB, and their take on the big data landscape. “It’s a very noisy market,” Bruce told a packed room at a recent VoltDB event. “It’s like shopping in a mall at Christmas time when there’s a lot of noise and a lot of information about a lot of technologies. We are trying to work with the marketplace to understand what you are trying to accomplish. Instead of using market maps based on technologies, we are looking at use cases.”

“Use case” is technology-speak for the list of requirements for achieving a specific goal, requirements that are embodied in the software that allows the user to achieve that goal. In other words, specialized software focused on addressing some unique need. VoltDB is focused on time (or data velocity) and believes, to quote Bruce, that “the whole world is trying to get as close to real-time as possible because that’s where the greatest value is of a single point of data.” Or, in the words of VoltDB’s website, companies are “devising new ways to identify and act on fast-moving, valuable data,” and VoltDB helps them “narrow the ‘ingestion-to-decision’ gap from minutes, or even hours, to milliseconds.” Which is why they see the “Data Value Chain” like this:

And describe the “Database Universe” like this:

This is the first attempt I’ve seen to map big data technologies based on what these technologies are trying to achieve and the type of data involved–is it unique (an individual item) or is it a part of a collection of data?–along three dimensions: Time, the value of the data, and application complexity.

The insight behind these charts is that the value of an individual piece of data goes down with time and the value of a collection of data goes up with time. Maybe this should be called “Stonebraker Law.” Mike Stonebraker is the database legend (forty years and counting) behind VoltDB and other big data startups. You can watch him, Bruce, and John Piekos, VoltDB’s  VP of Engineering, here.

[Originally published on Forbes.com]

“One of my biggest challenges,” Keith Collins told me recently, “is helping SAS understand how to communicate to IT organizations. We present workloads which look odd and different. IT does not know how to have an SLA (Service Level Agreement) around them.  We take all of the compute and I/O capacity that they can give us.”

SAS, the largest independent vendor in the business intelligence market, used to be a prime example of “shadow IT,” the purchasing of information technology tools by business users without the knowledge and approval of the central IT organization. But this is changing in the era of big data. The collection and analysis of data are becoming a very large part of many business activities and the IT organization is asked to provide support, even leadership, in tying together these disparate efforts.

Collins is SVP and CTO at SAS, where he has spent almost 30 years, helping the company grow with the market through a number of phases (and buzzwords)—statistical analysis, decision-support, data mining, knowledge and risk management, business intelligence, and business analytics.  Now SAS is helping its customers, including CIOs and their IT teams, address the challenges of big data. Collins has seen this movie before: “People are all hyped up about Hadoop.  But what is it, really? It is big and wide record sizes, big block sizes, designed specifically for high-volume, sequential processing. Just like a SAS data set in 1968… The only difference between a SAS data set and Hadoop is that now the disks are cheap enough that you can do replication.”  The following is an edited transcript of our conversation.

Gil Press:  Indeed, many people talk about Hadoop as a replacement for tape.

Keith Collins:  We love that people get that as a pattern now, because it really helps them understand SAS.  So it is a really good time for us to have the conversation with IT about it. But they are still struggling.  They see it as “what is my next big data repository?”  They do not see it as “this is my next big way to answer questions.”

Continue reading →