Two new research reports on big data flash out its early impact on enterprise IT. Continue reading
Machines vs. Models, Noise vs. Signal
An excerpt from Nassim Taleb’s forthcoming book, Antifragile, was posted yesterday on the Farnam Street blog. In “Noise and Signal,” Taleb says that “In business and economic decision-making, data causes severe side effects —data is now plentiful thanks to connectivity; and the share of spuriousness in the data increases as one gets more immersed into it. A not well discussed property of data: it is toxic in large quantities—even in moderate quantities…. the best way… to mitigate interventionism is to ration the supply of information, as naturalistically as possible. This is hard to accept in the age of the internet. It has been very hard for me to explain that the more data you get, the less you know what’s going on, and the more iatrogenics you will cause.” Continue reading
IIA, Forrester, IDC, and Gartner on the Future of Big Data Analytics and Cognitive Computing
Big data analytics is the next trillion-dollar market, says Michael Dell. IDC has a more modest and specific prediction, forecasting the market for big data technology and services to grow at a 23.1% compound annual growth rate, reaching $48.6 billion in 2019.
The larger market for business analytics software and business intelligence solutions which now includes the new disciplines of data science and cognitive computing (e.g., IBM Watson) is at least 5 times bigger. But a much larger market, which may indeed approach a trillion dollar sometime in the not-distance future, includes the revenues companies in any industry will generate from “monetizing” their data and algorithms.
Here’s my summary of predictions for big data analytics and cognitive computing from the International Institute for Analytics (IIA), Forrester, IDC, and Gartner.
Big data analytics will be embedded everywhere
IIA predicts that computing will become increasingly microservice-enabled, where everything – including analytics – will be connected via an API. IDC predicts that by 2020, 50% of all business analytics software will include prescriptive analytics built on cognitive computing functionality and that Cognitive Services will be embedded in new apps. Embedded data analytics will provide U.S. enterprises $60+ billion in annual savings by 2020.
Goodbye data preparation, hello data science
IIA predicts that automated data curation and management will free up analysts and data scientists to do more of the work they want to do. Forrester says that in 2016, machine learning will begin to replace manual data wrangling and data governance dirty work, and vendors will market these solutions as a way to make data ingestion, preparation, and discovery quicker. Through 2020, according to IDC, spending on self-service visual discovery and data preparation tools will grow 2.5x faster than traditional IT-controlled tools for similar functionality.
The meager supply of people with the right data analysis skills will continue to baffle experts
Automated data preparation will help address the limited supply of analysts and data scientists. However, opinions differ regarding when supply will start meeting demand. The talent crunch, says IIA, will ease as many new university programs come online and it will stop being a challenge for large corporations—they will find ways to address their requirements for number-crunching, model-spewing staff.
No, says IDC, the shortage of skilled staff will persist and extend from data scientists to architects and experts in data management. As a result, the market for big data professional services will expand rapidly, with a CAGR of 23% through 2020. Forrester agrees that the “huge demand” will not be met in the short term, “even as more degree programs launch globally.” In 2016, Forrester predicts, firms will turn to insights-as-a-service providers and data science- as-a-service firms and to labor-savings options such as algorithm markets and self-service advanced analytics tools.
There’s risk in them thar data hills
Gartner predicts that due to the volume and variety of data and the sophistication of advanced analytics capabilities, the risks associated with big data analytics projects will continue to be larger than those associated with typical IT projects. In addition, by 2018, 50% of business ethics violations will occur through improper use of big data analytics, according to Gartner. Forrester highlights some of the risks associated with the ever-changing big data vendor hype, predicting that half of all “big data lake” investments will stagnate or be redirected. Forrester also warns that immature data science teams will improperly exploit algorithm markets, and spend precious time either developing an algorithm they could have bought or trying to apply an algorithm incorrectly.
We will have a new buzzword
Cognitive technology will become the follow-on to automated analytics, predicts IIA. For many enterprises, the association between cognitive computing and analytics will solidify in much the same way that businesses now see similarities between analytics and big data. IIA adds to the mix yet another term, predicting also that data science and predictive/prescriptive analytics will become one and the same.
How about going back to “data mining”?
Data monetization will take off
By 2020, IDC predicts, data monetization efforts will result in enterprises increasing the marketplace’s consumption of their own data by 100-fold or more. Also by 2020, the amount of data that is worth analyzing will double. Forrester predicts that as firms will try to sell their data, “many will sputter.” In 2016, an increasing number of firms will look to drive value and revenue from their “data exhaust.” Only 10% of enterprises took their data to market in 2014, but 30% reported data commercialization efforts in 2015, a 200% increase.
Forrester declares that “all companies are in the data business now.” IDC predicts that by 2020, organizations able to analyze all relevant data and deliver actionable information will achieve an extra $430 billion in productivity benefits over their less analytically oriented peers. A similar figure for revenues associated with data monetization will get us closer to Michael Dell’s trillion-dollar prediction. In the same interview, Dell described the current state of data mining/predictive analytics/data science/prescriptive analytics/cognitive computing: “If you look at companies today, most of them are not very good at using the data they have to make better decisions in real time.”
Sources
IIA
2016 analytics priorities and predictions webinar
2016 analytics priorities and predictions research brief
Forrester
Predictions 2016: The Path From Data To Action For Marketers
IDC
IDC On-Demand Webcasts: Worldwide Big Data and Analytics 2016 Predictions
Gartner
Gartner Says Customer Data Has Monetary Value but Many Organizations Ignore It
Originally posted on Forbes.com
McKinsey Updates Estimates of Big Data Potential Value
Source: McKinsey, 2011
[In 2011], we estimated the potential for big data and analytics to create value in five specific domains. Revisiting them today shows uneven progress and a great deal of that value still on the table (exhibit). The greatest advances have occurred in location-based services and in US retail, both areas with competitors that are digital natives. In contrast, manufacturing, the EU public sector, and healthcare have captured less than 30 percent of the potential value we highlighted five years ago. And new opportunities have arisen since 2011, further widening the gap between the leaders and laggards.
Source: McKinsey, 2016
Most Hyped Technologies: Self-Driving Cars, Self-Service Analytics, IoT; No More Big Data Buzz
Gartner just released its 2015 Hype Cycle for Emerging Technologies report. It’s our most reliable buzz bellwether, annually defining what’s in and what’s out. At the peak of inflated expectations just two years ago, Big Data was dethroned by the Internet of Things last year (but it was still estimated to be five to ten years from the Plateau of Productivity), only to completely disappear from Gartner’s hype radar this year (the 2010-2014 hype cycles are at the bottom of this post). Big data is out. So what’s in?
Gartner, August 2015
The Internet of Things is still at the top of the list, with self-driving cars (“autonomous vehicles”) ascending from pre-peak to the peak of the hype cycle. But there is an intriguing new category—“advanced analytics with self-service delivery”—sharing with them top billing. I guess one could hype all three in one emerging technology package of “The Internet of Autonomous Vehicles Delivering Advanced Analytics“ as the solution to all our transportation problems.
These technologies at the peak of the hype cycle also highlighted for me what’s missing from this year’s report. Given that the most hyped news out of Black Hat and Defcon conferences earlier this month were demonstrations of how to hack into cars (self-driving or not) and take control of them remotely, it is interesting that Gartner does not list any specific cybersecurity-related emerging technologies. It does mention, however, two general categories—“digital security” and “software-defined security” —both described as pre-peak, 5 to 10 years to the Plateau of Productivity. This may simply reflect the hype-less status of cybersecurity technologies. Given the daily news about data breaches, one could only hope that next year’s report will include some specific emerging solutions to what is promising to be a growing economic burden.
Another emerging technology showing promise last year—data science—has disappeared from this year’s report. It is replaced by “citizen data science” which Gartner thinks, as it did regarding data science last year, is only 2 to 5 years from the plateau. This could turn out to be the most optimistic prediction in this year’s report. A related category—machine learning—is making its first appearance on the chart this year, but already past the peak of inflated expectations. A glaring omission here is “deep learning,” the new label for and the new generation of machine learning, and one of the most hyped emerging technologies of the past couple of years.
It all boils down to what Gartner calls digital humanism: “New to the Hype Cycle this year is the emergence of technologies that support what Gartner defines as digital humanism—the notion that people are the central focus in the manifestation of digital businesses and digital workplaces.”
For the last 21 years Gartner has published the Hype Cycle report, of which Lee Rainie of the Pew Research Center has said: “There are sometimes disputes about where on the curve any individual innovation might rest, but there have been few challenges to the general trends it outlines.” I remember attending a Gartner Conference just before it started publishing this report and listening to a presentation by the analyst responsible at the time for Gartner’s emerging technologies research. He started his presentation by declaring: “Those who live by the crystal ball, die eating broken glass.”
The charts below show the evolution of Gartner’s crystal ball over the last five years and allow us to track the hype around Big Data over that period. It made its first appearance in August of 2011 as “‘Big data’ and extreme information processing and management” with 2 to 5 years to the Plateau of Productivity,then just made it into the Peak of Inflated Expectations in 2012, then rose to the top of most hyped technologies (together with consumer 3D printing and Gamification) in 2013, then started to descend into the Trough of Disillusionment in 2014, only to completely vanish in 2015. I guess Big Data is no longer an emerging technology.
Gartner Hype Cycle 2014
Gartner, August 2014
Gartner Hype Cycle 2013
Gartner, August 2013
Gartner Hype Cycle 2012
Gartner, August 2012
Gartner Hype Cycle 2011
Gartner, August 2011
Gartner Hype Cycle 2010
Gartner, August 2010
An earlier version of this post was published on Forbes.com
China: Increasing Investments in AI, Big Data and Digital Health
Gartner Hype Cycle for ICT in China, 2016
Despite the slowdown in GDP growth to 6.9 percent in 2015, China is still making aggressive investments to drive the adoption of high technology by local enterprises and organizations, according to Gartner, Inc…
The massive consumer base and the number of internet users in China (estimated at 650 million internet and 980 million mobile internet users in 2016) present the most-promising big data opportunities. Led by hyperscale internet companies such as Baidu (internet traffic data), Alibaba (supply chain and transaction data) and Tencent (social data), approximately 25 percent of businesses have been pursuing the value of big data.
“The government-sponsored strategy ‘Internet Plus’ is targeted at boosting economic growth through digital transformation,” said Jie Zhang, research director at Gartner. “It has issued a detailed action plan for 11 key industries in 2015, mandating the necessity of digital business transformation by leveraging big data and cloud technologies.”
The biggest buzz in China’s internet industry isn’t about besting global tech giants by better adapting existing business models for the Chinese market. Rather, it’s about competing head-to-head with the U.S. and other tech powerhouses in the hottest area of technological innovation: artificial intelligence.
Venture capitalists have been pouring money into startups focused on AI, which broadly refers to efforts to make computers emulate human cognitive functions such as recognizing speech or images. Chinese tech companies such as search giant Baidu have been investing heavily in the technology, and poaching high-level talent from foreign rivals.
Enthusiasts of the technology in China say those resources, along with some particular advantages in China, such as the sheer volume of data generated by its enormous population of internet users, makes this an area where China can excel.
“China is poised to be a leader in AI because of its great reserve in AI talent, excellent engineering education and massive market for AI adoption,” says Kai-Fu Lee, a former Microsoft and Google executive who is now chief executive of Sinovation Ventures. The firm, formerly known as China’s Innovation Works, has invested $100 million in 25 AI-related startups in the U.S. and China in the past three years.
In total, over $1.1B has been deployed across 21 deals to Chinese digital health companies in the first six months of the year. It’s worth noting, though, that three investments each totaled over $100M in financing over the period including Ping An Insurance-backed medical services app Ping An Good Doctor, Beijing-based mobile healthcare app maker Spring Rain Software, and health data mining startup iCarbonX.
The chart above highlights how mega-rounds have propelled China’s digital health investment since 2012. Deal activity in the first half of 2016 was nearly equivalent with that of all of 2015.
Posted in Artificial Intelligence, Big Data, AI in Healthcare
Tagged CB Insights, Gartner
Leave a comment
On Brontobyte Data and Other Big Words
Source: Datafloq
Paul McFedries in IEEE Spectrum:
When Gartner released its annual Hype Cycle for Emerging Technologies for 2014, it was interesting to note that big data was now located on the downslope from the “Peak of Inflated Expectations,” while the Internet of Things (often shortened to IoT) was right at the peak, and data science was on the upslope. This felt intuitively right. First, although big data—those massive amounts of information that require special techniques to store, search, and analyze—remains a thriving and much-discussed area, it’s no longer the new kid on the data block. Second, everyone expects that the data sets generated by the Internet of Things will be even more impressive than today’s big-data collections. And third, collecting data is one significant challenge, but analyzing and extracting knowledge from it is quite another, and the purview of data science.
Just how much information are we talking about here? Estimates vary widely, but big-data buffs sometimes speak of storage in units of brontobytes, a term that appears to be based on brontosaurus, one of the largest creatures ever to rattle the Earth. That tells you we’re dealing with a big number, but just how much data could reside in a brontobyte? I could tell you that it’s 1,000 yottabytes, but that likely won’t help. Instead, think of a terabyte, which these days represents an average-size hard drive. Well, you would need 1,000,000,000,000,000 (a thousand trillion) of them to fill a brontobyte. Oh, and for the record, yes, there’s an even larger unit tossed around by big-data mavens: the geopbyte, which is 1,000 brontobytes. Whatever the term, we’re really dealing in hellabytes, that is, a helluva lot of data.
Wrangling even petabyte-size data sets (a petabyte is 1,000 terabytes) and data lakes (data stored and readily accessible in its pure, unprocessed state) are tasks for professionals, so not only are listings for big-data-related jobs thick on the ground but the job titles themselves now display a pleasing variety: companies are looking for data architects (specialists in building data models), data custodians and data stewards (who manage data sources), data visualizers (who can translate data into visual form), data change agents and data explorers (who change how a company does business based on analyzing company data), and even data frackers (who use enhanced or hidden measures to extract or obtain data).
But it’s not just data professionals who are taking advantage of Brobdingnagian data sets to get ahead. Nowhere is that more evident than in the news, where a new type of journalism has emerged that uses statistics, programming, and other digital data and tools to produce or shape news stories. This data journalism (or data-driven journalism) is exemplified by Nate Silver’s FiveThirtyEight site, a wildly popular exercise in precision journalism and computer-assisted reporting (or CAR).
And everyone, professional and amateur alike, no longer has the luxury of dealing with just “big” data. Now there is also thick data (which combines both quantitative and qualitative analysis), long data (which extends back in time hundreds or thousands of years), hot data (which is used constantly, meaning it must be easily and quickly accessible), and cold data (which is used relatively infrequently, so it can be less readily available).
In the 1980s we were told we needed cultural literacy. Perhaps now we need big-data literacy, not necessarily to become proficient in analyzing large data sets but to become aware of how our everyday actions—our small data—contribute to many different big-data sets and what impact that might have on our privacy and security. Let’s learn how to become custodians of our own data.
10 Most Successful Big Data Technologies
As the big data analytics market rapidly expands to include mainstream customers, which technologies are most in demand and promise the most growth potential? The answers can be found in TechRadar: Big Data, Q1 2016, a new Forrester Research report evaluating the maturity and trajectory of 22 technologies across the entire data life cycle. The winners all contribute to real-time, predictive, and integrated insights, what big data customers want now.
Here is my take on the 10 hottest big data technologies based on Forrester’s analysis:
- Predictive analytics: software and/or hardware solutions that allow firms to discover, evaluate, optimize, and deploy predictive models by analyzing big data sources to improve business performance or mitigate risk.
- NoSQL databases: key-value, document, and graph databases.
- Search and knowledge discovery: tools and technologies to support self-service extraction of information and new insights from large repositories of unstructured and structured data that resides in multiple sources such as file systems, databases, streams, APIs, and other platforms and applications.
- Stream analytics: software that can filter, aggregate, enrich, and analyze a high throughput of data from multiple disparate live data sources and in any data format.
- In-memory data fabric: provides low-latency access and processing of large quantities of data by distributing data across the dynamic random access memory (DRAM), Flash, or SSD of a distributed computer system.
- Distributed file stores: a computer network where data is stored on more than one node, often in a replicated fashion, for redundancy and performance.
- Data virtualization: a technology that delivers information from various data sources, including big data sources such as Hadoop and distributed data stores in real-time and near-real time.
- Data integration: tools for data orchestration across solutions such as Amazon Elastic MapReduce (EMR), Apache Hive, Apache Pig, Apache Spark, MapReduce, Couchbase, Hadoop, and MongoDB.
- Data preparation: software that eases the burden of sourcing, shaping, cleansing, and sharing diverse and messy data sets to accelerate data’s usefulness for analytics.
- Data quality: products that conduct data cleansing and enrichment on large, high-velocity data sets, using parallel operations on distributed data stores and databases.
Forrester’s TechRadar methodology evaluates the potential success of each technology and all 10 above are projected to have “significant success.” In addition, each technology is placed in a specific maturity phase—from creation to decline—based on the level of development of its technology ecosystem. The first 8 technologies above are considered to be in the Growth stage and the last 2 in the Survival stage.
Forrester also estimates the time it will take the technology to get to the next stage and predictive analytics is the only one with a “>10 years” designation, expected to “deliver high business value in late Growth through Equilibrium phase for a long time.” Technologies #2 to #8 above are all expected to reach the next phase in 3 to 5 years and the last 2 technologies are expected to move from the Survival to the Growth phase in 1-3 years.
Finally, Forrester provides for each technology an assessment of its business value-add, adjusted for uncertainty. This is based not only on potential impact but also on feedback and evidence from implementations and market reputation. Says Forrester: “If the technology and its ecosystem are at an early stage of development, we have to assume that its potential for damage and disruption is higher than that of a better-known technology.” The first 2 technologies in the list above are rated as “high” business value-add, the next 2 as “medium,” and all the rest “low,” no doubt because of their emerging status and lack of maturity.
Why did I add to the list of hottest technologies two that are still in the Survival phase—data preparation and data quality? In the same report, Forrester also provides the following data from its Q4 2015 survey of 63 big data vendors:
What is the level of customer interest in each of the following capabilities? (% answering “very high”)
Data preparation and discovery 52%
Data integration 48%
Advanced analytics 46%
Customer analytics 46%
Data security 38%
In-memory computing 37%
While Forrester predicts that a few standalone vendors of data preparation will survive, it believes this is “an essential capability for achieving democratization of data,” or rather, its analysis, letting data scientists spend more time on modeling and discovering insights and allowing more business users to have fun with data mining. Data Quality includes data security from the table above, in addition to other features ensuring decisions are based on reliable and accurate data. Forrester “expects that data quality will have significant success in the coming years as firms formalize a data certification process. Data certification efforts seek to guarantee that data meets expected standards for quality; security; and regulatory compliance supporting business decision-making, business performance, and business processes.”
“Big Data” as a topic of conversation has reached mainstream audiences probably far more than any other technology buzzword before it. That did not help the discussion of this amorphous term, defined for the masses as “the planet’s nervous system” (see my rant here) or as “Hadoop” for technical audiences. Forrester’s report helps clarify the term, defining big data as the ecosystem of 22 technologies, each with its specific benefits for enterprises and, through them, consumers.
Big data, specifically one its attributes, big volume, has recently gave rise to a new general topic of discussion, Artificial Intelligence. The availability of very large data sets is one of the reasons Deep Learning, a sub-set of AI, has been in the limelight, from identifying Internet cats to beating a Go champion. In its turn, AI may lead to the emergence of new tools for collecting and analyzing data.
Says Forrester: “In addition to more data and more computing power, we now have expanded analytic techniques like deep learning and semantic services for context that make artificial intelligence an ideal tool to solve a wider array of business problems. As a result, Forrester is seeing a number of new companies offering tools and services that attempt to support applications and processes with machines that mimic some aspects of human intelligence.”
Prediction is difficult, especially about the future, but it’s a (relatively) safe bet that the race to mimic elements of human intelligence, led by Google, Facebook, Baidu, Amazon, IBM, and Microsoft, all with very deep pockets, will change what we mean by “big data” in the very near future.
Originally posted on Forbes.com
10 New Big Data Observations from Tom Davenport
[youtube https://www.youtube.com/watch?v=DdHhD4n3iFE?rel=0]
The term “big data” has become nearly ubiquitous. Indeed, it seems that every day we hear new reports of how some company is using big data and sophisticated analytics to become increasingly competitive. The topic first began to take off in late 2010 (at least according to search results from Google Trends) and, now that we’re approaching a five-year anniversary, perhaps it’s a good time to take a step back and reflect on this major approach to doing business. This article describes 10 of my observations about big data.
See also Tom Davenport’s Guide to Big Data
Big Data, Small World: Kirk Borne at TEDxGeorgeMasonU (Video)
Kirk Borne is Professor of Astrophysics and Computational Science in the George Mason University School of Physics, Astronomy, and Computational Sciences (SPACS). Turns out he is the father of the term “unknown unknowns” – things we do not know we don’t know – popularized by former secretary of defense Donald Rumsfeld and later by Avinash Kaushik as “the unique space in which big data analysts should actually play.”
Posted in Big Data
Leave a comment