Jeopardy champion Jennings on how a computer beat him at his own game (Video)

[youtube https://www.youtube.com/watch?v=b2M-SeKey4o?rel=0]

Jennings in Slate:

…there’s no shame in losing to silicon, I thought to myself as I greeted the (suddenly friendlier) team of IBM engineers after the match. After all, I don’t have 2,880 processor cores and 15 terabytes of reference works at my disposal—nor can I buzz in with perfect timing whenever I know an answer. My puny human brain, just a few bucks worth of water, salts, and proteins, hung in there just fine against a jillion-dollar supercomputer.

“Watching you on Jeopardy! is what inspired the whole project,” one IBM engineer told me, consolingly. “And we looked at your games over and over, your style of play. There’s a lot of you in Watson.” I understood then why the engineers wanted to beat me so badly: To them, I wasn’t the good guy, playing for the human race. That was Watson’s role, as a symbol and product of human innovation and ingenuity. So my defeat at the hands of a machine has a happy ending, after all. At least until the whole system becomes sentient and figures out the nuclear launch codes. But I figure that’s years away.

Posted in Artificial Intelligence | Leave a comment

Recruiting Data Scientists to Mine the Data Explosion

DigitalUniverse_WSJ

 

Wes Hunt, Chief Data Officer (CDO) at Nationwide Mutual Insurance Co. on recruiting data scientists:

Finding talent is my largest challenge. Someone who understands our business, who has quantitative skills, who has the technical skills to create the models, and who is able to persuade others that the insights they’ve come up with are ones you can trust and take action on. The hardest part is persuasion. You get the quantitative skills, but there’s a struggle in that ability to communicate effectively. We’ll often pair people together, but we’d really like to grow the talent.

When I was in marketing, we put a focus on liberal-arts-educated individuals, because abstract thinking where there are ambiguous data sets is an area where they are comfortable. Ph.D.s in psychology were a great recruiting pool. A psych Ph.D. has a fair amount of statistical training. We created a program to recruit Ph.D.s.

There’s not yet an educational discipline and curriculum that produces data scientists at the scale that would clear the market. So the way we’ve focused on it is to find people with innate curiosity and critical thinking. You can teach the other skills. On my team, I have a pathologist, a bioengineering student who trained in doing heart research, an M.B.A., and someone who is trained in traditional data architecture. I also have a landscape construction engineer and a psychology Ph.D.

 

Posted in Data Science | Leave a comment

Doug Cutting on Hadoop, October 2014 (Video)

[youtube https://www.youtube.com/watch?v=0GOxDBR6VAU?rel=0]

Posted in Misc | Leave a comment

Home Monitoring mHealth Wearable Devices 2013-2019

Wearables_health_ABI

ABI Research:

Over the next 5 years, a new generation of elderly home care services will drive wearable device shipments to more than 44 million in 2019 up from just 6 million in 2013. In 2014 alone, shipments of wearable devices linked to elderly care systems will more than double over those in 2013, finds the latest ABI Research analysis of the mHealth market.

Growing adoption comes as tech savvy families increasingly turn to home monitoring offerings for assurance their aging parents and family members are safe and well. In addition, new offerings are boosting and extending a market that has long been the territory of dedicated, “Help! I’ve fallen and I can’t get up”-type personal emergency response systems. A host of niche players including BeClose, GrandCare Systems, Independa and others have all emerged to capitalize on a combination of market demand and the potential to leverage connected devices and systems.

In the past few months alone, one start-up, Live!y, has revamped and re-launched its offering to include a watch that offers activity tracking alongside personal emergency response services, while AT&T has added elderly care monitoring to its Digital Life smart home package. These players reflect how device manufacturers and service providers alike are increasingly targeting the elder care market and doing so with more feature rich offerings.

These findings are part of ABI Research’s mHealth Wearables, Platforms and Services Market Research which looks at the rapidly developing market for wearable wireless sensors, by device, connectivity and region across sports, fitness and wellbeing, home care monitoring, remote patient monitoring, and on-site professional healthcare markets.

Posted in Misc | Leave a comment

Most companies analyze a mere 12% of their data (Infographic)

Big_Data_Platfora_infographic

 

Source: Platfora

 

Posted in Big Data | Leave a comment

Gartner on Top Trends for 2015 and Beyond

PredictionsOn the occasion of its sold-out Symposium/ITxpo this week, Gartner revealed its predictions for top technology trends, the impact of technology on businesses and consumers, and the continuing evolution of the IT organization and the role of the CIO. Here is my summary of Gartner’s press releases:

Digital disruption will give rise to new businesses, some created by machines

By 2017, says Gartner, a significant disruptive digital business will be launched that was conceived by a computer algorithm. The most successful startups will mesh the digital world with existing physical logistics to create a consumer-driven network—think Airbnb and Uber—and challenge established markets consisting of isolated physical units.

The meshing of the digital with the physical will impact how we think about product development. By 2015, more than half of traditional consumer products will have native digital extensions and by 2017, 50% of consumer product investments will be redirected to customer experience innovations. The wide availability of product, pricing and customer satisfaction information has eroded the competitive advantage afforded in the past by product innovation and is shifting attention to customer experience innovation as the key to a lasting brand loyalty.

The meshing of the digital with the physical also means the rise of the Internet of Things. “This year,” says Gartner, “enterprises will spend over $40 billion designing, implementing and operating the Internet of Things.” That’s still a tiny slice of worldwide spending on IT which is projected to surpass $3.9 trillion in 2015, a 3.9% increase from 2014. But much of this spending will be driven by the digital economy and the Internet of Things is in the driver seat.  Gartner defines digital business as new business designs that blend the virtual world and the physical worlds, changing how processes and industries work through the Internet of Things.

Consider this: Since 2013, 650 million new physical objects have come online; 3D printers became a billion dollar market10% of automobiles became connected; and the number of Chief Data Officers and Chief Digital Officer positions have doubled. But fasten your seat belts: In 2015, Gartner predicts, all of these things will double again.

Managing and leading the digital business, mostly by humans

Gartner’s survey of 2,800 CIOs in 84 countries showed that CIOs are fully aware that they will need to change in order to succeed in the digital business, with 75% of IT executives saying that they need to change their leadership style in the next three years.

“The exciting news for CIOs,” says Gartner, “is that despite the rise of roles, such as the chief digital officer, they are not doomed to be an observer of the digital revolution.” According the survey, 41% of CIOs are reporting to their CEO. Gartner notes that this is a return to one of the highest levels it has ever been, no doubt because of the increasing importance of information technology to all businesses.

Still, reporting to the CEO does not necessarily mean leading the digital initiatives of the business. Reports from the Symposium highlighted another Gartner finding: While CIOs say they are driving 47% of digital leadership only 15% of CEOs agree that they do so. Similarly, while CIOs estimate that 79% of IT spending will be “inside” the IT budget (up slightly from last year), Gartner says that 38% of total IT spending is outside of IT already, and predicts that by 2017, it will be over 50%. This is a “shift of demand and control away from IT and toward digital business units closer to the customer,” says Gartner. It further estimates that 50% of all technology sales people are actively selling direct to business units, not IT departments.

Gartner sees the established behaviors and beliefs of the IT organization, the “best practices” that have served it well in previous years, as the biggest obstacle for CIOs in their pursuit of digital leadership. Process management and control our not as important as vision and agility. Compounding the problem of obsolete leadership style and inadequate skills, is a fundamental requirement of the digital business: It must be unstable. By 2017, 70% of successful digital business models will rely on deliberately unstable processes designed to shift as customer needs shift. “This holistic approach,” says Gartner, “blending business model, processes, technology and people will fuel digital business success.”

The rise of smart machines

By 2015, there will be more than 40 vendors with commercially available managed services offerings leveraging smart machines and industrialized services. By 2018, the total cost of ownership for business operations will be reduced by 30% through smart machines and industrialized services.

Smart machines are an emerging “super class” of technologies that perform a wide variety of work, of both the physical and the intellectual kind. Smart machines will automate decision making. Therefore, they will not only affect jobs based on physical labor, but they will also impact jobs based on complex knowledge worker tasks. “Smart machines,” says Gartner, “will not replace humans as people still need to steer the ship and are critical to interpreting digital outcomes.” But these humans will have new types of jobs.

Top digital jobs

By 2018, Gartner predicts, digital business will require 50% less business process workers and 500% more key digital business jobs, compared with traditional models. The top jobs for digital over the next seven years will be:

•             Integration Specialists

•             Digital Business Architects

•             Regulatory Analysts

•             Risk Professionals

Gartner: “You must build talent for the digital organization of 2020 now. Not just the digital technology organization, but the whole enterprise. Talent is the key to digital leadership.”

Where things can go wrong

Gartner highlights two areas of potential vulnerabilities as business pursue digital opportunities: Lack of portfolio management skills and inadequate risk management.

By year-end 2016, 50% of digital transformation initiatives will be unmanageable due to lack of portfolio management skills, leading to a measurable negative lost market share. The digital business brings with it vastly different and higher levels of risk, say 89% of CIOs and 69% believe that the discipline of risk management is not keeping up.

Gartner: “CIOs need to review with the enterprise and IT risk leaders whether risk management is adapting fast enough to a digital world.”  This is also an urgent tasks, I might add, for CEOs and the board of directors.

The pursuit of longer life and increased happiness (i.e., better shopping experience)

Gartner predicts that all these connected devices will have a positive impact on our health. By 2017, the use of smartphones will reduce by 10% the costs for diabetic care. By 2020, life expectancy in the developed world will increase by 0.5 years due to widespread adoption of wireless health monitoring technology.

The retail industry could be the industry most impacted by the digital tsunami, drastically altering our shopping experience. By year-end 2015, mobile digital assistants will have taken on tactical mundane processes such as filling out names, addresses and credit card information. By year-end 2016, more than $2 billion in online shopping will be performed exclusively by mobile digital assistants. Yearly autonomous mobile assistant purchasing will reach $2 billion dollars annually, representing about 2.5 percent of mobile users trusting assistants with $50 a year.

By 2017, U.S. customers’ mobile engagement behavior will drive mobile commerce revenue in the U.S. to 50% of U.S. digital commerce revenue. A renewed interest in mobile payments will arise in 2015, together with a significant increase in mobile commerce. By 2016, there will be an increase in the number of offers from retailers focused on customer location and the length of time in store. By 2020, retail businesses that utilize targeted messaging in combination with internal positioning systems (IPS) will see a five percent increase in sales.

These trends seem to be a logical extension of current technologies. But 3D printing will bring us a completely new shopping experience and will expand widely what’s on offer. By 2015, more than 90% of online retailers of durable goods will actively seek external partnerships to support the new “personalized” product business models and by 2017, nearly 20% of these retailers will use 3D printing to create personalized product offerings.

Gartner: “The companies that set the strategy early will end up defining the space within their categories.”

This statement, which Gartner made specifically about the personalized products business, is true for all types of digital businesses and for all the new management processes and attitudes that all organizations should put in place, sooner rather than later.

Sources:

Gartner Reveals Top Predictions for IT Organizations and Users for 2015 and Beyond

Gartner Survey of More Than 2,800 CIOs Reveals That CIOs Must “Flip” Their Leadership Styles to Grasp the Digital Opportunity

Gartner Says Digital Business Economy is Resulting in Every Business Unit Becoming a Technology Startup

Gartner Identifies the Top 10 Strategic Technology Trends for 2015

[Originally published on Forbes.com]

Posted in Predictions | Leave a comment

Oren Etzioni on Building Intelligent Machines

[youtube https://www.youtube.com/watch?v=E_6AZ8slivc?rel=0]
“There are more things in AI than classification… the entire paradigm of classification, which has fueled machine learning and data mining, is very limited… What we need is a process that is structured, multi-layered, knowledge-intensive, much more like kids playing Lego, instead of a karate chop that divides things into categories… Current knowledge bases are fact-rich but knowledge poor…’You can’t play 20 questions with Nature and win’ (Allen Newell, 1973)… What we need is knowledge, reasoning, and explanation.”

Slides (from KDD Keynote, scroll all the way down the page) are here 

From GigaOm:

Oren Etzioni, executive director of the Allen Institute of Artificial Intelligence (formerly founder of Farecast and Decide.com), takes a contrarian view of all the deep learning hype. Essentially, he argues, while systems that are better than ever at classifying images or words are great, they’re still not “intelligent.” He describes work underway to build systems that can truly understand content, including one capable of passing fourth-grade short-answer exams.

Etzioni on reddit:

“I think that poeple are often confusing computer autonomy with computer intelligence. computer viruses are autonomous, dangerous, but not particularly intelligent. Chess playing programs are intelligent (in a sense) but very limited. They don’t even play checkers!”

“I love star trek and particularly the star trek computer because AI is used there as a tool to help and inform Captain Kirk and the crew. That’s a much better model than fear mongering in movies like HER and Transcendence. AI can be used to help us and enhance our abilities. For example, we are all inundated with huge amounts of text, articles, technical papers, and nobody can keep up! How about if your doctor had a tool that would help him or her to figure out the latest studies and procedures relevant to your condition? Even better—what if you had a tool to help figure out what’s going on that’s much better [than] google or webmd.”

[Why do you enjoy working on AI? What first motivated you to get into the field?] “It is one of the most fundamental intellectual problems and it’s really, really hard. I find computers so rigid, so stupid that it’s infuriating. My goal is to fight “artificial stupidity” and to build AI programs that help scientists, doctors, and regular folks make sense of the world and the tsunami of information that we all face every day.”

The Turing test is about tricking someone to believe that a computer is human. At AI2 we are working on programs that will try to pass tests in science & math which requires the program to understand the questions (hard) utilize background knowledge (even harder) and accumulate that knowledge automatically (great big challenge).”

[Would the IBM computer that played Jeopardy be called intelligent by your metrics?] “Watson was an impressive demonstration but it was narrowly targeted at Jeopardy and exhibited very little semantic understanding. Now Watson has become an IBM brand for any knowledge based activity they do. The intelligence is largely in their PR department.”

 

 

Posted in Artificial Intelligence, Machine Learning | Leave a comment

The New Apple Wristop Computer: Not Designed for the Internet of Things

MIT Media Lab cofounder Nicholas Negroponte observed at a recent TED event that “I look today at some of the work being done around the Internet of Things and it’s kind of tragically pathetic.”

The “tragically pathetic” label has been especially fitting for wearables, considered the hottest segment of the Internet of Things.  Lauren Goode at Re/Code wrote back in March: “Let me guess: Your activity-tracking wristband is sitting on your dresser or in a drawer somewhere right now, while it seems that every day there’s a news report out about an upcoming wearable product that’s going to be better, cooler, smarter.”

All of this was going to change when Apple finally entered the category with its smart watch. Many observers hoped that Apple’s design principles, obsession with simplicity, and track record of delighting users with easy-to-use products, are going to finally give the world a useful and fun wearable.

Instead, we got a good-looking wrist-top computer. Not a simple, intuitive, and focused device but a generic, complex product with too many functions and options. Kevin McCullagh wrote in fastcodesing.com: “I can’t help but think Steve Jobs would have stopped the kitchen sink being thrown in like this. Do we really need photos and maps on a stamp-sized screen, when our phones are rarely out of reach? For all the claims of a ‘thousand no’s for every yes,’ the post-Jobs era is shaping up to be defined by less ruthless focus.” Back in June, Adam Lashinsky already made this general observation about the potential loss of the famed product development discipline: “Apple, once the epitome of simplicity, is becoming the unlikely poster child for complexity.”

“Complexity,” however, does not tell the whole story. By introducing a watch that is basically a computer on your wrist, Apple missed an opportunity not just to reorient the wearables market to something much better than “tragically pathetic,” but also to define the design and usability principles for the Internet of Things.

In his TED talk, Negroponte highlighted what he called “not a particularly enlightened view of the Internet of Things.” This is the tendency to move the intelligence (or functionality of many devices) into the cell phone (or the wearable), instead of building the intelligence into the “thing,” whatever the thing is – the oven, the refrigerator, the road, the walls, all the physical things around us. More generally, it is the tendency to continue evolving the current computer paradigm—from the mainframe to the laptop to the wristop computer—instead of developing a completely new Internet of Things paradigm.

The new paradigm should embrace and evolve the principles of what was once called “ubiquitous computing.” The history of that vision over the last two decades may help illuminate where the Internet of Things is today and where it may or may not go.

In 1991, Mark Weiser, then head of the Computer Science Lab at Xerox PARC, published an article in Scientific American titled “The Computer for the 21st Century.” The article opens with what should be the rallying cry for the Internet of Things today: “The most profound technologies are those that disappear. They weave themselves into the fabric of everyday life until they are indistinguishable from it.”

Weiser went on to explain what was wrong with the personal computing revolution brought on by Apple and others: “The arcane aura that surrounds personal computers is not just a ‘user interface’ problem. My colleague and I at the Xerox Palo Alto Research Center think that the idea of a ‘personal’ computer itself is misplaced and that the visions of laptop machines, dynabooks and ‘knowledge navigators’ is only a transitional step toward achieving the real potential of information technology.  Such machines cannot truly make computing an integral, invisible part of people’s lives.”

Weiser understood that, conceptually, the PC was simply a mainframe on a desk, albeit with easier-to-use applications.  He misjudged, however, the powerful and long-lasting impact that this new productivity and life-enhancing tool would exert on millions of users worldwide. Weiser wrote: “My colleagues and I at PARC believe that what we call ubiquitous computing will gradually emerge as the dominant mode of computer access over the next 20 years. … [B]y making everything faster and easier to do, with less strain and fewer mental gymnastics, it will transform what is apparently possible. … [M]achines that fit the human environment instead of forcing humans to enter theirs will make using a computer as refreshing as taking a walk in the woods.”

Ubiquitous computing has not become the “dominant mode of computer access” mostly because of Steve Jobs’ Apple. It successfully invented variations on the theme of the Internet of Computers: The iPod, the iPhone, the iPad. All of them beautifully designed, easy-to-use, and useful. All of them cementing and enlarging the dominance of the Internet of Computers paradigm. Now Apple has extended the paradigm by inventing a wristop computer. That the Apple Watch is more complex and less focused than Apple’s previous successful inventions matters less than the fact that it continues in their well-trodden path.

While the dominant paradigm has been reinforced and expanded by the successful innovations of Apple and others, the vision of ubiquitous computing has not died. Today, when we are adding intelligence to things at an accelerating rate, it is more important than ever. Earlier this year, I asked Bob Metcalfe what is required to make us happy with our Internet of Things experience. “Not so much good UX, but no UX at all,” he said. “The IoT should disappear into the woodwork, even faster than Ethernet has.” Metcalfe invented the Ethernet at Xerox PARC at the time Weiser and others were working on making computers disappear.

Besides ubiquity, there are at least two other dimensions to the new paradigm of the Internet of Things. One is seamless connectivity. In response to the same question, Google’s Hal Varian told me, “I think that the big challenge now is interoperability. Given the fact that there will be an explosion of new devices, it is important that they talk to each other. For example, I want my smoke alarm to talk to my bedroom lights, and my garden moisture detector to talk to my lawn sprinkler.” No more islands of computing, a hallmark of the Internet of (isolated) Computers.

Another important dimension of the new paradigm is useful data. Not big or small, nor irrelevant or trapped in a silo, just useful. The value of the “things” in the Internet of Things paradigm is measured by how well the data they collect is analyzed and how quickly useful feedback based on this analysis is delivered to the user.

Disappearing into the woodwork. All things talking to all things. Useful data. It may not be Apple, but the company or companies that will master these will usher in the new era of the Internet of Things where we finally get over our mainframe/PC/Wristop computer habit.

[Originally published on Forbes.com]

Posted in Internet of Things | Leave a comment

What’s the Big Data? 12 Definitions

Last week I got an email from UC Berkeley’s Master of Information and Data Science program, asking me to respond to a survey of data science thought leaders, asking the question “What is big data”? I was especially delighted to be regarded as a “thought leader” by Berkeley’s School of Information, whose previous dean, Hal Varian (now chief economist at Google, answered my challenge fourteen years ago and produced the first study to estimate the amount of new information created in the world annually, a study I consider to be a major milestone in the evolution of our understanding of big data.

The Berkeley researchers estimated that the world had produced about 1.5 billion gigabytes of information in 1999 and in a 2003 replication of the study found out that amount to have doubled in 3 years. Data was already getting bigger and bigger and around that time, in 2001, industry analyst Doug Laney described the “3Vs”—volume, variety, and velocity—as the key “data management challenges” for enterprises, the same “3Vs” that have been used in the last four years by just about anyone attempting to define or describe big data.

The first documented use of the term “big data” appeared in a 1997 paper by scientists at NASA, describing the problem they had with visualization (i.e. computer graphics) which “provides an interesting challenge for computer systems: data sets are generally quite large, taxing the capacities of main memory, local disk, and even remote disk. We call this the problem of big data. When data sets do not fit in main memory (in core), or when they do not fit even on local disk, the most common solution is to acquire more resources.”

In 2008, a number of prominent American computer scientists popularized the term, predicting that “big-data computing” will “transform the activities of companies, scientific researchers, medical practitioners, and our nation’s defense and intelligence operations.” The term “big-data computing,” however, is never defined in the paper.

The traditional database of authoritative definitions is, of course, the Oxford English Dictionary (OED). Here’s how the OED defines big data: (definition #1) “data of a very large size, typically to the extent that its manipulation and management present significant logistical challenges.”

But this is 2014 and maybe the first place to look for definitions should be Wikipedia. Indeed, it looks like the OED followed its lead. Wikipedia defines big data (and it did it before the OED) as (#2) “an all-encompassing term for any collection of data sets so large and complex that it becomes difficult to process using on-hand data management tools or traditional data processing applications.”

While a variation of this definition is what is used by most commentators on big data, its similarity to the 1997 definition by the NASA researchers reveals its weakness. “Large” and “traditional” are relative and ambiguous (and potentially self-serving for IT vendors selling either “more resources” of the “traditional” variety or new, non-“traditional” technologies).

The widely-quoted 2011 big data study by McKinsey highlighted that definitional challenge. Defining big data as (#3) “datasets whose size is beyond the ability of typical database software tools to capture, store, manage, and analyze,” the McKinsey researchers acknowledged that “this definition is intentionally subjective and incorporates a moving definition of how big a dataset needs to be in order to be considered big data.” As a result, all the quantitative insights of the study, including the updating of the UC Berkeley numbers by estimating how much new data is stored by enterprises and consumers annually, relate to digital data, rather than just big data, e.g., no attempt was made to estimate how much of the data (or “datasets”) enterprises store is big data.

Another prominent source on big data is Viktor Mayer-Schönberger and Kenneth Cukier’s book on the subject. Noting that “there is no rigorous definition of big data,” they offer one that points to what can be done with the data and why its size matters:

(#4) “The ability of society to harness information in novel ways to produce useful insights or goods and services of significant value” and “…things one can do at a large scale that cannot be done at a smaller one, to extract new insights or create new forms of value.”

In Big Data@Work, Tom Davenport concludes that because of “the problems with the definition” of big data, “I (and other experts I have consulted) predict a relatively short life span for this unfortunate term.” Still, Davenport offers this definition:

(#5) “The broad range of new and massive data types that have appeared over the last decade or so.”

Let me offer a few other possible definitions:

(#6) The new tools helping us find relevant data and analyze its implications.

(#7) The convergence of enterprise and consumer IT.

(#8) The shift (for enterprises) from processing internal data to mining external data.

(#9) The shift (for individuals) from consuming data to creating data.

(#10) The merger of Madame Olympe Maxime and Lieutenant Commander Data.

#(11) The belief that the more data you have the more insights and answers will rise automatically from the pool of ones and zeros.

#(12) A new attitude by businesses, non-profits, government agencies, and individuals that combining data from multiple sources could lead to better decisions.

I like the last two. #11 is a warning against blindly collecting more data for the sake of collecting more data (see NSA). #12 is an acknowledgment that storing data in “data silos” has been the key obstacle to getting the data to work for us, to improve our work and lives. It’s all about attitude, not technologies or quantities.

What’s your definition of big data?

See here for the compilation of Big data definitions from 40+ thought leaders.

[Originally published on Forbes.com]

Posted in Big Data | Leave a comment

The nature of data (Infographic)

The nature of data (Infographic)
Posted in Big Data, Infographics | Leave a comment