Whats the Big Data

By Hal Varian, Chief Economist, Google

Published on IMF.org

A computer now sits in the middle of virtually every economic transaction in the developed world. Computing technology is rapidly penetrating the developing world as well, driven by the rapid spread of mobile phones. Soon the entire planet will be connected, and most economic transactions worldwide will be computer mediated.

Data systems that were once put in place to help with accounting, inventory control, and billing now have other important uses that can improve our daily life while boosting the global economy.

Transmission routes

Computer mediation can impact economic activity through five important channels.

Data collection and analysis: Computers can record many aspects of a transaction, which can then be collected and analyzed to improve future transactions. Automobiles, mobile phones, and other complex devices collect engineering data that can be used to identify points of failure and improve future products. The result is better products and lower costs.

Personalization and customization: Computer mediation allows services that were previously one-size-fits-all to become personalized to satisfy individual needs. Today we routinely expect that online merchants we have dealt with previously possess relevant information about our purchase history, billing preferences, shipping addresses, and other details. This allows transactions to be optimized for individual needs.

Experimentation and continuous improvement: Online systems can experiment with alternative algorithms in real time, continually improving performance. Google, for example, runs over 10,000 experiments a year dealing with many different aspects of the services it provides, such as ranking and presentation of search results. The experimental infrastructure to run such experiments is also available to the company’s advertisers, who can use it to improve their own offerings.

Contractual innovation: Contracts are critical to economic transactions, but without computers it was often difficult or costly to monitor contractual performance. Verifying performance can help alleviate problems with asymmetric information, such as moral hazard and adverse selection, which can interfere with efficient transactions. There is no longer a risk of purchasing a “lemon” car if vehicular monitoring systems can record history of use and vehicle health at minimal cost.

Coordination and communication: Today even tiny companies with a handful of employees have access to communication services that only the largest multinationals could afford 20 years ago. These micro-multinationals can operate on a global scale because the cost of computation and communication has fallen dramatically. Mobile devices have enabled global coordination of economic activity that was extremely difficult just a decade ago. For example, today authors can collaborate on documents simultaneously even when they are located thousands of miles apart. Videoconferencing is now essentially free, and automated document translation is improving dramatically. As mobile technology becomes ubiquitous, organizations will become more flexible and responsive, allowing them to improve productivity.

Let us dig deeper into these five channels through which computers are changing our lives and our economy.

Data collection and analysis

We hear a lot about “big data” (see “Big Data’s Big Muscle,” in this issue of F&D), but “small data” can be just as important, if not more so. Twenty years ago only large companies could afford sophisticated inventory management systems. But now every mom-and-pop corner store can track its sales and inventory using intelligent cash registers, which are basically just personal computers with a drawer for cash. Small business owners can handle their own accounting using packaged software or online services, allowing them to better track their business performance. Indeed, these days data collection is virtually automatic. The challenge is to translate that raw data into information that can be used to improve performance.

Large businesses have access to unprecedented amounts of data, but many industries have been slow to use it, due to lack of experience in data management and analysis. Music and video entertainment have been distributed online for more than a decade, but the entertainment industry has been slow to recognize the value of the data collected by servers that manage this distribution (see “Music Going for a Song,” in this issue of F&D). The entertainment industry, driven by competition from technology companies, is now waking up to the possibility of using this data to improve their products.

The automotive industry is also evolving quickly by adding sensors and computing power to its products. Self-driving cars are rapidly becoming a reality. In fact, we would have self-driving cars now if it weren’t for the randomness introduced by human drivers and pedestrians. One solution to this problem would be restricted lanes for autonomous vehicles only. Self-driving cars can communicate among themselves and coordinate in ways that human drivers are (alas) unable to. Autonomous vehicles don’t get tired, they don’t get inebriated, and they don’t get distracted. These features of self-driving cars will save millions of lives in the coming years.

Personalization and customization

Twenty years ago it was a research challenge for computers to recognize pictures containing human beings. Now free photo storage systems can find pictures with animals, mountains, castles, flowers, and hundreds of other items in seconds. Improved facial recognition technology and automated indexing allow the photos to be found and organized easily and quickly.

Similarly, just in the past few years voice recognition systems have become significantly more accurate. Voice communication with electronic devices is possible now and will soon become the norm. Real-time verbal language translation is a reality in the lab and will be commonplace in the near future. Removing language barriers will lead to increased foreign trade, including, of course, tourism.

Continuous improvement

Observational data can uncover interesting patterns and correlations in data. But the gold standard for discovering causal relationships is experimentation, which is why online companies like Google routinely experiment and continuously improve their systems. When transactions are mediated by computers, it is easy to divide users into treatment and control groups, deploy treatment, and analyze outcomes in real time.

Companies now routinely use this kind of experimentation for marketing purposes, but these techniques can be used in many other contexts. For example, institutions such as the Massachusetts Institute of Technology’s Abdul Latif Jameel Poverty Action Lab have been able to run controlled experiments of proposed interventions in developing economies to alleviate poverty, improve health, and raise living standards. Randomized controlled experiments can be used to resolve questions about what sorts of incentives work best for increasing saving, educating children, managing small farms, and a host of other policies.

Contractual innovation

The traditional business model for advertising was “You pay me to show your ad to people, and some of them might come to your store.” Now in the online world, the model is “I’ll show your ad to people, and you only have to pay me if they come to your website.” The fact that advertising transactions are computer mediated allows merchants to pay only for the outcome they care about.

Consider the experience of taking a taxi in a strange city. Is this an honest driver who will take the best route and charge me the appropriate fee? At the same time, the driver may well have to worry whether the passenger is honest and will pay for the ride. This is a one-time interaction, with limited information on both sides and potential for abuse. But now consider technology such as that used by Lyft, Uber, and other ride services. Both parties can see rating history, both parties can access estimates of expected fares, and both parties have access to maps and route planning. The transaction has become more transparent to all parties, enabling more efficient and effective transactions. Riders can enjoy cheaper and more convenient trips, and drivers can enjoy a more flexible schedule.

Smartphones have disrupted the taxi industry by enabling these improved transactions, and every player in the industry is now offering such capabilities—or will soon. Many people see the conflict between ride services and the taxi industry as one of innovators versus regulators. However, from a broader perspective, what matters is which technology wins. The technology used by rideshare companies clearly provides a better experience for both drivers and passengers, so it will likely be widely adopted by traditional taxi services.

Simply being able to capture transaction history can improve contracts (see “Two Faces of Change,” in this issue of F&D). It is remarkable that I can walk into a bank in a new city, where I know no one and no one knows me, and arrange for a mortgage worth millions of dollars. This is enabled by credit rating services, which dramatically reduce risk on both sides of the transaction, making loans possible for people who otherwise could not get them.

Communication and coordination

Recently I had some maintenance work done on my house. The team of workers used their mobile phones to photograph items that needed replacement, communicate with their colleagues at the hardware store, find their way to the job site, use as a flashlight to look in dark places, order lunch for delivery, and communicate with me. All of these formerly time-consuming tasks can now be done quickly and easily. Workers spend less time waiting for instructions, information, or parts. The result is reduced transaction costs and improved efficiency.

Today only the wealthy can afford to employ executive assistants. But in the future everyone will have access to digital assistant services that can search through vast amounts of information and communicate with other assistants to coordinate meetings, maintain records, locate data, plan trips, and do the dozens of other things necessary to get things done (see “Robots, Growth, and Inequality,” in this issue of F&D). All of the big tech companies are investing heavily in this technology, and we can expect to see rapid progress thanks to competitive pressure.

Putting it all together

Today’s mobile phones are many times more powerful and much less expensive than those that powered Apollo 11, the 1969 manned expedition to the moon. These mobile phone components have become “commoditized.” Screens, processors, sensors, GPS chips, networking chips, and memory chips cost almost nothing these days. You can buy a reasonable smartphone now for $50, and prices continue to fall. Smartphones are becoming commonplace even in very poor regions.

The availability of those cheap components has enabled innovators to combine and recombine these components to create new devices—fitness monitors, virtual reality headsets, inexpensive vehicular monitoring systems, and so on. The Raspberry Pi is a $35 computer designed at Cambridge University that uses mobile phone parts with a circuit board the size of a pack of playing cards. It is far more powerful than the Unix workstations of just 15 years ago.

The same forces of standardization, modularization, and low prices are driving progress in software. The hardware created using mobile phone parts often uses open-source software for its operating system. At the same time, the desktop motherboards from the personal computer era have now become components in vast data centers, also running open-source software. The mobile devices can hand off relatively complex tasks such as image recognition, voice recognition, and automated translation to the data centers on an as-needed basis. The availability of cheap hardware, free software, and inexpensive access to data services has dramatically cut entry barriers for software development, leading to millions of mobile phone applications becoming available at nominal cost.

The productivity puzzle

I have painted an optimistic picture of how technology will impact the global economy. But how will this technological progress show up in conventional economic statistics? Here the picture is somewhat mixed. Take GDP, for example. This is usually defined as the market value of all final goods and services produced in a given country in a particular time period. The catch is “market value”—if a good isn’t bought and sold, it generally doesn’t show up in GDP.

This has many implications. Household production, ad-supported content, transaction costs, quality changes, free services, and open-source software are dark matter as far as GDP is concerned, since technological progress in these areas does not show up directly in GDP. Take, for example, ad-supported content, which is widely used to support provision of online media. In the U.S. Bureau of Economic Analysis National Economic Accounts, advertising is treated as a marketing expense—an intermediate product—so it isn’t counted as part of GDP. A content provider that switches from a pay-per-view business model to an ad-supported model reduces GDP.

One example of technology making a big difference to productivity is photography. Back in 2000, about 80 billion photos were taken worldwide—a good estimate since only three companies produced film then. In 2015, it appears that more than 1.5 trillion photos were taken worldwide, roughly 20 times as many. At the same time the volume exploded, the cost of photos fell from about 50 cents each for film and developing to essentially zero.

So over 15 years the price fell to zero and output went up 20 times. Surely that is a huge increase in productivity. Unfortunately, most of this productivity increase doesn’t show up in GDP, since the measured figures depend on the sales of film, cameras, and developing services, which are only a small part of photography these days.

In fact, when digital cameras were incorporated into smartphones, GDP decreased, camera sales fell, and smartphone prices continued to decline. Ideally, quality adjustments would be used to measure the additional capabilities of mobile phones. But figuring out the best way to do this and actually incorporating these changes into national income accounts is a challenge.

Even if we could accurately measure the number of photos now taken, most are produced at home and distributed to friends and family at zero cost; they are not bought and sold and don’t show up in GDP. Nevertheless, those family photos are hugely valuable to the people who take them.

The same thing happened with global positioning systems (GPS). In the late 1990s, the trucking industry adopted expensive GPS and vehicular monitoring systems and saw significant increases in productivity as a result. In the past 10 years, consumers have adopted GPS for home use. The price of the systems has fallen to zero, since they are now bundled with smartphones, and hundreds of millions of people use such systems on a daily basis. But as with cameras, the integration of GPS with smartphones has likely reduced GDP, since sales of stand-alone GPS systems have fallen.

As in the case of cameras, this measurement problem could be solved by implementing a quality adjustment for smartphones. But it is tricky to know exactly how to do this, and statistical agencies want a system that will stand the test of time. Even after the quality adjustment problem is worked out, the fact that most photos are not exchanged for cash will remain—that isn’t a part of GDP, and technological improvements in that area are just not measured by conventional statistics.

Will the promise of technology be realized?

When the entire planet is indeed connected, everyone in the world will, in principle, have access to virtually all human knowledge. The barriers to full access are not technological but legal and economic. Assuming that these issues can be resolved, we can expect to see a dramatic increase in human prosperity.

But will these admittedly utopian hopes be realized? I believe that technology is generally a force for good—but there is a dark side to the force (see “The Dark Side of Technology,” in this issue of F&D). Improvements in coordination technology may help productive enterprises but at the same time improve the efficiency of terrorist organizations. The cost of communication may drop to zero, but people will still disagree, sometimes violently. In the long run, though, if technology enables broad improvement in human welfare, people might devote more time to enlarging the pie and less to squabbling over the size of the pieces.