Big Data: Friend, Phobia or Fad?

May 5, 2014

As is often the case with new technologies, the reactions to big data have fallen into two camps. 

The enthusiasts see big data as a panacea. It creates economic value but also eliminates all existing problems, whether with the scientific method or good, old-fashioned uncertainty and messy data.

The pessimists agree that big data boosts productivity, but argue that it kills jobs. Smart technologies render superfluous that one last domain of human skills: cognitive and abstract abilities. We should be careful about embracing this new trend, which they say will lead to a “hollowing out” of the employment hierarchy, leaving only high-skill jobs for a few, and low-skill jobs for most.

The new White House report on big data, an "important conclusion" of which “is that big data technologies can cause societal harms beyond damages to privacy, such as discrimination against individuals and groups,” could spur some more pessimism or fear about the implications of the Big Data Revolution.

A third camp has also gathered steam, pointing to the failures of the much heralded Google Flu predictions. These deflationists dissent from the view that there is a revolution at all. Big data, they say, may be useful, but its impact will be limited. The dangers of data—ranging from security and privacy risks to “big data hubris” to misleading correlations—could ultimately outweigh the benefits. Businesses should be reluctant to throw everything out and capitulate to this new Silicon Valley fashion.

None of these camps are right. To be sure, there are important practical, political, and moral questions about whether, how, and to what extent to implement any new technology. There are even political and moral questions about whether a new technology is worth embracing in the first place (think weapons technologies, genetic engineering, or others). Too often we talk as if the widespread adoption of new technologies is inevitable, implicitly granting a technological determinism that few would defend if pressed.

But when it comes to big data, we should take stock of where things stand. First, there are vast amounts of data being stored, communicated, and transmitted at this very moment—so much that current metrics strain to make sense of it all. Most of this is “dark data” meaning that, as in the cosmos, what we know is vastly outstripped by what we don’t know. This is an empirical fact, not a prediction about a utopian or dystopian technological future.

Second, the near term will witness an explosion of even more data. Take that prediction to the bank. One reason is that data are already being commoditized and volumes are expanding apace. As businesses (and governments) increasingly realize, new analytics tools enable unexpected—even serendipitous—discoveries from large data sets. A given enterprise may not know exactly for what purposes its data could be used, but what was once dead data to be stored is now a treasure trove of potential information to be parsed, packaged, or sold. There is a growing incentive to hoard data.

Perhaps more importantly, smart technology is finding its way into ever more domains of enterprise and social life. Oil companies record staggering amounts of data to know what is happening with their operations in real time. GE alone records astronomical amounts—again, not only because the data might have secondary value, but also because managers want to know what is happening on the factory floor or with their aircraft turbines while in use. As smart technology becomes ubiquitous, the amount of data generated will make current metrics appear quaint.

Prudence may indeed be necessary as we embrace these trends. But we must distinguish between two things: first, whether or not there is already a shift in our economic mode of production associated with big data. Second, whether and how one (as an individual or a corporate body) ought to take advantage of the tools made possible by this transformation.

All three camps are right in part: the enthusiasts about the magnitude of the change; the pessimists about some of the challenges; and the deflationists when they point out that new data tools will not magically change the world.

But the enthusiasts fail to see that technology cannot change human nature: we are fallible and finite creatures. In the empirical realm, our knowledge is always accompanied by an asterisk: “until further notice.” And we will never know everything. Consequently, good sense and intuition are critical. A holistic approach to data analytics is indispensable because we are human.

The pessimists are wrong that this transformation will mean the end of work, declining employment, or more undesirable jobs. There will be challenges—whether technical (how to store and parse), political (how to secure), moral (how to protect), or social (how to adapt). Addressing them will take time and effort. But techno-economic creative destruction in the past has always produced more employment and an overall increase in income in the long term. This time is no different when it comes to people who argue “this time is different.” Accordingly, proposed solutions to big data challenges should take care not to stifle this process, e.g., with regulatory schemes aimed at limiting the collection of non-governmental data, even while targeting specific areas for improvement.

The deflationists are wrong because they see the trees for the forest. Big data isn’t about making predictions (or not); it’s about something—forgive the pun—much bigger. It represents an emerging techno-economic paradigm. The rise of the railroad and telegraph changed not only transportation, travel, and communication, but also business and management practices, regulatory regimes, and social mores. The era of big data will do no less. 

While recognizing the limits—whether technical, political, moral, or human—of the new tools furnished by big data, we ought to embrace this new paradigm as the locus of much-needed economic growth and the engine for the next cycle of innovation.