In antiquity, much of the world's recorded knowledge was located in one spot: the Library of Alexandria in Egypt. Nearly every culture and empire of that era recognized the library as the epicenter of scholarship. Today, however, information is increasingly decentralized. Alexandria's vast repositories of papyrus scrolls could now fit onto a single flash drive. In the 21st century, digital information is being created, analyzed, and stored at an astonishing rate. Consider that 90% of the world's data has been produced in just the last two years. This explosion of information is known as “Big Data,” and it is completely transforming the world around us.
Big Data is already an integral part of every sector in the global economy—as essential a factor of production as physical and human capital. Much of our modern economic activity simply could not function without it. The main driver of Big Data is the nearly incalculable amount of transactional data being produced by companies, financial institutions, and online intermediaries. This includes trillions of bytes of information about buyers, suppliers, and operations of critical interest to businesses and financial analysts.
As important as Big Data is today, its impact will be even more pronounced in the near future. The Big Data market is expected to grow to $16.9 billion in 2015. Gartner predicts that the expansion of the information economy will produce 4.4 million tech jobs globally. This growth will be amplified by a multiplier effect: each IT job created by Big Data will generate three more non-IT positions.
Yet working with Big Data can be intimidating, even for tech veterans and savvy startups. Because Big Data is so, well, big, it can be a difficult concept to comprehend, let alone explain. To be certain, Big Data is more of a dynamic than a “thing.” An oft-cited McKinsey Global Institute (MGI) report defines it as “large pools of data that can be captured, communicated, aggregated, stored, and analyzed.” We might also add that this data is often so large that it exceeds the processing capacity of traditional database systems and analytics software.
Unlocking the potential of Big Data is a puzzle for business executives and entrepreneurs, but also an opportunity. Large data sets and sophisticated analytics can create new products, enhance existing services, significantly improve decision-making, mitigate and minimize risks, and produce valuable insights about operations and consumer sentiment.
Social, Mobile, and the Cloud
The next few years will represent a big tipping point for Big Data in terms of technology, market share, and economic impact. Companies already adept at data accumulation and analysis will secure their market share, while start-ups and disruptors will push aside also-rans. Peter Sondergaard argues that this transformation is being driven by the convergence and mutual reinforcement of mobile, social, and cloud technologies.
Social networks generate a tremendous amount of personalized, location-specific data about users that is tremendously valuable to businesses and marketers alike. Although networks such as Facebook are finally reaching the boundaries of audience growth, many businesses are only now integrating social media into the core of their operations. Gartner predicts that in three years, at least ten organizations will each spend more than $1 billion on social media. That investment should pay off as social network data is linked with offline and localized data to create new, highly-optimized marketing opportunities.
The convergence between these trends becomes clear when you consider that mobile devices are increasingly the point of entry for social networks and many other applications. The Economist notes that the number of mobile phones and tablets surpassed the number of laptops and PCs for the first time in 2011. By 2020, the number of mobile connected devices operating across the globe should reach 10 billion. The data emanating from mobile phones holds particular interest because it can be linked directly to individual users in identified locations.
The transformation of mobile devices into mobile wallets represents the most important link between Big Data and the consumer experience. Apple, Google and Microsoft are all making investments in near field communication (NFC), a short-range wireless technology that allows for the automatic transmission of data between devices in close proximity. With a single tap, an NFC-enabled phone can download product specifications and coupons, complete payments at the register, and update shopping points in loyalty programs. The technology still has room to improve, but retailers and credit card companies see an opportunity to develop in-store shopping apps and flexible mobile payments options.
Cloud computing is a perfect complement to Big Data (and mobile data) since it offers unlimited storage space on demand. Cloud users do not have to commit to expensive software or a hardware infrastructure expansion—they can simply rent the resources they need as they go, only paying for what they use when they use it. Cloud computing usage has grown exponentially over the last few years. According to a recent IBM survey, most companies anticipate the cloud will replace on-premise servers for most or all of their needs by 2015. Most importantly, the combination of mobile devices and cloud servers makes Big Data completely decentralized. Big Data is now truly everywhere—and on the move.
Along with increased mobility comes context-aware computing—a mix of location awareness, social network integration, and smartphone hardware such as cameras and microphones. These components currently operate separately, but the next leap in mobile computing will occur as they begin to interoperate and connect. Today, less than 1% of all physical objects are connected to the digital network. Yet that number will only grow as connected devices become ubiquitous, and previously low-tech industries become digitized. This new wave of tech development will enhance the way we interact with all digital devices—but it will also put a new focus on existing privacy and security concerns.
Personal and private data is collected about consumers all of the time with little to no publicity, leading some observersto cite the specter of Orwell’s Big Brother. Digital skeptic Nicholas Carr argues that Big Data could soon produce a “nightmare world” where “moment-by-moment behavior of human beings—analog resources—is tracked by sensors and engineered by central authorities to create optimal statistical outcomes.” The promise of Google Glass also suggests a world in which we may be recorded at any time, without our knowledge. Alistair Croll at O'Reilly Radar makes the case that Big Data is a civil rights issue: the same technology that can personalize marketing can also be used to discriminate against particular demographics. And as one might suspect, this could eventually result in restrictive federal regulations.
There is certainly historical precedent for such policy. In the 1960s, Congress was forced to take action in response to public outcry about “redlining”—the banking practice of marking a red line on a map to delineate neighborhoods where they would not invest. The Fair Housing Act of 1968 prohibited redlining based on race, religion, sex, familial status, disability, or ethnic origin. Complaints concerning telemarketing calls to homes led to the establishment of the National Do Not Call Registry in 2004. Three years later, 72% of Americans had registered their phone numbers on the list—completely decimating the telemarketing industry. Companies dependent upon Big Data could experience a similar decline should data collection and application become restricted behind red tape and litigation.
Businesses would be well advised to stay in the good graces of both the public and legislators by developing internal privacy best practices that aim to keep your databases secure, limit the exposure of your customer information, and make sure the information you collect on consumers doesn't turn into a public relations disaster.
The significance of Big Data to businesses, marketers, and the entire global economy is well established. One important caveat hasn't been emphasized enough though: without strategic vision and careful planning, even a treasure trove of Big Data will only serve to distract business operations rather than drive business success. Indeed, the problems previously associated with “small” data—unreliability, lack of context, relative value—are only amplified as they grow larger. As The Economist has sagely observed, Big Data is ultimately ”no substitute for sound intuition and wise judgment.”
Business leadership, then, is the critical link between data and data-driven success. Skillful executives will seek out the best tools, the best people, and the best information. Through forward thinking planning, today's companies can become tomorrow's data leaders.