Big Data Can't Replace the Human Factor

June 18, 2014

There is sometimes an unwarranted fear of Big Data, a concern that information produced by the Internet of Things and all the technologies we use every day is reducing us to sets of 1s and 0s. Every day there seems to be more talk about what to fear about data rather than looking at the facts and good it provides to us.  The fear for many boils simply down to this question - Could Big Data come to replace human intelligence? 

The simple truth is data doesn’t work that way. The fact is data contributes to informed decision making, and as a recent study shows, that is only part of the equation. 

A recent report from the Economist Intelligence Unit, “Decisive Action: How Businesses Make Decisions and How They Could do it Better,” investigated how intuition fits into business executives’ decision-making process. In a survey of company leaders, the study found that 42% of respondents characterized their decision-making style as data driven, while 17% noted a primarily empirical (testing hypotheses) decision-making process. Just 10% reported a largely intuitive decision-making style. 

Yet, when asked what they would do if data contradicted a “gut feeling,” nearly 60% said they would reanalyze the data; 30% said they would collect more data; and, a meager 10% would ignore that little voice inside and do what the data says. 

What this tells us is that even as data-driven decision making is a growing private sector trend, it does not trump good, old fashioned human intuition. Fortunately, this is exactly how we should be approaching our ever-growing capacity to use data science in business operations, in part because sometimes the story the data tells can be wrong.

In a New York Times op-ed, Gary Marcus and Ernest Davis argue that data analysis can yield inaccurate results. They write: “If you look 100 times for correlations between two variables, you risk finding, purely by chance, about five bogus correlations that appear statistically significant — even though there is no actual meaningful connection between the variables. Absent careful supervision, the magnitudes of big data can greatly amplify such errors.”

Data quality also remains a challenge for analysts. Increasingly, businesses, scientists and Big Data advocates are focused on what some (such as Anjul Bhambhri, IBM's VP of Big Data Products) have called a fourth V of Big Data: veracity. 

This refers to data accuracy as well as source reliability, the context out of which the data comes, the methods for sorting and storing information, and a range of factors that can influence the data’s validity. This is already a large and time-consuming challenge. The Harvard Business Review reports that workers can spend up to 50% of their time looking for data, fixing errors, and trying to validate the numbers they have on hand.

While this shows the ongoing challenge of acquiring high-quality data, it also reveals just how important intuition remains, despite supposedly cold, hard facts. It is the uniquely human intuitive hunch, the gut feeling that can push a business leader to pause before acting on data analysis that just doesn’t add up. Without that human knowledge and wisdom, we could end up chasing Big Data red herrings. Instead, the data and the decision-maker must work together to produce the much-desired groundbreaking innovation or business insight.  That is very much a human role and one that cannot, nor should be replaced.

Being a good leader is about making good decisions. In every setting, leaders must use a mix of reliable information and experience to decide the best course of action. The growing saturation of data-generating technologies does contribute to an ocean of information that, when analyzed, can reveal new connections, trends and opportunities. Yet, no matter how good our data or ability to analyze it becomes, it will never replace the human mind. In the end, it will always be a person with a heartbeat (not an algorithm) that makes a decision.