Opportunity Ahead – Capitalizing on the High Demand for Big Data Skillsets

The great need for science, technology, engineering and mathematics (STEM) education in this country has been clear for years. From the school house to the White House, America’s leaders, businesses, and teachers have emphasized the growing skills gap that leaves U.S. companies without the high-tech talent they need. With STEM jobs growing rapidly and STEM skills not growing fast enough, the implications for the Big Data revolution are concerning. 

Data does nothing on its own; it is the people who analyze it and put it to work who allow businesses and governments to realize the potential from it. This can only happen, however, if there are enough people with the right skills. Yet, the notion of STEM skills is abstract. Saying America needs more scientists and engineers is like saying we need regulatory and tax reform. Of course we do, but the specifics are unique to the many different challenges business and individuals face. When it comes to realizing the fruits of data-driven business and innovation, there are specific skillsets and educational backgrounds needed to put to work insights gleaned through massive amounts of information. 

A report from McKinsey, “Big Data: The Next Frontier for Innovation, Competition, and Productivity,” found that, by 2018, the demand for statisticians, engineers and other “deep analytical talent” will exceed U.S. supply by 50% or even 60% (or between 140,000 and 190,000 positions). What is more, there will be a need for 1.5 million more managers who can understand various data sets and put it to work. As the report authors wrote, the United States “cannot fill this gap simply by changing graduate requirements and waiting for people to graduate with more skills or by importing talent…It will be necessary to retrain a significant amount of the talent in place.” 

These numbers are sobering. The kind of knowledge that is needed, however, is not a single set of technical skills but many sets. Businesses don’t just need a big data employee; they need big data teams that can provide a full aperture understanding of what data can tell us. Chief Scientist for Lithium Technologies, Dr. Michael Wu, told InformationWeek that there are three core types of data positions. In the simplest terms:

    • The Data Engineer collects customer or business data and processes it; they are the starting point for data-driven insights and require the technical expertise to work with “back-end” servers, systems and data streams.
    • The Machine Learning Expert takes this information and builds technical models that reveal the much-championed fruits of Big Data; they require skills in statistics, working with algorithms, and an ability to translate numbers into comprehensible insights the rest of the business can use.
    • The Business Analyst then uses tools and programs to integrate data insights into the company’s functions; they have some programming skills, a keen understanding of how data informs decision-making, and an appreciation for the business’ operation.


As noted in the McKinsey report, businesses also need managers who can oversee and guide the data teams and how their insights are implemented. Good managers need not be able to do everything their subordinates do, but they need the knowledge and understanding that allows them to effectively guide their team. As McKinsey stated, this takes training but not the formal education and years of study required for the core data positions. 

This is a glass-is-half-full moment for America. There will be a shortfall in the number of people ready to work with data, but this creates an opportunity to outcompete other companies and job seekers (and America runs on competition). Businesses can find innovative ways to attract and retain talent, deriving data insights their competitors cannot. For future employees, recognizing the need for data talent means they can find formal degree programs that build the skills that are in high demand, which bodes well for landing a job after school. People already in the workforce can leverage the need for data skills to foster job security by seeking out the training that will allow them to be effective managers or business leaders in a data-driven world. 

There remains a serious need for the U.S. educational system to improve its ability to instill STEM skills, and it is important that our culture and public opinion also shift to appreciate how important, interesting, and lucrative STEM-based careers can be. In the meantime, the workers and companies that recognize the expanding and permanent presence of data-driven business and innovation will enjoy great benefits, growing their businesses and the opportunities that go with them. Hopefully their success will encourage everyone else to catch up and allow the United States as a whole to realize the benefits and increased competitiveness Big Data can offer to our future.