7 Ways the Government Can Improve its Open Data Policies

October 21, 2014

The process of making government data easily available to the public is likely to be long and complicated. Yet it is tremendously important.

A recent survey by GovLab identified over 500 private businesses that depend on government data for their business success. Many more are expected in the future. These businesses are likely to add a great deal of social value merely by taking something that the government already collects and making it more accessible, combining it with other data, and/or using it to make better decisions on problems ranging from air traffic control, farm production, plant location, and export opportunities. Efforts to increase data transparency also provide the Administration with a welcome contrast to the unmonitored and sometimes illegal data practices of the intelligence and law-enforcement agencies.

An Executive Order issued May 9, 2013 and a series of follow-up memoranda by the White House and the Office of Management and Budget require agencies to collect and present information in a way that supports its later use and dissemination. As a first step toward increasing the public value of government information the Department of Commerce held an Open Data Roundtable with 21 private companies and 4 non-profits to explore ways to increase the value of open data. It was perhaps natural that Commerce should take the lead.

The Department of Agriculture held a similar event on August 1, 2014) in this effort since it oversees a number of major data agencies including the Census, the Patents and Trademarks Office (PTO), the Bureau of Economic Analysis, and the National Oceanic and Atmospheric Administration. Its current Secretary is also a former private company CEO well attuned to the needs of the business community.

Earlier this month GovLab released a summary of this event and follow up actions. GovLab identified seven key areas where government data policies could be improved:

  • Data Discovery and Findability – Private interests cannot use government data if they do not know that it exists or where to find it. Given the vast amount of data that the government collects and the large number of agencies that collect at least some data, even experts often are not aware of all the data available about their industry. Efforts here include making datasets easy to find, providing metadata in a form that can be queried programmatically, and providing the proper context and documentation for each dataset, including a contact number for the person responsible for maintaining the database.


  • Data Access – The needs surrounding data access are varied. Different users require different ways of accessing data. Providing different methods of access will help insure equality of access regardless of a user’s technological capabilities. Agencies should therefore allow downloads in multiple formats. They should also move from Portable Document Format (PDF) systems to all-digital text-searchable formats.


  • Data Quality – Agencies also need to address concerns about the completeness, validity and accuracy of the data. In general, government data is often more comprehensive than other sources, although it does sometimes suffer from a time lag. However, in many cases, such as Census data, the government does not release all of the data it has because of privacy concerns. Yet this detailed information is often the most valuable to private users. Attempts to bridge these concerns need to continue. In other cases, such as PTO, private users would benefit if the agency collected more information such as the identity and location of current patent holders.


  • Data Collection and Sharing – The report suggested that government could also increase the value of its survey data if it collected it more frequently and shared it more broadly through public-private collaborations that combined private and government data. Because certain benefits may only come from combining data collected by both the private sector and government, the report characterizes the act of doing so as an opportunity for “data philanthropy.” However, while it is true that private entities collect a wide amount of data, they often regard it as proprietary and may be reluctant to share it with others at little or no cost, even though the combined data sets may be much more useful than either one alone.


  • Data Interoperability – Data is more valuable when it can be combined with other data. In order for this to occur, agencies have to promote and use standards and promote application programming interfaces. One suggestion was to develop a better, common taxonomy and metadata standards that match industry standards. For instance, it would help if different agencies used the same geographical boundaries or income thresholds when collecting data that are broken down by these variables.


  • Data Storage and Dissemination – The report recognized that agencies face financial and operational challenges in storing and making accessible their data. However, it suggested that partnerships with the private sector could address these needs. Specifically, a system where the government publishes data but relies on the private sector to develop the applications needed to use the data may increase data availability without increasing government expense.


  • Data Users as Customers – Perhaps the most important need is for government to view the private sector as customers of its data use. This need applies equally to the intelligence and law enforcement agencies. Rather than view companies and individuals as potential terrorists, opportunists looking for an excuse to criticize government, or the undeserving beneficiaries of data the government provides, all agencies should view themselves as serving the public and as needing public support in order both to increase the value they deliver to society and to build support for obtaining the budgets necessary to deliver that value. Virtually all individuals and companies share an interest in maximizing the value of government data. Yet it is difficult for them to support government efforts if they do not know what data is being collected or how it is being used and if they are not able to use the data for their own purposes.


Working with GovLabs, the Commerce Department has been a leader in the efforts to increase the social value of government data. But it has a long way to go. Other agencies also need to be equally determined to respond to the recommendations listed above. Finally, the Administration needs to realize that, in order to be credible, its efforts to address public concerns about openness, privacy and the use of government data m apply equally to all agencies, not just some.