As part of the Public Policy Forum’s university lecture series, Mel Cappe, the former Cabinet Secretary to Jean Chretien (1999 to 2002) challenged a large audience at the University of Ottawa in late October to contemplate the implications of the current supply and demand for public policy in the federal government. In his presentation, Cappe argued that at this time there is an oversupply of policy analytical capacity within the public service, but not sufficient demand from the political level to maintain a comfortable point of equilibrium. One concern with this unstable arrangement is that over time the expertise and supply of policy analysts will atrophy and ultimately disappear.
At the same time as the issue of the supply of government policy analysts has become a matter of public debate, there are a number of new technology-related developments that are currently reshaping the world of analysis. The most notable of these transformative developments is a phenomenon known as big data. The term refers to new techniques for collecting and analyzing huge bodies of data that are contributing to more effective ways of making sense of our world in ways that we are just starting to appreciate. At its core, big data is improving our ability to predict outcomes with more certainty than ever before and is changing decision-making in every sector of society.
Most of us are aware of the importance of the so-called information revolution. However, its scale is so massive and the move to digital format has been so rapid that it is difficult to appreciate how much information we are talking about. For example, in 2000, only a quarter of the stored information in the world was digital and the balance was on paper, film, vinyl LP records, magnetic cassette tapes, and so on. Today, in 2013, the amount of stored information in the world is estimated to be around 1,200 exabytes (an exabyte is one billion gigabytes and a gigabyte is a billion bytes) of which less than two percent is non-digital. Thus, the world is now a digital one with massive amounts of data available for analysis to anyone with the wherewithal to access the information.
Most people are familiar with big data but have not necessarily recognized it as a new, game changing phenomenon. A good illustration of the power of big data is Amazon’s uncanny ability to provide immediate and relevant suggestions for further purchases once a consumer has placed an order for a book or CD. Amazon’s access to past purchases and to people with profiles that match individual consumers allows them to estimate instantaneously and in a seamless way what we would like to purchase next. Their predictive model of consumer behaviour is a very simple illustration of the power of big data.
The implications of big data for government are profound. With access to large databases from disparate sources around the world and the astounding number crunching capacity of super computers, big data will revolutionize policy analysis and will have disruptive effects on all those who have “policy analyst” in their job description.
There are several reasons for this. First, since big data represents vast amounts of data brought to bear on a given issue, analysts will no longer have to rely on sampling and statistical tests to verify their results. Second, given the massive size of the data sets, future analysts will be less interested in “exactitude” in their data and will be more accepting of “messy” results because the large number of data points will allow for probabilistic outcomes. Third, and most important of all, future policy analysts will have to forget most of what they learned in graduate school about causality and measurement and develop a new mindset where correlations and predictability are more important than hypothesis testing. One consequence of this development is that there will be less demand for experts in given policy fields and greater reliance placed on people who can aggregate information news websites such as Zite, which has no journalists on staff.
Cappe called on the policy community to create more demand for their work by doing things differently. Big data gives the policy community ample opportunity to hone new skill sets for a time when the demand for policy analysis increases. Now is a good time to experiment with big data, to train the next generation of policy analysts with new skills and to anticipate the many implications of bringing big data to government policy analysis, including the privacy concerns of citizens and their loss of anonymity.
