Evaluation and Government Decision-Making - Canadian Government Executive
Uncategorized
September 25, 2018

Evaluation and Government Decision-Making

The Canadian federal government has had an internal program evaluation system in place for over 35 years. Virtually all departments and agencies with budgets over one billion dollars have an evaluation unit that regularly examines the relevance and performance of their programs. Of the federal organizations with smaller budgets, about half appear to have a program evaluation function.

Internal evaluation studies help federal organizations identify ways in which specific programs can be improved or program portfolios adjusted. However, the internal evaluation system rarely produces the broad, high-level evaluative information needed for major program and policy decisions by Parliament or the Prime Minister and Cabinet. We perceive a need for specific structures to provide strategic evaluative information to government decision makers.

The Evaluator General concept

In 2007, Canadian Government Executive published an article outlining the idea of an Evaluator General for Canada. The authors envisioned an agent of Parliament who would provide evaluative evidence related to issues under debate. The Evaluator General position would parallel that of the Auditor General. Whereas the Auditor General investigates major issues in terms of compliance with accounting rules, an Evaluator General would investigate issues in terms of expected results achievement. Like the Auditor General, the Evaluator General would aim to provide Parliament with timely, strategic-level information on high profile issues.

Over the past 10 years, advocates of the Evaluator General idea have sought to increase its visibility by meeting with Parliamentarians, making presentations, writing newspaper articles and creating a website. 

Recent developments

In recent years, there have been clear indications of interest in improving systems that provide evidence to support decision making in both Parliament and Cabinet.

In 2017, revised legislation strengthened the role of the Parliamentary Budget Officer. This key, arms-length advisor is now a full officer of Parliament and reports directly to the Speaker of the House.

Government has also created and staffed a position of Chief Science Advisor. The mandate for that position includes advising on processes to ensure that scientific analyses are considered in government decisions.

A “Policy on Results” came into effect in July 2016. It combined the former Policy on Evaluation with policies on performance measurement and reporting. Some observers suggest that the combining of policies has led to an overemphasis on program metrics and less attention to the in-depth evaluation of programs that could provide information useful at the strategic level, such as comparisons of program cost-effectiveness. However, a directive on experimentation subsequently released by government could, in principle, increase the availability of strategically useful information. The directive, issued in the context of government’s plan to devote a fixed percentage of program funds to experimenting with new approaches, sets out expectations for serious, professional measurement of the impact of novel programming.

New systems have been set in place for tracking the progress of government commitments and ongoing programs. At this stage, the publicly-accessible information that is being generated by these tracking systems is primarily descriptive. Progress tends to be reported using terms such as “underway on track” or “attention required” rather than with hard data that could be useful to Parliament. However, there is some evidence that the tracking systems may be providing Cabinet Ministers with useful data on results.

In 2016, an article promoting the idea of an Evaluator General for Australia appeared in The Mandarin magazine. The vision for an Evaluator General presented in the article differs in many respects from the one being advocated in Canada, but it reflects the same interest in improving the usefulness of evaluation to strategic decision making. The article has influenced thinking within government. For example, in 2018, officials of the Department of Industry, Innovation and Science, when commenting on a draft review of the Public Governance, Performance and Accountability Act, called for a stronger focus on evaluation, rather than audit, as the function for assessing program impact. They proposed the establishment of an Australian National Evaluation Office headed by an Evaluator General.

Parliament needs to access the world’s evaluative information

According to one estimate, there are over 1,600 program evaluation reports from departments and agencies in an online, open-access database maintained by the Treasury Board Secretariat. If these reports on internal evaluations by departments and agencies contained enough information to support strategic decision making, our Parliamentarians could simply ask their staff to search the public database, extract reports and summarize findings.

However, issues debated by Parliament tend to be complex. They usually transcend individual programs and require information on cost-effectiveness that internal evaluations may not provide. Subjects under debate by Parliament are frequently interdepartmental, interjurisdictional or international in scope. Relevant information lies not only in internal evaluations of federal programs but also in evaluation reports from around the world.

Extraction of useful evidence from the vast amount of available evaluative information requires third-party expertise in assessing the credibility of source evaluations, the validity and reliability of their impact measures, and their applicability to specific Canadian contexts. Further, Parliament may sometimes need quick, original evaluative studies to provide additional evidence before voting on an issue. Members of Parliament may also request longer-term evaluation studies to provide support for future decision making.

The time is right for a new mechanism to provide Parliamentarians with strategic level advice on evidence related to the proven or potential effectiveness of major programs or policies that government or private members put forward for consideration. With a triumvirate comprised of the Auditor General, the Parliamentary Budget Officer and an Evaluator General, Parliament would have arms-length advice on the propriety of government spending, the credibility of government budgets, and the likelihood that programs and policies will achieve desired objectives.

Evaluative advice for the Cabinet

Just as Parliament at large needs the advice of an Evaluator General, so the Prime Minister and Cabinet need a Chief Evaluation Advisor, a position parallel to that of Chief Science Advisor. Compared to other Members of Parliament, the Prime Minister and Cabinet already have considerably greater access to providers of evidence. They are supported by the Privy Council Office and by Deputy Ministers with their networks of policy advisors, program managers and Chief Evaluation Officers. But currently they have no one who has a global perspective on evaluation and the capacity to lead rapid, highly focused studies to complement the existing evidence base. A Chief Evaluation Advisor reporting to the Prime Minister and the President of the Treasury Board could fill the gap.

A Parliamentary Evidence Officer?

Some thinkers envisage an Agent of Parliament who could assemble evidence for members of the House from many sources, not only from evaluation studies but also from scientific research and traditional knowledge. A “Parliamentary Evidence Officer” would have an expert staff of evaluators, science brokers and keepers of traditional knowledge to offer advice to Members of Parliament on the evidence base underlying proposals. But whatever the structure – Evaluator General, Parliamentary Evidence Officer or Chief Evaluation Advisor – greater access to evidence from systematic evaluation studies conducted around the world would improve the evidence base for strategic level decision making.

We need arms-length structures for generating advice on evidence from evaluation and supplying it to Parliamentarians, the press and the public in a timeframe and in a format that will make it useful to all concerned.

Michael Obrecht is an Ottawa-based Credentialed Evaluator with experience in federal public service and management consulting.

Michel Laurendeau is a leading consultant in performance measurement and evaluation of federal programs and policies.

Nicholas Gruen is a policy economist, entrepreneur and commentator, founder of Lateral Economics, Visiting Professor at King’s College London Policy Institute and Adjunct Professor at UTS Business School.

Related posts:

0 comments

There are no comments for this post yet.

Be the first to comment. Click here.

Uncategorized
 
The Canadian federal government has had an internal program evaluation system in place for over 35 years. Virtually all departments and agencies with budgets over one billion dollars have an evaluation unit that regularly examines the relevance and performance of their programs. Of the federal organizations with smaller budgets, about half appear to have a...
 
The federal government has dramatically increased the scope and scale of its consultations and engagement in the past two years. The goal is to make it easier for public servants, citizens and stakeholders to work together to solve common problems and improve outcomes. My team at the Privy Council Office monitors the success and impact...
 
“Local government touches all of us intimately every day of our lives: the water we drink, the transit system that takes us to work, the first responders who keep our cities safe. Those services are only as good as the people who deliver them, and the quality of the people at the working level is...
 
Yesterday, Prime Minister Justin Trudeau and Nova Scotia Premier Hon. Stephen McNeil reached a bilateral agreement to invest in the province’s wastewater and public transit systems....
 
The association representing more than 42,000 physicians and medical students in Ontario has voted against a doctor’s fee deal proposed by the province. Following weeks of debate over the proposal, 63.1 per cent of the Ontario Medical Association’s members voted against the tentative Physician Services Agreement (PSA). Related posts: Good for business: Investing in Aboriginal prosperity Cuts to TTC will compromise service, increase air pollution says union Province asked to build fentanyl monitoring, coordination system...
 
Thank you for accessing Canadian Government Executive. You can now access the articles within the site....
 
In this month’s blog, I would like to test out an idea. I have been toying with this idea for some time now. As I share it around, some people seem to think it is interesting albeit disruptive, and others get excited about its potential. Today, I would like to make a pitch to you...
 
Ducks look like they have it All Together. Above the water, all is calm. They glide, expressionless and serene. Below the water, they’re paddling madly trying to stay afloat and to avoid getting eaten. It’s like they’re in an endless, hidden panic. We’re not so different from the ducks. If the world discovered that we...
 
I blogged in November 2012 about seeing Dr. Bernard Meyerson, VP Global Innovation, IBM, at that year’s GTEC. Dr. Meyerson spoke to why innovation matters and how it is linked to growth and growth happens when good people pursue grand challenges to accomplish the seemingly impossible. Last week while watching an episode of Second regard on tou.tv, a similar comment came up in an...
 
In many respects, the future looks pretty grim through the eyes of Canada’s public service leaders despite the country’s much heralded economic resilience....
 
Although we have much to be proud of, I’ve come to the conclusion that we’re missing a sense of pride and identity in the profession of public service in Canada....
 
What can the cloud do for you? IT in Canada is holding a free webinar series that will answer exactly that question. The first in the series will be held on February 19 and will feature Waterfront Toronto, an organization that oversees a revitalization project funded by the governments of Canada and Ontario, and the...
 
The solution revolution is a global trend and phenomenon which relates to the coming together of non-governmental players in trying to address and overcome some of society’s toughest problems. Related posts: Organizations must adopt policies to secure their BYOD data In interview with: Benoit Gauthier and the 'International Year of Evaluation' Canada adopts U.N. declaration of rights of Indigenous peoples...
 
Some of the world’s leading thinkers on globalization are pointing to a fundamental change emerging regarding how global problems can be solved, and perhaps how we govern ourselves globally....
 
The Natural Resources Canada Resource Wiki has set in motion a culture change at NRCan, enabling a more open, dynamic, innovative and collaborative way for employees to work together. ...
 
The last few months have been busy in Ottawa. Since June, Canadians have witnessed a series of events that bring into question the role of the federal government and the evolving status of the public service....
 
The public sector faces an issue that threatens its historic role and future utility. If the issue is not addressed effectively, we will witness an ever more rapid deterioration in relations between senior members of the civil service and their politic masters....
 
In 2010, I had the pleasure of leading an international research network of senior leaders to explore the new frontiers of public administration....
 
Crises are anything but new and novel. Why then, in recent decades, has the management of risk and crises engendered so much research in business schools and occupied so much time in corporate boardrooms?...
 
Some title Some author
Some excerpt
 
Some title Some author
Some excerpt
 
Some title Some author
Some excerpt
 
Some title Some author
Some excerpt
 
Some title Some author
Some excerpt
The Canadian federal government has had an internal program evaluation system in place for over 35 years. Virtually all departments and agencies with budgets over one billion dollars have an evaluation unit that regularly examines the relevance and performance of their programs. Of the federal organizations with smaller budgets, about half appear to have a...