The Canadian federal government has had an internal program evaluation system in place for over 35 years. Virtually all departments and agencies with budgets over one billion dollars have an evaluation unit that regularly examines the relevance and performance of their programs. Of the federal organizations with smaller budgets, about half appear to have a program evaluation function.
Internal evaluation studies help federal organizations identify ways in which specific programs can be improved or program portfolios adjusted. However, the internal evaluation system rarely produces the broad, high-level evaluative information needed for major program and policy decisions by Parliament or the Prime Minister and Cabinet. We perceive a need for specific structures to provide strategic evaluative information to government decision makers.
The Evaluator General concept
In 2007, Canadian Government Executive published an article outlining the idea of an Evaluator General for Canada. The authors envisioned an agent of Parliament who would provide evaluative evidence related to issues under debate. The Evaluator General position would parallel that of the Auditor General. Whereas the Auditor General investigates major issues in terms of compliance with accounting rules, an Evaluator General would investigate issues in terms of expected results achievement. Like the Auditor General, the Evaluator General would aim to provide Parliament with timely, strategic-level information on high profile issues.
Over the past 10 years, advocates of the Evaluator General idea have sought to increase its visibility by meeting with Parliamentarians, making presentations, writing newspaper articles and creating a website.
Recent developments
In recent years, there have been clear indications of interest in improving systems that provide evidence to support decision making in both Parliament and Cabinet.
In 2017, revised legislation strengthened the role of the Parliamentary Budget Officer. This key, arms-length advisor is now a full officer of Parliament and reports directly to the Speaker of the House.
Government has also created and staffed a position of Chief Science Advisor. The mandate for that position includes advising on processes to ensure that scientific analyses are considered in government decisions.
A “Policy on Results” came into effect in July 2016. It combined the former Policy on Evaluation with policies on performance measurement and reporting. Some observers suggest that the combining of policies has led to an overemphasis on program metrics and less attention to the in-depth evaluation of programs that could provide information useful at the strategic level, such as comparisons of program cost-effectiveness. However, a directive on experimentation subsequently released by government could, in principle, increase the availability of strategically useful information. The directive, issued in the context of government’s plan to devote a fixed percentage of program funds to experimenting with new approaches, sets out expectations for serious, professional measurement of the impact of novel programming.
New systems have been set in place for tracking the progress of government commitments and ongoing programs. At this stage, the publicly-accessible information that is being generated by these tracking systems is primarily descriptive. Progress tends to be reported using terms such as “underway on track” or “attention required” rather than with hard data that could be useful to Parliament. However, there is some evidence that the tracking systems may be providing Cabinet Ministers with useful data on results.
In 2016, an article promoting the idea of an Evaluator General for Australia appeared in The Mandarin magazine. The vision for an Evaluator General presented in the article differs in many respects from the one being advocated in Canada, but it reflects the same interest in improving the usefulness of evaluation to strategic decision making. The article has influenced thinking within government. For example, in 2018, officials of the Department of Industry, Innovation and Science, when commenting on a draft review of the Public Governance, Performance and Accountability Act, called for a stronger focus on evaluation, rather than audit, as the function for assessing program impact. They proposed the establishment of an Australian National Evaluation Office headed by an Evaluator General.
Parliament needs to access the world’s evaluative information
According to one estimate, there are over 1,600 program evaluation reports from departments and agencies in an online, open-access database maintained by the Treasury Board Secretariat. If these reports on internal evaluations by departments and agencies contained enough information to support strategic decision making, our Parliamentarians could simply ask their staff to search the public database, extract reports and summarize findings.
However, issues debated by Parliament tend to be complex. They usually transcend individual programs and require information on cost-effectiveness that internal evaluations may not provide. Subjects under debate by Parliament are frequently interdepartmental, interjurisdictional or international in scope. Relevant information lies not only in internal evaluations of federal programs but also in evaluation reports from around the world.
Extraction of useful evidence from the vast amount of available evaluative information requires third-party expertise in assessing the credibility of source evaluations, the validity and reliability of their impact measures, and their applicability to specific Canadian contexts. Further, Parliament may sometimes need quick, original evaluative studies to provide additional evidence before voting on an issue. Members of Parliament may also request longer-term evaluation studies to provide support for future decision making.
The time is right for a new mechanism to provide Parliamentarians with strategic level advice on evidence related to the proven or potential effectiveness of major programs or policies that government or private members put forward for consideration. With a triumvirate comprised of the Auditor General, the Parliamentary Budget Officer and an Evaluator General, Parliament would have arms-length advice on the propriety of government spending, the credibility of government budgets, and the likelihood that programs and policies will achieve desired objectives.
Evaluative advice for the Cabinet
Just as Parliament at large needs the advice of an Evaluator General, so the Prime Minister and Cabinet need a Chief Evaluation Advisor, a position parallel to that of Chief Science Advisor. Compared to other Members of Parliament, the Prime Minister and Cabinet already have considerably greater access to providers of evidence. They are supported by the Privy Council Office and by Deputy Ministers with their networks of policy advisors, program managers and Chief Evaluation Officers. But currently they have no one who has a global perspective on evaluation and the capacity to lead rapid, highly focused studies to complement the existing evidence base. A Chief Evaluation Advisor reporting to the Prime Minister and the President of the Treasury Board could fill the gap.
A Parliamentary Evidence Officer?
Some thinkers envisage an Agent of Parliament who could assemble evidence for members of the House from many sources, not only from evaluation studies but also from scientific research and traditional knowledge. A “Parliamentary Evidence Officer” would have an expert staff of evaluators, science brokers and keepers of traditional knowledge to offer advice to Members of Parliament on the evidence base underlying proposals. But whatever the structure – Evaluator General, Parliamentary Evidence Officer or Chief Evaluation Advisor – greater access to evidence from systematic evaluation studies conducted around the world would improve the evidence base for strategic level decision making.
We need arms-length structures for generating advice on evidence from evaluation and supplying it to Parliamentarians, the press and the public in a timeframe and in a format that will make it useful to all concerned.
Michael Obrecht is an Ottawa-based Credentialed Evaluator with experience in federal public service and management consulting.
Michel Laurendeau is a leading consultant in performance measurement and evaluation of federal programs and policies.
Nicholas Gruen is a policy economist, entrepreneur and commentator, founder of Lateral Economics, Visiting Professor at King’s College London Policy Institute and Adjunct Professor at UTS Business School.
