After the Auditor General’s (AG) report was released on April 30, the national media were quick to draw attention to the more than $3 billion in anti-terrorism spending that was unaccounted for and the large number of overpayments to employment insurance recipients.
Sadly, the chapter entitled, “Status Report on Evaluating the Effectiveness of Programs,” attracted no media attention although it arguably could be characterized as the most important chapter for two reasons.
First, this chapter updates an earlier assessment of how well the federal government monitors the effectiveness of its more than $280 billion annual spending. Second, the evaluations that are being audited in this chapter form the basis, in theory, for either justifying continuing program expenditures or for abandoning underperforming programs. As a result, the health of the evaluation function is more fundamental to good government practices than problems with individual program spending.
Program evaluation has a long-standing and proud place in Canadian public administration. The federal government was among the first in the world during the 1970s to issue an evaluation policy and to create evaluation units in most departments. Since that time, the Treasury Board Secretariat (TBS) has revised the policy four times to reflect changes in performance measurement systems, comptrollership functions, and deputy minister accountabilities.
The 2009 revision to the policy staked out an ambitious program with three objectives. These were, first, to review all ongoing programs of grants and contributions in order to assess their relevance and effectiveness every five years. Second, to evaluate all direct program spending for relevance and performance every five years. And, third, to address five core questions in those areas of direct program spending: whether there is a need to continue the program; if the program aligns with government priorities; if the program falls under federal jurisdiction; whether the program has achieved expected outcomes; and if there is evidence of the program’s efficiency and economy.
Within this context the evaluation chapter in the Auditor General’s report sends a cautionary note to Parliamentarians. In essence, the chapter examines whether the federal government, after four years of preparation, is in a position to fully implement the 2009 evaluation policy.
Uncharacteristically, the Auditor General’s study is a rather modest effort given the far-reaching importance of the audit. Instead of looking at all federal departments, the audit team limited its work to scrutinizing the TBS’s evaluation group and three of the six departments that had previously been audited in 2009. While this is an unrepresentative sample of the full range of federal departments, it does send a “canary in the mineshaft” message that the new policy does not appear to have taken hold within the departments. As a consequence, there is a high probability that when Parliamentarians finally turn their attention to program performance, they will not have the evidence they need to assess program spending.
Written in the most careful way, the AG is signaling that departments are not fully complying with the 2009 evaluation policy. More specifically, the AG notes that there has been some significant progress on grants and contributions but poor progress has been made on evaluating major departmental programs. The AG also suggests that the current self-evaluation reporting practice of how much has been accomplished is misleading and “the TBS should review the tools it uses to monitor evaluation coverage to ensure that they provide an accurate perspective on departmental progress.”
Most important, the evaluation findings are not being used to support spending decisions and “the departments are not fully realizing the potential benefits of presenting evaluation results in Treasury Board submissions.” Finally, the Auditor General also advised the Centre of Excellence for Evaluation in TBS to provide greater leadership in tracking evaluation coverage of direct program spending, in assessing efficiency and economy, and in conducting neutral assessments of their evaluation functions.
One day, parliamentarians might be interested in linking program performance to ministerial requests for annual spending appropriations. While it is not likely to happen soon given the way in which the House of Commons holds the government to account, it is still imperative for the public service to have this crucial information available to legislators.
In the meantime, deputy ministers might take a close look at the performance of their Departmental Audit Committees (DACs) and look for ways in which the DAC might serve as a model for ensuring that the 2009 evaluation policy is properly implemented.
