Performance Measurement
September 27, 2012

Program evaluation: Meeting a new challenge

Beginning on April 1, 2013, and over a five-year cycle, the evaluation function in large federal departments and agencies will face a significant challenge, namely to contribute to budget planning by assessing program effectiveness for all direct program spending and major statutory spending.

While this challenge originates in the 2009 Treasury Board Policy on Evaluation, it impacts deputy heads as much as accounting officers. The scenario is clear: as deputies implement review targets under strategic review and the Deficit Reduction Action Plan, they want the best available evidence to explain and defend program results. Hence a robust demand for program evaluation.

The extent to which this scenario will play out is unknown, because the budgeting process is subject to Cabinet and budget secrecy. However, studies of the budgeting process have rated the federal government poorly in terms of attention to program performance and effectiveness: the spending departments focus on developing expenditure proposals and linking them to government priorities, rather than program performance. It is well known that the program evaluation function did not make much of a contribution to earlier program reviews in the mid-1980s and 1990s. Moreover, there is some indication that departments have lacked evidence about program performance when preparing for the current strategic reviews.

To meet the challenge this situation will have to change. The question is what influences the integration of evaluation into budgeting: what is required?

This article draws its inspiration from a government-wide audit of evaluation (Chapter 1, “Evaluating the Effectiveness of Programs,” Fall 2009 Auditor General’s Report), and a 2010 Treasury Board of Canada Secretariat (TBS) report on the health of the evaluation function. Below are five factors influencing the integration of evaluation into budgeting: capacity, quality, coverage, timelines and utility.  
 
Capacity is the question of how to ensure sufficient, qualified evaluation staff and funding to meet the needs for program evaluation. In the three years following the enactment of the Federal Accountability Act, departments increased professional staff in evaluation units, and received some central funding to implement the Act, with its requirement for evaluation of grants and contributions programs. Despite these gains, departments have experienced a shortage of experienced evaluators, and have faced difficulties finding experienced professional staff, particularly at senior levels.    

Quality can be assessed in terms of coverage of issues, validity and reliability of methods, and the independence and objectivity of reporting. While studies sponsored by TBS have rated most program evaluations as acceptable in quality, the assessment of program effectiveness also depends upon reliable data on performance. In many cases, evaluation has been hampered by unavailable or low-quality data.

Coverage requirements for grants and contributions are being met, in compliance with the legal requirement. In terms of overall direct program spending, it remains a challenge for departments to evaluate an average of 20 percent of spending each year. In particular, full coverage calls for a change in planning practices, since evaluation efforts can no longer be targeted on the basis of risks.  

Timeliness is the extent to which evaluation reports can be made available at the appropriate time in the annual budget cycle or when needed for other types of expenditure review. It has been identified as an underlying reason for the lack of use of evaluation findings in budgeting. There is a need for coordination with evaluation planning, which is carried out on a rotating five-year basis and requires Treasury Board approval. In addition, evaluations of larger, more complex programs can take more than one year to complete, potentially rendering them unavailable for a given annual cycle.

Utility refers to the use of evaluation findings in budgetary decision making: the question of fit or alignment. Some officials have viewed program evaluation as detailed, longer-term research, not always well suited to the less formal evaluations used in budgeting. In addition, budgeting tends to focus on fiscal aggregates, notably the budgetary balance and the elimination of the deficit; however, expenditure review targets can bring it back to the level of programs.  

In conclusion, the successful integration of evaluation in budgeting is a work in progress. Many of the necessary steps are known, and achievable. Overall, program evaluation can be better aligned with the budgeting, by tailoring evaluation studies to the needs of the budgetary process.


Tom Wileman, a principal with the Office of the Auditor General of Canada (retired), led the 2009 government-wide audit of program evaluation. He is a board member of the Performance and Planning Exchange (PPX).

About this author

0 comments

There are no comments for this post yet.

Be the first to comment. Click here.

Performance Measurement
 
Leadership, service delivery, change management, performance measurement, big data, transparency, digital...
 
Recently, there has been considerable interest in what is being referred...
 
Over the years, governments have been adopting Lean management principles to...
 
As the resettlement of Syrian Refugees exemplifies, government programs, currently running...
 
With businesses and government agencies increasingly operating in a highly information-based...
 
Risk is always present in any undertaking, no matter the size...
 
The Government of Canada’s newly announced Policy on Results replaces the...
 
In this episode, J. Richard Jones share information about the Canadian...
 
Rules and accountability are helpful in developing and standardizing processes but...
 
For several months now, the Liberal government ministers have been using...
 
Global market trends are accelerating to increase the pressure on commercial...
 
As a consultant, I work primarily with private sector organizations including...
 
Here’s some bad news for organizations contemplating projects that have something...
 
If the purpose of the Government of Canada’s performance measurement system...
 
Performance auditing can lead to more efficient, effective, and economical program...
 
The latest Auditor General’s report on Shared Services Canada (SSC) and...
 
Public sector innovation, once dismissed as a contradiction in terms, is...
 
The necessity for government executives to manage change, and achieve results,...
 
Every organization does it, but often with ambivalence: the performance measurement...
 
There is an apparent disconnect in performance measurement. Today, agencies measure...
 
The government of Canada’s transformation agenda continues to drive change across...
 
Cost savings in government programs and services used to be achieved...
 
As we move into 2015 – with a looming federal election...
 
Over the last four decades, public sector organizations have institutionalized program...
 
When you purchase clothes, do you like off-the-rack, or do you...
 
All governments, including Canada’s, strive to manage taxpayers’ dollars more efficiently....
 
Getting a clear definition of policy processes and functions can be...
 
While government agencies do not have “bottom lines” in the way...
 
Most of you already know that project management is a means...
 
The taxpayers are wary. Politicians worry about retaining their jobs. Everyone...
 
There has never been a time since the introduction of program...
 
The management of resources has always been taken for granted in...
 
In the last two decades, managing for results has gained momentum...
 
For the first time, the Consortium of Universities for Evaluation Education...
 
The government of New Zealand, and the State Services Commission in...
 
While it has been around for over 130 years, the concept...
 
Performance measurement and public reporting are intended to achieve two complementary...
 
Canada has over 400 administrative tribunals that are probably the least...
 
Last summer, the Vice Chief of the Defence Staff (VCDS) signed...
 
The core mandate of the Consortium of Universities for Evaluation Education...
 
the author fails to provide a complete picture (the good and...
 
Why talk about ethics at all right now? After all, Canada’s...
 
we now have what one federal ADM described to me as...
 
The reduction or mitigation of risks or harms is a major...
 
What are Canadian universities doing to maintain their relevance and innovative...
 
Managing for results has gained significant momentum in the public sector...
 
“Is performance management still relevant?” was the thought-provoking question posed by...
 
Within the last year, the Treasury Board of Canada Secretariat (TBS)...
 
Results matter in the public sector, particularly in these challenging times...
 
Today’s public sector environment is one in which many managers and...
 
Just as with public financial reporting, performance reporting is here to...
 
Being a high-performing organization is like being a high-performing individual: it...
 
With the current reality of deficit reduction, cost cutting and overall...
 
This article is New and Improved! After an exhaustive review cycle...
 
Canada was at the table when the “Commonwealth of Nations” was...
 
If you believe that performance measurement is a vital part of...
 
The last great international effort to refashion the public sector took...
 
Beginning on April 1, 2013, and over a five-year cycle, the...
 
Governments are increasingly using benchmarking to improve organizational performance; to determine...
 
Senior public sector managers, deputy heads of departments and parliamentarians need...
 
Canada is a hockey-loving nation and now is the time of...
 
Too few public sector organizations have truly embedded performance thinking into...
 
CGE Vol.13 No.3 March 2007 Governing is never easy. And in...
 
CGE Vol.13 No.2 February 2007 Improving organizational health and employee engagement...
 
CGE Vol.13 No.1 January 2007 The proposed Federal Accountability Act is...
 
CGE Vol.14 No.2 February 2008 The passage of the Federal Accountability...
 
CGE Vol. 14 No.4 April 2008 The Tellier-Mazankowski advisory committee on...
 
Performance management leads to either disdain or glee in public sector...
 
Les administrations publiques utilisent de plus en plus l’analyse comparative pour...
 
The redesign of corporate tax administration for Ontario is a major...
 
Coincidental with the push for more attention to be paid to...
 
The BC Public Service has had an intense focus on improving...
 
La reconception de l’administration de l’imposition des sociétés en Ontario est...
 
Dans son budget de lutte contre la récession (et d’économies dans...
 
The Harper government’s recession fighting (and government saving) budget heralded an...
 
Many governments have bought into the notion of the Balanced Scorecard,...
 
David P. Norton est le cocréateur, avec Robert Kaplan, du Tableau...
 
Together with Robert Kaplan, David P. Norton is the creator of...
 
Il y a dix ans, la Société canadienne du sang (SCS)...
 
Début septembre, la fonction publique de l’Ontario (FPO) célébrait le 10e...
 
Bon nombre d’administrations publiques ont adopté le tableau de bord prospectif,...
 
Today’s public sector managers encounter increasingly complex corporate issues that call...
 
Performance management seems to be everywhere these days, with much of...
 
The two key tools used by the Canadian government to measure...
 
The principle is unassailable: Canadians have the right to know whether...
 
L’administration publique s’efforce d’obtenir des résultats tangibles en mettant en œuvre...
 
Une étude récente commandée par l’OCDE conclut que, malgré le fait...
 
A recent study commissioned by the OECD concludes that while the...
 
Some title Some author
Some excerpt
 
Some title Some author
Some excerpt
Leadership, service delivery, change management, performance measurement, big data, transparency, digital...