Deficient data fails program evaluation – Canadian Government Executive

NEWS

SEARCH

AccountabilityManagement
May 7, 2012

Deficient data fails program evaluation

 

In a few weeks, the Minister of Finance will deliver the most anticipated federal budget since 1995 when the Chretien government tabled its Program Review budget. The 2012 budget has two related objectives: to eliminate the federal deficit by 2014 and, second, to cut back on federal programs and activities that are deemed to no longer be part of the government’s future agenda. 

This will be a defining budget for Stephen Harper. It will chart the financial course of the federal government for years to come. The current worldwide financial crisis will depress global economic activity and as a result will limit Canada’s export sales and simultaneously increase support payments to unemployed Canadians.

Given Canada’s great success with Program Review more than 15 years ago, it is expected that Treasury Board ministers will make good use of lessons learned from the earlier exercise by drawing on the numerous studies and evaluations that have been produced by the government over the past decade as it considers the Deficit Reduction Action Plan (DRAP). This is the magic moment for policy analysts inside and outside of government to play a significant role in the review process by providing robust and unassailable evidence of program effectiveness to the DRAP committee.  

Under the best of circumstances program evaluation is difficult to do. Reliable measurement is a challenge and determining long-term effectiveness depends on appropriate time frames, well-designed studies and clearly defined objectives. As a result, it is impossible to generate valid studies without years of planning and foresight.  

Interestingly, the Treasury Board made evaluation a priority in 2001 when it revised its policy framework for program evaluation and did so again in 2009 about the same time as the Auditor General was preparing a chapter on evaluation in one of her last annual reports. At that time, the new policy transferred the responsibilities for program evaluation to the deputy ministers and the program manager community in an effort to allocate responsibility for evaluation to the most appropriate level in the hierarchy. In addition, the Treasury Board also required that every federal program including grants and contributions be subject to an evaluation every five years. Stated another way, the 2009 policy requires that 20 percent of all federal government programs be subjected to a summative evaluation, each year, to determine whether the programs are providing value for money.

Unfortunately, it appears that the government’s evaluation policy is not having its desired effect. Recent Treasury Board analysis suggests that, despite the existence of an evaluation policy since 2001, the government carried out only about 170 evaluations per year representing between five and 13 percent of all federal program spending that took place between 2004 and 2008, instead of 100 percent as was envisaged with the policy.  

In fact, the shortcomings might be more serious than limited coverage. In her 2009 report to Parliament, the Auditor General noted many evaluations were plagued with serious measurement problems. She also concluded that many were late, provided inconclusive results, used rudimentary methodology, and contained findings that were often unreliable. 

Evaluation serves many important functions in addition to providing input into resource allocation exercises that happen once every 15 years. It is a useful tool to hold government to account, to improve the performance of ongoing programs, to free up funds for new efforts, and to provide lessons to other comparable programs.      

High quality evaluation work used to be a strength of the Canadian federal public service. In the 1970s, policy analysts were internationally recognized for their policy training programs, for the expertise of effectiveness and efficiency divisions in the Treasury Board Secretariat, and for the analytical strength of the evaluation units in many federal government departments.    

At this juncture when the government needs to make serious spending cuts to meet its fiscal targets, it appears the data are not there to address the key question regarding the value of the programs being reviewed. Not only is this unfortunate, it also represents a management system failure.

Nothing is more fundamental to good government than knowing how well tax money is being spent on behalf of citizens. If the role of Parliament is to hold government to account, then it is imperative that the appropriate House of Commons and Senate Committees demand valid and up-to-date evaluations. It is simply not sufficient to have rules and policies without a demand from legislators for their appropriate use.   

David Zussman holds the Jarislowsky Chair in Public Sector Management in the Graduate School of Public and International Affairs and at the Telfer School of Management at the University of Ottawa (dzussman@uottawa.ca).

 

About this author

0 comments

There are no comments for this post yet.

Be the first to comment. Click here.

Accountability
 
Cybersecurity professionals have sounded the alarm for years, and they are...
 
Cost estimation is becoming an extremely important skill within government due...
 
In this special episode of CGE Radio, your host John Jones...
 
Security professionals have an obligation to communicate risks and recommendations to...
 
Risk is always present in any undertaking, no matter the size...
 
Cost overruns have become institutionalized in the federal government, according to...
 
Last year, procurement Minister Jody Foote was prompted by the swirling...
 
Professionals, managers, and executives in the cost estimation industry can gain...
 
In this episode, hear from Carl Hammersburg, Manager, Government and Healthcare...
 
A new study from the Conference Board of Canada gives our...
 
In the world that we are living in today, free and...
 
The delivery method developed by Sir Michael Barber, chief adviser to...
 
Rules and accountability are helpful in developing and standardizing processes but...
 
Canadian doctors were told that climate change impacts human health and...
 
Even as talks between the government and federal workers affected problems...
 
The largest effort in 20 years to seek public input on...
 
Ottawa has overhauled the process by which justices are picked for...
 
Please to view this Content. (Not a member? Join Today! )...
 
In this episode, editor-in-chief, Patrice Dutil talks about the need for...
 
As much a 20 per cent of grade seven students in...
 
Please to view this Content. (Not a member? Join Today! )...
 
Upon receiving numerous complaints regarding add-on fees that turn making economy...
 
Are you absolutely clear what the government wants to achieve? Are...
 
Please to view this Content. (Not a member? Join Today! )...
 
Please to view this Content. (Not a member? Join Today! )...
 
Please to view this Content. (Not a member? Join Today! )...
 
Please to view this Content. (Not a member? Join Today! )...
 
Please to view this Content. (Not a member? Join Today! )...
 
Written By Jason McNaught The Public Service Alliance of Canada was...
 
Independence has long been regarded as a cornerstone of the auditing...
 
Canada is a diverse nation, in language, culture, geography, and, ultimately,...
 
Public sector organizations are under increasing pressure to identify all risks...
 
The government of Canada has implemented several measures over the past...
 
Whether at the territorial, provincial or federal government level, internal audit...
 
An organization’s reputation can take years to build but it can...
 
On October 30, Ontario began second reading of the Public Sector...
 
In the well-known children’s story, an Emperor falls victim to the...
 
The Nova Scotia Office of the Ombudsman is a small operation...
 
I think the ombudsman needs to be independent, because without independence...
 
Today’s business environment changes rapidly to adjust to evolving conditions and...
 
The best internal auditors actually are really good managers first. I...
 
The recent controversy about the actions of some staff members in...
 
Most professionals don’t need more than a sentence at a cocktail...
 
Recent research by the Institute of Internal Auditors Canada aims to...
 
When is it that a politician becomes part of the governing...
 
It’s been a busy couple of weeks on the information, privacy...
 
In 1996, a new budget watchdog, the Parliamentary Budget Officer, was...
 
The Ontario government is moving forward with the creation of a...
 
The Office of the Public Sector Integrity Commissioner of Canada (PSIC)...
 
After the Auditor General’s (AG) report was released on April 30,...
 
In the U.K. system, Permanent Secretaries are what we call Deputy...
 
We are living in a period of rapid change and limited...
 
US public sector employees don’t trust their management to do the...
 
The news of Mark Carney’s nomination as the new Governor of...
 
Following Singapore’s independence in 1965, the controversial leadership of Lee Kuan...
 
Over the past few years, the preparation and delivery of the...
 
In healthcare, cost-cutting can result in cutting what is valued most...
 
For over 20 years Colin Bennett has been exploring issues of...
 
Even before controversy shook the organization to its foundation, Ornge was...
 
It will be the largest international multi-sport event ever held on...
 
We’ve all seen the headlines – BC Ferries, Ornge, la Caisse...
 
It can happen, and it’s noteworthy when it does. Government, business...
 
Kevin Page’s mandate as the first Parliamentary Budget Officer comes to...
 
Governments are challenged to meaningfully mitigate the effects of the financial...
 
Canada is facing a huge financial challenge brought on by massive...
 
For the past one hundred years, democratic states have been moving...
 
It’s so much easier and less painful to learn from the...
 
CGE Vol.13 No.7 September 2007 "If the Public Service, as a...
 
When pondering leadership, we immediately think of exercising our influence downward...
 
CGE Vol.13 No.1 January 2007 "How can I be held accountable...
 
CGE Vol.13 No.2 February 2007 Canada’s Performance 2006 is the sixth...
 
CGE Vol.14 No.1 January 2008 The furor over the $300,000 that...
 
CGE Vol.14 No.2 February 2008 Let’s say you’re a senior manager,...
 
CGE Vol.13 No.1 January 2007 Perhaps it’s a legacy of the...
 
The Independent Blue Ribbon Panel on Grants and Contributions called for...
 
Au Canada, le secteur bénévole et à but non lucratif vit...
 
As the global economy struggles to regain some forward momentum, Canadian...
 
This will be a defining budget for Stephen Harper. It will...
 
It is difficult to determine when the debate about the need...
 
For the next few years, the federal government’s overarching agenda will...
 
Much of the current conversation about the federal government’s economic agenda...
 
Bill Greenlaw is the elected president of the Institute for Public...
 
Have you ever asked yourself the question: ‘How would I evaluate...
 
Last fall, Alberta’s Employment and Immigration department posted online the workplace...
 
In 2006 the world was feeling the aftershocks of a number...
 
CGE Vol.13 No.4 April 2007 Robert Parkins, editorial director, met recently...
 
In the past two decades, the nature of the state has...
 
Please to view this Content. (Not a member? Join Today! )...
 
Some title Some author
Some excerpt
 
Some title Some author
Some excerpt
 
Some title Some author
Some excerpt
 
Some title Some author
Some excerpt
 
Some title Some author
Some excerpt
Cybersecurity professionals have sounded the alarm for years, and they are...

Member Login

Forgot Password?

Join Us

Password Reset
Please enter your e-mail address. You will receive a new password via e-mail.