Performance Measurement
November 28, 2014

Of the essence: Coordinated training for government evaluators

There has never been a time since the introduction of program evaluation into our public services when training evaluators has been more important.

The problems in the field of Canadian evaluation are significant, not the least of which is its continued inability to deliver on its promises to provide real evidence of program effectiveness. The second challenge has been to provide consistent and rigorous training to evaluators in a way that meets the competencies identified by the Canadian Evaluation Society (CES), and most recently by the Treasury Board of Canada, for federal evaluation.

With regard to the first problem, the Auditor General reported in 2009 that departments were ill-equipped to evaluate all of their programs over a five-year period as required under the new Evaluation Policy. More important, the AG reported that the quality of evaluations was well below their coverage targets, but that “the rate of coverage was even lower because many of the effectiveness evaluations we reviewed did not adequately assess program effectiveness.” These conclusions were reinforced in the fall 2013 report, concluding that federal departments had made some progress toward meeting their coverage targets, but that again, “program evaluators noted constraints in their ability to address program effectiveness” (s.1.84).

Scholar Donald Savoie has written that despite significant federal investment in evaluation, the “contribution to the government’s policy and decision-making was negligible” and that “little has changed…apart from the fact that more money and staff are now being dedicated to the evaluation industry.” Most damning, however, he contends that managers and decision makers attribute limited currency to evaluation to value programs appropriately and realistically due to various conceptual problems in the field.

I argue that federal evaluation, at least, does not engage in the right questions nor does it have the capacity or training to engage appropriately in those questions, especially effectiveness, leading to major concerns about the entire legitimacy of the federal evaluation function.

These effectiveness challenges are perpetuated by the second problem. The 2009 Auditor General report states that, “despite having increased funding and staffing, the audited departments found it challenging to hire enough qualified, experienced evaluation staff to meet the needs for effectiveness evaluation.”

Funding for federal evaluation has since dropped considerably, but the capacity problem has always loomed large. In fact, we know that of the approximately 500 or so federal internal evaluators, at least half possess less than five-years of experience, and most of that experience is in project management, not actual evaluation research.

The combination of lack of effectiveness and inexperienced evaluators has resulted in a pervasive lack of confidence in the function and the field. Some of this can be attributed to a flawed federal policy, but in fairness these are field-wide issues. The pressure on the field to find evidence-based analysis to guide public policy mounts as provinces, territories and municipalities are now turning to evaluation as a way to guide budgetary decisions.

The challenge with our approach to training is that there is no approach. It is an assemblage of multiple actors working at cross-purposes in an open market. These professional bodies, private and non-profit sector entities, quasi-academic and academic institutions adhere to their own mandates and approaches vigorously. When such actors have come together to share their expertise, it has led to important improvements including the CES Credentialing Program. But without some Canadian vision for addressing a growing demand for evaluation training, we are inviting others to step in where we are divided with products that cost much and offer little.

As Canada inches toward the next federal election, Canadians will be asking what the record has been on some important matters, including social health, environment, employment and the economy. When governmental evaluators are asked what they’ve contributed, can they say more than simply they found “savings” in their existing programs?

Our governmental evaluation units have obsessed with cost reductions long enough: it is time to get back to the knitting and answer questions Canadians actually care about, including whether our tax dollars have been used to resolve public problems.

Getting qualified evaluators into our governmental internal units who understand this must be the first order of business. Providing a competency-based, coordinated, and Canadian approach to training is paramount if this is to happen.

The status quo will not hold: no one, including evaluators themselves, has the patience for it any longer, and the longer this drags on it does a disservice to us all.

0 comments

There are no comments for this post yet.

Be the first to comment. Click here.

Performance Measurement
 
Leadership, service delivery, change management, performance measurement, big data, transparency, digital...
 
Recently, there has been considerable interest in what is being referred...
 
Over the years, governments have been adopting Lean management principles to...
 
As the resettlement of Syrian Refugees exemplifies, government programs, currently running...
 
With businesses and government agencies increasingly operating in a highly information-based...
 
Risk is always present in any undertaking, no matter the size...
 
The Government of Canada’s newly announced Policy on Results replaces the...
 
In this episode, J. Richard Jones share information about the Canadian...
 
Rules and accountability are helpful in developing and standardizing processes but...
 
For several months now, the Liberal government ministers have been using...
 
Global market trends are accelerating to increase the pressure on commercial...
 
As a consultant, I work primarily with private sector organizations including...
 
Here’s some bad news for organizations contemplating projects that have something...
 
If the purpose of the Government of Canada’s performance measurement system...
 
Performance auditing can lead to more efficient, effective, and economical program...
 
The latest Auditor General’s report on Shared Services Canada (SSC) and...
 
Public sector innovation, once dismissed as a contradiction in terms, is...
 
The necessity for government executives to manage change, and achieve results,...
 
Every organization does it, but often with ambivalence: the performance measurement...
 
There is an apparent disconnect in performance measurement. Today, agencies measure...
 
The government of Canada’s transformation agenda continues to drive change across...
 
Cost savings in government programs and services used to be achieved...
 
As we move into 2015 – with a looming federal election...
 
Over the last four decades, public sector organizations have institutionalized program...
 
When you purchase clothes, do you like off-the-rack, or do you...
 
All governments, including Canada’s, strive to manage taxpayers’ dollars more efficiently....
 
Getting a clear definition of policy processes and functions can be...
 
While government agencies do not have “bottom lines” in the way...
 
Most of you already know that project management is a means...
 
The taxpayers are wary. Politicians worry about retaining their jobs. Everyone...
 
There has never been a time since the introduction of program...
 
The management of resources has always been taken for granted in...
 
In the last two decades, managing for results has gained momentum...
 
For the first time, the Consortium of Universities for Evaluation Education...
 
The government of New Zealand, and the State Services Commission in...
 
While it has been around for over 130 years, the concept...
 
Performance measurement and public reporting are intended to achieve two complementary...
 
Canada has over 400 administrative tribunals that are probably the least...
 
Last summer, the Vice Chief of the Defence Staff (VCDS) signed...
 
The core mandate of the Consortium of Universities for Evaluation Education...
 
the author fails to provide a complete picture (the good and...
 
Why talk about ethics at all right now? After all, Canada’s...
 
we now have what one federal ADM described to me as...
 
The reduction or mitigation of risks or harms is a major...
 
What are Canadian universities doing to maintain their relevance and innovative...
 
Managing for results has gained significant momentum in the public sector...
 
“Is performance management still relevant?” was the thought-provoking question posed by...
 
Within the last year, the Treasury Board of Canada Secretariat (TBS)...
 
Results matter in the public sector, particularly in these challenging times...
 
Today’s public sector environment is one in which many managers and...
 
Just as with public financial reporting, performance reporting is here to...
 
Being a high-performing organization is like being a high-performing individual: it...
 
With the current reality of deficit reduction, cost cutting and overall...
 
This article is New and Improved! After an exhaustive review cycle...
 
Canada was at the table when the “Commonwealth of Nations” was...
 
If you believe that performance measurement is a vital part of...
 
The last great international effort to refashion the public sector took...
 
Beginning on April 1, 2013, and over a five-year cycle, the...
 
Governments are increasingly using benchmarking to improve organizational performance; to determine...
 
Senior public sector managers, deputy heads of departments and parliamentarians need...
 
Canada is a hockey-loving nation and now is the time of...
 
Too few public sector organizations have truly embedded performance thinking into...
 
CGE Vol.13 No.3 March 2007 Governing is never easy. And in...
 
CGE Vol.13 No.2 February 2007 Improving organizational health and employee engagement...
 
CGE Vol.13 No.1 January 2007 The proposed Federal Accountability Act is...
 
CGE Vol.14 No.2 February 2008 The passage of the Federal Accountability...
 
CGE Vol. 14 No.4 April 2008 The Tellier-Mazankowski advisory committee on...
 
Performance management leads to either disdain or glee in public sector...
 
Les administrations publiques utilisent de plus en plus l’analyse comparative pour...
 
The redesign of corporate tax administration for Ontario is a major...
 
Coincidental with the push for more attention to be paid to...
 
The BC Public Service has had an intense focus on improving...
 
La reconception de l’administration de l’imposition des sociétés en Ontario est...
 
Dans son budget de lutte contre la récession (et d’économies dans...
 
The Harper government’s recession fighting (and government saving) budget heralded an...
 
Many governments have bought into the notion of the Balanced Scorecard,...
 
David P. Norton est le cocréateur, avec Robert Kaplan, du Tableau...
 
Together with Robert Kaplan, David P. Norton is the creator of...
 
Il y a dix ans, la Société canadienne du sang (SCS)...
 
Début septembre, la fonction publique de l’Ontario (FPO) célébrait le 10e...
 
Bon nombre d’administrations publiques ont adopté le tableau de bord prospectif,...
 
Today’s public sector managers encounter increasingly complex corporate issues that call...
 
Performance management seems to be everywhere these days, with much of...
 
The two key tools used by the Canadian government to measure...
 
The principle is unassailable: Canadians have the right to know whether...
 
L’administration publique s’efforce d’obtenir des résultats tangibles en mettant en œuvre...
 
Une étude récente commandée par l’OCDE conclut que, malgré le fait...
 
A recent study commissioned by the OECD concludes that while the...
 
Some title Some author
Some excerpt
 
Some title Some author
Some excerpt
Leadership, service delivery, change management, performance measurement, big data, transparency, digital...