Performance Measurement
February 19, 2014

Purposeful performance stories or pointless propaganda?

On December 3, 2013 the Performance and Planning Exchange held a workshop entitled, “Putting Performance Reporting to the Test: The 2013 DPR Rating Bee.” It was an opportunity for students in the Diploma in Policy and Program Evaluation program at Carleton University to put their evaluative skills to the test.

They were asked to assess five Departmental Performance Reports (DPR) according to two lenses: two of 13 criteria (reliability and validity, and relevance) from the Guide to Preparing Public Performance Reports (May 2007) developed by the Public Sector Accounting Board of Canada (PSAB); and, whether overall, they thought the DPRs told a coherent performance story according to John Mayne’s excellent discussion paper to the Office of the Auditor General of Canada and the guidance provided by the Report of the Standing on Public Accounts in September 2010, where the committee made recommendations for the improvement of departmental performance reports.

Mayne sets out five elements of a coherent performance story: i) sufficient context to allow readers to understand the report, including a results chain, and risks faced; ii) a description of planned spending linked to expected outputs and outcomes; iii) a comparison of expected results with actual results; iv) indication of lessons learned, and how the organization will benefit from these in future iterations of programming; and, v) whether steps are in place to ensure the quality of the data used in the report.

The students came to several important observations, but here are some of their major themes.

First, in terms of Mayne’s first element, there is no clear sense of purpose for these reports. The result, according to one student, is an “incoherent jumble of data streams that are not linked in any understandable way to plain language outcomes.” The reports ought to inform public debate about governmental action in addition to informing on the expenditure of public funds under the Financial Administration Act.

Second, the reports do not present a sense of vision and mission that provides the reader with where programming is aimed. According to another student commenting on Mayne’s second element, the reports “provided a ton of information on political tactics…which is not all that important to [citizens]. We need to start talking about high level policy goals and objectives that are relevant and agreed upon by all stakeholders.” These reports demonstrate little coherence on overall progress toward public policy objectives such as reducing poverty or crime.

Third, the reports lack evidentiary rigour. The measures tend to be at the activities level and do not shed light on results or outcomes. According to one student commenting on Mayne’s third and fourth elements, “a review of the performance indicators…does not yield much useful information on how [the department] used the information to inform decision-making and shape programs.” That is, the links between activities and results are unclear, which also leads one to question whether the links between planned and actual spending is reliable and valid. The lack of results chains, logic models, or links to vision statements show a clear divergence between departmental activities and collective progress toward coherently expressed results.

Fourth, given the political climate of the day, all of the reports examined provide a good news story that is unsupported by the limited evidence. According to a student commenting on Mayne’s fifth element, “most of these reports spend more time describing the few specific things they are doing well, while purposely avoiding shedding any bad light on what they are not doing well.”

Several improvements were forwarded by the students including:

• Prepare separate sections and messages for specific audiences, including citizens;
• Greater use of third party assessors, within/without government, to gauge report accuracy;
• Coherent guidance for departments on how to prepare DPRs in a standard format;
• DPRs should be focused more on results, and progress toward long-term policy outcomes in a coordinated and multi-jurisdictional way; and
• Use of plain language using valid evidentiary performance stories.

If trained students in evaluation are having difficulty with the limited rigour, coherence and readability of these reports, one can only imagine what the general public must think. The assumption is that Canadians want to engage in public policy conversations in an informed way. That we have a vehicle such as the DPR is a positive contribution, but in their current state, one must question their value for democratic discussion. If this workshop told us anything, it is that we can collectively do much better.

0 comments

There are no comments for this post yet.

Be the first to comment. Click here.

Performance Measurement
 
Leadership, service delivery, change management, performance measurement, big data, transparency, digital...
 
Recently, there has been considerable interest in what is being referred...
 
Over the years, governments have been adopting Lean management principles to...
 
As the resettlement of Syrian Refugees exemplifies, government programs, currently running...
 
With businesses and government agencies increasingly operating in a highly information-based...
 
Risk is always present in any undertaking, no matter the size...
 
The Government of Canada’s newly announced Policy on Results replaces the...
 
In this episode, J. Richard Jones share information about the Canadian...
 
Rules and accountability are helpful in developing and standardizing processes but...
 
For several months now, the Liberal government ministers have been using...
 
Global market trends are accelerating to increase the pressure on commercial...
 
As a consultant, I work primarily with private sector organizations including...
 
Here’s some bad news for organizations contemplating projects that have something...
 
If the purpose of the Government of Canada’s performance measurement system...
 
Performance auditing can lead to more efficient, effective, and economical program...
 
The latest Auditor General’s report on Shared Services Canada (SSC) and...
 
Public sector innovation, once dismissed as a contradiction in terms, is...
 
The necessity for government executives to manage change, and achieve results,...
 
Every organization does it, but often with ambivalence: the performance measurement...
 
There is an apparent disconnect in performance measurement. Today, agencies measure...
 
The government of Canada’s transformation agenda continues to drive change across...
 
Cost savings in government programs and services used to be achieved...
 
As we move into 2015 – with a looming federal election...
 
Over the last four decades, public sector organizations have institutionalized program...
 
When you purchase clothes, do you like off-the-rack, or do you...
 
All governments, including Canada’s, strive to manage taxpayers’ dollars more efficiently....
 
Getting a clear definition of policy processes and functions can be...
 
While government agencies do not have “bottom lines” in the way...
 
Most of you already know that project management is a means...
 
The taxpayers are wary. Politicians worry about retaining their jobs. Everyone...
 
There has never been a time since the introduction of program...
 
The management of resources has always been taken for granted in...
 
In the last two decades, managing for results has gained momentum...
 
For the first time, the Consortium of Universities for Evaluation Education...
 
The government of New Zealand, and the State Services Commission in...
 
While it has been around for over 130 years, the concept...
 
Performance measurement and public reporting are intended to achieve two complementary...
 
Canada has over 400 administrative tribunals that are probably the least...
 
Last summer, the Vice Chief of the Defence Staff (VCDS) signed...
 
The core mandate of the Consortium of Universities for Evaluation Education...
 
the author fails to provide a complete picture (the good and...
 
Why talk about ethics at all right now? After all, Canada’s...
 
we now have what one federal ADM described to me as...
 
The reduction or mitigation of risks or harms is a major...
 
What are Canadian universities doing to maintain their relevance and innovative...
 
Managing for results has gained significant momentum in the public sector...
 
“Is performance management still relevant?” was the thought-provoking question posed by...
 
Within the last year, the Treasury Board of Canada Secretariat (TBS)...
 
Results matter in the public sector, particularly in these challenging times...
 
Today’s public sector environment is one in which many managers and...
 
Just as with public financial reporting, performance reporting is here to...
 
Being a high-performing organization is like being a high-performing individual: it...
 
With the current reality of deficit reduction, cost cutting and overall...
 
This article is New and Improved! After an exhaustive review cycle...
 
Canada was at the table when the “Commonwealth of Nations” was...
 
If you believe that performance measurement is a vital part of...
 
The last great international effort to refashion the public sector took...
 
Beginning on April 1, 2013, and over a five-year cycle, the...
 
Governments are increasingly using benchmarking to improve organizational performance; to determine...
 
Senior public sector managers, deputy heads of departments and parliamentarians need...
 
Canada is a hockey-loving nation and now is the time of...
 
Too few public sector organizations have truly embedded performance thinking into...
 
CGE Vol.13 No.3 March 2007 Governing is never easy. And in...
 
CGE Vol.13 No.2 February 2007 Improving organizational health and employee engagement...
 
CGE Vol.13 No.1 January 2007 The proposed Federal Accountability Act is...
 
CGE Vol.14 No.2 February 2008 The passage of the Federal Accountability...
 
CGE Vol. 14 No.4 April 2008 The Tellier-Mazankowski advisory committee on...
 
Performance management leads to either disdain or glee in public sector...
 
Les administrations publiques utilisent de plus en plus l’analyse comparative pour...
 
The redesign of corporate tax administration for Ontario is a major...
 
Coincidental with the push for more attention to be paid to...
 
The BC Public Service has had an intense focus on improving...
 
La reconception de l’administration de l’imposition des sociétés en Ontario est...
 
Dans son budget de lutte contre la récession (et d’économies dans...
 
The Harper government’s recession fighting (and government saving) budget heralded an...
 
Many governments have bought into the notion of the Balanced Scorecard,...
 
David P. Norton est le cocréateur, avec Robert Kaplan, du Tableau...
 
Together with Robert Kaplan, David P. Norton is the creator of...
 
Il y a dix ans, la Société canadienne du sang (SCS)...
 
Début septembre, la fonction publique de l’Ontario (FPO) célébrait le 10e...
 
Bon nombre d’administrations publiques ont adopté le tableau de bord prospectif,...
 
Today’s public sector managers encounter increasingly complex corporate issues that call...
 
Performance management seems to be everywhere these days, with much of...
 
The two key tools used by the Canadian government to measure...
 
The principle is unassailable: Canadians have the right to know whether...
 
L’administration publique s’efforce d’obtenir des résultats tangibles en mettant en œuvre...
 
Une étude récente commandée par l’OCDE conclut que, malgré le fait...
 
A recent study commissioned by the OECD concludes that while the...
 
Some title Some author
Some excerpt
 
Some title Some author
Some excerpt
Leadership, service delivery, change management, performance measurement, big data, transparency, digital...