Performance Measurement
May 7, 2012

Measuring performance in development projects

The principle is unassailable: Canadians have the right to know whether their foreign aid money is being effectively spent.

To meet this challenge, the Canadian International Development Agency (CIDA) uses a sophisticated results-based management (RBM) methodology that, in principle, shows how each project’s inputs (money) produce outputs (x teachers trained) that translate into outcomes (percentage increase in enrolment).

Much time and effort goes into preparing, reporting on, and evaluating these schematics. But do they reveal whether our foreign aid money is being well spent? Numerous obstacles preclude an unequivocal “yes” response to this question. RBM systems are anything but simple, typically requiring lots of investment and years of refinement.

In practice, RBM reports to donors have significant credibility problems. One reason is overload: development projects typically place significant time demands on counterparts above and beyond those of their day jobs. The incentives to treat the labour-intensive reporting requirements as anything more than a pro forma obligation are minimal. In one particularly egregious case, district administrators in Uganda were reporting monthly on as many as 2,000 performance indicators.

Another obstacle is technical. A single project is just one influence among many. The desired outcome will inevitably be affected by numerous actions and actors outside the confines of the project. This is particularly true when RBM enthusiasts overreach as they move up the results chain. Projects supporting a modest number of exchanges between two governments somehow miraculously, at least in their results matrix, lay claim to increased GDP.

Relevance can also be problematic. To be meaningful, monitoring, reporting and evaluation must inform decision making. Project progress reports and assessments can easily become lost in an internal review cycle involving consultants and mid-level government and donor officials, and never come near a real decision maker. Although the determination of outcomes and impacts tends to occur in the medium term, the majority of evaluations are performed at a project’s conclusion, before most of the intended outcomes have been realized.

What then is to be done? The opposite is hardly enticing: abandon RBM, explain to Canadians that development results are just too hard to measure, and accept that Homer Simpson was right when he said “trying is the first step towards failure.” In fact, the results game must be played.

The answer is in how rigorously the rules are applied; think of it as quantitative easing.

Commonsense accommodations might include: a focus on project outputs that the government has already identified (often in budget submissions); considering “collectability” and “reporting burden” when choosing measures; linking reporting to the government’s own reporting cycles; using common indicators when multiple donors are working in the same area; being realistic about the project’s reach (it will not single-handedly deliver a Millennium Development Goal commitment); deferring the project evaluation until a few years after its conclusion; considering the political relevance of what is to be reported and finding ways to present conclusions in an appropriate format for decision makers (we would never expect a Canadian minister to read a 50-page report or decipher a log frame).

Most important, recognize that RBM is not the only results game in town.

Beyond RBM lies the world of ethics, relationships and networks. Harried senior managers in developing countries generally will tolerate projects, but will only embrace them when it is clear that they are offering value for time. True value does not come from a PowerPoint presentation, but from the personal rapport and trust that are built with one’s counterpart over a sustained period of time.

This is not to suggest that RBM frameworks should devise performance indicators to capture the degree to which project managers suss out, befriend and support influential change agents. Tangible outputs will still need to be identified and monitored.

But back to the main question: will we ever know if our foreign aid money is being effectively deployed? Well, probably not. But to the degree that the Canadian public and private sectors offer good models and people willing to engage counterparts at a professional and personal level, the likelihood that something positive will result, however unquantifiable, remains strong.

 

Gordon Evans is an international consultant who has worked in over 25 countries focusing on government decision making, strategic planning and policy formulation. He co-authored Helping Governments Keep Their Promises for the World Bank.

 

SIDEBAR

Progress through relationships

In the late 1990s, a project supporting the introduction of Ontario’s business planning system in newly-independent Lithuania veered outside its terms of reference when it agreed to assist the deputy speaker of Lithuania’s parliament reform the parliamentary committee system. A good rapport was struck between the project and the deputy speaker. In 1999, the deputy speaker was included in a senior Lithuanian delegation which saw, first hand, how Ontario’s planning and decision-making system functioned. Highly enthused, he concluded that this was something his country badly needed. In 2000, the former deputy speaker, now Prime Minister, introduced an Ontario-style strategic planning and cabinet committee system. The Lithuanian model has since been widely praised by the World Bank and remains in place today.

About this author

0 comments

There are no comments for this post yet.

Be the first to comment. Click here.

Performance Measurement
 
Leadership, service delivery, change management, performance measurement, big data, transparency, digital...
 
Recently, there has been considerable interest in what is being referred...
 
Over the years, governments have been adopting Lean management principles to...
 
As the resettlement of Syrian Refugees exemplifies, government programs, currently running...
 
With businesses and government agencies increasingly operating in a highly information-based...
 
Risk is always present in any undertaking, no matter the size...
 
The Government of Canada’s newly announced Policy on Results replaces the...
 
In this episode, J. Richard Jones share information about the Canadian...
 
Rules and accountability are helpful in developing and standardizing processes but...
 
For several months now, the Liberal government ministers have been using...
 
Global market trends are accelerating to increase the pressure on commercial...
 
As a consultant, I work primarily with private sector organizations including...
 
Here’s some bad news for organizations contemplating projects that have something...
 
If the purpose of the Government of Canada’s performance measurement system...
 
Performance auditing can lead to more efficient, effective, and economical program...
 
The latest Auditor General’s report on Shared Services Canada (SSC) and...
 
Public sector innovation, once dismissed as a contradiction in terms, is...
 
The necessity for government executives to manage change, and achieve results,...
 
Every organization does it, but often with ambivalence: the performance measurement...
 
There is an apparent disconnect in performance measurement. Today, agencies measure...
 
The government of Canada’s transformation agenda continues to drive change across...
 
Cost savings in government programs and services used to be achieved...
 
As we move into 2015 – with a looming federal election...
 
Over the last four decades, public sector organizations have institutionalized program...
 
When you purchase clothes, do you like off-the-rack, or do you...
 
All governments, including Canada’s, strive to manage taxpayers’ dollars more efficiently....
 
Getting a clear definition of policy processes and functions can be...
 
While government agencies do not have “bottom lines” in the way...
 
Most of you already know that project management is a means...
 
The taxpayers are wary. Politicians worry about retaining their jobs. Everyone...
 
There has never been a time since the introduction of program...
 
The management of resources has always been taken for granted in...
 
In the last two decades, managing for results has gained momentum...
 
For the first time, the Consortium of Universities for Evaluation Education...
 
The government of New Zealand, and the State Services Commission in...
 
While it has been around for over 130 years, the concept...
 
Performance measurement and public reporting are intended to achieve two complementary...
 
Canada has over 400 administrative tribunals that are probably the least...
 
Last summer, the Vice Chief of the Defence Staff (VCDS) signed...
 
The core mandate of the Consortium of Universities for Evaluation Education...
 
the author fails to provide a complete picture (the good and...
 
Why talk about ethics at all right now? After all, Canada’s...
 
we now have what one federal ADM described to me as...
 
The reduction or mitigation of risks or harms is a major...
 
What are Canadian universities doing to maintain their relevance and innovative...
 
Managing for results has gained significant momentum in the public sector...
 
“Is performance management still relevant?” was the thought-provoking question posed by...
 
Within the last year, the Treasury Board of Canada Secretariat (TBS)...
 
Results matter in the public sector, particularly in these challenging times...
 
Today’s public sector environment is one in which many managers and...
 
Just as with public financial reporting, performance reporting is here to...
 
Being a high-performing organization is like being a high-performing individual: it...
 
With the current reality of deficit reduction, cost cutting and overall...
 
This article is New and Improved! After an exhaustive review cycle...
 
Canada was at the table when the “Commonwealth of Nations” was...
 
If you believe that performance measurement is a vital part of...
 
The last great international effort to refashion the public sector took...
 
Beginning on April 1, 2013, and over a five-year cycle, the...
 
Governments are increasingly using benchmarking to improve organizational performance; to determine...
 
Senior public sector managers, deputy heads of departments and parliamentarians need...
 
Canada is a hockey-loving nation and now is the time of...
 
Too few public sector organizations have truly embedded performance thinking into...
 
CGE Vol.13 No.3 March 2007 Governing is never easy. And in...
 
CGE Vol.13 No.2 February 2007 Improving organizational health and employee engagement...
 
CGE Vol.13 No.1 January 2007 The proposed Federal Accountability Act is...
 
CGE Vol.14 No.2 February 2008 The passage of the Federal Accountability...
 
CGE Vol. 14 No.4 April 2008 The Tellier-Mazankowski advisory committee on...
 
Performance management leads to either disdain or glee in public sector...
 
Les administrations publiques utilisent de plus en plus l’analyse comparative pour...
 
The redesign of corporate tax administration for Ontario is a major...
 
Coincidental with the push for more attention to be paid to...
 
The BC Public Service has had an intense focus on improving...
 
La reconception de l’administration de l’imposition des sociétés en Ontario est...
 
Dans son budget de lutte contre la récession (et d’économies dans...
 
The Harper government’s recession fighting (and government saving) budget heralded an...
 
Many governments have bought into the notion of the Balanced Scorecard,...
 
David P. Norton est le cocréateur, avec Robert Kaplan, du Tableau...
 
Together with Robert Kaplan, David P. Norton is the creator of...
 
Il y a dix ans, la Société canadienne du sang (SCS)...
 
Début septembre, la fonction publique de l’Ontario (FPO) célébrait le 10e...
 
Bon nombre d’administrations publiques ont adopté le tableau de bord prospectif,...
 
Today’s public sector managers encounter increasingly complex corporate issues that call...
 
Performance management seems to be everywhere these days, with much of...
 
The two key tools used by the Canadian government to measure...
 
The principle is unassailable: Canadians have the right to know whether...
 
L’administration publique s’efforce d’obtenir des résultats tangibles en mettant en œuvre...
 
Une étude récente commandée par l’OCDE conclut que, malgré le fait...
 
A recent study commissioned by the OECD concludes that while the...
 
Some title Some author
Some excerpt
 
Some title Some author
Some excerpt
Leadership, service delivery, change management, performance measurement, big data, transparency, digital...