Reaching organizational goals through improved evaluation capacity - Canadian Government Executive
Performance Measurement
January 30, 2015

Reaching organizational goals through improved evaluation capacity

Over the last four decades, public sector organizations have institutionalized program and policy evaluation practices to varying degrees. As a management tool, evaluation has the potential to provide timely, useful information to decision-makers concerned with resource optimization and program outcomes.

However, piecemeal approaches to evaluation are commonplace in various public administrations, including the federal government. Such approaches tend to produce operational recommendations that hinder the ability of evaluators to support management and contribute meaningfully to organizational learning, accountability and improvement.

Building capacity
Organizations interested in developing a more strategic approach to program evaluation typically focus their efforts on staff training and budgetary allocations. Although these measures are an important starting point, the institutionalization of evaluation requires additional effort.

Researchers have turned their attention to this issue over the last decade and have dubbed the process “organizational evaluation capacity building.” Evaluation capacity can take on different forms, depending on an organization’s context; our own research has focused, therefore, on the development of an empirically-based description of organizational evaluation capacity in the federal government.

Our organizational evaluation capacity framework includes six main dimensions. The first three focus on the organization’s ability to produce high-quality evaluation studies, while the others describe the organization’s ability to use evaluation for organizational learning and accountability purposes.

Human Resources, the first dimension, refers to the technical and interpersonal competencies of evaluators as well as evaluation leadership within the organization. The second dimension, Organizational Resources, focuses on the systems and structures available to evaluators to carry out their duties, such as the financial resources allocated to evaluation studies and the availability of quality performance measurement data. Evaluation Planning and Activities, the third dimension, describes how evaluation is planned and conducted in the organization. This dimension includes, for example, the use of consultants and other external resources to complement internal capabilities and skill sets.

Evaluation Literacy is the fourth dimension and the first to describe organizational capacity to use evaluation. It refers to the widespread knowledge of evaluation and its associated concepts, such as results-based management, across the organization. Organizational Decision Making characterizes the fifth dimension and focuses on the integration of evaluation findings in broader organizational processes and decision-making, such as program resourcing or deficit reduction activities. Finally, Learning Benefits describes how evaluation is used for learning and improvement in the organization, which is the ultimate objective of evaluation itself and evaluation capacity building more specifically.

In addition to the six dimensions and their associated sub-dimensions, the framework also outlines four levels of evaluation capacity: low, developing, intermediate and exemplary capacity. Each of the sub-dimensions, therefore, is described qualitatively at each of those four levels in order to profile and highlight the incremental steps required to move from one level of capacity to the next.

Managers interested in assessing their own organization’s evaluation capacity may use this organizational profile framework to situate their organization across the four levels for all six dimensions. Alternatively, they may also be interested in using an organizational self-assessment instrument, which operationalizes the original framework and facilitates the assessment process.

Although the instrument is meant to be used by federal government departments and agencies, it is possible to adapt the tool to other organizational contexts. We have already adapted the instrument to the not-for-profit sector and the Quebec provincial government, and we are continuing this work to render the tool suitable for use in the Ontario public health sector. Fundamentally, the six dimensions apply to most public sector settings and it is possible to use the instrument in all of these, albeit in a slightly modified form that accounts for local contexts and terminology.

Organizational improvement
Our research shows that organizations that have a good sense of their evaluation capacity are better able to devise strategies to address any major gaps and continue on their path towards institutionalization.

We also know, based on our own sectoral comparative analyses, that organizations with higher evaluation capacity get the greatest “bang for their evaluation buck”; not only are their studies of greater quality and more timely, they also better anticipate senior decisio- makers’ information needs. This, in turn, leads to greater evaluation use and organizational learning.

The link to improved program design and administration, and to a more effective and efficient use of organizational resources, makes this endeavour wholly appealing, not only to evaluators, but to managers across all levels of public sector organizations.

About this author

0 comments

There are no comments for this post yet.

Be the first to comment. Click here.

Performance Measurement
 
As the resettlement of Syrian Refugees exemplifies, government programs, currently running on RBM, can be made more accountable if they were managed using PM principles, as well. The Organization for Economic Cooperation and Development (OECD) has outlined five accountability criteria for evaluating projects: relevance, effectiveness, efficiency, sustainability, and impact. Setting the latter two aside, for...
 
With businesses and government agencies increasingly operating in a highly information-based economy and technologically-linked society, knowing the implications of various social network connections that link individuals and organization will become essential, according to John Burrett. Burrett is president and principal consultant at Haiku Analytics. He is an expert in using data graphics, social network analysis,...
 
Risk is always present in any undertaking, no matter the size or duration of the project. Risk can jack up program costs, foul up schedules and draw out projects beyond their stated deadlines, and even ultimately derail a program. Related posts: In Defence of Direct Negotiation Moving to the Cloud: A strategic approach October issue: From procurement to commissioning...
 
The Government of Canada’s newly announced Policy on Results replaces the former Policy on Evaluation. One new and welcome feature is that it declares in its Directive on Results that heads of evaluation are responsible for “ensuring that departmental evaluators have opportunities to develop their competencies and to earn evaluation-related designations or certifications from recognized...
 
In this episode, J. Richard Jones share information about the Canadian Government Executive Summit and why you should attend, especially if you want to know more about putting deliverology into practise. He speaks witth Dave Telka from Accenture about a most recent report that has identified the need to make careers within public sector more...
 
Rules and accountability are helpful in developing and standardizing processes but there are instances when they can be a hindrance to innovation. Finding the right balance between accountability and innovation is essential in the public sector, according to Tony Dean , professor at the School of Public Policy and Governance at the University of Toronto , and adviser in public administration and public reforms to various governments in...
 
For several months now, the Liberal government ministers have been using as a guide, the theories and practices embodied in deliverology, the controversial result-management philosophy developed by British political adviser Michael Barber. Related posts: Manager development: Matching competency to complexity Delivering on Deliverology On the cusp of a dramatic, generational change...
 
Global market trends are accelerating to increase the pressure on commercial and government organizations. It’s time to stop dabbling in discrete customer-facing digital projects and take the plunge towards digitization. The annual CGI outlook has amassed the results of in-person conversations with leaders from 10 industries and 20 countries. According to CGI President and CEO...
 
As a consultant, I work primarily with private sector organizations including the “big-five” banks, professional governing bodies and regulators for accountants, lawyers and trustees in bankruptcy. These businesses have deeply rooted performance measurement cultures, but are often unfamiliar with program evaluation. Managing evaluations within these organizations often entails accentuating evaluation capacity building while simultaneously enhancing...
 
Here’s some bad news for organizations contemplating projects that have something to do with the Internet of Things. Research firm Gartner, predicts that by 2020 more than half of major new business process and systems will incorporate some element of IoT. However, despite the growing prevalence of connected systems in the private and public sectors, Gartner...
 
If the purpose of the Government of Canada’s performance measurement system is to keep people busy, then it works beautifully. If it’s supposed to show how programs make life better for Canadians, it’s a dud. More than 2,500 measures are just too many, period! Related posts: Leader as evaluator: The power of analysis The Performance and Planning Exchange (PPX) 2016 Symposium Learning from the Past – Preparing for th... Connecting the Dots between M, E, RBM….and Deliverology...
 
Performance auditing can lead to more efficient, effective, and economical program delivery and stronger internal controls within local governments. Surprisingly, many audit functions within local governments in Canada are still conducting few or no performance audits. To understand why they are lagging behind their U.S. counterparts, researchers surveyed auditors within all the major cities in...
 
The latest Auditor General’s report on Shared Services Canada (SSC) and the Social Security Tribunal could be an instructive document for IT managers on how not to run an IT department. The latest report of Auditor General Michael Ferguson revealed a huge backlog delaying the processing of Canada Pension Plan disability claims and failure of...
 
Public sector innovation, once dismissed as a contradiction in terms, is now increasingly seen as an imperative. Last November, the Organisation for Economic Co-operation and Development (OECD) hosted an international conference on “Innovating the Public Sector” and announced that it has been building a database on public sector innovation. Many federal government departments are setting-up...
 
The necessity for government executives to manage change, and achieve results, is an enduring quality of government work. Two fundamental assumptions support this view. Every government program, regardless of its scale or scope, exists to serve a purpose… Related posts: A collision of one too many good ideas? How an executive sponsor contributes to project management Benefits management: A better method for maximizing performance...
 
Every organization does it, but often with ambivalence: the performance measurement process has become a mechanical exercise dutifully executed to satisfy corporate requirements. Related posts: A collision of one too many good ideas? Management is one thing, measurement is another Savings without reduced service...
 
There is an apparent disconnect in performance measurement. Today, agencies measure two discrete things: the performance of the agency and the performance of the individual. The thought is that those are connected… Related posts: Reaching organizational goals through improved evaluation capacity Benefits management: A better method for maximizing performance Trudeau’s five deliverology targets...
 
The government of Canada’s transformation agenda continues to drive change across the public sector, requiring delivery of a large number of increasingly complex, high-profile multi-million dollar projects. Related posts: Complex project management a strategic imperative...
 
Cost savings in government programs and services used to be achieved simply by cutting budgets. But is there a way to find savings and eliminate inefficiencies without reducing service? Related posts: Prevention-focused people and risk aversion The logical way to manage change and achieve results Breaking through the digital miasma with creativity...
 
As we move into 2015 – with a looming federal election and a first annual report on progress on implementation of Blueprint 2020 (now Destination 2020) – it is appropriate to reflect upon the future state of the public service in Canada, and consider what might result from this major change agenda. Related posts: Blueprint 2020: Raising expectations for real change Acknowledging elephants...
 
Over the last four decades, public sector organizations have institutionalized program and policy evaluation practices to varying degrees. As a management tool, evaluation has the potential to provide timely, useful information to decision-makers… Related posts: A collision of one too many good ideas? Savings without reduced service Pinning Waves Upon the Sand...
 
When you purchase clothes, do you like off-the-rack, or do you prefer your garments custom-made? Each has its advantages, of course. Related posts: Strong project management framework leads to organizational success Trudeau’s ‘deliverology’ guy Lean solutions to 5 deliverology challenges...
 
All governments, including Canada’s, strive to manage taxpayers’ dollars more efficiently. One challenge to that goal is complexity, embodied by projects involving multiple stakeholders that often take years to complete. Related posts: How an executive sponsor contributes to project management Managing Talent: It's not just a necessity, it's an investment for the future Trudeau’s ‘deliverology’ guy...
 
Getting a clear definition of policy processes and functions can be difficult. A quick perusal of popular definitions suggests that policy processes involve problem identification, policy development – which may include research, advice, consensus building and advocacy, policy implementation and evaluation – as well as other activities. Related posts: Research unit provides city with secret weapon Case competition: Practicing the art of policymaking The Policy Role of Non-Government Actors: In from the Cold?...
 
While government agencies do not have “bottom lines” in the way private sector organizations do, agency leaders are “under the microscope” at least as much (or possibly more) than their private sector colleagues to reduce or eliminate financial waste. Related posts: IoT projects face long delays: Gartner What you need to know about Deliverology and Implementation Trudeau’s five deliverology targets...
 
Most of you already know that project management is a means to provide better service for Canadian taxpayers at a lower cost. You might wonder what the next step is after instituting effective project management. For many organizations, that next step involves PMOs....
 
There has never been a time since the introduction of program evaluation into our public services when training evaluators has been more important. Related posts: From past to future: Contrasting perspectives on public sector management Student research: Impressive and making a difference Drug coverage for veterans poorly managed: Auditor General...
 
The management of resources has always been taken for granted in Canada’s public sector. However, now more than ever, Canada’s public service executives need to devote equal and formal attention to results as well as resources. Related posts: Red flags appearing in ethics regime Benefits management: A better method for maximizing performance Looking Forward to Looking Back...
 
In the last two decades, managing for results has gained momentum in the public sector, strengthening the disciplines of strategic planning, risk management, audit, evaluation, and performance measurement and reporting. Related posts: Threats to effective service Evaluation Goals: Implementing Evaluation Audit Practice in a Non-Profit Agency in Toronto Building Better Buy-in for Evaluation...
 
For the first time, the Consortium of Universities for Evaluation Education (CUEE) sponsored a student track of research presentations at the Canadian Evaluation Society’s national conference in June. Related posts: Creating innovative approaches to teaching evaluation Increasing opportunities for evaluation education Evaluation - Systematically gathering information. An interview with the President, Canadian Evaluat......
 
The government of New Zealand, and the State Services Commission in particular, is focused on improving services to citizens. Some of the goals are familiar: better public services and the creation of an all-of-government brand. What’s interesting is that the government is actually willing to report openly on its progress. Related posts: The digital town hall: Meeting citizen expectations through government websites...
 
While it has been around for over 130 years, the concept of management through measurement is most often attributed to Peter Drucker, and his 1954 book, The Practice of Management – perhaps the first book to identify management as a professional discipline. Related posts: A collision of one too many good ideas? Talking about performance Metrics: A Cautionary Tale...
 
Performance measurement and public reporting are intended to achieve two complementary objectives: improve accountability and improve performance....
 
Canada has over 400 administrative tribunals that are probably the least understood government entity....
 
Last summer, the Vice Chief of the Defence Staff (VCDS) signed off on a departmental initiative known as the Project Management Competency Development initiative (PMCD) that will provide a structure for the necessary training project managers require to align their knowledge and skills with the risks and complexities of procurement programs. Related posts: Strong project management framework leads to organizational success ADM MAT adapts to new procurement strategy...
 
The core mandate of the Consortium of Universities for Evaluation Education (CUEE) is to grow capacity in Canada to meet the need for competent evaluation professionals. Part of realizing our goal is to make evaluation education more portable among CUEE members. Related posts: Creating innovative approaches to teaching evaluation Student research: Impressive and making a difference...
 
the author fails to provide a complete picture (the good and the bad) – one of the criticism of the DPRs. It could be said that the series of choices made...
 
Why talk about ethics at all right now? After all, Canada’s public service is arguably fully focused on undergoing another round of transformational changes, including Blueprint 2020, realignment of service delivery approaches (such as with Shared Services Canada), operating budget cuts, and a potentially fundamental reorientation of its human resources/labour relations legislative and policy regimes....
 
we now have what one federal ADM described to me as “”decision-based evidence making””. With the amount of information and knowledge growing exponentially...
 
The reduction or mitigation of risks or harms is a major priority of the government of Canada in matters related to Canadians’ health, safety, security and environmental protection and even economic ‘harms’ such as fraud....
 
What are Canadian universities doing to maintain their relevance and innovative edge in the teaching of program evaluation? Our response is: A great deal. Related posts: Student research: Impressive and making a difference Of the essence: Coordinated training for government evaluators Principled Evaluation: Who Cares?...
 
Managing for results has gained significant momentum in the public sector over the past two decades, strengthening the disciplines of strategic planning, risk management, audit, evaluation, and performance measurement and reporting....
 
“Is performance management still relevant?” was the thought-provoking question posed by Murray Kronick in the February edition. The article noted that while performance management is still highly relevant…...
 
Within the last year, the Treasury Board of Canada Secretariat (TBS) released two new requirements for federal departments relating to efficiency metrics....
 
Results matter in the public sector, particularly in these challenging times of fiscal restraint, expenditure reductions, and rethinking of program and service delivery....
 
Today’s public sector environment is one in which many managers and executives are navigating their way through uncharted territory looking for new and innovative ways to provide programs and internal services, the result of a global economic crisis and subsequent federal budgets in 2011 and 2012....
 
Just as with public financial reporting, performance reporting is here to stay. Just as with financial reporting, it will never be entirely free of controversy....
 
Being a high-performing organization is like being a high-performing individual: it requires a vision of excellence and a commitment to achieving it....
 
With the current reality of deficit reduction, cost cutting and overall cost constraints, are senior public sector managers still committed to measuring and managing performance?...
 
This article is New and Improved! After an exhaustive review cycle with an extensive team of researchers and editors, we’ve enhanced it from the previous version!...
 
Canada was at the table when the “Commonwealth of Nations” was reconstituted in 1949 and has remained one of its strongest supporters ever since....
 
If you believe that performance measurement is a vital part of feedback about whether what you are doing is reaching people and making the difference you want, then perhaps when times are tough, you need to do it more rather than less....
 
The last great international effort to refashion the public sector took place during the 1980s in tandem with efforts of conservative politicians to limit the size of government and to lower taxes in order to stimulate economic growth....
 
Beginning on April 1, 2013, and over a five-year cycle, the evaluation function in large federal departments and agencies will face a significant challenge…...
 
Governments are increasingly using benchmarking to improve organizational performance; to determine how well their organizations are performing through measuring and comparing their actual performance with others....
 
Senior public sector managers, deputy heads of departments and parliamentarians need to maintain a sound understanding of the role of effective oversight to positively influence public sector decision making and results for Canadians....
 
Canada is a hockey-loving nation and now is the time of year when performance in the National Hockey League comes to the forefront....
 
Too few public sector organizations have truly embedded performance thinking into everyday decision making. Performance monitoring and reporting is still often considered a ‘necessary evil’ to meet either corporate or central agency compliance reporting requirements....
 
CGE Vol.13 No.3 March 2007 Governing is never easy. And in this century, it’s getting a lot harder. Economic, social, environ...
 
CGE Vol.13 No.2 February 2007 Improving organizational health and employee engagement is one of the ten top challenges facing Dep...
 
CGE Vol.13 No.1 January 2007 The proposed Federal Accountability Act is the government’s response to the Auditor...
 
CGE Vol.14 No.2 February 2008 The passage of the Federal Accountability Act (FAA) is the federal government’s most recent effo...
 
CGE Vol. 14 No.4 April 2008 The Tellier-Mazankowski advisory committee on the federal public service delivered its second report to Prime Minister Stephen...
 
Performance management leads to either disdain or glee in public sector organizations. Both reactions rely on a faulty understanding on how to make it work....
 
Les administrations publiques utilisent de plus en plus l’analyse comparative pour améliorer le rendement organisationnel, et pour évaluer leur performance réelle en comparant les paramètres mesurés avec ceux d’autres organismes....
 
The redesign of corporate tax administration for Ontario is a major modernization initiative that transferred Ontario’s corporate income tax administration from the Ontario Ministry of Revenue (OMoR) to the Canada Revenue Agency (CRA)....
 
Coincidental with the push for more attention to be paid to performance management, the Supreme Court of Canada (SCC) has provided clarity and relief for public sector bodies dealing with the...
 
The BC Public Service has had an intense focus on improving employee engagement for three years. We’ve done that using one of the most probing and comprehensive engagement models applied anywhere in...
 
La reconception de l’administration de l’imposition des sociétés en Ontario est une initiative de modernisation d’envergure, dans le cadre de laquelle le ministère du Revenu de l’Ontario (MRO) a transféré...
 
Dans son budget de lutte contre la récession (et d’économies dans la fonction publique), le gouvernement Harper a prévu une série de dépenses sans précédent dans un vaste éventail de secteurs....
 
The Harper government’s recession fighting (and government saving) budget heralded an unprecedented flurry of spending in a vast range of areas....
 
Many governments have bought into the notion of the Balanced Scorecard, the system of performance management by Robert Kaplan and David Norton that is commonly used in the private sector....
 
David P. Norton est le cocréateur, avec Robert Kaplan, du Tableau de bord prospectif, un outil permettant de lier les opérations courantes d’un organisme à sa mission à long terme...
 
Together with Robert Kaplan, David P. Norton is the creator of The Balanced Scorecard, a means of linking an organization’s current actions to its long-term mission....
 
Il y a dix ans, la Société canadienne du sang (SCS) prenait les rênes d’un organisme en pleine déroute à la suite d’une sombre affaire de sang contaminé...
 
Début septembre, la fonction publique de l’Ontario (FPO) célébrait le 10e anniversaire de  Showcase Ontario, sa conférence annuelle d’apprentissage sur l’inf...
 
Bon nombre d’administrations publiques ont adopté le tableau de bord prospectif, une méthode de gestion du rendement mise au point par Robert Kaplan et David Norton et couramment employée dans le secteur privé....
 
Today’s public sector managers encounter increasingly complex corporate issues that call for empirically-based insight presented in a readily digestible way....
 
Performance management seems to be everywhere these days, with much of the push coming from central agencies....
 
The two key tools used by the Canadian government to measure program or policy “results” are performance monitoring and evaluation....
 
The principle is unassailable: Canadians have the right to know whether their foreign aid money is being effectively spent....
 
L’administration publique s’efforce d’obtenir des résultats tangibles en mettant en œuvre des politiques et des services bien ciblés....
 
Une étude récente commandée par l’OCDE conclut que, malgré le fait louable que le secteur public oriente aujourd’hui la gestion de la performance sur l’obtention de résultats…...
 
A recent study commissioned by the OECD concludes that while the focus of performance management in today’s public sector is rightly on delivering results, particularly service quality, governments in the 80 or so democratic regimes around the world do not appear to be meeting citizen expectations....
 
Some title Some author
Some excerpt
 
Some title Some author
Some excerpt
 
Some title Some author
Some excerpt
 
Some title Some author
Some excerpt
 
Some title Some author
Some excerpt
As the resettlement of Syrian Refugees exemplifies, government programs, currently running on RBM, can be made more accountable if they were managed using PM principles, as well. The Organization for Economic Cooperation and Development (OECD) has outlined five accountability criteria for evaluating projects: relevance, effectiveness, efficiency, sustainability, and impact. Setting the latter two aside, for...