Over the last four decades, public sector organizations have institutionalized program and policy evaluation practices to varying degrees. As a management tool, evaluation has the potential to provide timely, useful information to decision-makers concerned with resource optimization and program outcomes.
However, piecemeal approaches to evaluation are commonplace in various public administrations, including the federal government. Such approaches tend to produce operational recommendations that hinder the ability of evaluators to support management and contribute meaningfully to organizational learning, accountability and improvement.
Building capacity
Organizations interested in developing a more strategic approach to program evaluation typically focus their efforts on staff training and budgetary allocations. Although these measures are an important starting point, the institutionalization of evaluation requires additional effort.
Researchers have turned their attention to this issue over the last decade and have dubbed the process “organizational evaluation capacity building.” Evaluation capacity can take on different forms, depending on an organization’s context; our own research has focused, therefore, on the development of an empirically-based description of organizational evaluation capacity in the federal government.
Our organizational evaluation capacity framework includes six main dimensions. The first three focus on the organization’s ability to produce high-quality evaluation studies, while the others describe the organization’s ability to use evaluation for organizational learning and accountability purposes.
Human Resources, the first dimension, refers to the technical and interpersonal competencies of evaluators as well as evaluation leadership within the organization. The second dimension, Organizational Resources, focuses on the systems and structures available to evaluators to carry out their duties, such as the financial resources allocated to evaluation studies and the availability of quality performance measurement data. Evaluation Planning and Activities, the third dimension, describes how evaluation is planned and conducted in the organization. This dimension includes, for example, the use of consultants and other external resources to complement internal capabilities and skill sets.
Evaluation Literacy is the fourth dimension and the first to describe organizational capacity to use evaluation. It refers to the widespread knowledge of evaluation and its associated concepts, such as results-based management, across the organization. Organizational Decision Making characterizes the fifth dimension and focuses on the integration of evaluation findings in broader organizational processes and decision-making, such as program resourcing or deficit reduction activities. Finally, Learning Benefits describes how evaluation is used for learning and improvement in the organization, which is the ultimate objective of evaluation itself and evaluation capacity building more specifically.
In addition to the six dimensions and their associated sub-dimensions, the framework also outlines four levels of evaluation capacity: low, developing, intermediate and exemplary capacity. Each of the sub-dimensions, therefore, is described qualitatively at each of those four levels in order to profile and highlight the incremental steps required to move from one level of capacity to the next.
Managers interested in assessing their own organization’s evaluation capacity may use this organizational profile framework to situate their organization across the four levels for all six dimensions. Alternatively, they may also be interested in using an organizational self-assessment instrument, which operationalizes the original framework and facilitates the assessment process.
Although the instrument is meant to be used by federal government departments and agencies, it is possible to adapt the tool to other organizational contexts. We have already adapted the instrument to the not-for-profit sector and the Quebec provincial government, and we are continuing this work to render the tool suitable for use in the Ontario public health sector. Fundamentally, the six dimensions apply to most public sector settings and it is possible to use the instrument in all of these, albeit in a slightly modified form that accounts for local contexts and terminology.
Organizational improvement
Our research shows that organizations that have a good sense of their evaluation capacity are better able to devise strategies to address any major gaps and continue on their path towards institutionalization.
We also know, based on our own sectoral comparative analyses, that organizations with higher evaluation capacity get the greatest “bang for their evaluation buck”; not only are their studies of greater quality and more timely, they also better anticipate senior decisio- makers’ information needs. This, in turn, leads to greater evaluation use and organizational learning.
The link to improved program design and administration, and to a more effective and efficient use of organizational resources, makes this endeavour wholly appealing, not only to evaluators, but to managers across all levels of public sector organizations.
