As a consultant, I work primarily with private sector organizations including the “big-five” banks, professional governing bodies and regulators for accountants, lawyers and trustees in bankruptcy. These businesses have deeply rooted performance measurement cultures, but are often unfamiliar with program evaluation. Managing evaluations within these organizations often entails accentuating evaluation capacity building while simultaneously enhancing the utility of the evaluation process and its findings.
In situations like these, I turn to Michael Quinn Patton’s Utilization-focused Evaluation. Patton emphasizes placing primary intended users (PIUs) at the centre of the evaluation. His approach describes multiple steps to managing evaluations while promoting the utility of the evaluation. Parts of his framework focuses on the need to build relationships that promote not only evaluation management, but also evaluation capacity.
Patton also describes how role definition is important not just for the external consultant but for the organization’s personnel. An external evaluator can be especially helpful in identifying the primary intended users (PIU) of the evaluation. Key to the approach is scoping out the primary purposes and intended uses of the evaluation. The external evaluator’s relationship with the PIUs is key to evaluation success. It entails meeting the company’s evaluation objectives. It must also ensure that evaluation findings can be used in program-related decisions. The external evaluator can also lead by example in engaging stakeholders in meaningful decisions linked to the evaluation function.
Patton contends that fostering evaluation is a personal mission. My experiences confirm this. Cultivating relationships with people in the organization that want to learn about the evaluation process and its ties to program improvement is a key to enhanced evaluation use. These people do not just represent the authority in the organization. They are strategically located. They are close to the programs. They are enthusiastic, committed, competent and curious about evaluation.
The adoption of a “user-friendly” evaluation tactic that emphasizes collaboration, participation and empowerment increases the odds that evaluations will be utilized, and deemed a valuable process in the company or corporation.
Figure 1 summarizes Patton’s approach to managing evaluations for enhanced utility. In my experience, the multi-step method serves best as a road map for the external consultant and private sector organization. The clear logic is that all interested parties must be convinced about what the evaluation process will deliver. They must know what their roles are. This goes a long way to creating comfort with evaluation, especially in an organizational culture where the reflex is not in that direction of evaluation. The framework is general enough to be flexible and it can be adapted. It can assist any organization build evaluation organizational capacity to adjust to the evaluation process.
Figure 1
Patton’s Methodology for Managing Evaluations for Greater Utility
Successful implementation
The first step in my implementation of evaluation activities and simultaneous capacity building is to first establish stakeholder needs. It must identify primary intended users. The external evaluator must clearly assist the organization in defining roles and responsibilities of all stakeholders. These include the PIUs such as (a) the organization’s leadership through a project advisory board/committee; (b) the evaluation proponent or facilitator within the organization–which I refer to as the “evaluation champion” –and (c) the program operation staff. Having clear roles and responsibilities allows the external evaluator to forge relationships of different natures with the various interested parties.
It also helps to scope the time-commitments of various parties. While the evaluator sports many hats in these relationships, s/he wears only one at a time. For example, in meetings with organizational leadership, the evaluator is the expert, hired by the organization to complete a project, and must convey progress on bringing that project to its successful completion. But when the evaluator interacts with the organization’s evaluation champion, the former becomes an enabler, empowering the champion to make sense of the evaluation process and resultant data, and to sustain evaluative efforts after project completion.
I have been often asked why is there the need for an evaluation management framework to build these types of relationships. My response is that the framework allows all involved parties to see where they are on the evaluation roadmap. Once roles and responsibilities are identified, each party within the company better understands the evaluation process and where they fit. The feeling of comfort and reassurance that this provides cannot be underestimated. The framework’s orientation function may seem elementary, but in organizations with no evaluation culture, it is immensely beneficial in building such capacity of the individuals involved.
Lessons learned
The multi-step framework and its use as a road map, especially in stakeholder analysis and identification of PIUs to clearly delineate roles and responsibilities, also helps in building of relationships within the organization. The collective effort made to understand the historical context of the organization and program is invaluable. It is important for external evaluators to know what efforts have been made before to bring about program improvement, how well they worked, and who was involved in the process.
Most people are naturally reluctant to venture down a road they have already unsuccessfully travelled. They are especially suspicious about being led by someone external to their company. The nature and strength of relationships forged via the framework also provides confidence to the evaluator in adherence to ethical and professional obligations without fear of reprisal during times of organizational pushback. The fostering of relationships within a context of professional conduct helps to ensure relevance of evaluation processes and utilization of results within the organization on an ongoing basis. Here, the external evaluator is viewed not only as the “expert” but also the “enabler.”
Though I have had much success using this PIU-focused evaluation framework, not all evaluations have run smoothly. Recently, my team and I completed a comprehensive evaluation of the national certification program for a group of financial professionals. The framework identified the course for the evaluation, clarified people’s roles, identified PIUs and marshalled the efforts of an internal evaluation champion. When it came time to make our recommendations, which would have improved and altered the course of the program, we received a great deal of organizational pushback. But the organization’s evaluation champion echoed our opinions. Further, the capacity we had been building with the executive steering committee allowed them to better understand the evaluative processes and resulting recommendations.
Because we fostered strong relationships within the organization, our suggestions were not viewed as threatening. In the end, our recommendations were implemented, some program operations were altered and an improved program emerged. I do not believe that any of this could have realized if not for the application of the multi-step evaluation framework and its focus on PIUs. It provides a clear path, where few alternatives exist, for evaluation management and capacity building within a private sector organization that lack an evaluation culture.
A. Sidiq Ali, PhD CE, is Managing Director of Research & Evaluation Consulting, Inc. His blog is www.rae-consult.com
