Building Better Buy-in for Evaluation - Canadian Government Executive
PerformanceProgram Evaluation
June 25, 2018

Building Better Buy-in for Evaluation

Evaluation may provide information relevant to broader public processes such as program improvement, decision-making or accountability. Nonetheless, evaluation’s positive contribution to these goals may be affected by resistance from stakeholders associated with the program being evaluated.

Resistance to evaluation may take various forms at different stages of an evaluation initiative. Prior to beginning the evaluation, resistance may result from anxiety about the process or an anticipation of the effects of evaluation. For example, program managers may be afraid that the evaluation will involve a judgment on the quality of their work. Resistance could manifest itself as a set of arguments around the program specific characteristics, contesting the relevance of conducting an evaluation.

During evaluation, resistance may affect – passively or actively – the conduct of the evaluative process. Resistant stakeholders may try to generate barriers to the implementation of evaluation activities, for instance by refusing to participate, distorting information or altering the evaluation procedures. Resistance may arise, among other sources, from first line intervening actors who consider the exercise a waste of time affecting the fulfillment of their mission. It may also arise from program managers who question the timing of the evaluation in relation to their project management agenda. At the end of the evaluation initiative, resistance may involve challenging the legitimacy of the evaluation results and discrediting the evaluation by attacking the methodology or skills of the evaluator. Again here, an anticipation of the effects of evaluation may cause resistance. For example, a director may want to preserve the notoriety of its organization and perceive that evaluation results will feed critics outside the organization or potentially involve cut in program funding. Municipal actors may fear that evaluation results affect their city’s reputation. Indeed, evaluation – like other organizational processes – is at the centre of different interests and information needs that may contradict or complement each other.

A good understanding of the dynamics associated with resistance and propensity to evaluation is important. It can help decision-makers and managers, as sponsors of evaluation processes, to develop evaluation procedures and strategies minimizing resistance and maximizing propensity – that is to say, a positive inclination toward evaluation – from stakeholders.

Building on empirical examples, we discuss prominent factors influencing stakeholders’ perceptions as well as suggestions to prevent resistance and encourage the development of a positive orientation toward evaluation.

Finding the Path of Least Resistance in Evaluation

Factors at the individual and contextual levels, as well as the characteristics of the evaluation process, may have an impact on stakeholders’ perceptions regarding evaluation.

Personal characteristics such as a fear of criticism or a negative experience in previous evaluation processes may influence individual feelings and expectations with regard to the evaluation process. For instance, in a study involving professionals from a health institution in Haiti, Pernelle Smits identified factors influencing propensity among the group of participants. Individuals characterized with a higher propensity for evaluation had past positive experiences linked with evaluation activities, professional responsibilities involving a supporting role for evaluation, and identified themselves with individual and organizational performance. They were also giving a particular importance to quality of work and had a strong orientation toward processes and the possibility to act upon them, as well as a longer-term perspective on results.

From another standpoint, a study in two public sector organizations in the United Kingdom identified barriers to evaluation associated with the organizational level. The researchers identified obstacles such as a lack of mutual trust between stakeholders and the evaluator, diverging interests among stakeholders, and a perception by actors at lower hierarchical levels that evaluation was an imposed and rigid process. Their results demonstrated that some characteristics of the organizational environment may contribute to inhibiting evaluation, such as prevailing institutional values regarding evaluation and the organizational and relational environment in which evaluation is implemented.

At the evaluation level, the role of the evaluator, the choice of an evaluation approach, the design of data collection instruments and the quality of information produced by the evaluation may influence stakeholders’ propensity or resistance. In a study reporting on the evaluation of a training program within an organization with established evaluation practices, a team of evaluators noted changes following the adoption of a new national vocational training policy. They observed how this external factor generated uncertainty regarding the future of the program and created fear regarding the potential consequences of the evaluation process in case of negative findings. As a result, it contributed to the development of resistance to the evaluation process. In this context, drafting a contract between the evaluators and program stakeholders contributed to reducing participants’ anxiety. Under this contractual agreement, both parties committed themselves to self-examination and mutual consultation upon manifestations of resistance. They also agreed on creating spaces for open discussion on the objectives of the evaluation, and exchange of ideas and concerns regarding the evaluation, its results and controversial issues.

The main lesson learned in this case concerned the relevance of supplementing the standard evaluation contract with a formalized agreement between stakeholders and the evaluator covering technical and ethical considerations. This recommendation also applies in situations where evaluation is an established practice in the organization. The occurrence of an unforeseen event (change in the policy environment) contributed to changes in stakeholders’ perceptions. The agreement provided a formal space for specifying collaboration rules related to problem resolution, communication lines and the nature of interactions in the context of the evaluation process.

Evaluation: It’s the Trust Thing 

The diversity of contexts in which evaluation is implemented prevents from adopting a “one-size-fits-all” solution to support stakeholders’ propensity for evaluation. Nevertheless, some general principles issued from research combining social and organizational psychology as well as evaluation can contribute to decision-making in these situations. First, situations in which individuals feel that evaluation represents a threat to their power, control, autonomy and self-image generate negative feelings about evaluation and increase the risks of developing resistance. Second, previous experiences with evaluation or similar processes have an influence on perceptions toward evaluation. Accordingly, positive experiences with evaluation will contribute to future positive anticipation about the role of evaluation. Third, conflict and competition related to individual and organizational goals may influence stakeholders’ attitudes and behaviours toward evaluation. Facilitating the development of a trusting relationship (among stakeholders and with the evaluator) and identifying common goals to which evaluation is contributing may provide conditions for a stronger support to the evaluation process.

To support stakeholders’ buy-in, different measures can be implemented before (or at the planning stage), during and after the evaluation process. At the beginning of an evaluation, tools such as evaluability assessments may help identify upstream relevant factors that could influence stakeholders’ resistance or support to the evaluation process. Taking time to understand manifestations of resistance (at any time during the evaluation) to discover their underlying causes and then identifying appropriate ways to deal with them is a key aspect to a successful management of the evaluation process.

From another standpoint, increased stakeholders’ participation in planning and technical decision-making regarding evaluation can contribute to generating positive feelings toward evaluation as participants acquire a greater knowledge of evaluation’s activities and purposes. It is also a good idea to begin with involving the most supportive or enthusiastic stakeholders in the evaluation process. They may act as change agents and contribute to creating a virtuous circle around evaluation. Participation is also relevant in the final phases of the evaluation process when presenting and diffusing results. Generating spaces for discussions and identifying learning opportunities on the basis of the evaluation’s results creates a stronger appropriation and increases the likeliness that key stakeholders will support the implementation of the recommendations from the evaluation.

 

Marie-Hélène L’Heureux is a doctoral candidate in Political Science at Laval University (Québec City, Canada) and a member of the Canadian Evaluation Society (CES). She is also a research assistant at PerfEval, a research laboratory on public policy performance and evaluation (www.perfeval.net).

Steve Jacob is a member of the Canadian Evaluation Society (CES), a full professor in the Department of Political Science at Laval University (Québec City, Canada), and the director of PerfEval, a research laboratory on public policy performance and evaluation (www.perfeval.net).

Pernelle Smits is past president of SQEP and acted as a board member of IOCE and RFE. She is a professor at the Department of Management at Laval University, teaches evaluation in China, Africa, among others, and supports the emergence and reinforcement of national evaluation systems.

 

 

 

About this author

Marie-Hélène L’Heureux, Steve Jacob, and Pernelle Smits

Marie-Hélène L’Heureux is a doctoral candidate in Political Science at Laval University (Québec City, Canada) and a member of the Canadian Evaluation Society (CES). She is also a research assistant at PerfEval, a research laboratory on public policy performance and evaluation (www.perfeval.net). Steve Jacob is a member of the Canadian Evaluation Society (CES), a full professor in the Department of Political Science at Laval University (Québec City, Canada), and the director of PerfEval, a research laboratory on public policy performance and evaluation (www.perfeval.net). Pernelle Smits is past president of SQEP and acted as a board member of IOCE and RFE. She is a professor at the Department of Management at Laval University, teaches evaluation in China, Africa, among others, and supports the emergence and reinforcement of national evaluation systems.

0 comments

There are no comments for this post yet.

Be the first to comment. Click here.

Performance
 
In this episode of CGE Radio, J. Richard Jones talks with...
 
Everyone wants to change, become more efficient, and drive better value...
 
A healthy Canadian economy and opportunities outside of North America are...
 
Evaluation may provide information relevant to broader public processes such as...
 
Did your high school valedictorian go on to achieve greatness? High...
 
Compared to audit professionals and their preoccupation with “independence”, evaluation specialists...
 
Having a clear definition of success is one of the key...
 
Does your organization have a very narrow view of what digital...
 
We are happy to share with you the September 2017 issue...
 
The design needed and practiced in government departments and the broader...
 
Behavioural Insights (BI) is a multidisciplinary approach that uses principles and...
 
One of the writers in this month’s issue started a note...
 
The Senate has long been controversial largely because it is unelected...
 
We have all heard the sayings: “Home, sweet home”; “There’s no...
 
The year 2016 saw two critical words circling the halls of...
 
Having the right talent is critical to delivering right results. But...
 
What we do in life echoes in eternity (GLADIATOR (2000) An...
 
An article in The Economist (March 8, 2015) notes that the...
 
Since 2010, experts and academics eager to connect “systems thinking” and...
 
There is already a surging literature on the application of design...
 
“The secret to happiness is to put the burden of proof...
 
In a word: Stocktakes. Mirriam-Webster defines stocktaking as “the act of estimating...
 
Today’s transformative movement towards a more digital and connected government is...
 
You’ve probably heard a lot in recent months about the Phoenix payroll disaster . Between...
 
Contrary to the stereotype of eagerness and politesse, the second Bold...
 
A study conducted by Robert Half, the world’s largest specialized staffing...
 
CFOs around the world are changing – but an EY survey...
 
As an executive you’ve probably experienced frustration when you ask for...
 
The Liberal government is taking steps to rein in the National...
 
When I find myself in times of trouble, Mother Mary comes...
 
Leadership BS By Jeffrey Pfeffer Harper Business, 259 pages, $36.99 If...
 
It almost goes without saying that good governance requires fair and...
 
At first glance, the title may seem counterintuitive. By their very...
 
Every reputable learning institution offers leadership development training nowadays. So why...
 
Executive compensation is considered to be a central component of corporate...
 
Rushing to keep up with deadlines but never getting ahead is...
 
A leader’s credibility begins with personal success. It ends with helping...
 
Written by Donald Farmer Too often, we base Business Intelligence today...
 
Written by  Gail Vallance Barrington Anyone who has commissioned a program...
 
Written by  Patrice Dutil It’s hard to believe we are still...
 
Public sector innovation, once dismissed as a contradiction in terms, is...
 
Water is essential to life, yet this resource is not being...
 
Agility is becoming a crucial element of a successful business. Technology...
 
Data is increasingly becoming fundamental to the world we live in....
 
Municipal governments have a number of responsibilities, but chief among them...
 
Last week, Rick Conlow and Doug Watsabaugh wrote in Businessweek that...
 
Colin Talbot’s recent book, Theories of Performance, Organizational and Service Improvement...
 
This time of year the Clerk of the Privy Council puts...
 
The British Columbia Public Service has taken significant steps to implement...
 
On October 29, General Tom Lawson accepted the responsibility of Chief...
 
CGE Vol.13 No.5 May 2007 "…however fast they went, they never...
 
CGE Vol.13 No.2 February 2007 The leadership literature is littered with...
 
CGE Vol. 14 No.4 April 2008 “Be In This Place” is...
 
Although a recent international poll indicates that Canada’s public sector is...
 
CGE Vol.14 No.2 February 2008 As David Dodge steps down today...
 
CGE Vol.13 No.9 November 2007 Organizational culture plays a strong role...
 
CGE Vol.13 No.8 October 2007 The New Brunswick Department of Transportation...
 
Quote of the week “The job we have here is to...
 
I think the idea that someday the department’s lights will turn...
 
What keeps you up at night? The Institute of Public Administration...
 
As I have approached the end of my term as Auditor...
 
Given the budgetary challenges facing most government organizations, the use of...
 
Some title Some author
Some excerpt
In this episode of CGE Radio, J. Richard Jones talks with...