Building Better Buy-in for Evaluation - Canadian Government Executive
PerformanceProgram Evaluation
June 25, 2018

Building Better Buy-in for Evaluation

Evaluation may provide information relevant to broader public processes such as program improvement, decision-making or accountability. Nonetheless, evaluation’s positive contribution to these goals may be affected by resistance from stakeholders associated with the program being evaluated.

Resistance to evaluation may take various forms at different stages of an evaluation initiative. Prior to beginning the evaluation, resistance may result from anxiety about the process or an anticipation of the effects of evaluation. For example, program managers may be afraid that the evaluation will involve a judgment on the quality of their work. Resistance could manifest itself as a set of arguments around the program specific characteristics, contesting the relevance of conducting an evaluation.

During evaluation, resistance may affect – passively or actively – the conduct of the evaluative process. Resistant stakeholders may try to generate barriers to the implementation of evaluation activities, for instance by refusing to participate, distorting information or altering the evaluation procedures. Resistance may arise, among other sources, from first line intervening actors who consider the exercise a waste of time affecting the fulfillment of their mission. It may also arise from program managers who question the timing of the evaluation in relation to their project management agenda. At the end of the evaluation initiative, resistance may involve challenging the legitimacy of the evaluation results and discrediting the evaluation by attacking the methodology or skills of the evaluator. Again here, an anticipation of the effects of evaluation may cause resistance. For example, a director may want to preserve the notoriety of its organization and perceive that evaluation results will feed critics outside the organization or potentially involve cut in program funding. Municipal actors may fear that evaluation results affect their city’s reputation. Indeed, evaluation – like other organizational processes – is at the centre of different interests and information needs that may contradict or complement each other.

A good understanding of the dynamics associated with resistance and propensity to evaluation is important. It can help decision-makers and managers, as sponsors of evaluation processes, to develop evaluation procedures and strategies minimizing resistance and maximizing propensity – that is to say, a positive inclination toward evaluation – from stakeholders.

Building on empirical examples, we discuss prominent factors influencing stakeholders’ perceptions as well as suggestions to prevent resistance and encourage the development of a positive orientation toward evaluation.

Finding the Path of Least Resistance in Evaluation

Factors at the individual and contextual levels, as well as the characteristics of the evaluation process, may have an impact on stakeholders’ perceptions regarding evaluation.

Personal characteristics such as a fear of criticism or a negative experience in previous evaluation processes may influence individual feelings and expectations with regard to the evaluation process. For instance, in a study involving professionals from a health institution in Haiti, Pernelle Smits identified factors influencing propensity among the group of participants. Individuals characterized with a higher propensity for evaluation had past positive experiences linked with evaluation activities, professional responsibilities involving a supporting role for evaluation, and identified themselves with individual and organizational performance. They were also giving a particular importance to quality of work and had a strong orientation toward processes and the possibility to act upon them, as well as a longer-term perspective on results.

From another standpoint, a study in two public sector organizations in the United Kingdom identified barriers to evaluation associated with the organizational level. The researchers identified obstacles such as a lack of mutual trust between stakeholders and the evaluator, diverging interests among stakeholders, and a perception by actors at lower hierarchical levels that evaluation was an imposed and rigid process. Their results demonstrated that some characteristics of the organizational environment may contribute to inhibiting evaluation, such as prevailing institutional values regarding evaluation and the organizational and relational environment in which evaluation is implemented.

At the evaluation level, the role of the evaluator, the choice of an evaluation approach, the design of data collection instruments and the quality of information produced by the evaluation may influence stakeholders’ propensity or resistance. In a study reporting on the evaluation of a training program within an organization with established evaluation practices, a team of evaluators noted changes following the adoption of a new national vocational training policy. They observed how this external factor generated uncertainty regarding the future of the program and created fear regarding the potential consequences of the evaluation process in case of negative findings. As a result, it contributed to the development of resistance to the evaluation process. In this context, drafting a contract between the evaluators and program stakeholders contributed to reducing participants’ anxiety. Under this contractual agreement, both parties committed themselves to self-examination and mutual consultation upon manifestations of resistance. They also agreed on creating spaces for open discussion on the objectives of the evaluation, and exchange of ideas and concerns regarding the evaluation, its results and controversial issues.

The main lesson learned in this case concerned the relevance of supplementing the standard evaluation contract with a formalized agreement between stakeholders and the evaluator covering technical and ethical considerations. This recommendation also applies in situations where evaluation is an established practice in the organization. The occurrence of an unforeseen event (change in the policy environment) contributed to changes in stakeholders’ perceptions. The agreement provided a formal space for specifying collaboration rules related to problem resolution, communication lines and the nature of interactions in the context of the evaluation process.

Evaluation: It’s the Trust Thing 

The diversity of contexts in which evaluation is implemented prevents from adopting a “one-size-fits-all” solution to support stakeholders’ propensity for evaluation. Nevertheless, some general principles issued from research combining social and organizational psychology as well as evaluation can contribute to decision-making in these situations. First, situations in which individuals feel that evaluation represents a threat to their power, control, autonomy and self-image generate negative feelings about evaluation and increase the risks of developing resistance. Second, previous experiences with evaluation or similar processes have an influence on perceptions toward evaluation. Accordingly, positive experiences with evaluation will contribute to future positive anticipation about the role of evaluation. Third, conflict and competition related to individual and organizational goals may influence stakeholders’ attitudes and behaviours toward evaluation. Facilitating the development of a trusting relationship (among stakeholders and with the evaluator) and identifying common goals to which evaluation is contributing may provide conditions for a stronger support to the evaluation process.

To support stakeholders’ buy-in, different measures can be implemented before (or at the planning stage), during and after the evaluation process. At the beginning of an evaluation, tools such as evaluability assessments may help identify upstream relevant factors that could influence stakeholders’ resistance or support to the evaluation process. Taking time to understand manifestations of resistance (at any time during the evaluation) to discover their underlying causes and then identifying appropriate ways to deal with them is a key aspect to a successful management of the evaluation process.

From another standpoint, increased stakeholders’ participation in planning and technical decision-making regarding evaluation can contribute to generating positive feelings toward evaluation as participants acquire a greater knowledge of evaluation’s activities and purposes. It is also a good idea to begin with involving the most supportive or enthusiastic stakeholders in the evaluation process. They may act as change agents and contribute to creating a virtuous circle around evaluation. Participation is also relevant in the final phases of the evaluation process when presenting and diffusing results. Generating spaces for discussions and identifying learning opportunities on the basis of the evaluation’s results creates a stronger appropriation and increases the likeliness that key stakeholders will support the implementation of the recommendations from the evaluation.

 

Marie-Hélène L’Heureux is a doctoral candidate in Political Science at Laval University (Québec City, Canada) and a member of the Canadian Evaluation Society (CES). She is also a research assistant at PerfEval, a research laboratory on public policy performance and evaluation (www.perfeval.net).

Steve Jacob is a member of the Canadian Evaluation Society (CES), a full professor in the Department of Political Science at Laval University (Québec City, Canada), and the director of PerfEval, a research laboratory on public policy performance and evaluation (www.perfeval.net).

Pernelle Smits is past president of SQEP and acted as a board member of IOCE and RFE. She is a professor at the Department of Management at Laval University, teaches evaluation in China, Africa, among others, and supports the emergence and reinforcement of national evaluation systems.

 

 

 

About this author

Marie-Hélène L’Heureux, Steve Jacob, and Pernelle Smits

Marie-Hélène L’Heureux is a doctoral candidate in Political Science at Laval University (Québec City, Canada) and a member of the Canadian Evaluation Society (CES). She is also a research assistant at PerfEval, a research laboratory on public policy performance and evaluation (www.perfeval.net). Steve Jacob is a member of the Canadian Evaluation Society (CES), a full professor in the Department of Political Science at Laval University (Québec City, Canada), and the director of PerfEval, a research laboratory on public policy performance and evaluation (www.perfeval.net). Pernelle Smits is past president of SQEP and acted as a board member of IOCE and RFE. She is a professor at the Department of Management at Laval University, teaches evaluation in China, Africa, among others, and supports the emergence and reinforcement of national evaluation systems.

0 comments

There are no comments for this post yet.

Be the first to comment. Click here.

Performance
 
Evaluation may provide information relevant to broader public processes such as program improvement, decision-making or accountability. Nonetheless, evaluation’s positive contribution to these goals may be affected by resistance from stakeholders associated with the program being evaluated. Resistance to evaluation may take various forms at different stages of an evaluation initiative. Prior to beginning the evaluation,...
 
In this episode, Marcello Sukhdeo fills in for your regular hosts. He speaks with Harry Cummings, President of the Canadian Evaluation Society. Hear about evaluation, who can be evaluators, why evaluate, why it is special and needed, challenges and the future for evaluation. To learn more and to download CES’ handy guides, visit their website  here...
 
Did your high school valedictorian go on to achieve greatness? High schools select their valedictorians because they show promise and exemplify the best the school has to offer. So it’s not unreasonable for us to expect them to achieve great things. Many achieve success in their future careers. But greatness tends to be rare. And...
 
Compared to audit professionals and their preoccupation with “independence”, evaluation specialists may seem unconcerned about ethical principles. Their fixation is on systematic processes, methods and techniques. Principles underlying evaluations rarely come up in evaluation planning or implementation, and they seldom figure in evaluation reports or follow-ups. Give a workshop to evaluators on “big data” or...
 
Having a clear definition of success is one of the key components in achieving the desired results of a project but oftentimes, many projects become too complex to manage. In many of these cases, major projects and programs which are considered initially “complicated” in nature, end up in “chaos”, and often subject to visceral media...
 
Does your organization have a very narrow view of what digital means in today’s economy? If so, then your agency may be losing out on the opportunity to unlock the true value of its digital assets. Overall, Canadians are upbeat about the principles of digital strategy. As many as seven out of 10 executives in...
 
We are happy to share with you the September 2017 issue of Canadian Government Executive. In this issue, you’ll find the following articles: Fostering Resilience: How to Effectively Manage Workplace Stress Mental Health in the Virtual Workplace: a Cross-country Checkup The Best Public Service in the World You Say? Leveraging Internal Audit Information Making Channel...
 
The design needed and practiced in government departments and the broader public sector is often of a systemic nature. Typically, planned actions require a substantial period of time for implementation and the actual outcomes from these actions can only be observed after an even longer period. This is the case in education, healthcare, economic development,...
 
Behavioural Insights (BI) is a multidisciplinary approach that uses principles and methodologies from the behavioural sciences such as behavioural economics, marketing, and psychology. It is attracting considerable attention from researchers and policy makers around the world as a valuable and promising tool to improve governments’ programs, service delivery processes and relevant client outcomes. In the...
 
One of the writers in this month’s issue started a note by saying “Did you know that some of the world’s most interesting analysis of employment programs is being conducted in Canada?” It struck me as a strange line. It had a familiar, if odd, lacquer; a mix of pride and frustration.  I call it...
 
On the morning of October 22, 2014, I lived an extraordinary event on Parliament Hill. We were having our weekly management meeting when we were brusquely interrupted and informed that a ceremonial guard at the Canadian National War Memorial, across the street from our location, had been shot. Being security practitioners, we all ran down...
 
The Senate has long been controversial largely because it is unelected and proves difficult to reform. The expenses scandal that broke out in 2012 made it all the more so. The allegations of misuse of public funds for personal rather than parliamentary business captured media and public attention to an unprecedented degree. In June 2013,...
 
We have all heard the sayings: “Home, sweet home”; “There’s no place like home”; “Home is where the heart is.” Home is a word that conjures up thoughts of peace, comfort, and joy: A sanctuary, or a place of relaxation where one is at ease and generally happy. Anyone who has experienced a hospitalization will...
 
Canada has made great strides to make public spaces accessible to persons with disabilities over the past several decades. Look around your government building today and you’re likely to see many amenities to support the 4.82 million Canadians with a disability. But we must remind ourselves that our world now spans beyond the physical. Government...
 
The year 2016 saw two critical words circling the halls of Ottawa – ‘results’ and ‘deliverology.’ The first is not new to the federal public service, though occasionally it is touted as a major breakthrough (let’s not forget that Results for Canadians was the centrepiece of the government’s management framework at the turn of the...
 
Having the right talent is critical to delivering right results. But in public sector, government systems are overwhelmingly designed for earlier times which is optimised for a stable group of subject matter experts that doesn’t allow the government to unlock, mobilise or even identify its available talent – regardless of whether that talent is inside...
 
What we do in life echoes in eternity (GLADIATOR (2000) An allegory is a poem, story, or picture that can be interpreted to reveal a hidden moral or political meaning. For example, Pilgrim’s Progress is an allegory of the spiritual journey. We find allegories in storied lives everywhere — legends without ending, just a place...
 
An article in The Economist (March 8, 2015) notes that the term ‘Deliverology’ was originally used to poke fun at Sir Michael Barber’s approach to managing the Delivery Unit formed by the Blair government in 2001. Sir Michael adopted the term and the rest, as they say, is history. The term was subsequently adopted by...
 
Since 2010, experts and academics eager to connect “systems thinking” and policy design have been meeting annually to share their research and insights. For the first three years, Oslo was the convener. For the second year in a row, Canada has been the meeting point. In October, they gathered in Toronto for the 5th “Relating...
 
There is already a surging literature on the application of design thinking (DT) to government services. Ressler and others have argued that DT pays dividends to many: to citizens, who benefit from more sharply focused programs and to elected governments as well, who may benefit from more satisfied voters. In many jurisdictions, the main institutional...
 
“The secret to happiness is to put the burden of proof on unhappiness.” Robert Brault When public administration students insist upon a working definition of accountability, textbooks rarely satisfy. They are puzzled when accountability is equated with responsibility, liability, or answerability. Whatever interpretation is assumed gets probed and debated for the rest of the course....
 
In a word: Stocktakes. Mirriam-Webster defines stocktaking as “the act of estimating a situation at any given moment”. That’s not so bad right? Unfortunately, in business the underlying risk of frequent stocktakes is the unspoken mindset that: if we set targets, measure performance against the target, introduce consequence for missing the target then people will be...
 
PPX’s Annual Symposium was held May 17–19, 2016 in Ottawa, with the theme “Planning and Performance: Learning from the Past, Preparing for the Future”. This year’s Symposium saw a record 233 participants in attendance, and was particularly special as PPX celebrated its 20th anniversary. In keeping with the “Preparing for the Future” portion of this...
 
Today’s transformative movement towards a more digital and connected government is driven by the technological trends of big data, social networks, the “Internet of Things,” machine learning and bots. While these technologies can interact with one another, they all share the common foundation of cloud computing. Governments see the benefit of these new technological trends...
 
You’ve probably heard a lot in recent months about the Phoenix payroll disaster . Between unpaid wages, dodging emails and a deteriorating payroll system, the situation is nightmarish for many in the public sector. While Scott Brison, the Liberal’s Treasury Board president, claims his party has “inherited a mess”, he cannot deflect all the blame for the disaster. The...
 
Contrary to the stereotype of eagerness and politesse, the second Bold Thinking Report from NATIONAL Public Relations has found Canadians to be scornful and cynical towards businesses and governments. Along with this cynicism comes mistrust, and this mistrust blocks relationships. “Cynicism is a cancer that eats at relationships and is not sustainable,” says Kim West, Chief...
 
A study conducted by Robert Half, the world’s largest specialized staffing firm, has examined how and why companies are covering employee credential costs. Greg Scileppi, President of Robert Half’s International Staffing Operations, claims that providing and supporting ongoing training benefits company and employee productivity. “Many employers support ongoing training because they’re aware of how valuable...
 
CFOs around the world are changing – but an EY survey says Canadians might be getting left behind. Related posts: Going forward: CFOs share their strategies for advancement Training that pays for itself: survey...
 
As an executive you’ve probably experienced frustration when you ask for something and it takes weeks to appear. There is a good chance that your processes are too complex, have too many steps and require more effort than they should to get the job done. It’s known as Excessive Processing and it is one of...
 
The Liberal government is taking steps to rein in the National Shipbuilding Procurement Strategy (NSPS) which has gone over budget. As part of its new game plan, the government will not release estimates of the costs associated with what is possibly the military’s largest looming maritime purchase to date until it has the government has...
 
When I find myself in times of trouble, Mother Mary comes to me, speaking words of wisdom … JOHN LENNON & PAUL MCCARTNEY (1970): LET IT BE Let It Be was The Beatles’ twelfth and final studio album.The title song of reconciliation was Paul’s tribute to his dear departed mother, Mary.Two years earlier, The Beatles...
 
Leadership BS By Jeffrey Pfeffer Harper Business, 259 pages, $36.99 If you think that much of what you read and hear from leadership gurus is BS, you have a supporter in Jeffrey Pfeffer. At first glance he’s an unlikely backer since he’s a leadership guru himself. But he’s a professor of organizational behaviour at the...
 
It almost goes without saying that good governance requires fair and reliable performance reporting by public sector entities. Several OECD nations have developed performance reporting principles, while the Canadian government has called for concise, credible, reliable and balanced information. However, as Colin Talbot, a noted expert on performance has pointed out, these principles tend to...
 
At first glance, the title may seem counterintuitive. By their very nature, high potentials (HiPos) are assumed to be, or even defined as, very engaged. Yet, when we look at the evidence, this is not necessarily the case. A 2009 study by the Corporate Leadership Council (CLC) discovered that 25% of HiPos were planning to...
 
Every reputable learning institution offers leadership development training nowadays. So why did the Office of the Comptroller General of Canada feel compelled to develop its own Internal Audit Leadership Development Pilot Program in partnership with the Canadian Institute of Internal Auditors (IIA) and the University of Ottawa? The short answer: To develop the right kind...
 
Executive compensation is considered to be a central component of corporate governance mechanisms. It is believed that an effective and prudently designed executive compensation package can motivate managers/executives to perform their duties diligently and to look after the interests of firm owners (or shareholders). However, there is a widespread belief among the public, regulators, media...
 
Rushing to keep up with deadlines but never getting ahead is commonplace in today’s public service. In traditional thinking, we never seem to have enough capacity–people, time, budget–to keep up. There are ways to free up capacity to beat this crunch. And some of the clues are right in front of us and within our...
 
A leader’s credibility begins with personal success. It ends with helping others achieve personal success. – John Maxwell In 1966, U.S. Senator Robert Kennedy made an influential visit to South Africa. He offered words of hope to opponents of apartheid in his famous “Ripple of Hope” speech at the University of Cape Town: “Each time...
 
Written by Donald Farmer Too often, we base Business Intelligence today on a simple premise: that people are logical decision-makers. In this light, common BI practices promise that the right data, in the right format, at the right time, will help you toward better outcomes. However, as human beings, we just don’t work that way....
 
Written by  Gail Vallance Barrington Anyone who has commissioned a program evaluation knows what it is like to have a sleepless night. The reason is simple: you are concerned about having lost track of the original question. The hyper-caffeinated effect is made worse by the slow and measured process of the evaluation. Staring at a...
 
Written by  Patrice Dutil It’s hard to believe we are still talking about this, but the numbers confirm the anecdotal evidence. The reality is that government departments, even after years of efforts in weeding them out, harbour more jerks than what is endurable. We’re talking about abuse, but it is more than that. It is...
 
Public sector innovation, once dismissed as a contradiction in terms, is now increasingly seen as an imperative. Last November, the Organisation for Economic Co-operation and Development (OECD) hosted an international conference on “Innovating the Public Sector” and announced that it has been building a database on public sector innovation. Many federal government departments are setting-up...
 
On May 6, 2014, the Telfer School of Management at the University of Ottawa announced the launch for the first time in Canada of two Executive Master of Business degrees, one in Complex Program Leadership and the other in Strategic Procurement, with 80 percent common content. Related posts: Effective project management goes right to your bottom line What you need to know about Deliverology and Implementation Solving Complex Problems...
 
Whether in the public or private sector, a critical role of managers and supervisors is the management of employee performance. The new, public service-wide Directive on Performance Management, which recently came into effect, highlights the importance of this role for managers across departments and agencies. Related posts: A possible counter to Canadian risk aversion Collaboration works: A case study with social media Corporate Activism...
 
Water is essential to life, yet this resource is not being managed in a way that is sustainable. Cities are faced with aging infrastructure and stress on water systems. Moreover, an alarming percentage of water is wasted each year despite an increased demand by growing populations....
 
Agility is becoming a crucial element of a successful business. Technology now evolves at a faster pace than it did even ten years ago, and there is greater competition for talent now that people have the mobility to take their skills to the highest global bidder....
 
Data is increasingly becoming fundamental to the world we live in. Governments use it on a daily basis to provide us services, to protect us, and to keep our society running smoothly. But data sets are multiplying in size almost too quickly for our infrastructure to keep pace...
 
Municipal governments have a number of responsibilities, but chief among them is ensuring the safety of citizens, especially in emergency situations. On a day-to-day basis, emergency services have to deal with all sorts of scenarios -house fires, robberies, car accidents, and more....
 
Last week, Rick Conlow and Doug Watsabaugh wrote in Businessweek that 50 to 60 percent of managers fail in what they set out to do. They suggest that what causes these failures is a lack of understanding of how soft skills can affect a workplace. Related posts: Tactics for greater collaboration Professional development for the CIO Five ways governments can keep their staff — without a pay rise...
 
Colin Talbot’s recent book, Theories of Performance, Organizational and Service Improvement in the Public Domain (Oxford University Press, 2010), offers us a way forward: with better evidence about what works and why, a new approach to performance would improve…...
 
This time of year the Clerk of the Privy Council puts out his annual report to the Prime Minister on the state of the public service as he sees it. It always provides a good sense of where the most senior public servant in the land sees the institution going....
 
The British Columbia Public Service has taken significant steps to implement a new and innovative performance management model for our employees....
 
On October 29, General Tom Lawson accepted the responsibility of Chief of the Defence Staff from General Walt Natynczyk during a ceremony at the Canadian War Museum....
 
CGE Vol.13 No.5 May 2007 "…however fast they went, they never seemed to pass anything. ‘You’d generally get...
 
CGE Vol.13 No.2 February 2007 The leadership literature is littered with pithy maxims that claim to guide executives to career su...
 
CGE Vol. 14 No.5 May 2008 Why do only 30 percent of organizational initiatives achieve their objectives? The complexity of human resis...
 
CGE Vol. 14 No.4 April 2008 “Be In This Place” is the slogan for New Brunswick – and Premier Shawn Graham’s ambitious target of returning the province to prosperit...
 
Although a recent international poll indicates that Canada’s public sector is in better shape than other nations to face the current economic downturn, there is nevertheless an important opportunity to take quick and effective action to help it recover from budget deficits....
 
CGE Vol.14 No.2 February 2008 As David Dodge steps down today from his post as Governor of the Bank of Canada, he does so with top...
 
CGE Vol.13 No.9 November 2007 Organizational culture plays a strong role in encouraging performance-oriented behaviour. In the pri...
 
CGE Vol.13 No.8 October 2007 The New Brunswick Department of Transportation is implementing an asset-management program unlike any...
 
Quote of the week “The job we have here is to try to either incentivise them or enforce reductions in areas that we think the general public would support.” ...
 
I think the idea that someday the department’s lights will turn out and cease to exist is really a myth. There will always be a part of the executive branch of the government of Canada that deals with Aboriginal people....
 
What keeps you up at night? The Institute of Public Administration of Canada has been asking deputy ministers and chief administrative officers of municipalities this question since 1994....
 
As I have approached the end of my term as Auditor General of Canada, I have reflected upon a decade that I consider remarkable in many ways for the Office of the Auditor General....
 
Given the budgetary challenges facing most government organizations, the use of performance management techniques is expected to become even more important in the future....
 
Some title Some author
Some excerpt
 
Some title Some author
Some excerpt
 
Some title Some author
Some excerpt
 
Some title Some author
Some excerpt
Evaluation may provide information relevant to broader public processes such as program improvement, decision-making or accountability. Nonetheless, evaluation’s positive contribution to these goals may be affected by resistance from stakeholders associated with the program being evaluated. Resistance to evaluation may take various forms at different stages of an evaluation initiative. Prior to beginning the evaluation,...