Principled Evaluation: Who Cares? - Canadian Government Executive
ManagementPerformanceProgram Evaluation
January 18, 2018

Principled Evaluation: Who Cares?

The time is right for the Canadian evaluation community to have a robust and vital discourse on the ‘ties that bind’ us together as professionals, and that set the stage for principled practice. In the absence of such, a key cornerstone of evaluation excellence may begin to erode.

Compared to audit professionals and their preoccupation with “independence”, evaluation specialists may seem unconcerned about ethical principles. Their fixation is on systematic processes, methods and techniques. Principles underlying evaluations rarely come up in evaluation planning or implementation, and they seldom figure in evaluation reports or follow-ups.

Give a workshop to evaluators on “big data” or “data visualization” and you can fill an auditorium. Have a session on evaluation principles and ethics and the room is mainly empty. So, what’s going on? Perhaps the absence of discussion leads evaluators to conclude that ethical challenges do not exist. They then infer that they and their field must be, by their very nature, principled and ethical. Suspecting that it’s not that simple, we feel it’s time to ask: What dialogue on evaluation principles and ethics ever occurs? What ethical challenges are evaluators encountering? What are they doing about them?

As are any professionals, evaluators are first and foremost defined by the central values they hold. If their central tenets go unnoticed, then their practice fades into a parade alongside every other activity in the scientific management field. Indeed, no profession is immune from this risk. For example, a study of almost 1000 physicians in several countries found that although they were aware that their profession had a code of ethics, almost to a person they admitted to not referring to it and not using it. The time is right for the Canadian evaluation community to have a robust and vital discourse on the ‘ties that bind’ us together as professionals, and that set the stage for principled practice. In the absence of such, a key cornerstone of evaluation excellence may begin to erode.

To understand how evaluators implement ethics in their day-to-day practices, surveys were conducted of Canadian Evaluation Society (CES) members in 2010 (response rate 26 per cent, n = 455) and 2015 (response rate 12 per cent, n = 218). The 2015 study included qualitative case studies of eight evaluations conducted between 2013 and 2016.  See below for a summary of the results from these studies.

Ethics is a serious issue for Canadian evaluators. They associate ethics with ideas about integrity, quality and impartiality, values that are important to them as professionals. Start a dialogue about principles and ethics with evaluators and it quickly descends down a long chute of abstract concepts – accountability, competence, confidentiality, impartiality, independence, integrity, neutrality, objectivity, subjectivity, and the list goes on.

Comparing the 2010 and 2015 surveys suggests that evaluators are becoming more aware of and can identify more challenges in their work. As the 2010 and 2015 studies attest, the power of those who commission evaluations and their influence over evaluators’ treatment of ethics during the evaluation process is substantial. Evaluators are frequently called upon to deal with direct pressures from stakeholders to modify the evaluation process or findings. This can lead to potential bias in the evaluation, especially in terms of impartiality. It also increases evaluators’ anxiety. Evaluation clients, those who commission evaluations, and evaluators themselves hold important roles and responsibilities with regard to ensuring principled, ethical practice.

At the same time, evaluators seem shy to speak out on this topic. This has much to do with sensitivities related to conflicts of interest that are perceived to affect employment circumstances and opportunities. While evaluators do welcome occasions to provide feedback, there is a profound sensitivity around full ethical disclosure.

The 2010 survey revealed that three-quarters or 77 per cent of Canadian evaluators encountered at least one perceived ethical challenge in their work, but only 57 per cent were willing to provide a description or example. For them, the setting for such a discussion needs to be right, and the space safe. Both the 2010 and 2015 studies reveal that colleagues (professional evaluators or otherwise) are an important source of reflection and advice, i.e., a trusted viewpoint in a safe space.

As participants in the 2015 study observed, discussion about ethical challenges in evaluators’ daily practice is not systematic.  It raises the question of the extent to which evaluators can and feel comfortable raising these ethical issues with the client before, during, and or after the evaluation process.

The 2010 survey also revealed that there is room for a more active role from the Canadian Evaluation Society at 62 per cent. A vacuum currently exists and is something only the national association can fill. This may include professional development – workshops, formal training; regular surveillance among members; greater emphasis on ethics in the Credentialed Evaluator designation; more space in professional publications (e.g. Canadian Journal of Program Evaluation), and more dialogue at chapter level events and national conferences.

The Canadian Evaluation Society is indeed now taking on this role, starting with a review of its now 20-year old Ethics Principles of Competence, Integrity and Accountability. Since that document was produced, there have been major changes in the environment affecting Canadian evaluators’ work, including the introduction of national and provincial privacy legislation; the publication of the Government of Canada’s Tri-Council Policy Statement on research with Human Participants and its exclusion of program evaluation from ethical reviews; the publication of the CES Competencies for Canadian Evaluators, and its soon-to-be-released with a renewed focus on ethics.

Other national associations, notably the American Evaluation Association and the Australasian Evaluation Society, are also undertaking ethics renewal initiatives. As Harry Cummings, CES President noted, “Evaluators’ work, roles and fundamental obligations to their clients, program beneficiaries and society are only becoming more complex, and ethical challenges more subtle and intersectional. CES needs to step up and assume leadership in supporting and protecting Canadian evaluators and evaluation stakeholders in ethical practice.”

The Canadian Evaluation Society, National Capital Chapter (NCC) will be holding its Annual Learning Event in Ottawa on February 13, 2018. The theme of this event is Principled Evaluation: Should I give a s*#t? This title is admittedly provocative but one that will delve into the questions and concerns raised in this article.

To learn more or to register for this event, go here.

 

Wayne MacDonald MA, CEP – Director, Canadian Evaluation Society – National Capital Chapter and President, Infinity Consulting and Legal Services (ICLS). Wayne.macdonald@rogers.com

Emilie Peter, PhD – is a recipient of the Joseph-Armand-Bombardier Canada Graduate Scholarship (SSHRC, 2013) and recently completed her PhD in Public Administration at the École nationale d’administration publique (ÉNAP). Emilie.peter@enap.ca

Natalie Kishchuk PhD, CE, FCES – Vice-President, Canadian Evaluation Society and independent consultant located in Montréal, Québec. vicepresident@evaluationcanada.ca

 

 

About this author

0 comments

There are no comments for this post yet.

Be the first to comment. Click here.

Management
 
As the resettlement of Syrian Refugees exemplifies, government programs, currently running...
 
Over the last several months, as a participant in the OCG’s...
 
High in the Arctic, the Inuksuk is a timeless cultural symbol...
 
At the 2017 IPAC conference in Charlottetown, George Ross, Editor-in-Chief of...
 
Can government remember? Is it condemned to repeat mistakes? Or does...
 
‘Elephant in the room’ is a metaphorical idiom for an obvious...
 
Compared to audit professionals and their preoccupation with “independence”, evaluation specialists...
 
Having a clear definition of success is one of the key...
 
Time is of the essence for Alice’s White Rabbit. In business,...
 
Professional development is tremendously important to members of Canada’s public sector....
 
The International Civil Service Effectiveness (InCiSE) Index project, a collaboration between...
 
Yogi Berra, Maurice (The Rocket) Richard, Bill Russell, Michael Jordan, and...
 
Three years before his death in 2011, Jack Layton released a...
 
In today’s workplace, individuals increasingly face dynamic and difficult challenges that...
 
We are happy to share with you the September 2017 issue...
 
The health care system in Canada, known as Medicare, is publicly...
 
In March, the Canada Revenue Agency’s website was shut down by...
 
The answer my friends is blowin’ in the wind … BOB...
 
Water is essential to sustain all forms of life and more...
 
We are thrilled to offer the July/August issue of Canadian Government...
 
We are happy to share with you the May/June issue of...
 
The International Monetary Fund (IMF) in issuing its annual review of...
 
In about two weeks, APEX will be holding its 2017 Symposium...
 
As governments around the world move towards the digital transformation of...
 
Cost estimation is becoming an extremely important skill within government due...
 
We have all heard the sayings: “Home, sweet home”; “There’s no...
 
Risk is always present in any undertaking, no matter the size...
 
As Canada’s government agencies strive to improve delivery and efficiency under...
 
Last year, procurement Minister Jody Foote was prompted by the swirling...
 
Bill Gates once said that “Technology is just a tool. In...
 
Sustainable Development Goals (SDGs) are part of the new global “Agenda...
 
Laskey Hart guest hosts CGE Radio this week with a summary...
 
With baby boomers continuing to hang onto their office desks and...
 
In every industry, the competition for the talent pool is on...
 
The Conference Board of Canada recently published a report titled Leveraging...
 
While internal auditing has been considerably strengthened in most jurisdictions in...
 
Rules and accountability are helpful in developing and standardizing processes but...
 
Cyber-attacks threaten Canadian individuals and companies every minute of every hour....
 
Ontario’s provincial government needs to act fast in building a coordinated...
 
The bootleg fentanyl overdose crisis that is sweeping across Western provinces...
 
The association representing more than 42,000 physicians and medical students in...
 
A study conducted by Robert Half, the world’s largest specialized staffing...
 
A recent report suggests several strategies how governments and the private...
 
The latest information from IBM Cloud covers: Consolidating Complex Environments Consolidating...
 
In order to find out which strategies and priorities CFOs are...
 
A small but rapidly growing number of businesses have started hiring...
 
Sometimes it’s not the job that counts, but what comes after....
 
The possibility of mail delivery disruption on Friday this week loomed...
 
CFOs around the world are changing – but an EY survey...
 
There’s a fine line between confidence and naivety – a line...
 
Around the world, more and more workers are seeking employment. At...
 
Canada Post today warned the public about the potential for a...
 
In a little more than a year, Canadian banks and lending...
 
Digitizing certain aspects of the hiring process appeals to both employers...
 
As populations grow and age, the demand for services increases. As...
 
As a consultant, I work primarily with private sector organizations including...
 
On June 7, over 40 senior executives from within the public...
 
Dispersed organisations can be very powerful structures. There is ample evidence...
 
By Nicole Verkindt Despite growing up in a small town in...
 
The agency responsible for safeguarding the Pentagon and several other buildings...
 
The actions and decisions of public servants have consequences for the...
 
By Jason Zhang  The best way to cost effectively manage the retention...
 
Protests by irate taxi drivers, roadside scuffles involving cabbies and Uber...
 
He is often referred to as the federal government’s first “chief...
 
By Michael Murphy Not all assets can and should be equally...
 
When I find myself in times of trouble, Mother Mary comes...
 
People caught a glimpse of the work of partisan advisers in...
 
Copyright owners have struggled to protect their intellectual property since the...
 
Canada’s largest organization advocating for retirees from federal, police and military...
 
Here’s some bad news for organizations contemplating projects that have something...
 
The adoption last year by the Canada Border Services Agency of...
 
Two years ago, Bixi, a not-for-profit, para-municipal bicycle-share firm of the...
 
The melting Arctic ice caused by climate change is opening up...
 
The Prime Minister is now well into the consolidation phase of...
 
Canadians expect their public services to be delivered in a way...
 
It appears it was not just the Mounties that were affected...
 
Every reputable learning institution offers leadership development training nowadays. So why...
 
Many programs at all levels of government provide funds to community-based...
 
Nearly half of Canadian organizations are falling behind on implementation of...
 
If you want to improve your management procedures, search Google. No,...
 
A while back there have been numerous media reports about the...
 
Q: Where is Comptrollership going? Comptrollership in the Government of Canada...
 
Performance auditing can lead to more efficient, effective, and economical program...
 
Rushing to keep up with deadlines but never getting ahead is...
 
The idea of converged infrastructure and converged systems in the datacentre...
 
Hackers are zeroing in on users of SSL/TLS encryption and no...
 
That the new Liberal Government has embraced ‘open government’ is hardly...
 
Meet Bob Heart.  He is an outstanding employee who works hard...
 
A new study released yesterday, Securing the C-Suite, Cybersecurity Perspectives from...
 
The bloodletting at once-mighty Canadian mobile tech giant, BlackBerry continues as...
 
There is no shortage of examples of businesses that effectively used...
 
Rogers Media announced yesterday cost-cutting moves that will see some 200...
 
Microsoft, on Tuesday, ended support for older versions of Internet Explorer...
 
Written by Tim Wacker Almost a quarter century ago, when most...
 
Last week the Wall Street Journal, citing anonymous sources, reported that...
 
Written By Jason McNaught The Public Service Alliance of Canada was...
 
A leader’s credibility begins with personal success. It ends with helping...
 
Written by Donald Farmer Too often, we base Business Intelligence today...
 
Written by  Gail Vallance Barrington Anyone who has commissioned a program...
 
In late July, Amazon startled investors with the tiniest of profit....
 
Skills International, a global talent and career management firm, has launched...
 
When it comes to training, statistics show small businesses are more...
 
Today, TD Canada Trust released the results of a survey studying...
 
Management has its upsides, but no one said it would be...
 
Independence has long been regarded as a cornerstone of the auditing...
 
Congratulations. You’ve achieved professional recognition for your performance and the public...
 
In 2011, PwC began a conversation with global defence leaders on...
 
Canada is a diverse nation, in language, culture, geography, and, ultimately,...
 
Public sector organizations are under increasing pressure to identify all risks...
 
The government of Canada has implemented several measures over the past...
 
The economy is sluggish…oil prices are down…stock markets are up but...
 
Whether at the territorial, provincial or federal government level, internal audit...
 
On a cold, grey Sunday afternoon last November, about 2,500 residents...
 
An organization’s reputation can take years to build but it can...
 
Change definitely happens, and it is happening faster and faster as...
 
In the well-known children’s story, an Emperor falls victim to the...
 
The Nova Scotia Office of the Ombudsman is a small operation...
 
I think the ombudsman needs to be independent, because without independence...
 
Today’s business environment changes rapidly to adjust to evolving conditions and...
 
There’s a cloud looming over public sector employers in Canada. Baby...
 
The best internal auditors actually are really good managers first. I...
 
In the middle of an empty field on the edge of...
 
The recent controversy about the actions of some staff members in...
 
Most professionals don’t need more than a sentence at a cocktail...
 
Recent research by the Institute of Internal Auditors Canada aims to...
 
Today’s tasty hamburger bun is a marvel of industrial cuisine and...
 
In this last week before Christmas and the holidays, this is...
 
As part of the Public Policy Forum’s university lecture series, Mel...
 
Respecting, processing and cross-pollinating thoughts from diverse cultures will advance team...
 
When is it that a politician becomes part of the governing...
 
It is something that few senior managers spend much time thinking...
 
It’s been a busy couple of weeks on the information, privacy...
 
In 1996, a new budget watchdog, the Parliamentary Budget Officer, was...
 
The publication of Donald Savoie’s latest book, Whatever Happened to the...
 
The Ontario government is moving forward with the creation of a...
 
He blew up the factory! Jack Welch, as CEO of General...
 
There is no doubt that the challenges faced by governments are...
 
When printing press technology was diffused through Europe, adult literature preceded...
 
The Office of the Public Sector Integrity Commissioner of Canada (PSIC)...
 
After the Auditor General’s (AG) report was released on April 30,...
 
Entering 2013, Canadian employers and unions, public and private sector alike,...
 
In the U.K. system, Permanent Secretaries are what we call Deputy...
 
As both a management consultant and a Boston Marathoner who was...
 
I believe that the greatest challenge and opportunity for management is...
 
The Asian Development Bank estimates that the global demand for infrastructure...
 
n Saskatchewan, a drug shortage team in the Ministry of Health...
 
Three decades ago, the federal public service was proportionately larger, generally...
 
We are living in a period of rapid change and limited...
 
US public sector employees don’t trust their management to do the...
 
The news of Mark Carney’s nomination as the new Governor of...
 
Following Singapore’s independence in 1965, the controversial leadership of Lee Kuan...
 
Over the past few years, the preparation and delivery of the...
 
In healthcare, cost-cutting can result in cutting what is valued most...
 
For over 20 years Colin Bennett has been exploring issues of...
 
Even before controversy shook the organization to its foundation, Ornge was...
 
It will be the largest international multi-sport event ever held on...
 
We’ve all seen the headlines – BC Ferries, Ornge, la Caisse...
 
It can happen, and it’s noteworthy when it does. Government, business...
 
There seems little argument in the wake of the financial crisis...
 
Kevin Page’s mandate as the first Parliamentary Budget Officer comes to...
 
Governments are challenged to meaningfully mitigate the effects of the financial...
 
Canada is facing a huge financial challenge brought on by massive...
 
For the past one hundred years, democratic states have been moving...
 
It’s so much easier and less painful to learn from the...
 
Mandated by Congress in late 2010, the U.S. federal government is...
 
CGE Vol.13 No.4 April 2007 "The potential to lead is in...
 
CGE Vol.13 No.2 February 2007 The Connecting the Dots series is...
 
CGE Vol.13 No.2 February 2007 For too many managers, "crisis" is...
 
CGE Vol.14 No.3 March 2008 Organizations communicate "what the real game...
 
CGE Vol.14 No.2 February 2008 Community Safety and Correctional Services is...
 
CGE Vol.13 No.1 January 2007 Leadership How would you describe your...
 
CGE Vol.14 No.3 March 2008 If you spend much time at...
 
CGE Vol.13 No.5 May 2007 How often has a new employee...
 
CGE Vol.13 No.7 September 2007 "If the Public Service, as a...
 
When pondering leadership, we immediately think of exercising our influence downward...
 
CGE Vol.13 No.1 January 2007 "How can I be held accountable...
 
CGE Vol.13 No.2 February 2007 Canada’s Performance 2006 is the sixth...
 
CGE Vol.14 No.1 January 2008 The furor over the $300,000 that...
 
CGE Vol.14 No.2 February 2008 Let’s say you’re a senior manager,...
 
CGE Vol.13 No.1 January 2007 Perhaps it’s a legacy of the...
 
Managing public sector organizations requires living with high levels of complexity...
 
The significant challenges created by the global recession have had a...
 
Saskatchewan, like other provinces, spends almost half its budget on healthcare....
 
Building project management and business transformation capacity is critical to advancing...
 
You’ve heard it many times: public service leaders are accountable for...
 
Citizens want to talk to politicians and policymakers to explain what...
 
Dans le monde d’aujourd’hui, les entreprises, les organismes et leurs clients...
 
if possible I would love to ask you a few other...
 
En parcourant tous les diagrammes techniques qui couvrent des cartes murales...
 
On pense souvent que la fonction publique devra recruter un grand...
 
Jumeler les compétences opérationnelles avec le souci du service, les aptitudes...
 
La complexité des programmes publics ne date pas d’hier. Nul ne...
 
Après avoir consacré trois années à la mobilisation des employés et...
 
CGE Vol.13 No.9 November 2007 The new expenditure management system announced...
 
As the Public Service Commission of Canada celebrates its 100th anniversary...
 
With the hotly contested Canadian and American elections behind us, the...
 
CGE Vol.13 No.8 October 2007 A few weeks ago, the federal...
 
Henry Mintzberg is Canada’s most celebrated management thinker. He and colleagues...
 
Stephen M.R. Covey has sold 600,000 copies of The Speed of...
 
Comment les cadres et les élus des villes dirigentR09;ils les services...
 
A decade ago Canadian Blood Services took over a troubled agency...
 
Ne vous fiez à personne de plus de 30 ans, déclarait...
 
CGE Vol.13 No.10 December 2007 Demographics and technology, two of our...
 
Are you a good boss? Or a bad boss? Or a...
 
Most government executives, most of the time, are focused on dealing...
 
Like most people, public service leaders intuitively understand the value of...
 
Lee McCormack, ancien cadre supérieur du gouvernement, est directeur de la...
 
The public service strives to achieve positive outcomes with appropriately targeted...
 
The Independent Blue Ribbon Panel on Grants and Contributions called for...
 
As the global economy struggles to regain some forward momentum, Canadian...
 
Au Canada, le secteur bénévole et à but non lucratif vit...
 
This will be a defining budget for Stephen Harper. It will...
 
It is difficult to determine when the debate about the need...
 
Bill Greenlaw is the elected president of the Institute for Public...
 
For the next few years, the federal government’s overarching agenda will...
 
Have you ever asked yourself the question: ‘How would I evaluate...
 
Much of the current conversation about the federal government’s economic agenda...
 
Last fall, Alberta’s Employment and Immigration department posted online the workplace...
 
In 2006 the world was feeling the aftershocks of a number...
 
CGE Vol.13 No.4 April 2007 Robert Parkins, editorial director, met recently...
 
In the past two decades, the nature of the state has...
 
In recent months, the attention of Canadians has been focused on...
As the resettlement of Syrian Refugees exemplifies, government programs, currently running...