Program Evaluation: What keeps you up at night? – Canadian Government Executive

NEWS

SEARCH

BusinessManagementPerformance
October 7, 2015

Program Evaluation: What keeps you up at night?

Written by  Gail Vallance Barrington

Anyone who has commissioned a program evaluation knows what it is like to have a sleepless night. The reason is simple: you are concerned about having lost track of the original question. The hyper-caffeinated effect is made worse by the slow and measured process of the evaluation. Staring at a ceiling you can’t see, your mind sifts through the research questions, survey questions, interview questions, and analysis questions. Has the focus been blurred; is the core question lost altogether? What can you do about this?

With thirty years in the field as an independent evaluator, I have come to believe that sometimes the evaluation serves the research better than the program. This can be due to poor communications between evaluator and client. Certainly the hands-off approach preferred by some managers does not foster success. A manager who says “Just give me the bottom line” is courting disaster.

The disconnect between what you want and what you get can be avoided. This need not take a lot of your time, nor should you play the role of the researcher. Instead, if you develop a high level, on-going relationship with your evaluator, you can ensure whether the evaluation stays on track.

Here are some strategies:

Clarify the Policy Question in the Design Phase

Make sure the evaluator understands your policy question and the type of recommendations you need. Are they about:

  • Program improvement?
  • Sustainability?
  • Expansion?
  • Future funding?
  • Closure?

Policy questions are complex and difficult to unpack, so make sure all the relevant topics are articulated. Don’t just wait for them to show up in the research findings.

Determine the Type of Evidence Needed, then Match it to Methodology

What kind of evidence will support your decision? Will hard numbers be persuasive? Do you need to prove, in a narrow sense, some kind of cause-effect relationship? Is your question about why? If so, a quantitative approach should produce the right information. On the other hand, if you really want to understand a phenomenon, why it happens, and how it affects those involved, your question may be how or what? In this case, you need a qualitative approach.

And there is a third option. If you need to ask both kinds of questions, a mixed methods approach is best. This requires more thought because the sequence of research activities is important. Do you have a qualitative question which will highlight topics for more focused numbers or is it the reverse — quantitative data that needs further explanation and description? Whatever approach you choose, make sure that the policy question drives the methodology — not the other way around.

Review the Time Planned for Evaluation Tasks

Take a look at the evaluator’s proposed list of tasks. Extensive time may be allocated for tool design and data collection but data analysis and report writing can be collapsed into a couple of weeks. The evaluator may be trying to respond to the parameters of the Request for Proposals in which too much is being requested in too short a time frame. What this means is that in the rush to produce the final report, you will lose a lot of the meaning buried deep within the data. Be prepared to sacrifice a little on the data collection side. With more time for synthesis, interpretation and reflection, you will get much stronger findings.

Discuss the Findings with the Evaluator

It is important to prioritize the findings once the data has been analyzed. Ask the evaluator to marshal the evidence beside each of your original questions. What are the strongest findings (i.e., most likely to be true across most cases)? Where do the findings tend to cluster? Where are there inconsistencies or gaps? Which findings are likely to be the most politically sensitive? What conclusions fall out of these findings? Is there enough evidence for you to move ahead or do you need further analysis?

 Craft the Recommendations with Key Stakeholders

Before the report is finalized, work with key stakeholders on the recommendations. Schedule a special meeting long in advance to warn them of their expected involvement. Ask the evaluator to present an overview. Provide participants with thumbnail findings and key topics for recommendations. Have them consider such issues as feasibility, appropriateness, cost, intended and unintended consequences, and political impact. Ask the counterfactual — what happens if nothing is done? Once a draft list of recommendations is developed, consider the language carefully and re-check with stakeholders as needed. It’s better to deal with them before the report is finalized rather than afterwards.

If you really want the evaluation to have an impact, make sure your schedule allows you to invest your time and interest in the process to make sure that the results are really useful. You will sleep better too!

 

Gail Vallance Barrington is a Credentialed Evaluator who has conducted over 130 program evaluation studies since founding her consulting firm, Barrington Research Group, Inc.  gbarrington@barringtonresearchgrp.com

About this author

0 comments

There are no comments for this post yet.

Be the first to comment. Click here.

Business
 
Canadian Government Executive Media (CGE) and CATAAlliance , Canada’s one voice for...
 
Water is essential to sustain all forms of life and more...
 
Security professionals have an obligation to communicate risks and recommendations to...
 
During the past decade, the international university-level student population in Canada...
 
Professionals, managers, and executives in the cost estimation industry can gain...
 
Canadian Government Executive kicked off its CGE Leadership Series for 2017...
 
A new study from the Conference Board of Canada gives our...
 
As the 5th largest agricultural exporter on the planet, Canada plays...
 
In this episode, hear more about how Canada is a prime...
 
In the recent June issue, we published a captivating piece by...
 
The October issue of Canadian Government Executive on Focusing on Customer...
 
Please to view this Content. (Not a member? Join Today! )...
 
Please to view this Content. (Not a member? Join Today! )...
 
Please to view this Content. (Not a member? Join Today! )...
 
Please to view this Content. (Not a member? Join Today! )...
 
Please to view this Content. (Not a member? Join Today! )...
 
Executive compensation is considered to be a central component of corporate...
 
Open source was established around the ethos of sharing and collaboration....
 
In December last year, a privately owned aerial drone crashed into...
 
By Gregory Richards A recent study by McKinsey Global Institute suggests...
 
Last week the Wall Street Journal, citing anonymous sources, reported that...
 
Written by Donald Farmer Too often, we base Business Intelligence today...
 
Written By Chris Brown To deliver results that senior executives value,...
 
Written By Jason McNaught Contrary to what you may have heard,...
 
Written by  Gail Vallance Barrington Anyone who has commissioned a program...
 
“Public institutions are the cornerstone of our democratic system” is the...
 
Please to view this Content. (Not a member? Join Today! )...
 
A neuromarketing study examines exactly how much life direct mail continues...
 
As one of the U.S. Defense Department’s largest acquisition organizations, Program...
 
Many people say that managing projects is all about managing people...
 
Some of us can only dream of being 20 years old...
 
“With SMARTnership, we have to look into communication, being trustworthy, convincing...
 
Wreaths are placed at the naming ceremony for the new Canadian...
 
Barry Shepherd, the principal surveyor and client support manager for Central...
 
In 2012, the province of Prince Edward Island held a business...
 
The challenges city leaders now face require creativity and innovation. Many...
 
Depending on how you count, 10-20 percent of all Canadian public...
 
In a 2011 white paper published by the Internal Federation of...
 
For the last few years, governments around the world have been...
 
Compensation is often raised as a key tool that can either...
 
The Canadian mining industry is responsible for over 4.5 percent of...
 
Governments throughout Canada are facing a number of challenges, from reducing...
 
Since its founding, Canada has looked either east to Europe or...
 
It is a good management practice to review programs and activities...
 
Few things catalyze infrastructure faster than a major international sporting event....
 
What can you say about the future of long range public...
 
In Canada, it seems like any discussion of copyright quickly turns...
 
We live in a time when physical and digital infrastructures built...
 
How are leaders responding to a competitive and economic environment unlike...
 
We live in times of significant demographic and technological transition across...
 
Public-private partnerships (P3s) have become increasingly important in Canada at both...
 
Les citoyens souhaitent expliquer aux politiciens et aux décideurs ce qu’ils...
 
The need for better accountability has led Treasury Board to enforce...
 
It begs the question: Are federal real property professionals ready to...
 
With an annual deficit of $36.2 billion, the federal government is...
 
It now appears that Canada will emerge earlier than expected from...
 
Strong portfolio management is increasingly important in today’s interconnected world....
 
Early in 2011, the National Quality Institute (NQI) board of governors...
 
The Public Service Modernization Act, which came into force in 2005,...
 
In the current economic climate, the finance function role in the...
 
We live in exciting times: an unprecedented level of innovation is...
 
Saskatchewan’s oil and gas sector is booming, and the amount of...
 
The Economic Action Plan showed how much can be achieved when...
 
Public Works and Government Services Canada has a new and innovative...
 
Citizens may not always look forward to applying and paying for...
 
Maintaining and investing in water and waste water treatment facilities, roads,...
 
Stakeholders and media are currently focusing substantial attention on executive compensation....
 
The spectre of wrongdoing within a department can send chills down...
 
Il n’y a pas si longtemps, les Beatles chantaient : «...
 
The Beatles once sang, “You say you want a revolution? Well,...
 
An idea – that’s where it all starts. Entrepreneurs travel a...
 
Please to view this Content. (Not a member? Join Today! )...
 
Quote of the week politicians provide the energy generated by ideology,...
 
Some title Some author
Some excerpt
 
Some title Some author
Some excerpt
 
Some title Some author
Some excerpt
 
Some title Some author
Some excerpt
 
Some title Some author
Some excerpt

Member Login

Forgot Password?

Join Us

Password Reset
Please enter your e-mail address. You will receive a new password via e-mail.