Realist evaluation: Assessing national programs in unique communities – Canadian Government Executive

NEWS

SEARCH

AccountabilityManagement
May 25, 2015

Realist evaluation: Assessing national programs in unique communities

Canada is a diverse nation, in language, culture, geography, and, ultimately, the needs of its citizens. Such diversity can pose a challenge to a government that seeks to develop, implement, and evaluate national-level programming to local communities. Ultimately, programs and policies must be tailored to meet the unique needs of a community.

That presents a conundrum: the more a program is locally relevant, the harder it is to assess effectiveness at the national level.

In academic terms, this conundrum is part of the positivist vs. realist debate. Thankfully, there can be a reconciliation and way to move forward – where scientific rigour and local relevance can be of equal consideration in evaluation.

Realist evaluation offers a lens on establishing evidence; pushing beyond a focus on outcomes, and broadening to include both the context and processes (or mechanisms) affecting such outcomes. In fact, Health Canada has been exploring and researching this approach, particularly with respect to health promotion, and has added realist evaluation (RE) and syntheses to its methodological repertoire.

What is RE?
A mantra commonly used to describe RE is “what works, for whom, how, why, and under what circumstances.” This highlights the important role of context – namely, characteristics of the population or service provider, and of the setting. In addition, RE seeks to uncover the processes or mechanisms that bring certain outcomes. As a result, RE is frequently described as a theory-based approach to evaluation, including theory of change related to participants and theory of change related to the implementation of programs and policies.

The nuance of RE is the recognition that certain contexts may facilitate or hinder processes that bring about outcomes. This latter point is particularly relevant to diversity of the Canadian context – if evaluations uncover common mechanisms while capturing contextual features that support or hinder outcomes, one can identify successful processes that can be targeted across the contexts known to support outcomes (instead of applying them invariably across all contexts). Thus, realist evaluation has potential to focus on both the nation (common mechanisms) and the community (unique contextual features).

Current examples
An example of program implementation informed by a RE has been demonstrated by the Scottish government. The goal was to implement midwife-led care as a result of research demonstrating reductions in routine childbirth interventions and improvements in patient-reported care experiences.

Using interviews, focus groups, and multiple case studies, researchers found that consultant midwives varied in their approach to implementation of care in the team context according to the strength of the medical culture. In contexts with an ingrained medical culture (e.g., with little support for midwives), the appointment of midwives by administration demonstrated a high-level commitment to implementation of midwifery care and midwives relied on more robust change strategies (e.g., audits, debates, hands-on-training). In contexts where midwives were supported by obstetricians and physicians, midwife-lead care could emerge through empowerment of midwives to practice more autonomously.

Overall, the study was able to identify a set of mechanisms (e.g., commitment, resistance, feeling pressure) that would have differential impacts on outcomes according to the unique context of each hospital.

RE has also informed policy development. For example, the Australian Institute of Criminology undertook a realist review on effective crime prevention strategies. In addition to identifying which interventions appeared to be successful in preventing crimes, the authors were able to distill why and in what circumstances programs worked. Through a systematic realist review, they were also able to identify the necessary requirements for implementing these programs.

The findings from this review have led to the development of factsheets and handbooks that local governments can use to select, adapt, and implement interventions within their regions. In this way, if policymakers and practitioners understand how and why programs work in different contexts, better decisions can be made with regard to which programs or policies to use and how to adapt them in different contexts.

Limitations
Embarking on RE requires reflection, multidisciplinary collaboration and time. Not surprisingly there can be systemic barriers to such an approach that will need to be addressed, overcome, or circumvented. RE is inherently messy as it is about complexity – there can be multiple mechanisms operating in an orchestrated fashion. Finally, conclusions from RE are tentative directions and not considered stable and infallible.

As a nation recognizant of both unique and common identities, striking the right balance between both program and policy is crucial. RE offers a relevant framework that targets mechanisms/processes, as well as uncovers and documents contextual features responsible for supporting particular outcomes.

The broader application of RE could be facilitated through the identification of necessary infrastructure (e.g., cross discipline teams). Several scholars within the CUEE are actively pursuing research into this burgeoning evaluation approach to determine its application in governmental evaluation.

About this author

0 comments

There are no comments for this post yet.

Be the first to comment. Click here.

Accountability
 
In this episode, hear from Carl Hammersburg, Manager, Government and Healthcare...
 
A new study from the Conference Board of Canada gives our...
 
In the world that we are living in today, free and...
 
The delivery method developed by Sir Michael Barber, chief adviser to...
 
Rules and accountability are helpful in developing and standardizing processes but...
 
Canadian doctors were told that climate change impacts human health and...
 
Even as talks between the government and federal workers affected problems...
 
The largest effort in 20 years to seek public input on...
 
Ottawa has overhauled the process by which justices are picked for...
 
Please to view this Content. (Not a member? Join Today! )...
 
In this episode, editor-in-chief, Patrice Dutil talks about the need for...
 
Upon receiving numerous complaints regarding add-on fees that turn making economy...
 
Are you absolutely clear what the government wants to achieve? Are...
 
Please to view this Content. (Not a member? Join Today! )...
 
Please to view this Content. (Not a member? Join Today! )...
 
Please to view this Content. (Not a member? Join Today! )...
 
Please to view this Content. (Not a member? Join Today! )...
 
Please to view this Content. (Not a member? Join Today! )...
 
Written By Jason McNaught The Public Service Alliance of Canada was...
 
Independence has long been regarded as a cornerstone of the auditing...
 
Canada is a diverse nation, in language, culture, geography, and, ultimately,...
 
Public sector organizations are under increasing pressure to identify all risks...
 
The government of Canada has implemented several measures over the past...
 
Whether at the territorial, provincial or federal government level, internal audit...
 
An organization’s reputation can take years to build but it can...
 
On October 30, Ontario began second reading of the Public Sector...
 
In the well-known children’s story, an Emperor falls victim to the...
 
The Nova Scotia Office of the Ombudsman is a small operation...
 
I think the ombudsman needs to be independent, because without independence...
 
Today’s business environment changes rapidly to adjust to evolving conditions and...
 
The best internal auditors actually are really good managers first. I...
 
The recent controversy about the actions of some staff members in...
 
Most professionals don’t need more than a sentence at a cocktail...
 
Recent research by the Institute of Internal Auditors Canada aims to...
 
When is it that a politician becomes part of the governing...
 
It’s been a busy couple of weeks on the information, privacy...
 
In 1996, a new budget watchdog, the Parliamentary Budget Officer, was...
 
The Ontario government is moving forward with the creation of a...
 
The Office of the Public Sector Integrity Commissioner of Canada (PSIC)...
 
After the Auditor General’s (AG) report was released on April 30,...
 
In the U.K. system, Permanent Secretaries are what we call Deputy...
 
We are living in a period of rapid change and limited...
 
US public sector employees don’t trust their management to do the...
 
The news of Mark Carney’s nomination as the new Governor of...
 
Following Singapore’s independence in 1965, the controversial leadership of Lee Kuan...
 
Over the past few years, the preparation and delivery of the...
 
In healthcare, cost-cutting can result in cutting what is valued most...
 
For over 20 years Colin Bennett has been exploring issues of...
 
Even before controversy shook the organization to its foundation, Ornge was...
 
It will be the largest international multi-sport event ever held on...
 
We’ve all seen the headlines – BC Ferries, Ornge, la Caisse...
 
It can happen, and it’s noteworthy when it does. Government, business...
 
Kevin Page’s mandate as the first Parliamentary Budget Officer comes to...
 
Governments are challenged to meaningfully mitigate the effects of the financial...
 
Canada is facing a huge financial challenge brought on by massive...
 
For the past one hundred years, democratic states have been moving...
 
It’s so much easier and less painful to learn from the...
 
CGE Vol.13 No.7 September 2007 "If the Public Service, as a...
 
When pondering leadership, we immediately think of exercising our influence downward...
 
CGE Vol.13 No.1 January 2007 "How can I be held accountable...
 
CGE Vol.13 No.2 February 2007 Canada’s Performance 2006 is the sixth...
 
CGE Vol.14 No.1 January 2008 The furor over the $300,000 that...
 
CGE Vol.14 No.2 February 2008 Let’s say you’re a senior manager,...
 
CGE Vol.13 No.1 January 2007 Perhaps it’s a legacy of the...
 
The Independent Blue Ribbon Panel on Grants and Contributions called for...
 
Au Canada, le secteur bénévole et à but non lucratif vit...
 
As the global economy struggles to regain some forward momentum, Canadian...
 
This will be a defining budget for Stephen Harper. It will...
 
It is difficult to determine when the debate about the need...
 
For the next few years, the federal government’s overarching agenda will...
 
Much of the current conversation about the federal government’s economic agenda...
 
Bill Greenlaw is the elected president of the Institute for Public...
 
Have you ever asked yourself the question: ‘How would I evaluate...
 
Last fall, Alberta’s Employment and Immigration department posted online the workplace...
 
In 2006 the world was feeling the aftershocks of a number...
 
CGE Vol.13 No.4 April 2007 Robert Parkins, editorial director, met recently...
 
In the past two decades, the nature of the state has...
 
Please to view this Content. (Not a member? Join Today! )...
 
Some title Some author
Some excerpt
In this episode, hear from Carl Hammersburg, Manager, Government and Healthcare...

Member Login

Forgot Password?

Join Us

Password Reset
Please enter your e-mail address. You will receive a new password via e-mail.