AccountabilityManagement
May 25, 2015

Realist evaluation: Assessing national programs in unique communities

Canada is a diverse nation, in language, culture, geography, and, ultimately, the needs of its citizens. Such diversity can pose a challenge to a government that seeks to develop, implement, and evaluate national-level programming to local communities. Ultimately, programs and policies must be tailored to meet the unique needs of a community.

That presents a conundrum: the more a program is locally relevant, the harder it is to assess effectiveness at the national level.

In academic terms, this conundrum is part of the positivist vs. realist debate. Thankfully, there can be a reconciliation and way to move forward – where scientific rigour and local relevance can be of equal consideration in evaluation.

Realist evaluation offers a lens on establishing evidence; pushing beyond a focus on outcomes, and broadening to include both the context and processes (or mechanisms) affecting such outcomes. In fact, Health Canada has been exploring and researching this approach, particularly with respect to health promotion, and has added realist evaluation (RE) and syntheses to its methodological repertoire.

What is RE?
A mantra commonly used to describe RE is “what works, for whom, how, why, and under what circumstances.” This highlights the important role of context – namely, characteristics of the population or service provider, and of the setting. In addition, RE seeks to uncover the processes or mechanisms that bring certain outcomes. As a result, RE is frequently described as a theory-based approach to evaluation, including theory of change related to participants and theory of change related to the implementation of programs and policies.

The nuance of RE is the recognition that certain contexts may facilitate or hinder processes that bring about outcomes. This latter point is particularly relevant to diversity of the Canadian context – if evaluations uncover common mechanisms while capturing contextual features that support or hinder outcomes, one can identify successful processes that can be targeted across the contexts known to support outcomes (instead of applying them invariably across all contexts). Thus, realist evaluation has potential to focus on both the nation (common mechanisms) and the community (unique contextual features).

Current examples
An example of program implementation informed by a RE has been demonstrated by the Scottish government. The goal was to implement midwife-led care as a result of research demonstrating reductions in routine childbirth interventions and improvements in patient-reported care experiences.

Using interviews, focus groups, and multiple case studies, researchers found that consultant midwives varied in their approach to implementation of care in the team context according to the strength of the medical culture. In contexts with an ingrained medical culture (e.g., with little support for midwives), the appointment of midwives by administration demonstrated a high-level commitment to implementation of midwifery care and midwives relied on more robust change strategies (e.g., audits, debates, hands-on-training). In contexts where midwives were supported by obstetricians and physicians, midwife-lead care could emerge through empowerment of midwives to practice more autonomously.

Overall, the study was able to identify a set of mechanisms (e.g., commitment, resistance, feeling pressure) that would have differential impacts on outcomes according to the unique context of each hospital.

RE has also informed policy development. For example, the Australian Institute of Criminology undertook a realist review on effective crime prevention strategies. In addition to identifying which interventions appeared to be successful in preventing crimes, the authors were able to distill why and in what circumstances programs worked. Through a systematic realist review, they were also able to identify the necessary requirements for implementing these programs.

The findings from this review have led to the development of factsheets and handbooks that local governments can use to select, adapt, and implement interventions within their regions. In this way, if policymakers and practitioners understand how and why programs work in different contexts, better decisions can be made with regard to which programs or policies to use and how to adapt them in different contexts.

Limitations
Embarking on RE requires reflection, multidisciplinary collaboration and time. Not surprisingly there can be systemic barriers to such an approach that will need to be addressed, overcome, or circumvented. RE is inherently messy as it is about complexity – there can be multiple mechanisms operating in an orchestrated fashion. Finally, conclusions from RE are tentative directions and not considered stable and infallible.

As a nation recognizant of both unique and common identities, striking the right balance between both program and policy is crucial. RE offers a relevant framework that targets mechanisms/processes, as well as uncovers and documents contextual features responsible for supporting particular outcomes.

The broader application of RE could be facilitated through the identification of necessary infrastructure (e.g., cross discipline teams). Several scholars within the CUEE are actively pursuing research into this burgeoning evaluation approach to determine its application in governmental evaluation.

About this author

0 comments

There are no comments for this post yet.

Be the first to comment. Click here.

Accountability
 
Despite a rapid expansion in digital democracy initiatives around the globe,...
 
On October 22, 2018, elections were held in Ontario’s 400+ municipalities....
 
What are the key initiatives to support government innovation and change?...
 
Buying Social. Responsible sourcing. Ethical procurement. Sustainable procurement. Whatever name you...
 
In this episode, Lori Turnbull discusses the positive side of disruption,...
 
The Annual Financial Report of the Government of Canada Fiscal Year...
 
According to a 2018 study led by Dr. Michael McGuire, Senior...
 
Imagine your favorite world class service provider replacing the Board of...
 
When Mark Zuckerberg checked out of Harvard for California’s Silicon Valley,...
 
In his article, Gaming the System, How the misuse of data impedes...
 
My observation is people often take the view, “no one else...
 
The recent 2018 Senate Report examining the Government of Canada’s Phoenix...
 
Human services agencies play a vital role in society, providing a...
 
Tornado warnings broadcast a few weeks ago in Eastern Ontario and...
 
Canada’s international reputation for welcoming and integrating newcomers is unparalleled. At...
 
A few years back, consultants with ghSMART told us the biggest...
 
On today’s show, J. Richard Jones sits down for a chat...
 
Procurement modernization has been a Government of Canada (GC) policy priority...
 
It takes a lifetime to build a reputation and just moments...
 
Over the past two decades, codes of ethical conduct have become...
 
Earlier this year the Canadian government took a crucial step by...
 
On the global scene, technology has revolutionized and automated the work...
 
Over the past decade, major Canadian procurement projects have encountered increasing...
 
Prime Minister Justin Trudeau has, on many occasions, expressed his commitment...
 
In this episode, CGE radio show host J. Richard Jones speaks...
 
There is growing pressure on governments around the world to be...
 
Public servants are responsible for providing advice and support to the...
 
Today, the challenges facing governments are increasingly shifting away from traditional,...
 
Management in the Government of Canada is continuously searching for new...
 
Rankings of public sector entities has been big trend for quite...
 
Canadian Government Executive media through its upcoming TechGov event is providing the...
 
Ships docked in a harbour may not be going anywhere; however,...
 
The International Monetary Fund (IMF) in issuing its annual review of...
 
There’s no shortage of organizations claiming to have a digital transformation...
 
Cybersecurity professionals have sounded the alarm for years, and they are...
 
Cost estimation is becoming an extremely important skill within government due...
 
In this special episode of CGE Radio, your host John Jones...
 
Security professionals have an obligation to communicate risks and recommendations to...
 
Risk is always present in any undertaking, no matter the size...
 
Cost overruns have become institutionalized in the federal government, according to...
 
Last year, procurement Minister Jody Foote was prompted by the swirling...
 
Professionals, managers, and executives in the cost estimation industry can gain...
 
In this episode, hear from Carl Hammersburg, Manager, Government and Healthcare...
 
A new study from the Conference Board of Canada gives our...
 
In the world that we are living in today, free and...
 
The delivery method developed by Sir Michael Barber, chief adviser to...
 
Rules and accountability are helpful in developing and standardizing processes but...
 
Canadian doctors were told that climate change impacts human health and...
 
Even as talks between the government and federal workers affected problems...
 
The largest effort in 20 years to seek public input on...
 
Ottawa has overhauled the process by which justices are picked for...
 
July 27 was pay day some federal public workers that finally...
 
The Senate committee looking into Canada’s Syrian refugee program wants the...
 
In this episode, editor-in-chief, Patrice Dutil talks about the need for...
 
Yes, according to the former head of the Canadian Security Intelligence...
 
As much a 20 per cent of grade seven students in...
 
A Quebec superior court judge facing allegations that he helped commit...
 
In a move meant to the strengthen liability and compensation regime...
 
We often read news articles about rampant drug addiction and suicide...
 
Upon receiving numerous complaints regarding add-on fees that turn making economy...
 
Are you absolutely clear what the government wants to achieve? Are...
 
Veterans Affairs Canada is not adequately managing the drug component of...
 
Whether responding to emergencies back home or assisting regular troops abroad,...
 
Some public servants will have to request their departments for emergency...
 
A landmark Supreme Court ruling has paved the way for some...
 
The Canadian Radio-television and Telecommunication Commission (CRTC) yesterday heard from several...
 
At least one prime minister has resigned, another in under fire,...
 
It appears it was not just the Mounties that were affected...
 
A while back there have been numerous media reports about the...
 
Performance auditing can lead to more efficient, effective, and economical program...
 
Written By Jason McNaught The Public Service Alliance of Canada was...
 
Independence has long been regarded as a cornerstone of the auditing...
 
Canada is a diverse nation, in language, culture, geography, and, ultimately,...
 
Public sector organizations are under increasing pressure to identify all risks...
 
The government of Canada has implemented several measures over the past...
 
Whether at the territorial, provincial or federal government level, internal audit...
 
An organization’s reputation can take years to build but it can...
 
On October 30, Ontario began second reading of the Public Sector...
 
In the well-known children’s story, an Emperor falls victim to the...
 
The Nova Scotia Office of the Ombudsman is a small operation...
 
I think the ombudsman needs to be independent, because without independence...
 
Today’s business environment changes rapidly to adjust to evolving conditions and...
 
The best internal auditors actually are really good managers first. I...
 
The recent controversy about the actions of some staff members in...
 
Most professionals don’t need more than a sentence at a cocktail...
 
Recent research by the Institute of Internal Auditors Canada aims to...
 
When is it that a politician becomes part of the governing...
 
It’s been a busy couple of weeks on the information, privacy...
 
In 1996, a new budget watchdog, the Parliamentary Budget Officer, was...
 
The Ontario government is moving forward with the creation of a...
 
The Office of the Public Sector Integrity Commissioner of Canada (PSIC)...
 
After the Auditor General’s (AG) report was released on April 30,...
 
In the U.K. system, Permanent Secretaries are what we call Deputy...
 
We are living in a period of rapid change and limited...
 
US public sector employees don’t trust their management to do the...
 
The news of Mark Carney’s nomination as the new Governor of...
 
Following Singapore’s independence in 1965, the controversial leadership of Lee Kuan...
 
Over the past few years, the preparation and delivery of the...
 
In healthcare, cost-cutting can result in cutting what is valued most...
 
For over 20 years Colin Bennett has been exploring issues of...
 
Even before controversy shook the organization to its foundation, Ornge was...
 
It will be the largest international multi-sport event ever held on...
 
We’ve all seen the headlines – BC Ferries, Ornge, la Caisse...
 
It can happen, and it’s noteworthy when it does. Government, business...
 
Kevin Page’s mandate as the first Parliamentary Budget Officer comes to...
 
Governments are challenged to meaningfully mitigate the effects of the financial...
 
Canada is facing a huge financial challenge brought on by massive...
 
For the past one hundred years, democratic states have been moving...
 
It’s so much easier and less painful to learn from the...
 
CGE Vol.13 No.7 September 2007 "If the Public Service, as a...
 
When pondering leadership, we immediately think of exercising our influence downward...
 
CGE Vol.13 No.1 January 2007 "How can I be held accountable...
 
CGE Vol.13 No.2 February 2007 Canada’s Performance 2006 is the sixth...
 
CGE Vol.14 No.1 January 2008 The furor over the $300,000 that...
 
CGE Vol.14 No.2 February 2008 Let’s say you’re a senior manager,...
 
CGE Vol.13 No.1 January 2007 Perhaps it’s a legacy of the...
 
The Independent Blue Ribbon Panel on Grants and Contributions called for...
 
Au Canada, le secteur bénévole et à but non lucratif vit...
 
As the global economy struggles to regain some forward momentum, Canadian...
 
This will be a defining budget for Stephen Harper. It will...
 
It is difficult to determine when the debate about the need...
 
For the next few years, the federal government’s overarching agenda will...
 
Much of the current conversation about the federal government’s economic agenda...
 
Bill Greenlaw is the elected president of the Institute for Public...
 
Have you ever asked yourself the question: ‘How would I evaluate...
 
Last fall, Alberta’s Employment and Immigration department posted online the workplace...
 
In 2006 the world was feeling the aftershocks of a number...
 
CGE Vol.13 No.4 April 2007 Robert Parkins, editorial director, met recently...
 
In the past two decades, the nature of the state has...
 
In recent months, the attention of Canadians has been focused on...
 
Some title Some author
Some excerpt
Despite a rapid expansion in digital democracy initiatives around the globe,...