Canada is a diverse nation, in language, culture, geography, and, ultimately, the needs of its citizens. Such diversity can pose a challenge to a government that seeks to develop, implement, and evaluate national-level programming to local communities. Ultimately, programs and policies must be tailored to meet the unique needs of a community.
That presents a conundrum: the more a program is locally relevant, the harder it is to assess effectiveness at the national level.
In academic terms, this conundrum is part of the positivist vs. realist debate. Thankfully, there can be a reconciliation and way to move forward – where scientific rigour and local relevance can be of equal consideration in evaluation.
Realist evaluation offers a lens on establishing evidence; pushing beyond a focus on outcomes, and broadening to include both the context and processes (or mechanisms) affecting such outcomes. In fact, Health Canada has been exploring and researching this approach, particularly with respect to health promotion, and has added realist evaluation (RE) and syntheses to its methodological repertoire.
What is RE?
A mantra commonly used to describe RE is “what works, for whom, how, why, and under what circumstances.” This highlights the important role of context – namely, characteristics of the population or service provider, and of the setting. In addition, RE seeks to uncover the processes or mechanisms that bring certain outcomes. As a result, RE is frequently described as a theory-based approach to evaluation, including theory of change related to participants and theory of change related to the implementation of programs and policies.
The nuance of RE is the recognition that certain contexts may facilitate or hinder processes that bring about outcomes. This latter point is particularly relevant to diversity of the Canadian context – if evaluations uncover common mechanisms while capturing contextual features that support or hinder outcomes, one can identify successful processes that can be targeted across the contexts known to support outcomes (instead of applying them invariably across all contexts). Thus, realist evaluation has potential to focus on both the nation (common mechanisms) and the community (unique contextual features).
Current examples
An example of program implementation informed by a RE has been demonstrated by the Scottish government. The goal was to implement midwife-led care as a result of research demonstrating reductions in routine childbirth interventions and improvements in patient-reported care experiences.
Using interviews, focus groups, and multiple case studies, researchers found that consultant midwives varied in their approach to implementation of care in the team context according to the strength of the medical culture. In contexts with an ingrained medical culture (e.g., with little support for midwives), the appointment of midwives by administration demonstrated a high-level commitment to implementation of midwifery care and midwives relied on more robust change strategies (e.g., audits, debates, hands-on-training). In contexts where midwives were supported by obstetricians and physicians, midwife-lead care could emerge through empowerment of midwives to practice more autonomously.
Overall, the study was able to identify a set of mechanisms (e.g., commitment, resistance, feeling pressure) that would have differential impacts on outcomes according to the unique context of each hospital.
RE has also informed policy development. For example, the Australian Institute of Criminology undertook a realist review on effective crime prevention strategies. In addition to identifying which interventions appeared to be successful in preventing crimes, the authors were able to distill why and in what circumstances programs worked. Through a systematic realist review, they were also able to identify the necessary requirements for implementing these programs.
The findings from this review have led to the development of factsheets and handbooks that local governments can use to select, adapt, and implement interventions within their regions. In this way, if policymakers and practitioners understand how and why programs work in different contexts, better decisions can be made with regard to which programs or policies to use and how to adapt them in different contexts.
Limitations
Embarking on RE requires reflection, multidisciplinary collaboration and time. Not surprisingly there can be systemic barriers to such an approach that will need to be addressed, overcome, or circumvented. RE is inherently messy as it is about complexity – there can be multiple mechanisms operating in an orchestrated fashion. Finally, conclusions from RE are tentative directions and not considered stable and infallible.
As a nation recognizant of both unique and common identities, striking the right balance between both program and policy is crucial. RE offers a relevant framework that targets mechanisms/processes, as well as uncovers and documents contextual features responsible for supporting particular outcomes.
The broader application of RE could be facilitated through the identification of necessary infrastructure (e.g., cross discipline teams). Several scholars within the CUEE are actively pursuing research into this burgeoning evaluation approach to determine its application in governmental evaluation.
