DevelopmentInnovations
April 3, 2018

Evaluating Innovation in Practice: A GP for Me

Throughout Canada, healthcare systems are challenged by population growth, aging, and physician retirement and relocation. These, among other factors, have added demands to and limited the capacity of the healthcare system.

Such challenges require a combination of initiatives, including community-driven efforts to first understand and then respond to local and regional needs. In British Columbia, the Ministry of Health and Doctors of BC worked in partnership through the General Practice Services Committee (GPSC), to foster community action of this sort, through a three-year initiative named A GP for Me. Launched in 2013, A GP for Me had three goals:

  • Confirm and strengthen the continuous doctor and patient relationship, including better support for the needs of vulnerable patients.
  • Increase the capacity of the primary health care system.
  • Enable patients who want a family doctor to find one.

Funding for A GP for Me was provided in two ways: the creation of incentive fees to facilitate practice-level change with physicians; and investments in projects delivered by more than thirty Divisions of Family Practice, groups of family physicians in over 230 communities across British Columbia.

At the heart of it, A GP for Me was truly a community-driven, broad undertaking to foster innovation, quality and continuity in patient access and care.

The Task: Undertaking a Provincial Evaluation of A GP for Me 

Evaluating an initiative like that of A GP for Me is complicated. A few of the twists and turns of this endeavour centred on the following.

Relating Results: A GP for Me was one of many efforts in British Columbia to further health care system transformation. Additionally, divisions of family practice had been formed around the province, providing an alternative way for family physicians to collaborate with health authorities and other partners to identify and address local healthcare challenges. Over the course of A GP for Me, about 150 projects were started by divisions, each furthering its own form of innovation. This made the direct attribution and summary of results to A GP for Me challenging.

Harnessing Efforts: The purpose of the provincial evaluation was to not only gain an understanding of the effectiveness of A GP for Me from the experiences of those involved but also to share lessons learned from innovations that seek to enhance the quality of care.

In addition to the provincial evaluation, divisions were also completing assessments of the effectiveness of supported projects, to inform their own learning and innovation. The provincial evaluation had to harness these local efforts to allow for an integration of findings, while also not obstructing the unique work that was proceeding.

Dealing with Unknowns:  In 2015, a substantial Provincial Evaluation Plan was prepared for A GP for Me with the involvement of staff from the Doctors of BC and the Ministry of Health, a GPSC Evaluation Working Group, contracted specialists, a Divisions Reference Group, and evaluators hired by various divisions.

Following the completion of this Provincial Evaluation Plan, and by the time the provincial evaluation commenced, it became apparent that the awareness of stated outcomes and indicators of performance for A GP for Me had since diminished such that partners were challenging concepts, describing intentions and using measures in a variety of disparate ways. This complicated the ability to draw comparisons and roll-up findings across the province. In addition, as the provincial evaluation progressed, it became clear that baseline data was not always available for some of the indicators and planned data sources (e.g. a Core Provincial Patient Survey) would not be obtained in time.

The Answer: A Collaborative and Adaptive Approach to the Evaluation

The success of the provincial evaluation of A GP for Me was founded on the following key principles that guided the work of what was a hybrid evaluation team, made up of staff from the Doctors of BC as the contracting authority and MNPLLP as the contracted evaluator.

Collaboration: Members of the Doctors of BC team and the MNP team worked closely together, such that the provincial evaluation was a truly collaborative undertaking.  This extended beyond frequent, joint check-in meetings to include the co-facilitation of consultations as well as the assigning of work across the combined teams. It was also critical to coordinate with the work of other evaluators, be it the ones engaged by divisions or those working on components of the Provincial Evaluation Plan.

Integration: The provincial evaluation had to integrate the foundational work of the Divisions of Family Practice and that of several other external evaluators, as well as administrative data from the Ministry of Health.  With the divisions, a tiered, thematic analysis was carried out of thirty “Final Evaluation and Project Implementation Close-Out Reports” in the form of a high-level summary, division highlights, and direct report extracts.

In addition to these close-out reports and what was referenced above, there were the results of surveys (e.g. of family physicians, with the GPSC, and the Canadian Community Health Survey), interviews with health authorities and the BC Ministry of Health, and facilitated focus group sessions.

Adaptation: Allowances had to be made for adaptations to the Provincial Evaluation Plan to address gained insight, respond to any gaps in information, and ensure key perspectives were accounted for.

It was also critical to follow an iterative approach to validating early findings and interpretations. This took the form of a staged confirmation of the findings with a variety of stakeholders, covering the various components of the analyses (e.g., interpretations of the administrative data).

The Takeaways: Reflecting on the Experiences of the Evaluation

Just as A GP for Me has illustrated that there is not one solution to address the complex challenges of BC’s health care system, the provincial evaluation highlighted opportunities for future practice.

There are clear benefits to be gained from adopting hybrid evaluation teams, bringing together experienced staff and third-party contractors.  Yet, it is important to also keep in mind the maintenance of harmony and balance in teams, be it with respect to size, well-defined and agreed upon roles, and contributed areas expertise.

Having a well-thought-out plan, or framework, for an evaluation is another factor of success.  The value of this planning though is nicely summed up by Voltaire, “don’t let the perfect be the enemy of the good.” Ongoing socialization of an evaluation plan and allowing for moments for adaptation along the way should be embraced.

With initiatives that serve to foster innovation, especially across geographic and other communities of interest, there can be a concern of imposing structure, which can come in the form of evaluation.  Just as innovation depends on some arrangement for implementation, evaluators need to consider how findings will be combined in a coherent way.

Finally, the pursuit of an all-inclusive, retrospective evaluation may be better replaced with an iterative strategy that allows for the staging of evaluative studies over the course of a major long-term initiative like A GP for Me.  In this way, the evaluation approach will naturally evolve with the initiative, and the latter stages of evaluation will yield findings that are informed by refined priorities, strategic direction and focus.

During the final stages of the A GP for Me evaluation, the GPSC was looking forward and updating its strategic direction to include the patient medical home as the foundation of an integrated system of care. The provincial evaluation informed the evolution of the primary health care system in British Columbia, by providing important insights into local healthcare challenges; raising awareness of community-driven innovations that build on the important work of physicians, divisions of family practice, and health authorities; and furthering an environment for conversation on how to continue to improve patient access and care.

 

Bill Reid, Partner, Advisory Services, MNP and Past President, BC and Yukon Chapter, Canadian Evaluation Society, bill.reid@mnp.ca

Jasmina Fatic, Manager, Advisory Services, MNP, and Past Vice President, BC and Yukon Chapter, Canadian Evaluation Society, jasmina.fatic@mnp.ca

Petra Lolic, Executive Coordinator, GPSC, Doctors of BC, plolic@doctorsofbc.ca

0 comments

There are no comments for this post yet.

Be the first to comment. Click here.

Development
 
In the next year, one in three US federal government workers...
 
Despite a rapid expansion in digital democracy initiatives around the globe,...
 
In the show, J. Richard Jones talks with Colin Earp, Partner...
 
Deforestation interventions in Latin America and Asia may be more effective...
 
Below is a piece of design fiction focused on what it...
 
Service delivery is an integral part of the work of governments...
 
In recent years, growing evidence about the importance of early childhood...
 
In the past 18 months or so, new governments have taken...
 
Could robot doctors be on the horizon? A study published February...
 
We are excited to announce that the February/March 2019 issue of...
 
The use of “nudges” in policymaking has been a major trend...
 
There is an interesting paradox playing out with the public sector...
 
Last week, Perpetual Guardian, a New Zealand-based financial services company, announced...
 
Canadian Government Executive, in collaboration with Dalhousie University, is seeking your...
 
Imagine your favorite world class service provider replacing the Board of...
 
Lately, all levels of the Canadian public service have devoted much...
 
We have been hearing quite a lot about superclusters. The Government...
 
In this episode of CGE Radio, J. Richard Jones talks with...
 
In 2017, identity theft cost Americans $16.8 billion, and, in just...
 
In this episode of CGE Radio, J. Richard Jones interviews Alex...
 
It is time to reboot democracy. This year, the French constitution,...
 
We are excited to share with you the October/November 2018 edition...
 
Ontario Northland is a crown corporation that provides transportation solutions to...
 
Canada’s international reputation for welcoming and integrating newcomers is unparalleled. At...
 
In the three years to 2020, Canada will welcome almost a...
 
Mining embodies progress. Mining provides civilization with the materials required for...
 
On today’s show, J. Richard Jones sits down for a chat...
 
We are excited to share with you the May/June issue of...
 
In this episode of CGE Radio, J. Richard Jones speaks with...
 
As the world grows more digital – and increasingly mobile –...
 
Throughout Canada, healthcare systems are challenged by population growth, aging, and...
 
Over half a billion children live in countries that are falling...
 
According to a recent McKinsey study there is a potential for...
 
On the global scene, technology has revolutionized and automated the work...
 
Today on CGE Radio, we speak with Craig Szelestowski, President and...
 
The adoption of new technologies provides many opportunities, poses unique challenges...
 
“Culture eats strategy for breakfast.” This quote from management specialist Peter...
 
On a recent visit to Ottawa, I had the opportunity to...
 
Professional development is tremendously important to members of Canada’s public sector....
 
Over the last two years, an ever-growing number of organizations around...
 
Ships docked in a harbour may not be going anywhere; however,...
 
Canadian Government Executive Media (CGE) and CATAAlliance , Canada’s one voice for...
 
Do Labour Market Development Agreements (LMDAs) make a difference? Our evaluation...
 
The design needed and practiced in government departments and the broader...
 
As social media becomes ever-more prevalent in the lives of today’s...
 
Evaluation is performed for a range of reasons:  to improve programs,...
 
Radical candour sounds rather outré as a prescription for government executives....
 
In this episode of CGE Radio, J. Richard Jones is joined...
 
Behavioural Insights (BI) is a multidisciplinary approach that uses principles and...
 
In March, the Canada Revenue Agency’s website was shut down by...
 
How does a mandate for innovation challenge evaluation? Increasingly, the public...
 
As the face of the Association of Professional Executives of the...
 
One of the writers in this month’s issue started a note...
 
Excellence in leadership is instrumental in serving the public interest. New-age...
 
Canadian Government Executive Media (CGE) announced last Friday, that George Ross...
 
We are thrilled to offer the July/August issue of Canadian Government...
 
At this time a year ago, the 88,000 displaced people of...
 
“Former Metro banker stole $73,000.”  “Enbridge sues ex-employee.”  “Woman stole $51,000...
 
The recent introduction of the new Government of Canada (GC) IT...
 
So, what can be said about Canada’s public administration as the...
 
The way forward to engaging Millennials in the public sector Passionate,...
 
We are happy to share with you the May/June issue of...
 
The Canadian Advanced Technology Alliance (CATA Alliance) is Canada’s One Voice...
 
Since Confederation in 1867, more than 17 million immigrants landed in...
 
In today’s world of heightened focus on good governance and accountability...
 
Orchestrating the transition from present to emergent leadership Millennial is the...
 
As ‘month 13’ for Canada’s Syrian refugee arrivals comes and goes,...
 
As governments around the world move towards the digital transformation of...
 
The way organizations recruit, shape and manage their workforce to find...
 
The promise of web 2.0 was the emergence of a user-driven...
 
Commodity Management (the official term for life-cycle asset management of goods...
 
During the past decade, the international university-level student population in Canada...
 
We have all heard the sayings: “Home, sweet home”; “There’s no...
 
Imagine a Canadian government that shares best practices in service delivery....
 
There is a new buzz in policy circles. Advances in data...
 
The North Atlantic Treaty Organization, with its 28 independent member countries...
 
The internal audit function in the Canadian public sector finds itself...
 
As Canada’s government agencies strive to improve delivery and efficiency under...
 
To boldly go where no one has gone before STAR TREK:...
 
In recent months, Canadians have borne witness to a new cycle...
 
Professionals, managers, and executives in the cost estimation industry can gain...
 
Canadian Government Executive kicked off its CGE Leadership Series for 2017...
 
The rapid advances in digital technology, demographic and societal shift even...
 
The critical challenge facing the public service is changing its culture....
 
Having the right talent is critical to delivering right results. But...
 
Sustainable Development Goals (SDGs) are part of the new global “Agenda...
 
An Interview with Larry Ostola, Toronto’s Director of Museums and Heritage...
 
For all its subtleties and mysteries, the Westminster system of government...
 
When Marshall McLuhan coined the phrase “the medium is the message”...
 
Laskey Hart guest hosts CGE Radio this week with a summary...
 
The non-profit sector is operating within an era of increased accountability,...
 
Earlier this year the Ontario Ministry of Health announced that it...
 
Gladwell made Anders Ericsson famous—or, at least, his research work. A...
 
Canada’s top soldier said the Armed Forces continue to be locked...
 
BC Hydro said it is pushing through with it $9-billion, hydroelectric megaproject on...
 
Yesterday, Federal and Atlantic Provincial Ministers of the environment met to...
 
After writing the book “Megatrends: The Impact of Infrastructure on Ontario’s...
 
Governments around the world are seeking to tap technologies such as...
 
Signaling a realignment of Canada’s involvement with NATO, Prime Minister Justin...
 
As an executive you’ve probably experienced frustration when you ask for...
 
The Public Health Agency of Canada has long been aware of...
 
Dispersed organisations can be very powerful structures. There is ample evidence...
 
By Nicole Verkindt Despite growing up in a small town in...
 
Canada, today, became a full supporter of the United Nations Declaration...
 
A research project at that explores the design and feasibility of...
 
A landmark Supreme Court ruling has paved the way for some...
 
Years and years ago when I was unemployed, being able to...
 
Local job figures blew away expectations as records indicated that 41,000...
 
At first glance, the title may seem counterintuitive. By their very...
 
The Liberal government’s first federal budget laid out $11.9 billion over...
 
The Liberal government is expected to announce on Tuesday a new...
 
A leader’s credibility begins with personal success. It ends with helping...
 
Monday was the final day for Canadians to donate money to...
 
Huawei is bringing back to Canada and expanding its information communication...
 
Many women entrepreneurs in Canada struggle to access capital, technology, networks...
 
Written by Tim Wacker Almost a quarter century ago, when most...
 
Written by  Brady G. Wilson You may not realize it, but...
 
Continuous, life-long learning is the future of the public service. Building...
 
Mega-ConnEX is an annual speed networking event organized by the Health...
 
Canada’s contribution to development and humanitarian assistance is recognized around the...
 
Deliberations and negotiations in the United Nations are intensifying on what...
 
The UN consultations on the post-2015 development framework are focused on...
 
At the turn of the millennium, the international community made a...
 
Budgets are statements of political will and power. They are performance...
 
Decades of reform in developing countries point to four enduring dilemmas:...
 
The Lough Erne Accountability Report states that the G8 has played...
 
While there can be no singular solution for good governance, building...
 
it’s not the most original saying...
 
With the demographics of an aging workforce and a significant number...
 
Shakespeare’s “dark comedy,” Measure for Measure, deals with the issues of...
 
Reform is almost always about developing the capacity of government to...
 
CGE Vol.13 No.3 March 2007 Assembly of First Nations National Chief...
 
CGE Vol. 14 No.5 May 2008 Before dawn on Monday, 18...
 
CGE Vol. 14 No.4 April 2008 “What is a Foreign Service...
 
CGE Vol.14 No.3 March 2008 Most observers expect more than 40...
 
CGE Vol.14 No.3 March 2008 The Commonwealth Secretariat convened the first...
 
If you want to understand someone else’s position, walk a mile...
 
What makes a perfect mentor? Is it age and seniority? Perhaps...
 
Often, decisions about how to recruit are made based on habit...
 
It has been said that if everything is important, nothing is...
 
The Mosaic Index, by Professor Richard Florida, which measures the percentage...
 
Government policy agendas have long been fraught with complexity....
 
Looking at all the engineering diagrams overlaid on maps on the...
 
Gérés efficacement, les conflits peuvent favoriser la résolution conjointe des problèmes,...
 
Popular wisdom holds that the public service will need to recruit...
 
CGE Vol.14 No.2 February 2008 As I mentioned previously, one cannot...
 
CGE Vol.13 No.9 November 2007 What is the toughest job in...
 
CGE Vol.13 No.9 November 2007 Growth can mean opportunity. Significant increases...
 
The subtitle of this report, prepared by the U.S. National Governors...
 
We can learn from archetypes. They can help us lead, if...
 
Improving leadership development is widely recognized by HR leaders and senior...
 
Natural Resources Canada is accustomed to nurturing, protecting and growing Canada’s...
 
In an increasingly competitive and complex global marketplace, both employers and...
 
The transition from the private sector into an executive role in...
 
Some title Some author
Some excerpt
 
Some title Some author
Some excerpt
 
Some title Some author
Some excerpt
 
Some title Some author
Some excerpt
 
Some title Some author
Some excerpt
In the next year, one in three US federal government workers...