DevelopmentInnovationsKnowledge ManagementPublic Sector
July 14, 2017

Confronting the Wicked: Using Evaluation to Boost Innovation

Developmental evaluation has the potential to fill a unique niche to help accelerate the process of generating, testing and adjusting new ideas, and along the way, get better at the very processes of innovation.

How does a mandate for innovation challenge evaluation?

Increasingly, the public sector is becoming more intentional and systematic in stimulating innovation. New processes and structures, such as investments in innovation hubs and labs, are exploring ideas such as the future of the public service (e.g., NRCAN’s IN·spire), or experimenting with new approaches to complex policy and program challenges (e.g., the Privy Council’s Innovation Hub). The persistent issues of our time – social inequity, climate change, urbanization, for example – demand breakthrough ideas and novel solutions. Businesses, governments and community organizations face urgent pressures to adapt quickly, often in the face of great uncertainly and high complexity.

Challenges of Traditional Evaluation

Evaluation commonly asks questions such as: Was a program implemented as it was supposed to be? Did an intervention achieve a pre-determined result? Were benchmarks met? How can we refine the program to increase effects, reduce costs or make implementation easier? Evaluation’s purpose, in these circumstances, is to help make a judgment of merit or worth, assess the efficiency of goal attainment, or delineate with clarity the causal links for a well-defined technology or intervention to prepare for replication. These purposes are based upon linear, logical problem solving approaches, which work very well when the problem and solution well understood. Evaluation’s standardization, linearity and compliance-driven nature is a fit when there is an optimal solution with clear boundaries and a limited set of possible alternatives.

The challenge for evaluators, and for innovators, is that not all problems are bounded. They don’t all have optimal solutions or occur within stable parameters. These kinds of problems – often called complex, or “wicked” – are difficult to define. The pre-specificity, prediction and control of evaluation is often very frustrating for innovators. It does not mesh with the dynamic and rapid pace of adaptation. Too often, it has the effect of constraining the innovation process instead of supporting it.

Innovation + Evaluation = Real Time Solutions

Innovation brings novelty and creativity to the introduction of something new and useful. The very techniques that enable evaluation excellence in more static situations – standardization of inputs, consistency of treatment, uniformity of outcomes and clarity of causal linkages – are unhelpful, even harmful, to situations where there is a lot of uncertainty and ‘moving goalposts.’ With dynamic and unpredictable phenomena, narrowly defining and structuring evaluative questions can interfere with learning and adaptability. Innovation is often about breaking previous boundaries.  How can you judge if something was done as expected when you are in the midst of creating it? If you are doing something for the first time, there is no benchmark, so what do you measure against? What if we could blend the creative thinking of innovation with the critical thinking of evaluation?

Developmental evaluation is an approach to evaluation that has emerged in response to the need for adaptation in highly innovative situations. The premise of developmental evaluation is to integrate the elements of evaluation – being systematic and thorough in our thought process, collecting data, drawing upon evidence, and testing assumptions – in a way that supports the open-ended nature of adaptability.

Tracking Thinking as it Evolves

WellAhead is a national initiative of the J.W. McConnell Family Foundation to help strengthen the integration of social and emotional well-being in schools. The McConnell Foundation commonly involves a developmental evaluator in the earliest stages of their major social innovation initiatives, and they position the evaluator as part of the core program or ‘innovation’ team. In the early stages of idea development, the developmental evaluator acts as a learning partner, helping to frame the intervention, and supporting the innovation team’s evolving understanding of nature of the problem and what to do about it. As options are considered, program ideas developed, and decisions about what to do, much can be revealed about a group’s implicit theory of change, their desired impacts, and key assumptions or uncertainties. The developmental evaluation helps by reflecting back the thinking and framing up emerging working models to support the evolution and tightening of theory and concept.

Accountability for Learning

When the United Way Toronto (now the United Way Toronto and York Region) launched a city-wide initiative to engage a network of practitioners who shared a common goal for improving outcomes in youth educational attainment, they were trying out a new role as a granting organization. The initiative was a major shift from funding individual programs to acting as a systems level convener with the intent of amplifying the learning from a wide range of programs across the city. The developmental evaluation helped staff access real time feedback about what was taking shape in the initiative, so that they could continually adjust their strategies. The evaluation also supported United Way Toronto to tell the story of how this initiative was developing and what progress was being made. The insights and perspective from the evaluation helped to communicate to internal and external stakeholders what was happening, helping to strengthen the case for the approach being taken, and give the initiative a mechanism by which to manage risks related to an adaptive initiative. In being actively observant in a systematic way, a narrative is constructed that explains what is happening, as well as how and why, in a way that is grounded and credible.

Probe and Respond

In situations of high complexity, we learn most through taking action and gathering feedback. Because of this, small experiments – or prototypes – are often an essential feature of developmental evaluations (and of innovation labs). Experiments are usually low cost, low risk activities that serve the primary purpose of learning something. They differ from pilots, which are much more developed ideas or initiatives, done to demonstrate the efficacy of something before scaling, and to work out efficiencies for implementation. Developmental evaluators help innovators to construct small experiments, often in a series of iterations, or with purposeful variation in order to accelerate learning. What emerges? Do our assumptions play out? What criteria develop to tell the difference between “working” and “not working”? We also do prototypes to clarify our thinking. Expectations, priorities, and key concepts become clearer when we move from the theoretical to action. The key is not simply to do something, but to do something with a specific learning agenda with sufficient attention, data, and critical thinking to understand what emerges and inform what is next.

Conditions for Success

The private sector operates comfortably with an ethos of experimenting, quickly capturing lessons from failures, and rebuilding. This plays out in product, business or technological innovation. Similarly, NGOs and philanthropic foundations have embraced developmental evaluation in support of social innovation. What we have learned from experience of both the business sector and NGO community is that success in innovation – and by extension, the evaluation of innovation – requires organizational conditions that support adaptive thinking and change. Organizations that have succeeded with developmental evaluation are more learning centred, less risk adverse, willing to experiment and even accommodate failure.  If key stakeholders require a high degree of certainty, or there is a lack of openness to experimentation and reflection, or there is a low tolerance for risk, then a developmental evaluation is likely not a good fit.

The promise of innovation in a public sector context leads us to many places: more responsive programming, better use of resources, improved client reach, better outcomes. Innovation also introduces alternative ways of thinking and acting — often ones never previously considered or examined. This is inevitably disruptive, and the resulting uncertainty can prove challenging in an environment of hyper-accountability.  Where are the niches within the public sector that are ripe and ready for purposeful and systematic innovation? As these continue to emerge, so will the need to facilitate assessments of where novel ideas are at and how things are unfolding. Developmental evaluation has the potential to fill a unique niche to help accelerate the process of generating, testing and adjusting new ideas, and along the way, get better at the very processes of innovation.

 

Jamie Gamble is the Principal of Imprint Consulting and a member of the Canadian Evaluation Society. He is a pioneer in the field of developmental evaluation and has supported innovation and development in a wide range of issues including poverty reduction, environmental sustainability, food security, public health and safety, citizen engagement and the arts.

About this author

0 comments

There are no comments for this post yet.

Be the first to comment. Click here.

Development
 
Below is a piece of design fiction focused on what it...
 
Service delivery is an integral part of the work of governments...
 
In recent years, growing evidence about the importance of early childhood...
 
In the past 18 months or so, new governments have taken...
 
Could robot doctors be on the horizon? A study published February...
 
We are excited to announce that the February/March 2019 issue of...
 
The use of “nudges” in policymaking has been a major trend...
 
There is an interesting paradox playing out with the public sector...
 
Last week, Perpetual Guardian, a New Zealand-based financial services company, announced...
 
Canadian Government Executive, in collaboration with Dalhousie University, is seeking your...
 
Imagine your favorite world class service provider replacing the Board of...
 
Lately, all levels of the Canadian public service have devoted much...
 
We have been hearing quite a lot about superclusters. The Government...
 
In this episode of CGE Radio, J. Richard Jones talks with...
 
In 2017, identity theft cost Americans $16.8 billion, and, in just...
 
In this episode of CGE Radio, J. Richard Jones interviews Alex...
 
It is time to reboot democracy. This year, the French constitution,...
 
We are excited to share with you the October/November 2018 edition...
 
Ontario Northland is a crown corporation that provides transportation solutions to...
 
Canada’s international reputation for welcoming and integrating newcomers is unparalleled. At...
 
In the three years to 2020, Canada will welcome almost a...
 
Mining embodies progress. Mining provides civilization with the materials required for...
 
On today’s show, J. Richard Jones sits down for a chat...
 
We are excited to share with you the May/June issue of...
 
In this episode of CGE Radio, J. Richard Jones speaks with...
 
As the world grows more digital – and increasingly mobile –...
 
Throughout Canada, healthcare systems are challenged by population growth, aging, and...
 
Over half a billion children live in countries that are falling...
 
According to a recent McKinsey study there is a potential for...
 
On the global scene, technology has revolutionized and automated the work...
 
Today on CGE Radio, we speak with Craig Szelestowski, President and...
 
The adoption of new technologies provides many opportunities, poses unique challenges...
 
“Culture eats strategy for breakfast.” This quote from management specialist Peter...
 
On a recent visit to Ottawa, I had the opportunity to...
 
Professional development is tremendously important to members of Canada’s public sector....
 
Over the last two years, an ever-growing number of organizations around...
 
Ships docked in a harbour may not be going anywhere; however,...
 
Canadian Government Executive Media (CGE) and CATAAlliance , Canada’s one voice for...
 
Do Labour Market Development Agreements (LMDAs) make a difference? Our evaluation...
 
The design needed and practiced in government departments and the broader...
 
As social media becomes ever-more prevalent in the lives of today’s...
 
Evaluation is performed for a range of reasons:  to improve programs,...
 
Radical candour sounds rather outré as a prescription for government executives....
 
In this episode of CGE Radio, J. Richard Jones is joined...
 
Behavioural Insights (BI) is a multidisciplinary approach that uses principles and...
 
In March, the Canada Revenue Agency’s website was shut down by...
 
How does a mandate for innovation challenge evaluation? Increasingly, the public...
 
As the face of the Association of Professional Executives of the...
 
One of the writers in this month’s issue started a note...
 
Excellence in leadership is instrumental in serving the public interest. New-age...
 
Canadian Government Executive Media (CGE) announced last Friday, that George Ross...
 
We are thrilled to offer the July/August issue of Canadian Government...
 
At this time a year ago, the 88,000 displaced people of...
 
“Former Metro banker stole $73,000.”  “Enbridge sues ex-employee.”  “Woman stole $51,000...
 
The recent introduction of the new Government of Canada (GC) IT...
 
So, what can be said about Canada’s public administration as the...
 
The way forward to engaging Millennials in the public sector Passionate,...
 
We are happy to share with you the May/June issue of...
 
The Canadian Advanced Technology Alliance (CATA Alliance) is Canada’s One Voice...
 
Since Confederation in 1867, more than 17 million immigrants landed in...
 
In today’s world of heightened focus on good governance and accountability...
 
Orchestrating the transition from present to emergent leadership Millennial is the...
 
As ‘month 13’ for Canada’s Syrian refugee arrivals comes and goes,...
 
As governments around the world move towards the digital transformation of...
 
The way organizations recruit, shape and manage their workforce to find...
 
The promise of web 2.0 was the emergence of a user-driven...
 
Commodity Management (the official term for life-cycle asset management of goods...
 
During the past decade, the international university-level student population in Canada...
 
We have all heard the sayings: “Home, sweet home”; “There’s no...
 
Imagine a Canadian government that shares best practices in service delivery....
 
There is a new buzz in policy circles. Advances in data...
 
The North Atlantic Treaty Organization, with its 28 independent member countries...
 
The internal audit function in the Canadian public sector finds itself...
 
As Canada’s government agencies strive to improve delivery and efficiency under...
 
To boldly go where no one has gone before STAR TREK:...
 
In recent months, Canadians have borne witness to a new cycle...
 
Professionals, managers, and executives in the cost estimation industry can gain...
 
Canadian Government Executive kicked off its CGE Leadership Series for 2017...
 
The rapid advances in digital technology, demographic and societal shift even...
 
The critical challenge facing the public service is changing its culture....
 
Having the right talent is critical to delivering right results. But...
 
Sustainable Development Goals (SDGs) are part of the new global “Agenda...
 
An Interview with Larry Ostola, Toronto’s Director of Museums and Heritage...
 
For all its subtleties and mysteries, the Westminster system of government...
 
When Marshall McLuhan coined the phrase “the medium is the message”...
 
Laskey Hart guest hosts CGE Radio this week with a summary...
 
The non-profit sector is operating within an era of increased accountability,...
 
Earlier this year the Ontario Ministry of Health announced that it...
 
Gladwell made Anders Ericsson famous—or, at least, his research work. A...
 
Canada’s top soldier said the Armed Forces continue to be locked...
 
BC Hydro said it is pushing through with it $9-billion, hydroelectric megaproject on...
 
Yesterday, Federal and Atlantic Provincial Ministers of the environment met to...
 
After writing the book “Megatrends: The Impact of Infrastructure on Ontario’s...
 
Governments around the world are seeking to tap technologies such as...
 
Signaling a realignment of Canada’s involvement with NATO, Prime Minister Justin...
 
As an executive you’ve probably experienced frustration when you ask for...
 
The Public Health Agency of Canada has long been aware of...
 
Dispersed organisations can be very powerful structures. There is ample evidence...
 
By Nicole Verkindt Despite growing up in a small town in...
 
Canada, today, became a full supporter of the United Nations Declaration...
 
A research project at that explores the design and feasibility of...
 
A landmark Supreme Court ruling has paved the way for some...
 
Years and years ago when I was unemployed, being able to...
 
Local job figures blew away expectations as records indicated that 41,000...
 
At first glance, the title may seem counterintuitive. By their very...
 
The Liberal government’s first federal budget laid out $11.9 billion over...
 
The Liberal government is expected to announce on Tuesday a new...
 
A leader’s credibility begins with personal success. It ends with helping...
 
Monday was the final day for Canadians to donate money to...
 
Huawei is bringing back to Canada and expanding its information communication...
 
Many women entrepreneurs in Canada struggle to access capital, technology, networks...
 
Written by Tim Wacker Almost a quarter century ago, when most...
 
Written by  Brady G. Wilson You may not realize it, but...
 
Continuous, life-long learning is the future of the public service. Building...
 
Mega-ConnEX is an annual speed networking event organized by the Health...
 
Canada’s contribution to development and humanitarian assistance is recognized around the...
 
Deliberations and negotiations in the United Nations are intensifying on what...
 
The UN consultations on the post-2015 development framework are focused on...
 
At the turn of the millennium, the international community made a...
 
Budgets are statements of political will and power. They are performance...
 
Decades of reform in developing countries point to four enduring dilemmas:...
 
The Lough Erne Accountability Report states that the G8 has played...
 
While there can be no singular solution for good governance, building...
 
it’s not the most original saying...
 
With the demographics of an aging workforce and a significant number...
 
Shakespeare’s “dark comedy,” Measure for Measure, deals with the issues of...
 
Reform is almost always about developing the capacity of government to...
 
CGE Vol.13 No.3 March 2007 Assembly of First Nations National Chief...
 
CGE Vol. 14 No.5 May 2008 Before dawn on Monday, 18...
 
CGE Vol. 14 No.4 April 2008 “What is a Foreign Service...
 
CGE Vol.14 No.3 March 2008 Most observers expect more than 40...
 
CGE Vol.14 No.3 March 2008 The Commonwealth Secretariat convened the first...
 
If you want to understand someone else’s position, walk a mile...
 
What makes a perfect mentor? Is it age and seniority? Perhaps...
 
Often, decisions about how to recruit are made based on habit...
 
It has been said that if everything is important, nothing is...
 
The Mosaic Index, by Professor Richard Florida, which measures the percentage...
 
Government policy agendas have long been fraught with complexity....
 
Looking at all the engineering diagrams overlaid on maps on the...
 
Gérés efficacement, les conflits peuvent favoriser la résolution conjointe des problèmes,...
 
Popular wisdom holds that the public service will need to recruit...
 
CGE Vol.14 No.2 February 2008 As I mentioned previously, one cannot...
 
CGE Vol.13 No.9 November 2007 What is the toughest job in...
 
CGE Vol.13 No.9 November 2007 Growth can mean opportunity. Significant increases...
 
The subtitle of this report, prepared by the U.S. National Governors...
 
We can learn from archetypes. They can help us lead, if...
 
Improving leadership development is widely recognized by HR leaders and senior...
 
Natural Resources Canada is accustomed to nurturing, protecting and growing Canada’s...
 
In an increasingly competitive and complex global marketplace, both employers and...
 
The transition from the private sector into an executive role in...
 
Some title Some author
Some excerpt
 
Some title Some author
Some excerpt
 
Some title Some author
Some excerpt
Below is a piece of design fiction focused on what it...