Performance Measurement
December 29, 2014

How to measure influence on policy

Getting a clear definition of policy processes and functions can be difficult. A quick perusal of popular definitions suggests that policy processes involve problem identification, policy development – which may include research, advice, consensus building and advocacy, policy implementation and evaluation – as well as other activities. We understand that it relates to decision-making and its support, but that not all decisions represent policies.

Policy processes and functions are arguably even more difficult to monitor and evaluate. The Institute of Development Studies (IDS) in the U.K. has articulated several challenges:

• It can be difficult to determine the links between support activities, influencing activities and subsequent changes in policy;
• It is difficult to establish causality, especially because a counter factual is difficult to construct. It is also difficult to judge the contribution of one organization to any given change;
• A clear success in terms of achieving a specific expected change is quite rare;
• Policy change tends to occur over long time periods; and
• The political nature of policy makes it difficult to assess and interpret the accounts of different policy actors.

Given these difficulties, analysts have established various theoretical frameworks to help explain and examine policy functions. Could there be a way to explicitly link policy results to commonly understood theories of change? Can potential policy results be categorized and possibly organized into a logical results chain? Table 1, constructed by the author, draws on the work of various analysts and practitioners, as well as practice observation. It has been found to be useful in helping people to frame and to understand the nature of policy influence and results.

Table 1:  Summary Table of Policy Influence Impacts
Basic Result Description Policy Influence Impacts ‘Menu’
Decisions and Actions level and/or extentof adoption of new and/or specific policy content and/or changes to policies (content), practices, protocols, processes and/ordelivery. Changing institutions – influencing policy, strategy and resource allocations, including developing legal, regulatory and social frameworks.

Changes in the way policy is delivered – substantive change in the way policy is implemented and/or the way policy is delivered to intended recipients.

Changes in policy content – substantive changes in the content of policy and/or resources allocated.

Learning level of changes to awareness, knowledge, understanding, attitudes, skills, aspirations, commitments and /or priorities. Agenda setting – changes in policymakers’ priorities, with attention to previously underemphasized policy issues.

Developing capacity – improving high-level understanding of an issue, and improving how policymakers respond.

Shifts in policy framing – changes in the way that policymakers understand a problem or the possible responses to it.

Reactions type(positive or negative) and  level (extent of positive or negative)of thoughts, feelings and reactions from the experience Changing perceptions – increasing awareness and shaping public opinion.
Engagement type (e.g., network, cooperating, coordinating, collaborating),  extent (i.e., level, coverage of groups, dosage, intensity) and quality (degree of openness, information sharing, transparency among actors, communication directness/clarity, timeliness and “good faith” as appropriate to the desired type). Building networks and partnerships that support the delivery of change.

Promoting dialogue exchange and learning among (network) members.

Convening organizations or people.

The categories in this framework form a prospective checklist or “menu” of results which can be reviewed with study respondents, planners, managers and others. They also form a sort of results logic showing the extent and depth of change that may occur from any given set of policy activities or functions. In other words, one can judge one’s progress along the results chain of policy influence. In practice we have found that offering such categories actually provides a broader and heightened perspective on the impacts of any given policy initiative. The framework provides more options to find beneficial outcomes for policy support programs and functions.

In one recent case, it was found that virtually none of the specific recommendations from a study were adopted in a given recipient policy community. Upon review using this framework, it was clear that while the specific recommendations were not adopted due to organizational resistance and “politics,” the policy work had succeeded in reframing users’ understanding of a particular issue and they went on to profoundly change plans and actions as a result of that improved understanding. In this way we argued that learning and an agenda change had occurred even though specific recommendations were not adopted.

The results framework for policies in Table 1 can construct a success definition for analysts performing case studies, evaluations and reviews. The accumulation of the changes noted in this framework can be taken as progress markers for success.

The framework can be used as a good start to lay out one’s perspective on a policy function’s influence and results. It is expected that this framework will continue to evolve as it is informed by generative practice.

About this author

0 comments

There are no comments for this post yet.

Be the first to comment. Click here.

Performance Measurement
 
Leadership, service delivery, change management, performance measurement, big data, transparency, digital...
 
Recently, there has been considerable interest in what is being referred...
 
Over the years, governments have been adopting Lean management principles to...
 
As the resettlement of Syrian Refugees exemplifies, government programs, currently running...
 
With businesses and government agencies increasingly operating in a highly information-based...
 
Risk is always present in any undertaking, no matter the size...
 
The Government of Canada’s newly announced Policy on Results replaces the...
 
In this episode, J. Richard Jones share information about the Canadian...
 
Rules and accountability are helpful in developing and standardizing processes but...
 
For several months now, the Liberal government ministers have been using...
 
Global market trends are accelerating to increase the pressure on commercial...
 
As a consultant, I work primarily with private sector organizations including...
 
Here’s some bad news for organizations contemplating projects that have something...
 
If the purpose of the Government of Canada’s performance measurement system...
 
Performance auditing can lead to more efficient, effective, and economical program...
 
The latest Auditor General’s report on Shared Services Canada (SSC) and...
 
Public sector innovation, once dismissed as a contradiction in terms, is...
 
The necessity for government executives to manage change, and achieve results,...
 
Every organization does it, but often with ambivalence: the performance measurement...
 
There is an apparent disconnect in performance measurement. Today, agencies measure...
 
The government of Canada’s transformation agenda continues to drive change across...
 
Cost savings in government programs and services used to be achieved...
 
As we move into 2015 – with a looming federal election...
 
Over the last four decades, public sector organizations have institutionalized program...
 
When you purchase clothes, do you like off-the-rack, or do you...
 
All governments, including Canada’s, strive to manage taxpayers’ dollars more efficiently....
 
Getting a clear definition of policy processes and functions can be...
 
While government agencies do not have “bottom lines” in the way...
 
Most of you already know that project management is a means...
 
The taxpayers are wary. Politicians worry about retaining their jobs. Everyone...
 
There has never been a time since the introduction of program...
 
The management of resources has always been taken for granted in...
 
In the last two decades, managing for results has gained momentum...
 
For the first time, the Consortium of Universities for Evaluation Education...
 
The government of New Zealand, and the State Services Commission in...
 
While it has been around for over 130 years, the concept...
 
Performance measurement and public reporting are intended to achieve two complementary...
 
Canada has over 400 administrative tribunals that are probably the least...
 
Last summer, the Vice Chief of the Defence Staff (VCDS) signed...
 
The core mandate of the Consortium of Universities for Evaluation Education...
 
the author fails to provide a complete picture (the good and...
 
Why talk about ethics at all right now? After all, Canada’s...
 
we now have what one federal ADM described to me as...
 
The reduction or mitigation of risks or harms is a major...
 
What are Canadian universities doing to maintain their relevance and innovative...
 
Managing for results has gained significant momentum in the public sector...
 
“Is performance management still relevant?” was the thought-provoking question posed by...
 
Within the last year, the Treasury Board of Canada Secretariat (TBS)...
 
Results matter in the public sector, particularly in these challenging times...
 
Today’s public sector environment is one in which many managers and...
 
Just as with public financial reporting, performance reporting is here to...
 
Being a high-performing organization is like being a high-performing individual: it...
 
With the current reality of deficit reduction, cost cutting and overall...
 
This article is New and Improved! After an exhaustive review cycle...
 
Canada was at the table when the “Commonwealth of Nations” was...
 
If you believe that performance measurement is a vital part of...
 
The last great international effort to refashion the public sector took...
 
Beginning on April 1, 2013, and over a five-year cycle, the...
 
Governments are increasingly using benchmarking to improve organizational performance; to determine...
 
Senior public sector managers, deputy heads of departments and parliamentarians need...
 
Canada is a hockey-loving nation and now is the time of...
 
Too few public sector organizations have truly embedded performance thinking into...
 
CGE Vol.13 No.1 January 2007 The proposed Federal Accountability Act is...
 
CGE Vol.14 No.2 February 2008 The passage of the Federal Accountability...
 
CGE Vol.13 No.3 March 2007 Governing is never easy. And in...
 
CGE Vol.13 No.2 February 2007 Improving organizational health and employee engagement...
 
CGE Vol. 14 No.4 April 2008 The Tellier-Mazankowski advisory committee on...
 
L’administration publique s’efforce d’obtenir des résultats tangibles en mettant en œuvre...
 
Une étude récente commandée par l’OCDE conclut que, malgré le fait...
 
A recent study commissioned by the OECD concludes that while the...
 
The principle is unassailable: Canadians have the right to know whether...
 
La reconception de l’administration de l’imposition des sociétés en Ontario est...
 
Dans son budget de lutte contre la récession (et d’économies dans...
 
The Harper government’s recession fighting (and government saving) budget heralded an...
 
Many governments have bought into the notion of the Balanced Scorecard,...
 
Bon nombre d’administrations publiques ont adopté le tableau de bord prospectif,...
 
David P. Norton est le cocréateur, avec Robert Kaplan, du Tableau...
 
Together with Robert Kaplan, David P. Norton is the creator of...
 
Performance management leads to either disdain or glee in public sector...
 
Il y a dix ans, la Société canadienne du sang (SCS)...
 
Les administrations publiques utilisent de plus en plus l’analyse comparative pour...
 
Début septembre, la fonction publique de l’Ontario (FPO) célébrait le 10e...
 
The redesign of corporate tax administration for Ontario is a major...
 
Coincidental with the push for more attention to be paid to...
 
The BC Public Service has had an intense focus on improving...
 
Today’s public sector managers encounter increasingly complex corporate issues that call...
 
Performance management seems to be everywhere these days, with much of...
 
The two key tools used by the Canadian government to measure...
 
Some title Some author
Some excerpt
 
Some title Some author
Some excerpt
Leadership, service delivery, change management, performance measurement, big data, transparency, digital...