The year 2016 saw two critical words circling the halls of Ottawa – ‘results’ and ‘deliverology.’ The first is not new to the federal public service, though occasionally it is touted as a major breakthrough (let’s not forget that Results for Canadians was the centrepiece of the government’s management framework at the turn of the century). The second proved a real attention-getter. It was in part because of the newness and awkwardness of the word itself, but also because of its friends in high places. Based on an approach introduced by the Tony Blair government in the UK in 2001, the concept of deliverology was introduced onto the Canadian federal scene with the creation of a centre of government ‘results and delivery unit’ to be based in the Privy Council Office (PCO). The intent is laudable – to put a sharper focus on government priorities to help ensure that government delivers on its commitments, goals are met and ‘results’ delivered to Canadians. As is so often the case with best intentions aligned to new approaches, though, the devil is in the detail, and government, in instituting the deliverology model, will need to take account of the Canadian context if it is to work effectively.
What does it mean to ‘measure results’?
Of critical importance to an effective deliverology process are the mechanisms/tools to measure and analyze ‘results’ – performance measurement/monitoring (M) and evaluation (E). (Internal Audit (IA) is also an important public sector tool that traditionally examines issues of efficiency and compliance, and not effectiveness of programs or policies.) They are complementary to one another: M is best suited to measuring outputs and often short-term outcomes of a program, while E is generally the more cost-effective tool to measure, analyze and understand medium- and longer-term outcomes and impacts. All represent ‘results’ in terms of a government intervention, though the need for such information and the underlying detail required likely varies across a range of ‘users’ of results information.
One might think of a whole hierarchy of potential users of ‘results’ information that varies, in part, by the type of information they may need for their particular use (See Chart 1). At an operational end, from the perspective of good governance, management and accountability, government managers need to understand how and how well their programs are operating: are they delivering on results expected and, if not, why not and what needs to be done to alleviate or improve the situation? They also represent value for money. Most of this information and understanding needs to be gleaned from a systematic and objective assessment of program performance, including delivery on results – in other words, via an evaluation. Monitoring information that examines trends or comparative statistics across a few select indicators will generally not provide an adequate level of information needed to understand performance associated with most social or economic programs.
At the other end of the hierarchy, Parliamentarians and Canadians likely don’t need the same level of detail in most reporting on results – some select indicators and an overview analysis may be sufficient to inform the most interested on how and how well results associated with particular goals are being delivered. What they do need, though, as do all users of results information along such a hierarchy, is an assurance that expectations regarding public sector governance, good management practices and accountability are being met. To do so demands that ‘measuring and reporting on results’ needs to start with what Chart 1 refers to as ‘foundational information to understand results,’ and thus needs to incorporate both M and E as tools to measure and analyze ‘results.’
Canadian Experience to date with Measuring ‘Results’
Unlike most other countries or jurisdictions where deliverology has been introduced, Canada has at the federal level a more comprehensive and systematically institutionalized system of monitoring and evaluation for measuring ‘results.’ A formalized Evaluation function in government has evolved over the past four decades, with all major departments and agencies devoting resources to assess the effectiveness, efficiency and continued need for public programs. (The federal government currently produces some 125 evaluation studies annually dealing with results of various public programs. All are easily accessed on public departmental websites.) Indeed, this has facilitated the introduction of a more systematic and results-oriented approach to Performance Measurement/Monitoring across government. (There is no doubt that the introduction of Results for Canadians in 2000 as the government’s management framework gave added impetus to the drive to build a more results-oriented approach to both M and E as the key tools to be used by departments and agencies.) The Canadian model could succinctly be described as ‘central leadership with departmental/agency delivery,’ with the Treasury Board Secretariat (TBS) setting the policy, standards and guidelines that underlie the use of both E and M across government.
But, while Canada is regarded internationally as one of the world leaders in Evaluation, measuring and using results information is not without its challenges. Indeed the Evaluation Policy has not remained static, but has been altered some four times over the past four decades, in part to address issues of measurement, as well as issues of experience and use, with an increasing focus on ‘results’ over this period. The most recent adjustment occurred in July 2016 with the introduction of the government’s Policy on Results that incorporated new standards and guidelines for both E and M into the Policy.
In theory, therefore, some of the potential challenges that deliverology typically faces in other jurisdictions – shortage of needed skills, capacity, data and experience in translating aspirational goals into measurable outcomes – should readily be handled in the Canadian federal scene, given the lengthy experience and evolution with the use of M and E to measure results. This of course hinges on whether or not the Canadian context is readily incorporated into the deliverology process.
What can we Learn about the Deliverology Process from the International Arena?
While different versions of the deliverology concept have been introduced into various governments and jurisdictions around the world, there is limited knowledge on its effectiveness. Indeed, the Telfer School of Management at the University of Ottawa conducted a systematic literature review and concluded that “there is, at this point in time, no definitive research that points to the success or failure of the concept [but that] success or failure depends a great deal on the ‘how’ of implementation” (see Greg Richards et al “Does Deliverology Deliver?”, CGE, December 2016, p. 5.) More comprehensive research of the World Bank has concluded that the deliverology approach ought not to be viewed as a ‘magic bullet,’ noting that “Each country has its own public service values, reform program and institutional pattern and a Delivery Unit must fit within that context if it is to effectively support improvement and reform.” (See R. Shostak et al, “When Might the Introduction of a Delivery Unit Be the Right Intervention?” Governance and Public Sector Management Practice Note, World Bank: June 2014.) Understanding and aligning with the Canadian context would thus seem to be critically important as deliverology gets rolled out across the federal landscape. But, what would that entail?
Examining international experience with deliverology can be informative, since it has pointed to both potential pros and cons in the way it is being implemented. On the positive side, deliverology, given its link to political power, generally brings with it authority, resources, flexibility and a striving for provision of timely advice and quick turnaround (i.e. a sense of urgency that can potentially cut through bureaucratic roadblocks to action). Potential downsides with deliverology which might have relevance to the Canadian scene include: a tendency to rely on scorecards and key performance indicators (KPIs), ignoring evaluation as a tool to measure and understand ‘results’ (it is worth noting that in many cases where deliverology has been introduced, a capacity for systematic evaluation either did not exist or was in its infancy). There is a potential for conflicts between departmental managers and the requirements imposed by deliverology that may adversely impact the building of M and E into the organizational culture and operations and, by extension, decision-making and planning processes. Thirdly, the potential burden imposed by the deliverology process on targeted departments associated with government priorities could impair the effort. Finally, there may be a perceived lack of openness. What does Canada need to do to maximize the positives and minimize the potential negatives?
Some Considerations in Going Forward
Chart 2 depicts the relationships being established among the major players implicated with the introduction of deliverology in Canada – PCO; TBS; departments/agencies; and, the political support of the government.
Their smooth functioning however is not guaranteed, and so some considerations addressing practical implementation issues are offered. They address both the tools to measure results and the institutional arrangements, including roles, responsibilities and coordination across major players.
(i) Recognizing the importance of E in measuring and understanding ‘results’: What the lengthy experience with both M and E in the Canadian federal scene has demonstrated is that the expectation for M, as a tool to measure ‘results,’ is likely overstated. To a large extent, ‘results’ are still not being measured by M, in spite of a significant investment and a concerted level of effort to develop monitoring schemes. This is a function, in part, of unrealistic expectations regarding the ability of M to deliver a cost-effective approach to measuring outcomes.
(ii) Working with the current M and E infrastructure, adjusting as needed: As noted above, unlike most previous experiences with deliverology, Canada offers a relatively mature M and E infrastructure, though it is not without its issues. The new Policy on Results is in theory addressing these, but this is an opportunity to find the right balance between ‘learning’ and ‘accountability’ in demanding and using results/M and E information.
(iii) Incorporating evaluative thinking and adaptive management into the measurement and use of results information: Associated with the above, more flexibility, rather than command and control, would allow an enabling environment for evidence-based learning and innovation, i.e. regularly using information on results to improve delivery.
(iv) Collaboration and coordination between PCO (RDU) and TBS (Results Policy) – Aligning and streamlining the measurement and reporting requirements: Both PCO and TBS have an interest in measuring ‘results.’ PCO is the central authority behind deliverology focusing on a few priorities of government, TBS has a broader scope as it rolls out its new Policy on Results. Both are imposing new requirements insofar as results measurement and reporting is concerned. To avoid burden on the system, ongoing collaboration and coordination between the two central agencies, as well as the occasional ‘health check’ on individual departments and the system in general, would help avoid the new schemes becoming overly burdensome.
(v) Addressing cross-boundary collaboration: Since government priorities likely involve several programs across several department (and, possibly different levels of government), accelerating cross-boundary collaboration could be an important entry point for deliverology. The Canadian experience of linking E to policy development (in departments and centrally) or horizontal initiatives of government has for the most part not been strong. The realities of results measurement typically find programs at different levels of maturity, M and E readiness, and data availability. These factors underline the importance of cooperation and coordination across several players.
(vi) Learning and adjusting as needed: Given the importance of implementation to the success of deliverology, a ‘formative’ type study would be useful in the short-term, offering advice on adjustments, as needed. In the medium-term, the system would benefit from a broader assessment of cost and effectiveness issues of deliverology, conducted by the Auditor General or Parliamentary Budget Officer, to ensure its objectivity and transparency.
While the above considerations will not guarantee success for the deliverology process – that would involve factors far broader than the issues of measurement and organization – they do represent a fundamental starting point in its implementation, and so working from the outset to getting it right ought to be paramount.
Robert Lahey is founding Head of Canada’s Centre of Excellence for Evaluation, the federal government’s policy centre for evaluation, and, over three decades had headed the evaluation function in four government departments and agencies in Canada. He is a member of the Canadian Evaluation Society’s (CES) Credentialing Board.
