Canadian Government Executive - Volume 23 - Issue 1

January 2017 // Canadian Government Executive / 19 Program Evaluation of enhanced program and policy devel- opment, informed strategic formulation, stronger decision-making or improved public reporting. These changes should be more than instrumental. While some- times offering conceptual support for learning or knowledge exchange, evalu- ations often contribute more to symbolic use or ‘legitimatizing’ an organization’s system or practices. By their very nature, evaluations should transcend functional processes. They should lead beyond docu- menting ‘transactions’ to something that is more ‘transformational.’ How big is the evaluation system at the Federal level? Program evaluation capacity within the Government of Canada is significant. The Center of Excellence for Evaluation (CEE) of the Treasury Board of Canada Secretar- iat’s (TBS) Evaluation of the 2009 Policy on Evaluation estimates that substantial hu- man and financial resources are invested in evaluation. From 2007-08 to 2012-13 the number of evaluations completed across large departments and agencies ranged between over 120 to165 annually. The number of staff employed to administer the function annually was between 400 to 475 full-time equivalents. When salaries, operations and maintenance, professional services and other costs were combined, the total federal financial expenditure for evaluation ranged between $55 million to just under $67 million annually. Over a six-year period, no fewer than 825 evalu- ations were completed at a price tag well over $300 million. If one extrapolates the number of recommendations connected with this ample repository of studies, con- servatively assuming 5 recommendations per evaluation, one could easily arrive at over 4,000 recommendations. An equal number of management response actions, if not double the number, would accom- pany these prescriptions. It leads to a key question, ‘so what?’ Are there opposing views? Donald J Savoie, Canada Research Chair in Public Administration and Governance at the University of Moncton, argues in his 2015 book What is Government Good At? that program evaluation is not one of the contenders. He contends that govern- ment is not good at evaluating activities (above the ‘fault line’), and that evalua- tion takes up an inordinate amount of time and resources. ‘Central agencies and many units in line departments in Ot- tawa must know that a good number of program evaluations are of little value to anyone, yet at a cost to the taxpayers’ (p. 182). He maintains that public spending on evaluation has contributed precious little other than contributing to the ‘fat’ of government. Program evaluation units in departments are of little help and have limited ability to assess the success of pol- icies and programs (p. 238). At the same time, the Federal Govern- ment (TBS-CEE) offers a different per- spective. It has examined the quality of evaluation and associated management response action plans. Until recently, an ‘acceptable’ designation was given to evaluation reports mostly completed by internal evaluators, established on criteria linked to the 2009 Policy and Standards on Evaluation. Based on a sample of 138 reports submitted for 2013-2015, TBS-CEE found that a majority of evaluation re- ports, completed by large departments and agencies, scored an ‘acceptable’ rat- ing for evaluation recommendations and for linked action plans. The proportion of reports for which the recommendations were deemed ‘acceptable’ increased from 86% in 2013-14 to 91% in 2014-15. So it would seem that program evaluation outputs were meeting some minimum measure of quality and adequacy. Federal departments were also asked to report on their tracking systems. Who was responsible for monitoring follow-up to evaluation recommendations? What was the status of implementation of the action items? The proportion of assessed reports, for which the action plans were deemed acceptable, remained relatively unchanged at 81 per cent. With respect to average implementation rates, only half of the management action plans were categorized as fully implemented. The By their very nature, evaluations should transcend functional processes. They should lead beyond documenting ‘transactions’ to something that is more ‘transformational.’