The reduction or mitigation of risks or harms is a major priority of the government of Canada in matters related to Canadians’ health, safety, security and environmental protection and even economic ‘harms’ such as fraud. While the goals are simple, the situations are usually complex, featuring multiple actors, dynamic environments and highly variable circumstances.
One emerging approach to address this phenomenon is a systems performance methodology that assesses the strengths and weaknesses of the system in terms of key outcomes, assesses the key factors that influence success, and comes up with key lessons for continuous improvement.
In order to plan, measure and evaluate system performance, agencies such as the Canadian Food Inspection Agency have developed regulatory systems performance models which map out the logical relationships between the use of resources, activities, outputs, the reach of key systems actors within its sphere of influence, and a sequence of outcomes related to those actors.
From the regulatory systems model, an agency can develop key performance indicators to check whether policies, procedures and practices critical to risk or harm mitigation are successful in achieving the desired results. These indicators typically represent behaviours of key actors in the system as follows:
1. Internal activities (sphere of agency operational control)
The key concerns of this level are the consistency and quality of delivery, how well the program design is working and operational tactical planning and management. Some questions include the extent that:
• work plans are delivered as planned;
• service standards are met;
• inspection, investigation and initiative steps have been followed;
• operational and tactical information is complete and accurate;
• decisions are consistent;
• internal processes are productive and efficient; and
• the internally defined level of quality is met.
2. Stakeholder engagement (sphere of direct influence)
This level involves the engagement and supportive actions, compliance and commitment (through behaviors) of governing / regulatory bodies, marketplace actors such as members of the supply chain, and consumers, users and citizens. Some key questions at this level involve the extent to which all stakeholders are constructively engaged and their reactions, capacity, commitment, compliance and/or appropriate supportive actions to address the harm risk or threat.
3. Mission achievement (sphere of indirect influence)
This level concerns the overall safety, security, environmental integrity, economic or marketplace ‘state’. It is typically measured by overall levels of harms or negative consequences. The key question at this level relates to harm (or reduced harm) level as a consequence of system behaviours. The three levels can be scaled to tell a performance story of any level. Most important, the systems are related and inextricably linked.
Behaviours like “delivery according to plan” would be tracked at the operational level (Level 1) followed by the adoption of supportive activities and regulated party compliance (Level 2) and finally the tracking of the state of the harm or threat (Level 3) over time.
While monitoring and on-going measurement can assist with systems performance assessment and management, deeper oversight activities like evaluation can assist agencies to continuously improve their policies, programs and operations.
Realistic evaluation and contribution analysis approaches can be applied to initiatives at both program and policy levels to address the essential question of what works and why. They can address the core issues of need, performance and cost-effectiveness and can frame and test specific assumptions in the delivery of programs.
Some key ‘to do’ principles which have emerged with regard to planning, monitoring, evaluating and managing complex regulatory initiatives include actions to:
• develop a common lens and language across functions and levels;
• recognize key systems actors in different spheres of influence;
• focus on behaviors in the actors and the integration of those behaviors to achieve higher level results;
• recognize and track engagement and relationships as well as traditional ‘performance’ metrics; and
• promote integrated systems thinking in all corporate and line functions.
While some lessons are emerging, much more remains to be learned.
An upcoming PPX breakfast learning event planned for January 2014 will further explore the ideas expressed in this article. Check out www.ppx.ca
