Evaluation of the quality management system – Canadian Government Executive

NEWS

SEARCH

AccountabilityManagement
May 7, 2012

Evaluation of the quality management system

Have you ever asked yourself the question: “How would I evaluate the Quality Management System (QMS) in my organization?” The answer really depends on why you are doing the evaluation. To answer this you need to engage stakeholders, know something about evaluation and preferably a little about Quality Management.

A quality management system (QMS) can be expressed as the organizational structure, procedures, processes and resources needed to implement Quality Management or Business Excellence.The QMS description starts with a statement of need describing the problem or opportunity that the QMS addresses and the results expected. The linkage to organizational mission, vision and strategic outcomes should be identified. The description should discuss what the QMS must accomplish to be considered successful, that is, the vision and objectives of the QMS, and its capacity to effect change, its stage of development, and how it fits into the larger organizational priorities. Resource descriptions should include the people, money and technology used to conduct QMS activities.

As decisions in the evaluation will be based on the description of the QMS, it should identify the activities that the system does to effect change and should display them in a logical model.

Engaging stakeholders
The evaluation cycle begins by engaging the people having a vested interest in what will be learned from an evaluation and what will be done with the knowledge. These stakeholders include: those responsible for the functional management of the QMS; those involved in using the QMS to improve program delivery; (e.g., managers, employees, administrators, partners, suppliers); those served or affected by the QMS (e.g., employees, clients, citizens, the government, central agencies, advocacy groups); and, primary users of the evaluation (e.g., the specific persons who are in a position to act on recommendations regarding the program).

Purpose of evaluation
The purpose of the evaluation must be aligned with the issues of greatest concern to stakeholders. Stakeholders need to know what they will do as a result of the evaluation. For example, they may want to understand, verify or increase impact of the QMS on products or services to satisfy clients/citizens. Or they may want to improve delivery processes to be more efficient – and there are a host of other possible decisions that can be reached.

Audience(s) for the evaluation
The stakeholders will also decide who the audiences are for the information obtained from the evaluation. The audience could be made up of clients/citizens, senior management, the Board of Directors, Treasury Board Secretariat or internal employees.

Information needed for evaluation

After the first two decisions are made by stakeholders, the kinds of information needed support decision-making and/or inform the intended audiences can be determined. You may want information to understand: the product/ service delivery process (its inputs, activities and outputs); benefits to clients/citizens (results); is the organization spending more than necessary on its QMS and if yes, why?; and, are there opportunities for improving the Quality Management System design and delivery (cost-effectiveness)?

Type of evaluation
Based on the purpose of the evaluation and the kinds of information needed, the stakeholders can state what type of evaluation should be planned. It may be developmental – to identify areas for improvement early in the program, organizational excellence-based, or perhaps comprehensive – a mixture of several types.

Sources of information and evidence
Stakeholders should be involved in defining data that they would find pertinent, so that they will more likely accept the evaluation’s conclusions and recommendations. Evaluators should collect this information / evidence which are perceived by stakeholders as believable and relevant for answering their questions. Such decisions depend on the evaluation questions being posed and the reasons for asking them.  There are many potential sources of information, including staff/employees, clients/citizens, Management Accountability Framework (MAF) assessments, Program Evaluation and Auditor General Reports.

Tools for collecting information

It is essential that the information be collected in a manner that facilitates collection, manipulation and analysis. The techniques for gathering evidence in an evaluation must be aligned with the organizational culture and data-collection procedures should be examined carefully to ensure that confidentiality of information and sources are ensured. The evaluator will likely use a number of tools  such as a written survey or questionnaire (e.g. handout, telephone, fax, mail, e-mail, or Internet), observation of clients/citizens/managers/employees, conducting focus groups among various stakeholders and others, so that there can be corroboration and multiple approaches to enhance credibility.

Conclusions

Evaluation conclusions are warranted when they are supported by the evidence gathered and assessed against criteria or standards agreed to by the stakeholders. Before stakeholders will agree to the recommendations, they must believe that conclusions are sound. Justifying conclusions involves the following steps:
     1.    Analysis of an evaluation’s findings to determine important findings and synthesis of various sources of information to reach a larger understanding.
     2.    Interpretation of the findings to make sense of the evidence so as to appreciate the relevance of what has been learned.
     3.    Judgments of the QMS by comparing the findings and interpretations against agreed criteria and standards.
     4.    Recommendations for continuing, improving, expanding, or terminating the QMS are separate from judgments, but based on them.

Ensuring evaluation recommendations are implemented

Having initiated the evaluation with the stakeholders and then continued with their engagement throughout the evaluation will go a long way to ensure that the evaluation recommendations are implemented. The following are critical elements for ensuring implementation of the evaluation recommendations:
·    Consult with the stakeholders as necessary to ensure their understanding and acceptance of the findings, conclusions and recommendations.
·    Provide the responsible managers with advice/interpretations/information in developing their Action Plan to deal with the recommendations.
·    Present the Evaluation Report, and have the responsible managers present their Action Plan at the same time, to the “Evaluation Board” of senior managers and external people who provide an oversight to the management of the department’s programmes.
·    Ensure commitments in the Action Plan are incorporated, as appropriate, in the Strategic Plan, Business Plans and Performance Management Agreements.
·    Make the evaluation report public, in particular management’s action plan and commitment to address the recommendations.

John Thomas is president of John F. Thomas Management Consulting and a director of the Canadian Public Sector Quality Association (CPSQA). Don Wilson is the executive director for the National Capital Region for the National Quality Institute (NQI).

About this author

0 comments

There are no comments for this post yet.

Be the first to comment. Click here.

Accountability
 
In this special episode of CGE Radio, your host John Jones...
 
Security professionals have an obligation to communicate risks and recommendations to...
 
Risk is always present in any undertaking, no matter the size...
 
Cost overruns have become institutionalized in the federal government, according to...
 
Last year, procurement Minister Jody Foote was prompted by the swirling...
 
Professionals, managers, and executives in the cost estimation industry can gain...
 
In this episode, hear from Carl Hammersburg, Manager, Government and Healthcare...
 
A new study from the Conference Board of Canada gives our...
 
In the world that we are living in today, free and...
 
The delivery method developed by Sir Michael Barber, chief adviser to...
 
Rules and accountability are helpful in developing and standardizing processes but...
 
Canadian doctors were told that climate change impacts human health and...
 
Even as talks between the government and federal workers affected problems...
 
The largest effort in 20 years to seek public input on...
 
Ottawa has overhauled the process by which justices are picked for...
 
Please to view this Content. (Not a member? Join Today! )...
 
In this episode, editor-in-chief, Patrice Dutil talks about the need for...
 
Upon receiving numerous complaints regarding add-on fees that turn making economy...
 
Are you absolutely clear what the government wants to achieve? Are...
 
Please to view this Content. (Not a member? Join Today! )...
 
Please to view this Content. (Not a member? Join Today! )...
 
Please to view this Content. (Not a member? Join Today! )...
 
Please to view this Content. (Not a member? Join Today! )...
 
Please to view this Content. (Not a member? Join Today! )...
 
Written By Jason McNaught The Public Service Alliance of Canada was...
 
Independence has long been regarded as a cornerstone of the auditing...
 
Canada is a diverse nation, in language, culture, geography, and, ultimately,...
 
Public sector organizations are under increasing pressure to identify all risks...
 
The government of Canada has implemented several measures over the past...
 
Whether at the territorial, provincial or federal government level, internal audit...
 
An organization’s reputation can take years to build but it can...
 
On October 30, Ontario began second reading of the Public Sector...
 
In the well-known children’s story, an Emperor falls victim to the...
 
The Nova Scotia Office of the Ombudsman is a small operation...
 
I think the ombudsman needs to be independent, because without independence...
 
Today’s business environment changes rapidly to adjust to evolving conditions and...
 
The best internal auditors actually are really good managers first. I...
 
The recent controversy about the actions of some staff members in...
 
Most professionals don’t need more than a sentence at a cocktail...
 
Recent research by the Institute of Internal Auditors Canada aims to...
 
When is it that a politician becomes part of the governing...
 
It’s been a busy couple of weeks on the information, privacy...
 
In 1996, a new budget watchdog, the Parliamentary Budget Officer, was...
 
The Ontario government is moving forward with the creation of a...
 
The Office of the Public Sector Integrity Commissioner of Canada (PSIC)...
 
After the Auditor General’s (AG) report was released on April 30,...
 
In the U.K. system, Permanent Secretaries are what we call Deputy...
 
We are living in a period of rapid change and limited...
 
US public sector employees don’t trust their management to do the...
 
The news of Mark Carney’s nomination as the new Governor of...
 
Following Singapore’s independence in 1965, the controversial leadership of Lee Kuan...
 
Over the past few years, the preparation and delivery of the...
 
In healthcare, cost-cutting can result in cutting what is valued most...
 
For over 20 years Colin Bennett has been exploring issues of...
 
Even before controversy shook the organization to its foundation, Ornge was...
 
It will be the largest international multi-sport event ever held on...
 
We’ve all seen the headlines – BC Ferries, Ornge, la Caisse...
 
It can happen, and it’s noteworthy when it does. Government, business...
 
Kevin Page’s mandate as the first Parliamentary Budget Officer comes to...
 
Governments are challenged to meaningfully mitigate the effects of the financial...
 
Canada is facing a huge financial challenge brought on by massive...
 
For the past one hundred years, democratic states have been moving...
 
It’s so much easier and less painful to learn from the...
 
CGE Vol.13 No.7 September 2007 "If the Public Service, as a...
 
When pondering leadership, we immediately think of exercising our influence downward...
 
CGE Vol.13 No.1 January 2007 "How can I be held accountable...
 
CGE Vol.13 No.2 February 2007 Canada’s Performance 2006 is the sixth...
 
CGE Vol.14 No.1 January 2008 The furor over the $300,000 that...
 
CGE Vol.14 No.2 February 2008 Let’s say you’re a senior manager,...
 
CGE Vol.13 No.1 January 2007 Perhaps it’s a legacy of the...
 
The Independent Blue Ribbon Panel on Grants and Contributions called for...
 
Au Canada, le secteur bénévole et à but non lucratif vit...
 
As the global economy struggles to regain some forward momentum, Canadian...
 
This will be a defining budget for Stephen Harper. It will...
 
It is difficult to determine when the debate about the need...
 
For the next few years, the federal government’s overarching agenda will...
 
Much of the current conversation about the federal government’s economic agenda...
 
Bill Greenlaw is the elected president of the Institute for Public...
 
Have you ever asked yourself the question: ‘How would I evaluate...
 
Last fall, Alberta’s Employment and Immigration department posted online the workplace...
 
In 2006 the world was feeling the aftershocks of a number...
 
CGE Vol.13 No.4 April 2007 Robert Parkins, editorial director, met recently...
 
In the past two decades, the nature of the state has...
 
Please to view this Content. (Not a member? Join Today! )...
 
Some title Some author
Some excerpt
In this special episode of CGE Radio, your host John Jones...

Member Login

Forgot Password?

Join Us

Password Reset
Please enter your e-mail address. You will receive a new password via e-mail.