GovernmentLeadershipPublic Sector
March 28, 2016

Improving the Utilization of Evaluation Reports by Focusing on Clients’ Needs

It is no secret that evaluation reports in the federal public service are not being utilized to the extent that they should to support decision-making. This can be explained in part by the requirements of the 2009 Policy on Evaluation, which stipulates that all direct program spending must be evaluated every five years. The very fact that each program must be evaluated in that five-year span might not coincide with the actual needs of senior management. With the Policy currently being revised, there may be more flexibility to undertake more strategic evaluations. Evaluations that are better aligned with management’s needs will not only serve as a useful tool to support decision-making, but they will also help address any concerns, priorities, or challenges that senior management might have.

Evaluators gain a great deal by choosing evaluation projects that are highly significant to their Department, notably by factoring in departmental and senior management priorities. To accomplish this task, evaluators must ask themselves first “what keeps senior managers up at night?”.

An innovative Approach: Using the Audit Process for Evaluation Planning

The Evaluation Division at Infrastructure Canada recently adopted an innovative approach, similar to planning approaches used in some other departments and agencies, to improve its planning process. Instead of relying only on a more traditional planning approach based on the development of a program inventory, our evaluation team embarked upon a special journey with the audit team, by participating in a joint four-phased risk-based process. What we were hoping to gain out of this unprecedented collaboration was a better plan that would be more useful to management and would help mitigate what keeps them up at night.

The first phase of the risk-based process focused on defining the universe of entities that could be evaluated. Defining our preliminary evaluation universe only required us to review the departmental program inventory. But we wanted to expand this universe of possibilities and we could only do it by learning more about our clients’ needs. This was made possible during phase two, where we jointly held consultations with several levels of management within the Department, from program managers to the Deputy Minister, and various counterparts in other departments. These consultations enabled us to better understand their top priorities, concerns, challenges, areas of risks, as well as their appetite in terms of future audits and evaluations reports. The valuable information obtained through the consultations was applied in phase three: the risk assessment, during which we created an expanded list of potential evaluable entities. In order to measure the level of risk of each entity, we developed a total of six risk criteria: Materiality, Previous Evaluations and Audits, Public Sensitivity, Recent Changes, Complexity, and Senior Management Interest. Each evaluable entity was defined against all of these criteria. The fourth and last phase was the identification of potential evaluation projects. In this phase, we took a number of essential factors in consideration: the program inventory, the highest priorities and concerns of senior management, the highest level of risk and the Policy requirements. The outcome was a list of potential evaluation engagements, in the form of a five-year calendar. We also added a rationale and a strategic evaluation approach for each project, to justify the reason behind each choice. We then followed up with senior management to present and validate the calendar of evaluation projects. This resulted in our new evaluation engagements.

The joint participation of the Evaluation and the Audit teams in this process also resulted in another product: the very first combined Risk-Based Audit and Evaluation Plan in the Department.

Success and Lessons learned

What contributed to the real success of this collaborative approach were the joint consultations held with multiple levels of management. By only contacting clients once, we were able to significantly reduce the burden on them while still obtaining strategic and valuable information. Also, by taking into account the clients’ true needs, challenges and risks, we ended up creating a list of evaluation engagements that are better aligned with current priorities, and thus, that have a higher chance of being more useful to decision-makers. As for the lessons learned, moving forward, extra time should be allocated right after the consultations so that the evaluation team can better define the preliminary objectives and scope of its next evaluation engagements.

 

 

Marie-Josée Courchesne is the Manager of Evaluation Services at Infrastructure Canada. She also completed a Master’s Degree in Public Administration specialized in Program Evaluation at the École Nationale d’Administration Publique (ENAP) in Gatineau. mjcourchesne@hotmail.com
Linda Vertefeuille is currently working as an Evaluation Analyst at Infrastructure Canada. She has recently obtained her Master’s Degree in Public Administration specialized in Program Evaluation at the École Nationale d’Administration Publique (ENAP) in Gatineau. vertefeuillelind@hotmail.com

0 comments

There are no comments for this post yet.

Be the first to comment. Click here.

Government
 
Digital is a big component of government services today. The drive...
 
What is transformation going to look like? That’s the question on...
 
We are excited to share with you the August/September 2019 issue...
 
That hypnotically red eye. That eerily meticulous, methodical, and measured voice....
 
What’s your organisation’s mission? Hopefully, you know what it does on...
 
As I’ve mentioned in previous articles, the Government needs to move...
 
The majestic sequoia is a species of redwoods that is among...
 
Thanks to the development of new technologies that make previously unimaginable...
 
Throughout the history of economic thought, government has long been seen...
 
The Ontario Public Service (OPS) is at a crossroads. The 2018...
 
What do low confidence in institutions and dissatisfaction with leadership have...
 
We live in a time of economic turbulence that is quickly...
 
In this edition of Canadian Government Executive, we take a look...
 
In June of last year, I returned to my roots. After...
 
We live in the age of paradox.  Never before in human...
 
In this episode, Lori Turnbull and J. Richard Jones talk about...
 
The evidence that design reaps multiple benefits for service delivery  is growing ....
 
In the next year, one in three US federal government workers...
 
Summary and Recommendations Labour markets in Canada and around the world...
 
Despite a rapid expansion in digital democracy initiatives around the globe,...
 
In the show, J. Richard Jones talks with Colin Earp, Partner...
 
There’s no doubt that digital transformation is a crucial topic, not...
 
Impact investment has taken Canada by storm, opening the doors to...
 
The discussion around where technology is going in government tends to...
 
In recent years, growing evidence about the importance of early childhood...
 
As the labour market shifts from full-time work toward short-term and...
 
We are excited to announce that the February/March 2019 issue of...
 
The use of “nudges” in policymaking has been a major trend...
 
In this episode, Lori Turnbull discusses the positive side of disruption,...
 
There is an interesting paradox playing out with the public sector...
 
In recent years there has been much hype surrounding blockchain technologies...
 
We all know that innovation involves taking risk and accepting the...
 
February 20, 2019, Aurora, ON – Canadian Government Executive Media (CGE)...
 
Canadian Government Executive, in collaboration with Dalhousie University, is seeking your...
 
Wouldn’t we all love to work in an open, transparent and...
 
Barring a massive emergency, on December 7,h 2018, Prime Minister Justin...
 
In November 2015, Prime Minister Justin Trudeau formed the first gender-balanced...
 
After the general federal election in October 2015, Prime Minister Justin...
 
In my previous post about Nesta’s  Digital Frontrunners , I introduced four challenges ...
 
While most of my focus here is on IT and digitization...
 
In his article, Gaming the System, How the misuse of data impedes...
 
Over the years, governments have been adopting Lean management principles to...
 
Canada is governed by multiple levels of governme­nt, which is evident...
 
We are excited to announce that the Dec/Jan issue of CGE...
 
In federal systems like Canada’s – where power, authority, and responsibility...
 
When James Cattell, a delivery manager at the UK’s Department for...
 
Kotter International found that 90 per cent of managers and employees...
 
It is time to reboot democracy. This year, the French constitution,...
 
The role of internal audit in the public sector has changed...
 
Welcome to the fall edition of CGE. Since our last issue,...
 
We are excited to share with you the October/November 2018 edition...
 
The recent 2018 Senate Report examining the Government of Canada’s Phoenix...
 
Why do the richest 1 per cent of Americans take 20...
 
George Ross, editor in chief of Canadian Government Executive spoke with...
 
Usually, the summer period is a relatively slow time for public...
 
The Québec government has initiated a historic engagement process. The 2017...
 
A healthy Canadian economy and opportunities outside of North America are...
 
We are excited to share with you the May/June issue of...
 
The UK Department for Education (DfE) doesn’t have an easy mandate:...
 
With the launch of the Canada Infrastructure Bank and US Congressional interest in Trump’s...
 
Ask a public servant if they know about “ deliverology ” and...
 
In this era of truth and reconciliation, what are the principles...
 
Innovation labs and units have become so fashionable in the public...
 
Can government remember? Is it condemned to repeat mistakes? Or does...
 
Over the past two decades, codes of ethical conduct have become...
 
In this episode, J. Richard Jones sits down with Martin Joyce,...
 
In this episode, J. Richard Jones, publisher of Canadian Government Executive...
 
The January/February 2018 issue of Canadian Government Executive is on the...
 
This year is proving to be an interesting one for all...
 
Recently, George Ross, Editor-in-Chief of Candian Government Executive sat down with...
 
As Executive Director of the UK’s Government Digital Service, Mike Bracken...
 
For over two days at the end of January this year,...
 
Prime Minister Justin Trudeau has, on many occasions, expressed his commitment...
 
Today, many connect with their banks, private companies, and friends in...
 
The Trudeau Government’s decision, in August 2017, to split the Indigenous...
 
In the last few years, we’ve seen various federal governments warning...
 
Rankings of public sector entities has been big trend for quite...
 
For over 20 years, Canadian Government Executive (CGE) has been a...
 
The health care system in Canada, known as Medicare, is publicly...
 
Do Labour Market Development Agreements (LMDAs) make a difference? Our evaluation...
 
Evaluation is performed for a range of reasons:  to improve programs,...
 
Radical candour sounds rather outré as a prescription for government executives....
 
Behavioural Insights (BI) is a multidisciplinary approach that uses principles and...
 
In March, the Canada Revenue Agency’s website was shut down by...
 
In 2016, Deloitte undertook a global internal audit research study on...
 
One of the writers in this month’s issue started a note...
 
During the general election in 2015, Justin Trudeau, the leader of...
 
So, what can be said about Canada’s public administration as the...
 
The way forward to engaging Millennials in the public sector Passionate,...
 
As the Liberal’s assumed office, three big democratic reform ideas were...
 
The International Monetary Fund (IMF) in issuing its annual review of...
 
In today’s world of heightened focus on good governance and accountability...
 
At a recent meeting, a colleague asked me how I became...
 
As ‘month 13’ for Canada’s Syrian refugee arrivals comes and goes,...
 
As governments around the world move towards the digital transformation of...
 
Cost estimation is becoming an extremely important skill within government due...
 
The Senate has long been controversial largely because it is unelected...
 
In this special episode of CGE Radio, your host John Jones...
 
In this episode J. Richard Jones examines the rise in international...
 
Currently, there are ten organizations at the federal level that function...
 
Imagine a Canadian government that shares best practices in service delivery....
 
Risk is always present in any undertaking, no matter the size...
 
There is a new buzz in policy circles. Advances in data...
 
The North Atlantic Treaty Organization, with its 28 independent member countries...
 
The internal audit function in the Canadian public sector finds itself...
 
As Canada’s government agencies strive to improve delivery and efficiency under...
 
While Canadians grapple with the meaning of the Trump presidency and...
 
In recent months, Canadians have borne witness to a new cycle...
 
In 2015, the Department of Innovation, Science and Economic Development (ISED)...
 
The year 2016 saw two critical words circling the halls of...
 
The critical challenge facing the public service is changing its culture....
 
In this episode, hear from Carl Hammersburg, Manager, Government and Healthcare...
 
As understanding of social behaviour increases through intensified research, many government...
 
Sustainable Development Goals (SDGs) are part of the new global “Agenda...
 
The Trudeau government was swept into office, in part, by promising...
 
An Interview with Larry Ostola, Toronto’s Director of Museums and Heritage...
 
From a policy perspective, it is a challenge to predict how...
 
The arrival of President Trump in the White House marks a...
 
In the first year of Prime Minister Justin Trudeau’s term in...
 
For all its subtleties and mysteries, the Westminster system of government...
 
While the incoming administration of President-elect Donald Trump is being buffeted...
 
Mental health problems are increasingly recognized today. By some calculations, one...
 
Faithful readers will remember that the cover story on last month’s...
 
Rehabilitating the West Block in considered the most complex restoration project...
 
Defending his government’s use of targets to drive UK public-sector reform...
 
In the world that we are living in today, free and...
 
Earlier this year the Ontario Ministry of Health announced that it...
 
While government watchers tend to concentrate on the big policy issues,...
 
The RCMP adopted a new media strategy earlier this month by...
 
Several factors are driving an increased interest in horizontal assurance in...
 
There is already a surging literature on the application of design...
 
In 2001, the OECD published The Hidden Threat to E-Government. The...
 
At the end of August, when most Canadians were enjoying their...
 
The Conference Board of Canada recently published a report titled Leveraging...
 
The Government of Canada’s newly announced Policy on Results replaces the...
 
While internal auditing has been considerably strengthened in most jurisdictions in...
 
The results-driven work ethics of Michael Barber, a top British political...
 
As team Trudeau would tell it, this past summer saw spirited...
 
The Trudeau Government has been off and running for almost one...
 
Cyber-attacks threaten Canadian individuals and companies every minute of every hour....
 
Today’s transformative movement towards a more digital and connected government is...
 
Deliverology is fast becoming a buzzword in the Canadian government scene....
 
For several months now, the Liberal government ministers have been using...
 
Ontario’s provincial government needs to act fast in building a coordinated...
 
The bootleg fentanyl overdose crisis that is sweeping across Western provinces...
 
Terrorism operates with deadly regularity. In June 2016, a gunman who...
 
It’s a common notion that young workers born in the mid-1990s...
 
BC Hydro said it is pushing through with it $9-billion, hydroelectric megaproject on...
 
Canadian doctors were told that climate change impacts human health and...
 
Even as talks between the government and federal workers affected problems...
 
At last the the nostalgic fun of the CNE decends upon...
 
Just as the federal government has begun consultations on cyber security,...
 
The association representing more than 42,000 physicians and medical students in...
 
Contrary to the stereotype of eagerness and politesse, the second Bold...
 
Yesterday, Federal and Atlantic Provincial Ministers of the environment met to...
 
Following three months of little change in the employment numbers, Statistics...
 
Regardless of whether you are an elected official, a public servant,...
 
Ottawa has overhauled the process by which justices are picked for...
 
July 27 was pay day some federal public workers that finally...
 
A recent report suggests several strategies how governments and the private...
 
After writing the book “Megatrends: The Impact of Infrastructure on Ontario’s...
 
Though many of Canada’s immigrants have above-average education, they often find...
 
Fred Vettese, Morneau Shepell’s Chief Actuary, has released a report titled...
 
The Phoenix payroll systems long-running technical glitches which have plagued public...
 
The local electric vehicle industry just got a boost from Ontario...
 
Signaling a realignment of Canada’s involvement with NATO, Prime Minister Justin...
 
The fact that Shared Services Canada (SSC) has struggled mightily under...
 
Prime Minister Justin Trudeau today announced the deployment of Canadian troops,...
 
The union representing Canadian postal workers has rejected a proposal from...
 
A small but rapidly growing number of businesses have started hiring...
 
The possibility of mail delivery disruption on Friday this week loomed...
 
The Senate committee looking into Canada’s Syrian refugee program wants the...
 
United States President Barack Obama, speaking before Parliament last night, urged...
 
Prime Minister Justin Trudeau said making it easier for goods and...
 
Canada Post today warned the public about the potential for a...
 
Even as British stocks took a plunge and the pound sterling...
 
Yes, according to the former head of the Canadian Security Intelligence...
 
Early this morning, Philippine police confirmed that the severed head found...
 
Computer software company Adobe, has migrated more than 11 million pages...
 
The recent retirement of seven federal deputy ministers (DMs) reminds us...
 
The Public Health Agency of Canada has long been aware of...
 
On June 7, over 40 senior executives from within the public...
 
All’s fair in love and war JOHN LILY (1578): EUPHUES: THE...
 
Whether it is convincing companies to build R&D centres, factories, or...
 
The fact that Shared Services Canada (SSC) has struggled mightily under...
 
The government has to move quickly in replacing its fleet of...
 
As much a 20 per cent of grade seven students in...
 
The Liberal government is taking steps to rein in the National...
 
Canada is severely in need of a maritime vessel that can...
 
The actions and decisions of public servants have consequences for the...
 
A Quebec superior court judge facing allegations that he helped commit...
 
He is often referred to as the federal government’s first “chief...
 
We often read news articles about rampant drug addiction and suicide...
 
Upon receiving numerous complaints regarding add-on fees that turn making economy...
 
Public sector design thinking has evolved from obscurity to something of...
 
The procurement group within the Government of Canada is undergoing a...
 
Government agencies, international businesses, as well as, European organizations that comply...
 
Canada, today, became a full supporter of the United Nations Declaration...
 
People caught a glimpse of the work of partisan advisers in...
 
Officials and first responders battling the raging wildfires in Fort McMurray...
 
Veterans Affairs Canada is not adequately managing the drug component of...
 
Whether responding to emergencies back home or assisting regular troops abroad,...
 
The Trudeau government is now six months into its mandate and...
 
Some public servants will have to request their departments for emergency...
 
The Royal Canadian Mounted Police (RCMP) is poised to launch an...
 
The murder of a Canadian citizen abroad and the uncertainty of...
 
Canada’s largest organization advocating for retirees from federal, police and military...
 
The government is spending $3.5 million to improve an existing immunization...
 
It almost goes without saying that good governance requires fair and...
 
The adoption last year by the Canada Border Services Agency of...
 
A research project at that explores the design and feasibility of...
 
One of the most important powers at the disposal of Canadian...
 
Two years ago, Bixi, a not-for-profit, para-municipal bicycle-share firm of the...
 
A landmark Supreme Court ruling has paved the way for some...
 
One of Canada’s largest integrated oil companies said it is not...
 
Following the various mandate letters from the Trudeau Administration, the Minister...
 
Conservation efforts in the Canadian Arctic will be obstructed and sensitive...
 
The Canadian Radio-television and Telecommunications Commission (CRTC) today begins a public...
 
Local job figures blew away expectations as records indicated that 41,000...
 
In dealing with the impact on the sharing economy on transportation,...
 
In this episode, Richard talks with Alex Willner, Assistant Professor from...
 
The melting Arctic ice caused by climate change is opening up...
 
At least one prime minister has resigned, another in under fire,...
 
Associates of Russian President Vladimir Putin, the king of Saudi Arabia,...
 
Now more than ever, organizations in both the public and private...
 
Things just get curiouser and curiouser. Lewis Carroll (1865): ’s Adventures...
 
Laid off workers in Canada will soon get some relief with...
 
The Federal Bureau of Investigation announced that it has managed to...
 
The Prime Minister is now well into the consolidation phase of...
 
It is no secret that evaluation reports in the federal public...
 
  The pressure is always on to make government services to...
 
The Department of Innovation, Science and Economic Development (ISED) Lab is...
 
The Liberal government’s first federal budget laid out $11.9 billion over...
 
Eight years ago, the always prolific University of Chicago Law School...
 
Canadians expect their public services to be delivered in a way...
 
The Liberal government is expected to announce on Tuesday a new...
 
The Trudeau Liberal platform of instituting “delivery” capabilities has garnered considerable...
 
The ambitious immigration plan of the Liberal government promises to zero...
 
Nearly half of Canadian organizations are falling behind on implementation of...
 
If the purpose of the Government of Canada’s performance measurement system...
 
The Ontario government and telecom gear maker Huawei Canada yesterday announced...
 
The facility of a top-secret military communications group in Borden, Ontario...
 
Q: Where is Comptrollership going? Comptrollership in the Government of Canada...
 
Following days of stalled negotiations and with the clock ticking on...
 
Last year, the Canada Revenue Agency rolled out a pilot program...
 
Monday was the final day for Canadians to donate money to...
 
A controversial investment protection clause in the Canada-European Union trade deal...
 
When the Liberals announce their first ever budget on March 22...
 
Prime Minister Trudeau is off on one of the most intense...
 
With the inexorable expansion of internet, and the advent of highly...
Digital is a big component of government services today. The drive...