Archived - Evaluation of the Economic Studies and Policy Analysis Division
Archived information is provided for reference, research or recordkeeping purposes. It is not subject to the Government of Canada Web Standards and has not been altered or updated since it was archived. Please contact us to request a format other than those available.
Internal Audit and Evaluation
Department of Finance Canada
Approved by the Deputy Minister of Finance on the recommendation of
Audit and Evaluation Committee on January 4, 2012
Table of Contents
This report presents the results of the pilot evaluation of the Economic Studies and Policy Analysis Division (ESPAD) of the Department of Finance Canada. The evaluation was undertaken by Internal Audit and Evaluation between May 2010 and June 2011. The evaluation study covers approximately $1.5 million in annual direct program spending and focuses primarily on the past five years of ESPAD’s operations.
ESPAD is the central policy research group in the Department and one of the most significant in the federal government as it initiates and coordinates research on key elements of the government’s medium-term economic agenda. ESPAD has a broad mandate to conduct economic research that is in-depth and policy-relevant to inform the Minister of Finance on issues that are of medium- and long-term significance to the Canadian economy. It examines and evaluates the medium-term economic challenges and risks facing Canada, and assesses and evaluates various policy options and proposals using complex economic models.
Evaluations of policy functions have rarely been conducted in the context of Canadian or other governments. Consequently, this evaluation is considered a pilot study. One of the key challenges of this type of evaluation is the difficulty in making a direct causal link between policy research, analysis and advice, on the one hand, and policy decisions, on the other, because these decisions are based on multiple factors. The methodology piloted for this type of evaluation sought to assess, to the extent possible, the quality and usefulness of ESPAD’s policy research outputs for departmental senior management.
The evaluation relied on four lines of evidence: file and document review; key informant interviews (with ESPAD staff, clients within the Department and external stakeholders); case studies; and a service survey. The following are the key findings and conclusions of the evaluation.
The evaluation found that ESPAD’s objectives and activities continue to be relevant and aligned with government and departmental priorities and roles and responsibilities, and that there is evidence of ongoing need for this in-house economic research function. ESPAD’s operational activities, including the conduct and communication of policy-relevant economic research and the ability to challenge the research findings and analysis of others, were considered essential functions required within the Department by virtually all groups of stakeholders who participated in the evaluation.
The evaluation found that ESPAD is highly respected throughout the Department of Finance Canada and by other government departments in terms of its research capacity and the quality of its work. ESPAD’s organizational structure, offering a collaborative work environment and a series of processes for verifying the methodological rigour of each research project, was found to support the generation of high-quality research. Its research and analysis outputs were found to be valued within the Department and useful for informing departmental senior management and policy decision makers.
ESPAD’s roles and responsibilities were found to have changed over the last five years. It is now better integrated into the Department, and this has led to some changes in its activities and outputs. There has been greater demand for ESPAD’s research expertise by both senior management and managers from other branches. More time is being spent on the development and maintenance of complex economic models that are necessary for quick answers to current policy questions, and outputs are increasingly communicated through vehicles that are easy to use for policy makers, such as presentations and short non-technical notes. There has been a commensurate decline in time spent producing research on economic issues intended for a more general audience and an associated decline in working paper publications and in ESPAD’s active participation in economic conferences.
The evaluation found that better departmental integration of ESPAD has benefited the Department by making ESPAD’s work more responsive to its needs. It also found that a balance in the conduct of policy-relevant research is beneficial over the long term. Continued work on current policy questions allows ESPAD to stay connected to current departmental issues, which in turn can help inform longer-term research projects that are more policy relevant. At the same time, maintaining in-depth research on general economic policy issues and building modeling and analytic capacity remains important; otherwise, management runs the risk of depleting the analytical and research capacity that ESPAD has built over the last 20 years. This capacity is the basis upon which it is able to provide timely answers to current policy questions.
The changes in roles and responsibilities over the last five years were found to have led to a difference of understanding within ESPAD regarding the expected responsibilities of research staff. Reviewing and reconciling ESPAD’s mission and activities with the outputs and balance of research work required by the Department will promote the right fit for ESPAD researchers.
ESPAD’s integration within the Department could be reinforced further with a more integrated research agenda based on departmental needs and priorities. Input into ESPAD’s annual research plan from departmental senior management and other departmental branches was found to be limited. Collaboration with other departments was generally found to be effective, though often described as ad hoc. The evaluation found that there is an interest among research groups within government for better communication of research plans and findings.
One of the most expensive activities in research divisions is modeling. The evaluation found that ESPAD’s costs are comparable or relatively lower than those of other departments. In recent years, ESPAD was found to be producing more research and analysis outputs demanded by the Department using existing resource levels.
Based on these findings, the evaluation recommends the following:
Recommendation 1: To enable ESPAD researchers to keep abreast of cutting-edge research, to maintain strong links with the external research community and to preserve the Department’s visible role as a leader in economic policy research, the Director of ESPAD should, within the financial limits of the division, ensure a minimum participation of ESPAD staff in relevant economic conferences and that information gained is disseminated to team members.
Recommendation 2: A review of ESPAD’s mandate and mission statement should be conducted by the Director of ESPAD to ensure it is aligned with ESPAD’s current work activities and the mandate of the Department. ESPAD’s management should clearly communicate this mandate to all employees, including potential and new employees, to promote realistic work expectations.
Recommendation 3: To increase the relevance of ESPAD’s research, the Director of ESPAD should put in place a systematic process for a) annual sharing and input from senior management across the Department into ESPAD’s annual research work plan; and b) communication of research plans with other governmental organizations to facilitate more effective horizontal collaboration.
This report presents the results of the pilot evaluation of the Economic Studies and Policy Analysis Division (ESPAD) of the Department of Finance Canada. The evaluation was undertaken by Internal Audit and Evaluation between May 2010 and June 2011.
The authority to conduct the evaluation is based on approval of the 2010–15 Departmental Evaluation Plan by the Deputy Minister on December 8, 2009. The Plan was developed in accordance with the 2009 Treasury Board Policy on Evaluation, which requires departments to evaluate 100 per cent of their direct program spending, including operations such as the policy functions. This is the first evaluation of a policy function in the Department; as such, it is considered to be a pilot study.
The evaluation risk for ESPAD was assessed as medium, primarily because no previous audit or evaluation exists and the work itself was assessed to be of medium public sensitivity. The evaluation was scheduled early in the Plan on the advice of departmental senior management. ESPAD’s focused mandate made it a suitable choice for a pilot study. Furthermore, it is not as heavily implicated in the annual budget process as other policy groups in the Department, allowing for some flexibility in timing various aspects of the evaluation.
ESPAD is one of three divisions of the Department’s Economic and Fiscal Policy Branch. This branch analyzes Canada’s economic and fiscal situation and outlook, and provides the Minister of Finance with analytical support on a wide range of economic and fiscal issues.
Mission statement and key activities
ESPAD represents the central and largest policy research group in the Department. It has a broad mandate to conduct economic research that is in-depth and policy-relevant to inform the Minister on issues that are of medium- to long-term significance (see the Department of Finance Canada website). According to ESPAD’s mission statement, its main objectives are excellence in conducting policy-relevant research to inform policy decisions; excellence in communicating policy-relevant research to senior management via user-friendly vehicles; and maintaining ongoing contact with the research community to share research findings with other researchers in the academic world and in government to help stimulate evidence-based public policy debate. ESPAD’s key activities include the following:
- Developing and maintaining several economic models;
- Examining and evaluating the medium-term economic challenges and risks facing Canada;
- Initiating and coordinating the development of the government’s medium-term economic agenda; and
- Evaluating various policy options and the economic proposals of other government departments, supporting the challenge function exercised in several branches of the Department.
ESPAD is also the main liaison between the Government of Canada and the Organisation for Economic Co-operation and Development (OECD), representing Canada’s economic interests in OECD economic meetings and facilitating the preparation of Canada’s Economic Survey conducted by the OECD. ESPAD is also involved in several policy-related committees, many of which are horizontal across several government departments.
Organizational structure and resources
ESPAD comprises three sections that conduct research on a given set of issues:
- The Macroeconomic Analysis team assesses and evaluates macroeconomic policy issues (e.g., fiscal stimulus, global imbalances, the inflation-targeting framework, macroeconomic risks and population aging).
- The Structural Analysis team assesses and evaluates social and labour market policy issues (e.g., the employment insurance system, barriers to labour mobility, the pension system, education, and tax and transfer measures).
- The Policy Analysis team focuses on the remaining economic areas and the coordination of the medium-term economic agenda (e.g., productivity, regulation and climate change).
There were 18 people working at ESPAD when the evaluation study began in May 2010. According to 2009–10 budget estimates, ESPAD had an annual operating budget of approximately $1.5 million.
The purpose of conducting this evaluation was to provide departmental senior management with an objective, evidence-based assessment of the relevance and performance (i.e., effectiveness, efficiency and economy) of ESPAD’s function and to identify opportunities for improvement.
The scope of the evaluation included all aspects of ESPAD’s activities. The evaluation covered approximately $1.5 million in annual direct program spending, and the analysis focused primarily on the past five years of ESPAD’s operations.
Evaluations of policy functions have rarely been conducted in the Canadian government or in other governments. However, the policy function, which includes policy research, analysis and advice, is an integral part of the Department of Finance Canada, with a significant portion of operating resources committed to its maintenance and support. Internal Audit and Evaluation conducted a thorough review of existing international and domestic practices, and engaged external experts to develop an approach to evaluating the policy function that would yield informative and useful results.
A logic model illustrating a results chain for ESPAD was developed for the evaluation (see Figure 1). In a typical program evaluation, the logic model is based on an underlying program theory of how a program’s activities are expected to lead to a desired outcome(s). However, in this evaluation, making a direct causal link between ESPAD’s policy-focused research outputs (e.g., models and simulations, working papers, analytical notes, briefings and presentations) and policy function outcomes, such as real changes in policy, poses several challenges. Research, analysis and advice provided by policy functions may inform or influence policy decisions, but these decisions ultimately rest with elected officials. Even policy research, analysis and advice of the highest quality may not be clearly or immediately reflected in policy decisions. This presents an attribution problem whereby policy decisions cannot be directly attributed to the policy function. Furthermore, the objective of policy functions in government, and particularly of policy research, is to inform and contribute to policy development, rather than to advance specific policies.
Figure 1. Logic Model
Consequently, the methodology developed for this evaluation examined the quality and usefulness of ESPAD’s policy-focused research outputs and the appropriateness or adequacy of ESPAD’s processes and resources for producing the research. The approach taken aimed to assess achievement of the immediate outcome of whether senior government officials have access to policy-focused research and analysis that is evidence-based, timely and of high quality. Intermediate outcomes, specifically ESPAD’s influence on policy development, were assessed to some extent by examining the extent to which ESPAD’s work is used and the extent to which it is valued. This approach is premised on the argument that if ESPAD’s work is relevant, useful and of high quality, then it is reasonable to suggest that it contributes to the achievement of departmental outcomes.
The quality assessment undertaken in the evaluation was based on criteria identified by Schacter (see Table 1) for high-quality policy advice. The evaluation approach also incorporated elements of organizational assessment inspired by the balanced scorecard methodology, assessing ESPAD’s relations with its clients and stakeholders, the appropriateness of its organizational structure and internal processes, and the level of its financial and human resources. This methodology was developed based on the premise that for an organization to achieve its goals cost-effectively, the needs of clients and stakeholders should be well understood and employees should have the required skills and resources to implement appropriate processes to meet client and stakeholder needs.
While the risk level for this evaluation was assessed as medium, considerable effort was invested in planning the evaluation since it was a pilot project in a new area—the evaluation of policy research and advice. Implementation of the evaluation approach, including choice of methods and scope of data collection (e.g., the number of interviewees, documents reviewed and cases studied) was determined in accordance with the level of risk.
|Is it timely?|
|Is it based on adequate consultations?|
|Does it articulate a clear purpose?|
|Is it logical, differentiating facts from assumptions and linking conclusions to recommendations?|
|Is it based on sound evidence?|
|Is it balanced, and does it include multiple viewpoints?|
|Does it present viable options for action?|
|Is it relevant to the actual situation, and does it anticipate related developments?|
|Is it well-organized and well-presented to the reader?|
The evaluation issues assessed are outlined in Table 2 and are in line with the core evaluation issues required by the 2009 Treasury Board Directive on the Evaluation Function. Each evaluation question is linked to the data sources expected to provide evidence, which are described in detail in section 3.3. All issues were examined with equal weight for this pilot evaluation of the policy function.
|Line of Evidence|
|Evaluation Issues||Document Review||Interviews||Case Studies||Service Survey|
| 1. To what extent are ESPAD’s objectives and activities still
relevant and consistent with the priorities and roles and
responsibilities of the federal government?
2. a) To what extent does the existing organizational structure
b) To what extent is the working environment and culture
c) To what extent are management and human resources
3. a) What structures, mechanisms and processes are in place
b) What process and mechanisms are in place to capture and
| 4. To what extent has ESPAD been able to meet its objectives and
commitments during the last five years? In particular, in what
ways and to what extent has it been able to inform policy
decisions of clients in and outside the Department?
|Performance—Efficiency and Economy|
5. a) To what extent have ESPAD’s mandate and roles and
b) Are ESPAD’s outputs commensurate with the level of inputs?
| 6. To what extent has ESPAD achieved its objectives efficiently
and economically? Are the costs of its inputs reasonable, relative
to the market costs?
A combination of quantitative and qualitative approaches was used in this evaluation. Quantitative methods included a service survey on the relative importance of, and satisfaction with, ESPAD’s outputs and financial, FTE and conference participation data. Qualitative data were collected through document reviews, interviews and case studies. Table 3 provides an overview of the people consulted during this evaluation. A detailed discussion of the data collection methods follows.
|Number of Participants|
|Participant Type||Interviews||Case Studies||Service Survey|
|ESPAD staff and management||11||5||10|
|Departmental senior management||2||0||2|
3.3.1 Document review
External documents reviewed included literature on the performance and evaluation of various types of policy functions, and evaluations of research functions in other organizations (e.g., the Bank of Canada, the World Bank, the International Monetary Fund and other organizations). Internal documents reviewed included Departmental Performance Reports; Reports on Plans and Priorities; integrated business plans; internal emails; work plans and research plans; ESPAD research papers, notes and presentations; departmental Internet and intranet websites; financial and full-time equivalent (FTE) data for 2001–10; and data on participation in Canadian Economics Association (CEA) conferences over the last 15 years.
A total of 24 interviews were conducted to obtain a comprehensive perspective of ESPAD’s relevance and performance. Interviews were conducted with internal ESPAD staff and management, departmental senior management, representatives of other branches or divisions within the Department of Finance Canada, and external stakeholders from other government departments. Internal Audit and Evaluation and ESPAD jointly prepared a list of potential interview candidates, and Internal Audit and Evaluation selected the internal interviewees to reflect appropriate representation in terms of level within the organization, years of experience in ESPAD and type of research conducted within ESPAD.
A supplementary set of four telephone or email interviews were conducted late in the evaluation process with departments that use general equilibrium (GE) models to obtain contextual and comparative information on the costs of developing, maintaining and using these models.
3.3.3 Case studies
Nine additional interviews were conducted in relation to two selected cases of research work done by ESPAD within the last five years in order to i) collect researcher, client and stakeholder perspectives on the quality of ESPAD research and research products and ii) provide in-depth understanding and illustration of processes, practices, opportunities and challenges associated with producing selected divisional outputs.
3.3.4 Service survey
A service survey that listed key ESPAD outputs (i.e., informal communication, analytical work, presentations, and models or simulations) was used in conjunction with the interviews to obtain feedback from ESPAD staff, management and clients, who were asked to rate these outputs in terms of importance and satisfaction. In terms of satisfaction, clients were asked to assess the outputs for defined quality attributes based on Schacter’s criteria.
Most of the service survey respondents were internal to the Department since external stakeholders were not sufficiently familiar with ESPAD’s outputs to provide valuable answers.
- A limited number of ESPAD research outputs were examined in depth through case study research, and the findings were used primarily to better understand and support findings from other lines of evidence.
- Benchmarking ESPAD performance against other research groups was limited to comparing modeling costs across government departments. The intangible nature of policy-relevant research and analysis outputs make benchmarking performance of policy functions difficult.
- Only half of the interviewees had more than five years of experience in ESPAD, and as a result, the information used for the evaluation is largely based on the last few years. However, information covering the last five years or longer was available from some ESPAD interviewees and from documents and financial records.
- The ratio of ESPAD versus non-ESPAD interviewees could be perceived to result in an over-representation of observations from within ESPAD. Given that the evaluation methodology incorporates an organizational assessment of the work environment and processes within ESPAD, it was important to have a sufficiently large sample size of staff. To achieve a balance in perspective on other evaluation issues, care was taken to select a sample size of departmental clients and external stakeholders in relatively equal weight to the sample size of ESPAD staff.
In this section, data gathered during the conduct of this evaluation were used to assess ESPAD’s relevance and performance. As much as possible, findings from multiple lines of evidence were triangulated to confirm and formulate conclusions and recommendations.
The evaluation found that ESPAD’s objectives and activities continue to be consistent with federal government and departmental priorities and roles and responsibilities, and that there is ongoing need for this in-house economic policy research function. There was general consensus that if departmental research is done externally, it would likely be more costly while being less timely, responsive and useful for the needs of departmental policy makers.
Alignment with federal government and Department of Finance Canada priorities and roles and responsibilities
Evidence gathered through interviews and document review indicated that ESPAD’s mandate and core activities of conducting and communicating policy-relevant economic research are in line with departmental and federal government priorities and roles and responsibilities.
The 2011–12 departmental Report on Plans and Priorities states that the Department of Finance Canada helps shape and deliver the Government of Canada’s agenda, particularly its fiscal and economic priorities. The Department is focused on a single strategic outcome: A strong and sustainable economy, resulting in increasing standards of living and improved quality of life for Canadians.ESPAD’s mandate is clearly aligned to the departmental priority of supporting sustainable economic growth by “providing high-quality research and effective and sound analysis and advice on economic and policy issues.” For example, one of the most recent economic initiatives of the federal government is the Economic Action Plan (EAP), as articulated in recent budget announcements and Speeches from the Throne. Interviews with senior management indicated that ESPAD provided background information on the impact of fiscal stimulus on the economy, as well as on the economic impact of the financial market crisis. Senior management also indicated that ESPAD supported the Minister of Finance on pre-budget consultation tours in the first and second years of EAP implementation.
Recent reports of the Privy Council Office outlining performance goals and priorities for the public service confirm the continued relevance and importance of conducting and communicating government research on policy issues of medium- and long-term significance. TheFifth Report of the Prime Minister’s Advisory Committee on the Public Service identifies providing policy advice to current and future governments to address the important issues of the day as a core responsibility of the public service. Investing in research, analysis and strategic thinking as a platform for longer-term policy thinking is viewed as critical to fulfill this role, particularly in the environment of economic uncertainty. The report recommends that the public service “continue to invest in the sustained examination of issues beyond the current agenda and in developing people with the skills to do this kind of work.”
The Clerk of the Privy Council’s Eighteenth Annual Report to the Prime Minister on the Public Service of Canada further affirms that excellence in policy means bringing evidence and analysis to bear in the development of advice to government and the importance of knowledge and analytical rigour. The report highlights the importance of horizontal collaboration within and outside the public service and notes that “knowledge and information are important government assets that should be systematically captured and shared among individuals and across organizations.”
All interviewees were unanimous in stating that ESPAD’s activities of communicating policy-relevant economic research were in line with federal government priorities and roles and responsibilities. Collaboration and exchange of information on policy issues with other departments was viewed as important by many of the interviewees, and seen to align with the Department’s mandate to “interact extensively with other federal departments and agencies and play a pivotal role in the analysis and design of public policy across a wide range of issues affecting Canadians.”
Ongoing need for ESPAD’s economic research
The evaluation found that there is ongoing need for a robust in-house research capacity and challenge function to support decision makers in their goal of advancing the economic health of the country. All departmental directors and senior management interviewed agreed that the Department should maintain a strong internal knowledge and capacity to conduct in-house policy-focused economic research that is timely, responsive and neutral, with results communicated to stimulate public policy debate and the ability to challenge policy research findings of other organizations.
The evaluation examined whether research conducted by ESPAD could be done by other government departments or external research groups (e.g., think tanks or academia). All groups of interviewees agreed that there may be some overlap between the research and analysis work of ESPAD and other research groups, but that this could not be considered duplication or a viable alternative to the in-house research function.
In general, confidentiality in sovereign economic policy development is considered critical to avoid disruptions in domestic markets. This aspect of conducting policy-relevant economic research was acknowledged in the interviews as the foremost justification for an in-house economic policy research function. Private sector or external organizations may not be in the best position to answer sensitive policy questions of senior management, particularly since the work may lead to policy or legislative changes.
It was pointed out in the interviews that the Department needs to conduct research from a neutral and relatively independent perspective in order to present viable, balanced policy options to senior management. It was noted that research done by external research organizations often has some mandate or bias specific to their own needs or the needs of their stakeholders. Line departments, for example, have a mandate to examine issues critical and specific to their respective sectors and to be responsive to the needs of their own minister.
Several internal and external interviewees stated that if economic research were contracted out, the Department would lose the expertise to conduct policy-focused economic research. In addition to informing policy development internally, the in-house research function was noted to provide the Department with the capacity to assess and understand the implications of external research and analysis. All internal interviewees agreed that the capacity to test and question the results and findings of other departments and sources external to government was critical for the Department to successfully fulfill its central agency challenge function. Senior management noted this to be an area where ESPAD has added value to the Department, emphasizing that external economic expertise is not as responsive and is more costly. They noted that with external contractors, the contracting process is lengthy. Further, contractors often do not give sufficient time for the frequent back and forth discussions necessary to present the context of the work, guide the direction of the research and ensure it meets government needs.
Results from the survey confirmed that departmental senior management considered ESPAD's ability to answer current policy questions as particularly important and useful. In interviews, ESPAD staff confirmed that they believe complex models must be built in-house so that they can be updated easily to analyze new scenarios and conduct estimates that respond quickly to policy questions posed by senior management.
The value within the Department of ESPAD's research capabilities was further confirmed through case study findings. For example, regarding external research commissioned by the Department on retirement income adequacy,  work done by ESPAD was described by a senior departmental official at that time as "an outstanding example of the importance and potential positive effect of quality economic analysis of a difficult policy issue that seems to be available only from within this Department."
The evaluation found that ESPAD is highly respected and valued, both internally and externally, for its research expertise and the high quality of its research and analysis.
Over the last two to five years, the demand for ESPAD’s research expertise from within the Department has increased, and a greater proportion of ESPAD’s researchers have been focused on research and analysis related to current policy issues. ESPAD is producing fewer working papers and presentations for external audiences and more models and outputs that are useful to conduct and communicate policy-relevant research to government policy makers. The evaluation found that these changes have led to a difference of understanding within ESPAD of the roles and responsibilities of the research staff.
The evaluation found that these changes have benefited the Department by making ESPAD’s work more responsive to its needs. At the same time, the evaluation found that a balance in the conduct of research is beneficial. Focus on current policy issues is important to maintain the relevance and usefulness of ESPAD’s work for the Department. Research effort into more general economic issues and model development remains important for building and maintaining ESPAD’s skill level and knowledge base because this work is often the foundation for providing timely research related to current policy issues.
The evaluation found that there is room for improvement in systematically obtaining Department-wide input to ESPAD’s research plan and in communicating research plans and findings across government to facilitate more effective horizontal collaboration.
126.96.36.199 Design and implementation
The evaluation examined the appropriateness of ESPAD’s organizational structure and work environment, human resources, and internal processes and practices in support of its activities. In this section, ESPAD’s roles and responsibilities in implementing its mandate are also examined.
Internal structures and processes
Information collected through interviews indicated no significant issues regarding ESPAD’s organizational structure. Staff described the culture at ESPAD as collaborative and open, facilitating the exchange of information and knowledge. Many ESPAD employees described their division as a small group of economists with an open-door policy who do not get lost in formality. The majority of ESPAD team members stated they did not feel that the hierarchical structure hindered their ability to seek out the knowledge and expertise of any member of the team. Most staff interviewed found the work environment motivating and ESPAD’s coverage of diverse economic topics appealing.
Work planning, quality control processes and monitoring processes were also found to be adequate. Documented meeting invitations showed that ESPAD’s management team meet every two weeks to discuss the progress of ongoing work and set the work agenda based on demands from senior management and other branches, as well as to review ideas for research projects from within ESPAD. It was stated that the Director attends weekly meetings with the Assistant Deputy Minister and the General Director to report progress on ESPAD’s priorities.
In conducting research, ESPAD was found to follow a rigorous process to ensure quality. When new research is proposed, an assessment of its purpose and objective to determine appropriate methodology and data requirements is reviewed and approved by the Director. ESPAD also occasionally works with academics to verify methodology, particularly on analytical work or aspects of the GE models. For ESPAD-initiated research, internal peer review is predominately used in the quality control process, including through internal workshops and seminars.
Management stated that ESPAD learns about the quality, timeliness and relevance of its current and past research predominantly from verbal and written feedback. Organizational developments and changes are also informally discussed among management and staff.
The ESPAD team’s level of expertise in the field of economics was described unanimously by all interviewees as diverse and of high caliber. According to ESPAD management, a master’s degree in economics is essential to work in ESPAD; an analysis of staff indicated that 50 per cent possess a doctorate in economics. A listing of past ESPAD directors posted on the Department’s intranet site indicates that many have gone on to senior-level positions both within and outside government, including academia.
In interviews, ESPAD management noted that it was difficult to attract “top talent” candidates with PhDs, since these candidates are in high demand in the private and academic sectors where entry-level salaries are generally higher than government wages. Universities also offer candidates the opportunity to conduct and publish research freely.
Retention of employees is primarily linked to their satisfaction at work, as well as to the availability of outside opportunities. The public service conducts a survey every three years for all employees to anonymously voice their opinions on their leadership, workforce and work environment. Economic and Fiscal Policy Branch results showed that in 2008, the last survey for which results were available at the time of this evaluation, 84 per cent of employees agreed that the Branch was a good place to work as compared with 89 per cent in 2005. In comparison, Department-wide results were 74 per cent (2008) and 87 per cent (2005); and government-wide results were 71 per cent (2008) and 82 per cent (2005). In general, the survey results up to 2008 indicate that Branch employees have been relatively satisfied with the work environment.
Training and development
Each year, ESPAD staff are asked to complete a learning plan whereby they articulate the precise skill area they would like to develop and select a corresponding training course. Many ESPAD economists pursue French language courses in order to be considered for promotions to higher levels in their classification group.
At the same time, conference participation was considered the optimal venue for staying abreast of the latest developments in economic research. Most public service economists from junior level staff to director level and external stakeholders interviewed agreed that conference attendance is important for skill development and maintenance in the field of advanced economics. It is also valuable for exchanging current economic findings and theories with others working in the field, such as private sector economists in the banking sector and think tanks, university academics and other members of the government community of economists and policy advisors.
According to interview findings, for many years ESPAD staff participated in the annual conferences of the Canadian Economics Association (CEA) and the American Economic Association (AEA). The CEA hosts an annual conference covering economic, econometric and policy research topics and has been recognized as the foremost economics conference in Canada. A review of data on CEA conference participation published on the CEA website indicates a declining trend in active participation from the Department over the period 1995–2010. Active participation is defined as participation as a presenter, a co-author, a session organizer and/or a session chair. It does not reflect attendance only, for which complete data were not available. During the last five years in particular, active participation of the Department of Finance Canada was found to have dropped significantly. Figure 2 illustrates the number of presentations by departmental staff over a 15-year period compared with three other government departments considered leaders in policy research and analysis—Industry Canada, Statistics Canada and Human Resources and Skills Development Canada (HRSDC) (formerly Human Resources Development Canada (HRDC)). Participation from these leading policy departments has generally risen over time and has been more or less maintained in recent years. For example, the Bank of Canada is reputed to conduct cutting-edge economic policy research and analysis, and is arguably the Department’s closest federal counterpart. In contrast with participation from the Department of Finance Canada, the Bank’s participation has increased steadily and significantly over time, as can be seen in Figure 3.
Further analysis of more detailed data available on attendance and active participation at the 2010 CEA conference confirmed ESPAD’s notable absence compared with strong participation by other leading government policy research groups.
Figure 2. Presentations by Leading Policy Research Departments at CEA Conferences 1995–2010
Figure 3. Comparison Between the Bank of Canada and the Department of Finance Canada’s Active Participation at CEA Conferences 1995–2010
A change in ESPAD’s roles and responsibilities over the last five years, which is discussed further in the next section, could be associated with the decline in conference participation. Increased demand for ESPAD’s expertise within the Department has meant that a greater proportion of analysis conducted by ESPAD is related to current policy development. This current policy research is more sensitive and is increasingly communicated through vehicles that are useful to departmental policy makers, such as presentations and short non-technical notes, than through working papers based on several more months of research. This has left less time and resources for external publications and presentations.
In interviews, staff as well as external stakeholders expressed concerns that declining conference participation may negatively affect the quality of the Department’s research and the reputation of the Department as a leader in economic research and knowledge. Internal and external researchers interviewed stated there was a risk that the quality of the work and the capabilities of analysts would decline if conference attendance and the publication of research papers are not taking place, since their work is not being challenged by external researchers. It was noted that passive participation would be useful if active participation is not possible. Interviewees considered reading academic journals to be a necessary activity as well; however, given that it takes approximately two years for research to be published, the information in academic journals could be considered to be outdated to some extent. Conference participation was viewed to improve the flow of knowledge of advanced methodology, which is of particular importance to those designing and updating economic models. Conference participation was also considered an efficient means to update researchers on emerging innovations and insights into policy research.
ESPAD’s roles and responsibilities
Based on interviews with ESPAD staff, the evaluation found that the research they undertake can be broadly defined by two categories: 1) responses to questions related to current departmental issues or work priorities and 2) production and publication of research papers on more general economic issues. The research papers generally analyze anticipated or potential policy requirements and are broader in scope, although they still have to align with departmental and government priorities.
Information collected through interviews and document review indicated a clear shift in ESPAD’s roles and responsibilities over the last five years, with ESPAD being more integrated into the Department and its analysis more in demand by senior management and other branches within the Department. As a result, a greater proportion of ESPAD’s researchers are focused on answering questions related to current policy issues. ESPAD staff are no longer expected to produce numerous working papers annually; instead, they are expected to provide policy-relevant research and analysis on issues related to current trends. Senior management noted that while the number of working papers has decreased, the number of presentations and briefings, which are considered more useful outputs for policy makers, has increased. Senior management also noted that ESPAD has significantly increased the number of models built and maintained in-house, enabling it to analyze short-, medium- and long-term trends, and quickly answer questions posed by the Deputy Minister and the Minister.
Senior management stated that the intent of these changes in ESPAD’s activities and outputs was to better use the analysts’ capabilities to answer policy-relevant questions.
Based on interviews with ESPAD management and staff, it was noted that these changes seem to have led to a difference of understanding regarding ESPAD’s mandate and objectives, and thus the roles and responsibilities of research staff. Several researchers interviewed expressed the view that increased demands for research and analysis on current policy issues have reduced the time allotted to model development and the production of more in-depth academic quality, publishable research papers on economic issues of longer-term significance. Many felt that this type of research better reflected ESPAD’s stated mandate and better utilized and maintained the type of technical research skills for which they were hired. Concerns were also raised by management that some employees were not adjusting to the new demands. At the same time, senior management acknowledged some of the risks inherent to these changes in relation to depleting human capital, such as loss of knowledge and skills, and staff motivation and retention.
The changes have made ESPAD more responsive to departmental needs. Information collected from interviews with ESPAD staff and client staff from other branches indicates that a balance in the conduct of policy relevant research is beneficial. Work on current issues provides assistance that is relevant to departmental priorities and at the same time allows ESPAD to stay connected to current departmental issues. This in turn can help inform more policy-relevant longer-term research projects. Ongoing research on more general economic issues of longer-term significance and model development remains important for building and maintaining ESPAD’s skill level and knowledge base, the foundation for effectively responding to current issue questions. In interviews, other divisions within the Department who are ESPAD clients indicated that they viewed in-depth research on longer-term economic issues as important to maintain within the Department. ESPAD’s existing body of research has been useful to feed into the shorter-term operational requests within the Department. In interviews, senior management acknowledged the importance of maintaining effort into capacity building.
Recommendation 1: To enable ESPAD researchers to keep abreast of cutting-edge research, to maintain strong links with the external research community and to preserve the Department’s visible role as a leader in economic policy research, the Director of ESPAD should, within the financial limits of the division, ensure a minimum participation of ESPAD staff in relevant economic conferences and that information gained is disseminated to team members.
Recommendation 2: A review of ESPAD’s mandate and mission statement should be conducted by the Director of ESPAD to ensure it is aligned with ESPAD’s current work activities and the departmental mandate. ESPAD management should clearly communicate this mandate to all employees, including potential and new employees, to promote realistic work expectations.
188.8.131.52 Achievement of expected outcomes
This section assesses ESPAD’s performance related to the objective of providing senior government officials with policy-focused research that is timely and of high quality on priority and emerging economic and financial issues. ESPAD’s contribution to the Department and its policy influence are also briefly examined.
Quality of the research
The evaluation used Schacter’s quality criteria to assess the quality of ESPAD’s work in both the case study interviews and in the service survey for participants and respondents.
The evaluation found that ESPAD’s work was viewed from researcher, client and stakeholder perspectives as being of high quality. It was noted by clients interviewed that ESPAD staff are highly engaged, accessible and responsive in terms of clarifying research expectations, explaining and answering questions related to their work, and sharing relevant research (subject to confidentiality and approval constraints). Particular strengths of ESPAD’s work, noted by virtually all clients and stakeholders, included economic rigour, evidence basis, clarity of assumptions, and logic and presentation (the work was easy for policy makers to understand). It was noted by ESPAD staff, clients and stakeholders alike that the ability to consult externally was limited due to political sensitivity and confidentiality of topics. Some ESPAD staff noted that time constraints often further limited the level of consultations.
The survey results confirmed these strengths of ESPAD’s research outputs, which was considered by all respondents to be of high quality, timely, satisfying operational requirements and important to departmental work. All respondents rated ESPAD’s analytical work as most important, giving it an average rating of 4 on a scale of 1 to 5. They also expressed that they were satisfied with the timeliness, empirical evidence, logic and relevance of ESPAD’s analytical work, with an average rating of +1 on a scale of -2 to +2. Among non-ESPAD respondents, the satisfaction rating for analytical work was even better: an average of +1.5. The other three research outputs assessed—informal communications, presentations, and models and simulations—were also viewed positively. The overall average rating was +3 for “importance” and +0.75 for perceived “satisfaction.” Senior management responded to the service survey questions by stating that all of ESPAD’s products and services were useful, highlighting the importance of having the modeling tools and policy-relevant research on hand in order to be ready to answer questions quickly and to challenge the estimates of others.
A review of email correspondence related to ESPAD research and analysis further supported the assessment that clients have been satisfied with ESPAD research outputs. Feedback on ESPAD research and analysis has been positive. In the last two years, ESPAD has been nominated by three different branches for awards to recognize their contribution on inter-branch work.
Expectations of other divisions and branches in relation to the timeliness and rigour of research and analytical work were believed to be better met by ESPAD than by others outside the Department. However, some branches that have quick turnaround time requirements expressed that ESPAD’s research has been thorough but, as a consequence, has not always met immediate needs. This highlights the importance of anticipating departmental research needs.
Communication of research plans and findings
Each year ESPAD completes a work plan that groups planned research projects into key issues or themes. These theme areas are proactively selected by ESPAD’s management and are related to issues of medium- and long-term significance for the Canadian economy. Staff noted that they also suggest specific projects that fit under each of the research themes and their respective areas of expertise. The plan is normally approved by senior management and has occasionally been posted on the Department’s intranet site.
Obtaining Department-wide input on ESPAD’s annual research plan was identified as an area for improvement to better integrate its work into the overall operational activities of the Department. It was noted in internal interviews that ESPAD’s annual work plan is not regularly circulated or reviewed by other departmental divisions and branches. There is no formal mechanism whereby the research issues of interest to the branches are communicated to ESPAD for incorporation into its annual work plan. In the last two years, ESPAD’s work plan was informally circulated to some divisional directors for their input and suggestions. These directors viewed the opportunity to provide input into the research plan as positive.
In terms of collaboration with other branches, interviews with ESPAD staff and clients indicated that instances of collaboration were generally seen to be effective, but were also characterized as ad hoc and issue-specific in areas of common interest. Communication channels were largely informal, but effective. The evaluation found through case studies that an important outcome of inter-branch cooperation has been the professional networks built between the researchers and policy makers across branches, opening doors to future collaboration.
Findings were similar for collaboration with research groups in other government departments. Departments collaborating with ESPAD expressed a willingness to share research findings in areas of common interest, stating that a collaborative approach on certain larger themes had been helpful. Other departments, however, indicated they had little access to Department of Finance Canada work, and that it would be helpful to have this access.
Communication of research findings to external audiences, such as academia, was found to be infrequent lately. ESPAD staff have rarely presented at conferences in recent years and have not published much of their work on the Department’s external and internal websites, nor in research journals as was the case in the past. The last working paper published on the Internet site was in 2008, and the last analytical note was published on the intranet site in May of 2007.
Contributions to the Department and to policy
Increased demand for ESPAD services within the Department is a clear indicator that its work is useful to the Department. In interviews, senior management and clients stated that research conducted by ESPAD has been reflected in topics examined in Memoranda to Cabinet (MC) and in briefing notes to the Minister of Finance. Many departmental directors interviewed indicated that they rely on ESPAD for the quantitative economic findings or estimates that go into these documents. Noted areas of policy influence were ESPAD’s work on the Advantage Canada plan and, more recently, ESPAD’s contributions to the government’s EAP.
Given that the impact of research on longer-term issues may not be immediately evident, researchers often do not see the contribution of their work directly. ESPAD does not track its contribution to policy documents. ESPAD staff and even management are not always aware of the contributions they have made to various MCs or briefing notes. ESPAD interviewees expressed some concern about this; they felt they could better direct and formulate their work if they were more aware of the extent to which their research is used.
Recommendation 3: To increase the relevance of ESPAD’s research, the Director of ESPAD should put in place a systematic process for a) annual sharing and input from senior management across the Department into ESPAD’s annual research work plan and b) communication of research plans with other governmental organizations to facilitate more effective horizontal collaboration.
4.2.2 Efficiency and economy
The evaluation found that ESPAD’s costs associated with modeling, one of the most expensive activities of research divisions, were comparable or lower than in other departments.
In recent years, ESPAD was found to be producing more research and analysis outputs that are demanded by the Department with existing resource levels.
Appropriateness of ESPAD’s resource levels
Financial data indicate that ESPAD spends most of its budget on personnel costs. Financial records for ESPAD and the Economic and Fiscal Policy Branch over the period from 2001–02 to 2009–10 (see Figures 4a and 4b) indicate that nominal budgets for both the Branch and ESPAD have not changed significantly over the last 10 years. Deflating this series by Statistics Canada’s Consumer Price Index (CPI) suggests that, in real terms, budgets have declined by 4.5 per cent for ESPAD and by 8.6 per cent for the Branch since 2001–02. This assumes that movements in CPI capture part of the upward annual increments and upward adjustments in pay that occur periodically.
While actual spending for the Branch overall has been close to budgeted resources, ESPAD’s actual spending has been more variable. Given that non-personnel operating costs have remained stable, this variation in ESPAD’s spending reflects changes in its level of FTEs year to year. FTE data for ESPAD indicate that the number of people working in ESPAD increased steadily over the last five years, following what appears to be a period of decreases from 2001 to 2005. In interviews with ESPAD management, it was stated that due to budget freezes and annual increments, the number of personnel would have to be reduced this year by 1 to 2 FTEs.
Figure 4a. Economic and Fiscal Policy Branch Budget, Actual Expenditures and FTEs from 2001–02 to 2009–10
Figure 4b. ESPAD Budget, Actual Expenditures and FTEs from 2001–02 to 2009–10
ESPAD management expressed the view that ESPAD’s efficiency has improved in recent years in terms of producing more research outputs demanded from within the Department. It was also clear that ESPAD is producing fewer time-intensive outputs, such as working papers. However, the findings of interviews within ESPAD indicate that the changes in the ESPAD’s activities and outputs are also affecting its organizational efficiency in the short-term as it adapts to meet these new demands. Capabilities of ESPAD staff to produce the research outputs that are increasingly in demand by senior management are uneven, with some staff better able than others to meet the varied research demands. This presents challenges when assigning workload in circumstances where there is a high demand for research outputs related to current issues. ESPAD is actively managing the staff to ensure they develop the skills set necessary to better meet client needs. This would enable ESPAD to produce more with its existing staff.
Costs of in-house general equilibrium models
Modeling was identified as one of the most expensive activities for research groups. ESPAD is currently updating, maintaining and using six dynamic GE models. GE models consist of a complex set of relationships between key economic variables that are used to analyze economic changes. With minor changes to a model, a set of policy options can be developed and a corresponding set of potential impacts can be determined. Updating and maintaining these models, however, requires specialized skills. ESPAD management and staff agreed that ESPAD has the capacity and expertise to develop, modify and update any GE model. ESPAD management has stated that the development of a model from scratch requires 2 FTEs for one year; approximately 1 FTE is required to maintain and update each model. On average, five to six ESPAD team members divide their time between model development and updates, and other activities, depending on operational requirements and work-task prioritization. Occasionally, academics have been contracted to provide assistance for a nominal fee.
The results of a short survey conducted in a few key departments that have or are planning to have in-house GE models, along with information collected through case study interviews, indicate that in other departments, model development from fundamentals was estimated to require between 2 and 4 FTEs and to take one to two years. In addition, external assistance from consultants is often required. If developed externally, the cost per model can range from $200,000 to $400,000 annually, depending upon the complexity. There are not many private firms capable of developing GE models or willing to develop or estimate policy options using GE models. It is a highly specialized field. Often the consulting groups hired are teams of academics from universities; this actually keeps the cost relatively low. Conducting a mini-research project with an academic is reported to cost between $10,000 and $25,000 for analysis of each policy issue, depending on the complexity of the question and whether the academic would need to revise a module of the GE model to answer the question. However, if contracted out externally, the expertise in modeling and analysis remains with the consultants; any further modifications would likely require hiring the consultants again; alternatively, a significant investment in time by ESPAD staff would be required to fully understand the changes made and their impact. Overall, the data collected on costs for modeling within government departments indicate that ESPAD costs for modeling and analysis are comparable, if not lower.
It was noted during case study interviews that obtaining resources for modeling is not just an issue of finding the financial resources to pay for the expertise. Finding the right expertise can also be a challenge. In government departments where economic modeling is meant to affect policy analysis and development, a rare combination of skills is required: the ability to build and run models, to interpret results and to communicate these results to policy makers effectively.
|Recommendation||Management Response||Planned Action||Lead||Target Date|
1. To enable ESPAD researchers to keep abreast of cutting-edge research, to maintain strong links with the external research community and to preserve the Department’s visible role as a leader in economic policy research, the Director of ESPAD should, within the financial limits of the division, ensure a minimum participation of ESPAD staff in relevant economic conferences and that information gained is disseminated to team members.
Management agrees with this recommendation.
Management will allocate about one third of ESPAD’s budget for goods and services for conference attendance, including conference fees and travel and accommodation costs (approximately $15,000 based on 2010–11 resource levels).
This will have to be dealt in accordance with the restrictions on travel and hospitality cap set up by the Department.
The Director of ESPAD will be responsible for the coordination and the planning of conference attendance.
Fiscal year 2012–13
2. A review of ESPAD’s mandate and mission statement should be conducted by the Director of ESPAD to ensure it is aligned with its current work activities and the departmental mandate. ESPAD management should clearly communicate this mandate to all employees, including potential and new employees, to promote realistic work expectations.
Management agrees with this recommendation.
In consultation with senior management, management will review and clarify ESPAD’s mandate. ESPAD staff will also be consulted.
This mandate will be communicated to all employees and posted on our website.
In consultation with senior management, the Director of ESPAD will update and clarify the mandate and objectives of the division.
3a) To increase the relevance of ESPAD’s research, the Director of ESPAD should put in place a systematic process for annual sharing and input from senior management across the Department into ESPAD’s annual research work plan.
Management agrees with this recommendation.
ESPAD’s annual research work plan will be circulated across the Department, and feedback will be sought.
Senior management of the Branch will circulate the work plan.
3b) To increase the relevance of ESPAD’s research, the Director of ESPAD should put in place a systematic process for communication of research plans with other government organizations to facilitate more effective horizontal collaboration.
Management agrees with this recommendation.
ESPAD’s annual work plan will be shared with other research groups. The high sensitivity of some projects might limit the amount of information provided to other departments or government agencies.
The Director of ESPAD will circulate the work plan after consultation with senior management.
Canada. Clerk of the Privy Council. Fifth Report of the Prime Minister’s Advisory Committee on the Public Service: A Public Service for Challenging Times. Ottawa, 2011.
Canada. Clerk of the Privy Council. Eighteenth Annual Report to the Prime Minister on the Public Service of Canada. Ottawa, 2011.
Canada. Department of Finance. Advantage Canada. Ottawa, 2006.
Canada. Department of Finance. 2011-2012 Report on Plans and Priorities.Ottawa, 2011.
Canada. Department of Finance. Human Resources Strategic Outlook 2011–12: Considerations for Managers during the 2011–12 Integrated Business Planning Process. Ottawa, 2010.
Canada. Department of Finance. Department of Finance University Recruitment Campaign 2003-2004. Final Report. Ottawa, 2004.
Canada. Department of Finance. Finance Canada University Recruitment. Ottawa. [Last modified September 3, 2009].
Canada. Department of Finance. Summary Report on Retirement Income Adequacy Research. Ottawa, 2009.
Canada. Office of the Auditor General of Canada. “Chapter 22—Attributes of Well-Managed Research Organizations.”1999 November Report of the Auditor General of Canada. Ottawa, 1999.
Canada. Policy Research Initiative. Townsend, Thomas and Bob Kunimoto. Capacity, Collaboration and Culture: The Future of the Policy Research Function in the Government of Canada. Ottawa, 2009.
Canada. Speech from the Throne, March 3, 2010. Ottawa, 2010.
Canada. Treasury Board of Canada Secretariat. Canada’s Performance: The Government of Canada’s Contribution— Annual Report to Parliament 2009–10. Ottawa, 2010.
Canada. Treasury Board of Canada Secretariat. Public Service Employee Survey. Ottawa,2005, 2008 and 2011.
Dobuzinskis, Laurent, Michael Howlett, and David Laycock, eds. Policy Analysis in Canada: The State of the Art. Toronto: University of Toronto Press, 2007.
Kaplan, Robert S. and David P. Norton. “The Balanced Scorecard—Measures that Drive Performance.” Harvard Business Review. 70, 1 (February 1992), pp. 71–79.
Schacter, Mark. “The Worth of a Garden”: Performance Measurement and Policy Advice in the Public Service – A Discussion Paper. Commissioned by the Treasury Board of Canada Secretariat. Ottawa, 2006.
 We would like to thank Russ Robinson and Iqbal Singh Jaswal for their valuable insights, discussions, written notes, presentations and initial work on the project.
 Internal Audit and Evaluation also participated in the interdepartmental Policy Program Evaluation Working Group chaired by the Centre of Excellence in Evaluation in order to develop best practices and obtain feedback on proposed evaluation methodologies.
 Mark Schacter, “The Worth of a Garden”: Performance Measurement and Policy Advice in the Public Service – A Discussion Paper (Commissioned by the Treasury Board of Canada Secretariat, March 7, 2006), p. 7.
 One interviewee was interviewed for two case studies. The number of distinct case study interviewees was eight.
 This observation was noted by the Branch Assistant Deputy Minister, and Internal Audit and Evaluation was in agreement.
 Department of Finance Canada, Report on Plans and Priorities 2011–12, pp. 3, 10.
 Budget 2009, 2010 and 2011; Speech from the Throne, March 3, 2010, and June 3, 2011.
 The Fifth Report of the Prime Minister’s Advisory Committee on the Public Service, p. 8.
 Clerk of the Privy Council, Eighteenth Annual Report to the Prime Minister on the Public Service of Canada, p. 22.
 Department of Finance Canada, Report on Plans and Priorities 2011–12, p. 3.
 Senior management were asked to respond to questions parallel to those found on the service survey form in order to facilitate their input on the importance and satisfaction of ESPAD’s products and services.
 Over the last four years from fall 2008 to winter 2011, ESPAD has conducted 25 seminars and workshops on various topics.
 Survey results can only be drilled down to the branch level; therefore, the results are generalized to the ESPAD team. The results are generalizing to ESPAD with a degree of caution given that the other two divisions in the Economic and Fiscal Policy Branch have larger teams. In 2008, ESPAD had 18 staff members, the Fiscal Policy Division had 24, and the Economic Analysis and Forecasting Division had 28. In addition, comparisons between the 2005 and 2008 survey need to be made with caution. The 2008 survey used a different sample size and a five-point scale (as opposed to a four-point scale in 2005) that included a neutral answer option in order to reduce bias in survey results. This most likely explains why the percentages were slightly higher in 2005 than in 2008.
 Jean-Pierre Voyer, “Chapter 9: Policy Analysis in the Federal Government: Building the Forward-Looking Policy Research Capacity” in Policy Analysis in Canada: The State of the Art (2007),discusses departmental policy research capacity in the federal government and identifies these federal departments (and policy research groups within) as leaders in terms of departments that have made strong investments in medium-term research work since the mid-1990s.
 Details on the quality criteria used in this report are provided in section 3, which describes the evaluation methodology.
 Advantage Canada was introduced in 2006 as the government’s economic plan for future prosperity.
 FTE data for 2001–02 to 2003–04 were obtained from divisional organization charts over the period.
 FTE data for the Branch were not available for fiscal year 2006–07 or for fiscal years 2001–02 to 2003–04.
 FTE utilization data for fiscal years 2004–05 to 2009–10 are calculated based on the total number of working days for each employee in the fund centre. Time allotted to the fund centre for each employee is assumed to be 100 per cent. Year-end FTE utilization data were not available for fiscal year 2006–07; the number used is from a 2006–07 mid-year employee forecast report for ESPAD. Year-end FTE data for fiscal years 2001–02 to 2003–04 were also not available; FTE numbers were estimated based on organizational charts for those years. Therefore, while the FTE numbers used are not directly comparable year to year, in conjunction with data on spending levels they provide an approximate picture of ESPAD’s resource levels over the years.