Archived - Evaluation of the Research and Policy Initiatives Assistance Program

Archived information

Archived information is provided for reference, research or recordkeeping purposes. It is not subject to the Government of Canada Web Standards and has not been altered or updated since it was archived. Please contact us to request a format other than those available.

Final Report
Prepared by
Internal Audit and Evaluation
Department of Finance Canada

Approved by the Deputy Minister of Finance on the recommendation of the Audit and Evaluation Committee on August 26, 2013

Table of Contents

Executive Summary

1.0 Background

2.0 Methodology

3.0 Evaluation Findings

4.0 Conclusion

Annex A – Logic Model: RPIA Program

Annex B – Management Response and Action Plan

Executive Summary

This report presents the findings of the evaluation of the Research and Policy Initiatives Assistance (RPIA) Program of the Department of Finance Canada, conducted by Internal Audit and Evaluation (IAE) between February 2013 and April 2013. The evaluation covered just below $100,000 in actual program spending from the fiscal years 2008–09 to 2012–13, and examined the relevance and performance of RPIA Program.

Relevance

The evaluation found that the RPIA Program addressed a demonstrable need and contributed to the work of both the sponsoring branch and the Department. The evaluation also found that the initiatives funded by the RPIA Program were aligned with government priorities, roles and responsibilities, and a clear link could be made between the funded initiatives and at least one departmental strategic objective.

Performance

Overall, the evaluation found that the RPIA Program was successful in achieving its expected immediate outcomes and contributed to meeting its intermediate outcomes. The RPIA Program added value in different ways. For example, it provided a channel for the Department to access the research in fields of interest; it helped sponsoring branches to expand their network of contacts; and it supported initiatives that informed the policy context, including macroeconomic policy, health care policy and tax policy.

The RPIA Program was successful in leveraging financial and human resources, as well as in taking advantage of the networks and processes in place in recipient organizations. This allowed the initiatives to be delivered more effectively and efficiently. As a class contribution program, the RPIA Program was found to be an efficient mechanism for funding small ad hoc initiatives.

The RPIA Program used financial resources economically, as Program management was diligent in approving funding for the different initiatives and the level of full-time equivalent required to manage the RPIA Program was consistent with the need for the RPIA Program.

The Financial Management Division (FMD) of the Corporate Services Branch was found to have administered the RPIA Program efficiently. FMD implemented the RPIA Program as originally designed, and ensured that the financial controls were in place. FMD also examined project proposals to ensure that they satisfied the terms and conditions of the program. The evaluation identified one opportunity for improvement. The evaluation found that performance information was not reported consistently across the different initiatives. As a result, it was not possible to systematically assess whether outcomes were achieved across the spectrum of funded projects. The following recommendation is given in the spirit of continuous improvement.

Recommendation: The RPIA Program management should consider improving the consistency of project evaluation reports.

1.0 Background

1.1 Introduction

This report presents the findings of the evaluation of the Research and Policy Initiatives Assistance (RPIA) Program of the Department of Finance Canada, conducted by Internal Audit and Evaluation (IAE) between February 2013 and April 2013. The evaluation was approved by the Deputy Minister (DM) as part of Department’s 2012–17 five-year evaluation plan. The evaluation plan was developed based on the Treasury Board Policy on Evaluation and Section 42.1 of the Financial Administration Act, which require departments to evaluate the relevance and performance of all ongoing programs of grants and contributions at least once every five years.

1.2 Profile of the RPIA Program

The RPIA Program is a class contribution program used for small ad hoc initiatives that contribute to the Department’s objectives. The RPIA Program supports research and policy type work in various formats such as educational events, workshops, seminars, conferences and studies. The program logic model is provided in the Annex and illustrates the linkages between RPIA Program activities, outputs and expected outcomes.

The Financial Management Division (FMD) of the Corporate Services Branch (CSB) administers the RPIA Program. Participation in the RPIA Program requires the recommendation of a Branch Head, and the sign-off of the Department’s Chief Financial Officer and of the DM.

1.3 Program Resources

A maximum of $100,000 can be paid out under the terms and conditions of the RPIA Program. Also, a single initiative could receive a maximum of $25,000 per year and up to $100,000 over a five year period for multi-year projects.

Over the period 2008–09 to 2012–13, a total of $93,500 had been approved through the RPIA Program. Included in this total are two projects funded during 2008–09 that were not covered by the 2009 evaluation of the RPIA Program.

Table 1: Annual spending on the RPIA Program covered by this evaluation
Fiscal Year 2008–09 2009–10 2010–11 2011–12 2012–13 Total
Spending $40,000 $3,000 $25,500 $10,000 $15,000 $93,500

2.0 Methodology

This section presents the evaluation methodology, including the evaluation scope and objective, the evaluation issues and questions, the data collection methods, as well as the limitations.

IAE calibrated the evaluation approach and methods, as well as the length of the evaluation report, with the size and associated low risk of the RPIA Program. To assess the risk, IAE considered that the RPIA Program was evaluated in 2009 and no major issues with relevance and performance were identified at that time. In addition, the costs associated with the RPIA Program continue to be low, both in terms of contribution amounts and resources to manage the program.

The underlying flows of program activities, outputs and outcomes were originally mapped out during the 2009 evaluation to form the logic model found in the Annex. The methodology for this evaluation used this logic model when assessing the extent to which the RPIA program achieved its expected outcomes.

Achievement of outcomes was assessed through interviews, as well as through the project evaluation reports. These two sources of evidence were complementary and useful in two ways. First, the project evaluation reports give a quick impression of the immediate results achieved. Second, the interviews took place specifically for the evaluation. The data gathered from these interviews provided a longer-term view on the usefulness of the project, and could be used to determine, to some extent, the contribution to the intermediate outcomes. Therefore, even though the report specifically focuses on the achievement of immediate outcomes, the link to intermediate outcomes is fairly direct. For example, once there is an increased access to the knowledge base on subjects of interest, it stands to reason that this would enhance policies and programs of the department.

2.1 Evaluation Scope and Objective

The evaluation covered the period between 2008–09 and 2012–13. For 2008–09 the evaluation included two projects that had not been covered in the 2009 evaluation of the RPIA Program.

The objective of this evaluation was to assess the relevance and performance (i.e. the effectiveness, efficiency and economy) of the RPIA program. Given the size and associated low risk of the RPIA Program, the evaluation focused on (1) the program design and implementation; and (2) the extent to which the RPIA Program contributed to the achievement of its expected immediate outcomes.

2.2 Evaluation Issues and Questions

Table 2 presents the broad evaluation issues and questions examined as part of this evaluation and associated data collection methods. The evaluation issues reflect the core issues identified in the Treasury Board Directive on the Evaluation Function.

Table 2: Evaluation Matrix
"X" indicates that the line of evidence was used in addressing that particular question or issue.
    Data Collection Methods
   
Evaluation Issues Evaluation Questions Review of administrative documents Review of evaluation reports Interviews Email survey
Relevance          
  • Continued need for the RPIA Program
  • Alignment with government priorities
  • Alignment with government roles and responsibilities
R1: To what extent are the objectives and activities of the RPIA Program still relevant and in line with the priorities and roles and responsibilities of the Department of Finance Canada and the federal government? X X X X
Performance (i.e., effectiveness, efficiency and economy)          
Program design and implementation P1: To what extent does the existing structure support the effective administration and delivery of the RPIA Program? X X X X
Achievement of expected outcomes P2: To what extent has the RPIA Program achieved its objectives and expected outcomes, in particular with respect to its immediate outcomes? X X
Demonstration of efficiency and economy P3: To what extent has the RPIA Program conducted its operations efficiently and economically? X X

2.3 Data Collection Methods

Using both primary and secondary data sources, IAE collected qualitative and quantitative data for this evaluation. Four lines of evidence were used to mitigate the limitations identified and associated with the evaluation methodology, and to validate to the extent possible the conclusions and recommendations. The four lines of evidence are as follows:

Review of administrative documents: The purpose of this review was to assess whether the RPIA Program addressed a demonstrable need, and to review the design and implementation of the RPIA Program. IAE reviewed key documents related to the RPIA Program, including financial documents, the RPIA Program’s Results-based Management and Accountability Framework, the relevant Treasury Board submission, the terms and conditions of the RPIA Program, and related memoranda.

Review of evaluation reports: IAE reviewed six project evaluation reports, one for each completed project during the review period,[1] with the exception of two projects funded in FY 2008–09 for which no project evaluation report were required. IAE also reviewed the 2009 evaluation report of the RPIA Program. The review of the project evaluation reports helped to assess whether the RPIA Program addressed a demonstrable need, to confirm the extent to which the recommendations of the 2009 evaluation had been implemented, and assess the extent to which the funded initiatives contributed to the achievement of the RPIA Program’s expected outcomes.

Interviews: IAE undertook a group interview with three members of the RPIA Program management (i.e., FMD) and five interviews with branch contacts. These interviews provided information that was not readily available through the documentation.[2] The branch interviews were comprehensive, as they covered five of the six project included in the review period.

Email survey: The purpose of the email survey of Departmental Coordinating Committee (DCC) members[3] was to seek their understanding and views on the RPIA Program.

2.4 Limitations

The evaluation methodology had the following three limitations:

Attribution of results: Since various factors other than the RPIA Program contributed to the achievement of the RPIA Program’s expected final outcomes, attribution issues must be taken into consideration when assessing these outcomes. To achieve more robust conclusions, the evaluation focused on whether the activities and outputs contributed to the achievement of the RPIA Program’s expected immediate and intermediate outcomes.

Inconsistent performance reporting: The performance information in the project evaluation reports were not reported consistently, with the reports differing in the type and depth of information provided. Specifically, the achievement of expected outcomes was well documented in some files, while lacking in some aspects in others. Overall, the six project evaluation reports were not presented in a consistent manner. IAE mitigated this issue by supplementing the data with branch interviewees who prepared the project evaluation reports. Therefore, the impact of this limitation on the evaluation findings was minimized. Recommendation 1 addresses the issue of inconsistent performance information.

Limited number of respondents to the email survey: IAE surveyed DCC members, of which four representatives of branches responded (about one third). As a result, their opinion may not be representative of all DCC members. However, the findings of the email survey were well supported by the branch contact interviews. Thus, the impact of this limitation on the evaluation findings was mitigated.

3.0 Evaluation Findings

This section presents the main findings by evaluation issue as outlined in the evaluation matrix (Table 2). IAE used the triangulation of data from the different lines of evidence to validate conclusions and recommendations, and to provide a high level of rigour for the findings.

3.1 Relevance

3.1.1 Continued need for the RPIA Program

The evaluation found that there is a continued need for the RPIA Program, as it addressed a demonstrable need and contributed to the work of both the sponsoring branch and the Department. The RPIA Program was found to contribute to the work of the Department by providing an additional channel for looking into emerging or long-term issues of interest. Evidence indicated that the need would not easily be filled by other means.

3.1.2 Alignment with government priorities

The evaluation found that funded initiatives were in line with government priorities. Examples include the harmonization of the sales tax and the reform of the Canadian retirement income system. The RPIA Program enabled the Department to participate in the selection of topics and issues addressed through funded research, conferences and workshops. This participation, overall, positively influenced the relevance of funded projects from a departmental perspective.

3.1.3 Alignment with government roles and responsibilities

The RPIA Program was found to be aligned with government roles and responsibilities. Although the Department does not have a general mandate to promote research or professional community development, the RPIA Program was found relevant as a clear link could be made between funded initiatives and at least one departmental strategic objective. For example, the Federal-Provincial Relations and Social Policy Branch supported a conference that explored how social policy in Canada will be shaped by both fiscal pressures and inter-governmental relations as the Canadian economy recovers from the financial crisis. There was a clear role for the Department to support this initiative given that the Federal-Provincial Relations and Social Policy Branch provides policy advice to the Minister of Finance on federal transfer programs that support various provincial and territorial health and social programs to ensure that they support long-term economic growth and are fiscally sustainable.

3.2 Performance

3.2.1 Program Design and Implementation

FMD implemented the RPIA Program as originally designed (as described in sections 1.2 and 1.3), and ensured that financial controls were in place. FMD also examined project proposals to ensure that they satisfied the terms and conditions of the program.

The evaluation found that the performance information contained in project evaluation reports was not reported consistently across the different projects. For example, the level of details as to whether funded initiatives contributed to the achievement of expected project outcomes varied greatly. As a result, it was difficult to consistently assess the outcomes achieved across the spectrum of funded projects.

Recommendation 1: The RPIA Program management should consider improving the consistency of project evaluation reports.

3.2.2 Achievement of expected outcomes

Overall, the evaluation found that the RPIA Program was successful in the achievement of its expected immediate outcomes and contributed to meeting its intermediate outcomes. This section provides detailed information for each expected outcome.

Increased access to expanded knowledge base on subjects of interest

The RPIA Program added value by providing a channel for the Department to access the research available in fields of interest. For example, the RPIA Program funded a portion of the cost of a conference on tax policy issues, which provided primary access to the research before it went to the public realm.

The RPIA Program also helped sponsoring branches expand their network of contacts. For example, the RPIA Program funded a research program on pension issues. This initiative provided an opportunity to have a representative of the sponsoring branch provide guidance on research themes and priority focus areas, and to meet with pension experts. The representative was able to foster connections with sponsoring organizations, such as the Ontario Teachers’ Pension Plan, the Alberta Investment Management Corporation, the Caisse de dépôt et placement du Québec, the Association of Canadian Pension Management, and the Pension Investment Association of Canada.

Better informed policy context

The RPIA Program was found to support initiatives that informed the policy context in fields of interest such as macroeconomic policy, health care policy and tax policy. FMD created a template for project proposals in which branch contacts would provide a clear statement of how the project would further the objectives of the Department. This provided assurance that the research and knowledge generated through the funded initiatives would feed into the policy cycle of the Department. For example, the RPIA Program funded a symposium on the Canadian retirement income system. According to the project evaluation report and the branch contact, this initiative allowed the exchange of research and ideas about new policy proposals, and helped convinced some jurisdictions that more research on the retirement income system was needed.

Increased leverage of resources or other capacities from external partners towards departmental objectives

The evaluation found that the RPIA Program increased the leverage of financial and human resources and other capacities such as the structures (i.e. networks and processes) in place in recipient organizations.

Financial Resources: For most of the initiatives, the RPIA Program financed a fraction of the total cost (Table 3), while the Department benefited from the entire initiatives.[4] For example, the RPIA Program provided 2% of the total cost of the conference organized by the Queen’s International Institute on Social Policy. According to the project evaluation report, this conference contributed to the advancement of research and analysis, including an international comparison of issues relevant to policy formulation in the Federal-Provincial Relations and Social Policy Branch. It also allowed for an exchange of ideas and discussions between government officials, academics and other experts.

Human Resources: Funded initiatives allowed the Department to draw on the human resources of external partners. Indeed, the funding recipients carried out the planning, development, implementation and delivery of the initiatives, while the Department played a liaison role and occasionally advised on direction. According to branch contact interviewees, being responsible for all of the phases of these initiatives would have been more time-consuming.

Structures: Through the RPIA Program, the Department took advantage of the networks and processes established by recipient organizations, allowing the initiatives to be delivered more effectively and efficiently. Because these organizations often organize conferences, workshops and research paper series, they already had the necessary framework in place.

Table 3: Percentage of Total Cost of Initiatives Funded Through the RPIA Program
Projects Year RPIA Program Contribution Total Cost of Initiative % of Total Cost
Contribution to support research on exploiting micro-data to assess the impact of government incentives for innovation (OECD) 2008–09 $15,000 $36,000 42%
C.D. Howe Institute Workshop on Sales Tax Harmonization 2008–09 $25,000 $25,000 100%
Financial contribution for Canadian Public Economics Group conference 2009–10 to 2010–11 $6,000 $14,000 43%
C.D. Howe Institute Pension Research Series 2010–11 to 2012–13 $15,000 N/A* N/A*
Reforming the Canadian Retirement Income System 2010–11 $15,000 $182,300 8%
Financial Contribution for Queen’s International Institute on Social Policy 2010–11 $2,500 $144,400 2%
Centre for the Study of Living Standards 2011–12 $5,000 $30,000 17%
The School of Public Policy – University of Calgary 2012–13 $10,000 N/A* N/A*
TOTAL   $93,500    
*This information was not available

3.2.3 Efficiency and Economy

3.2.3.1 Efficiency

FMD was found to have administered the RPIA Program efficiently. According to the branch contacts, FMD responded to funding requests in a timely manner, helped facilitate the preparation of documents required from branches seeking funding, followed-up with branch contacts, ensured that the project evaluation reports were received, and facilitated the sign-off by the Chief Financial Officer (CFO) and the Deputy Minister (DM). A few branch contacts found the two levels of sign-off cumbersome. However, sign-off by the CFO and DM is required to ensure that contribution payments meet the requirements of the Treasury Board Policy on Transfer Payments.

As a class contribution program, the RPIA Program was found to be an efficient mechanism for funding small ad hoc initiatives. One alternative mentioned was the use of a Treasury Board Submission for each small project, but branch contacts noted that the level of effort required by this mechanism would be too high. It was stated that without the RPIA Program small ad hoc initiatives were unlikely to receive funding.

3.2.3.2 Economy

The evaluation found that the RPIA Program used financial resources economically. Some interviewees stated that the amount of funding allocated to some projects was lower than originally requested, indicating that the RPIA Program management was diligent in approving funding for the different initiatives.[5] In addition, the level of full-time equivalent (FTE) involved in managing the RPIA Program remained stable at below 0.1 FTE, based on FMD estimate. This was also found to be consistent with the need for the RPIA Program.

4.0 Conclusion

The evaluation found that the RPIA Program addressed a demonstrable need and generally functioned well. It was successful in leveraging financial and human resources, as well as in taking advantage of the networks and processes established by recipient organizations.

Annex A – Logic Model: RPIA Program

Logic  Model: RPIA Program

View text version

Annex B

Management Response and Action Plan
Recommendation Management Response Planned Action Lead Target Date
The RPIA Program management should consider improving the consistency of project evaluation reports. Management agrees with the recommendation. Management will develop an evaluation template with standardized sections requiring completion. Senior Director, Financial Management Directorate, Corporate Services Branch March 31, 2014

[1] The project evaluation reports are brief summaries of the benefits realized through the RPIA Program.

[2] Two interview questionnaires were used, tailored for RPIA Program management and branch contacts.

[3] DCC members consist of senior managers at the General Director level (or equivalent) from each branch.

[4] The evaluation also found that the one initiative where the RPIA Program contributed to 100% of total cost would have required more financial and human resources if done in-house.

[5] The evaluation did not further investigate why these decisions were taken since this was outside the scope of the evaluation.