# Archived - Evaluation of the Advanced Policy Analyst Program

Archived information

Archived information is provided for reference, research or recordkeeping purposes. It is not subject to the Government of Canada Web Standards and has not been altered or updated since it was archived. Please contact us to request a format other than those available.

Final Report
Approved by the Program Steering Committee
March 3, 2015
Prepared by the Internal Audit and Evaluation Division
Department of Finance

## Table of Content

Executive Summary

1. Background

2. Evaluation Objectives and Scope

3. Methodology

4. Evaluation Findings

Annex A: Logic Model for the Advanced Policy Analyst Program

Annex B: Comparative Study

Annex C: Management Response & Action Plan

## Executive Summary

This report presents the results of the evaluation of the Advanced Policy Analyst Program (APAP or the Program). The APAP, previously known as the Accelerated Economist Training Program, is a recruitment and development program that prepares high potential recruits to excel in policy and leadership roles in the Government of Canada. APAP participants gain the expected experience and skills through four six-month assignments across the three central agencies—the Privy Council Office, the Department of Finance Canada, and the Treasury Board of Canada Secretariat—and a line department in the National Capital Region.

The evaluation assessed the relevance and performance (effectiveness, efficiency and economy) of the APAP. It covered the APAP’s activities and performance since its official launch in 1988 to March 31, 2014, with a particular focus on the past five years (2009–2014). The evaluation was carried out by the Internal Audit and Evaluation division of the Department of Finance Canada between February and October 2014.

The following summarizes the key findings of the evaluation.

### Relevance

The evaluation concluded that the APAP addresses a continued need for knowledgeable policy analysts, and its mandate, objectives and activities are consistent with government priorities, roles and responsibilities. The Program enables the government to recruit individuals with high-potential and to prepare them with the experience and skills to excel in policy and leadership roles across the federal government. The Program was found to be in line with the Public Service Renewal Action Plan, and current federal directives on human resources.

### Performance

The APAP has been successful in achieving its immediate (short-term) and intermediate (long-term) outcomes. The Program was found to be successful in recruiting high potential individuals and preparing them for policy roles in the federal government. A high proportion of Program participants have advanced to management and leadership roles in the government; and participants are more satisfied with their careers, relative to public servants who did not go through the program. A significant number of participants indicated that they chose to pursue a career in the federal government because of the APAP.

The current APAP structure and internal processes were found to be appropriate and to contribute to the achievement of the Program objectives. The deputy-head level oversight committee, alumni community and participating departments’ involvement in the administration of the Program were identified as having strengthened the Program structure. The participant recruitment process was found to be rigorous and efficient; the six-month assignment duration and the assignment composition were found to be appropriate; and, participants’ performance evaluations and subsequent promotions were found to be carried out rigorously and consistently. The Program was also found to conduct its operations and deliver its activities efficiently and economically. Over the past four years the average cost per participant has been stable over time, while total program costs have trended downwards.

Although most of the internal processes were found to be generally appropriate, the evaluation identified the following challenges: (1) The funding structure was found to be complex; (2) The quality of the 6-month assignments were found to be inconsistent; (3) the requirement for participants to return to their sponsoring organization after graduation was found to be a contentious issue for some; and, (4) the Program’s strong reliance on specific individuals, rather than institutions, was found to create risks to the smooth functioning of the Program’s administration in the future.

Given the above findings, the following recommendations are provided in the spirit of continuous improvement:

#### Recommendation 1

To facilitate the planning and administration of funds, Program administrators should explore options to improve the timing of fund transfers and cost recoveries, as well as to reduce the overall complexity of the funding structure.

#### Recommendation 2

To maintain high-quality assignments, Program administrators should consider:

1. Providing assignment supervisors with background information of their assignees early in the process, allowing them to arrange for suitable and high-quality assignments.
2. Establishing a method of tracking assignment quality, such as brief post-assignment surveys to be completed by the participant, in order to identify lessons-learned and opportunities for improvement.

#### Recommendation 3

To help promote a positive transition back to the sponsoring organization, Program administrators should consider the following. Some of these steps have been outlined in the latest MOU (2014-19). However, the implementation of these should be monitored annually.

1. Clearly communicating to the participants, at the outset, the requirement to return to their sponsoring organization (department/agency).
2. Encouraging the sponsoring organizations to foster a closer and more proactive relationship with the sponsored participants as outlined in section 3, Schedule B of the 2014-19 MOU.
3. Encouraging sponsoring organizations to take participants’ interests into consideration to a greater extent when preparing the three post-graduation job offers, as outlined in section 6 and 7, Schedule B of the 2014-19 MOU.

#### Recommendation 4

To promote seamless functioning of the Program in the future, Program administrators should consider putting in place a succession plan for key positions such as the Executive Advisor and the Organizational Coordinators.

## 1. Background

### 1.1 Introduction

This report presents the results of the evaluation of the Advanced Policy Analyst Program (APAP). The Internal Audit and Evaluation (IAE) division of the Department of Finance Canada conducted the evaluation, at the request of the APAP Steering Committee, on behalf of the APAP participating departments and agencies. In December 2013, the Deputy Minister of Finance authorized Finance IAE to conduct the evaluation. The evaluation was carried out between February and October 2014.

### 1.2 Profile of the Advanced Policy Analyst Program

The APAP, previously known as the Accelerated Economist Training Program, is a leadership development program that began as a pilot in 1986 and was officially launched in 1988. The objective of the Program is to prepare recruits with a high potential to excel in policy and leadership roles in the Government of Canada by offering them the opportunity to get exposure and to gain experience in policy making and policy development processes on a number of social, economic and international policy files.

Participants in the program are expected to gain:

• an understanding of the Cabinet and parliamentary systems, and of the roles of central agencies and line departments;
• exposure to the overall functioning of government, to a variety of policy areas and policy making, and to the expenditure management process;
• opportunities to hone soft skills; and
• an effective network of colleagues and program alumni across government.

In order to gain this knowledge and experience, participants are rotated through four six-month assignments across the three central agencies—the Privy Council Office, the Department of Finance Canada, and the Treasury Board of Canada Secretariat—and a line department in the National Capital Region.

Under the current design, a home organization sponsors a candidate who is typically hired into the Program at the EC-03 group and level.1 Based on performance, participants can then advance to the EC-04 level after one year, and then to the EC-05 level after 24 months. To advance, a program participant is required to obtain a fully satisfactory rating after each assignment and be successful at the promotion board interview, which is held after the first 12 months of participation. Upon graduation, participants are appointed into their sponsoring organization at the EC-05 level, where they are expected to remain for at least a full year.

Governance and Administration: From its inception to 2009 the Program was funded and operated centrally. Since 2009, the Program has been operating under a decentralized model with funding provided by the organizations that choose to participate in the Program. Under this model, participants are required to return to their sponsoring organization after graduation and stay there for at least one year. Prior to 2009, graduating participants accepted positions offered to them across government.

The Program’s highest decision-making body is the Steering Committee, which consists of deputy head-level representatives from each of the central agencies and up to two representatives from sponsoring line organizations, the Executive Advisor, and the Chair of the Alumni Advisory Board on an ex-officio basis. The Steering Committee has various responsibilities, including a) providing the overall direction for the Program, b) approving recommended candidates for admission into and graduation from the Program, as well as c) appointing the Executive Advisor.

The Executive Advisor is responsible for managing the Program’s administration and approving participant assignments, recommendations for promotion, recruitment documentation etc. The Executive Advisor is supported by a Program Coordinator who acts as the Program Secretariat and administrative hub.

The governance structure also includes the Alumni Advisory Board, whose main role is to support the Executive Advisor by providing insight and advice on the strategic development and improvement of the Program. The Alumni Advisory Board consists of alumni currently at the executive level (i.e., EX-03 and above), as well as a member of the Alumni Relations Committee and a recent Program Advisor.

APAP alumni are engaged in the Program via volunteering, networking activities and other events. Volunteer alumni are elected by their peers to act as Program Advisors. Program Advisors report to the Executive Advisor and provide guidance to the incoming cohorts, including conducting recruitment activities, generating program information, and liaising with placement managers and organizational coordinators for the purposes of recruitment activities, participant assignments, evaluations, board interviews, etc.

Sponsoring organizations have coordinators at the senior management level who work with Program Advisors to identify assignments within their organization. These organizational coordinators arrange for information sessions for the sponsored participant, as well as coordinate the job matching, recruitment and offer processes on behalf of their organizations. Sponsoring organizations also have the option of designating a Program Champion, typically at the Assistant Deputy Minister level, to serve as advocates for the Program within their organization.

Recruitment and Screening Activities: Each year, a maximum of approximately 14 participants are recruited, although more recently the cohorts have comprised about eight participants. Each cohort that enters the Program begins with an introductory orientation week held in June or July. Participants start their first assignment immediately after the orientation. A graduation ceremony is held after 24 months, upon successful completion of the Program.

Before 2009, recruitment was centrally funded. Since 2009, each participating organization contributes their own funds for the recruitment of candidates. Sponsoring organizations lead the recruitment activities on a rotational basis. For example, the Department of Fisheries and Oceans (DFO) led the recruitment of the 2011–13 and 2012–14 cohorts; and Industry Canada (IC) led the recruitment of the 2013–15 and 2014–16 cohorts.

Starting with the 2011–13 cohort, APAP-specific postings have been listed in the annual Government of Canada Public Service Recruitment campaign; and emails advertising the posting have been sent to universities with relevant post-graduate programs. Program alumni also promote the Program via presentations at selected universities. The lead organization (e.g., DFO, IC), along with Program Advisors and the APAP Secretariat, conduct the initial screening of the applications received. An interview is then administered and sponsoring organizations are matched with successful candidates. Successful candidates receive a letter of offer from their sponsoring organization with a start date in late June.

Program Expenditures: The 2014 MOU outlines the financial arrangements for the program. Recruitment costs are recovered from sponsoring organizations by the department or agency leading the recruitment. Administration costs for the orientation week, graduation and other activities are generally incurred by the Executive Advisor, who then recovers the costs from sponsoring organizations. Central agencies provide a fixed amount per participant per year for operating expenses to the sponsoring organizations (current amount is 15,000 per central agency). Participant salaries and associated costs (net of central agency contributions) are incurred directly by the sponsoring organization. ## 2. Evaluation Objectives and Scope The objective of the evaluation was to assess the relevance and performance (effectiveness, efficiency and economy) of the Program. The evaluation covered the Program’s activities and performance since its official launch in 1988 to March 31, 2014. It particularly focused on the Program’s activities over the past five years (2009–2014), following the Program’s transition to the decentralized administration model. The information collected through this evaluation is intended to inform the decision-making process by providing an objective and evidence-based assessment of the Program’s relevance, design and performance, as well as identifying potential opportunities for improvement. An Evaluation Working Group was established to advise the evaluation team on evaluation issues and to ensure that the needs and perspectives of the various stakeholders were incorporated in the evaluation. The Working Group consisted of the Chair of the Alumni Advisory Board, a representative from the APAP Secretariat, the Head of Evaluation for the Department of Finance, the manager of the evaluation, and two recent Program Advisors of the Program. ## 3. Methodology The methodology for the evaluation was developed based on the Program’s logic model, which is provided in Annex A. The logic model illustrates the expected results chain for the APAP, and shows how the Program’s activities and outputs are expected to lead to the outcomes. The assessment of relevance focused on the extent to which the Program continues to meet the needs of participating organizations and their respective deputy heads. The performance assessment focused on the extent to which it has achieved its short-term and long-term outcomes. The evaluation also examined the Program’s design and structure, specifically its decentralized and voluntary-based administration and internal processes, including recruitment activities, assignment periods (i.e., the four six-month assignments), assignment composition (three central agencies and one line department), and participant performance evaluation and promotion. The focus of the assessment was to determine whether the overall design of these processes were appropriate and contributing to the achievement of the Program objectives. The evaluation addressed the following questions: • To what extent is there still a continued need for the Program? In what way, and to what extent, does the Program meet the needs of the participating Departments? • To what extent is the Program aligned with the roles, responsibilities and priorities of the federal government? • To what extent has the Program been able to achieve its objectives and expected outcomes? • Has the implementation of the Program resulted in any significant unintended outcome? • What are the key strengths and weaknesses of the Program? • To what extent do the existing structure, model and internal processes support the effective execution of the Program mandate and the achievement of its objectives? • Are there any changes to the program design that could help better achieve the objectives? • To what extent is the Program implemented and delivered both efficiently and economically? ### 3.1 Data Collection Methods The following four data collection methods were used to address the key evaluation questions. Document and literature reviews: The evaluation team reviewed various documents, including background and administrative documents, as well as some documents on similar programs and other relevant literature, such as annual reports from the Clerk to the Prime Minister on the Public Service. The information collected through document review was used to develop the Program’s profile, to gain insight into its operations, as well as to assess its relevance, efficiency and economy. Interviews: A total of 42 interviews were conducted. 34 of these in-depth interviews were with the Program Steering Committee members, past and present participants, administrators, advisors and supervising managers. Eight additional interviews were conducted with HR advisors and the coordinators of other EC recruitment and development programs as part of the comparative study. Efforts were made to ensure that the interviewees represented a wide range of departments and perspectives (e.g., gender, language, position, cohort) to capture the diversity of the opinions and experiences that existed. Surveys: Three online surveys were developed and administered for the purposes of this evaluation. The surveys were issued to the following groups: • APAP participants – Selected using the complete list of program participants (1986–present) received from the APAP Secretariat. Individuals being interviewed for the evaluation, as well as those with no email address on file, were removed from the survey list. 205 survey invitations were issued to this group, 131 (63.9%) complete survey responses were received and used in the analysis. • Comparable Non-participants in the Public Service – Selection of the non-participant group began with a random selection covering every year from 1988 to 2013. For each year, public servants who were at the EC-05 (ES-04 and SI-05) occupation group and level were randomly selected from the total EC population of that year. The final group of invitees consisted of individuals for whom contact information was available. Respondents were then filtered to include only those with a graduate degree. 502 survey invitations were issued to this group, 163 (32.5%) complete survey responses were received and used in the analysis. • Supervisors of APAP candidates and graduates – Managers who had previously supervised APAP candidates during the rotational assignments or upon graduation were selected from a list provided to us by the APAP Secretariat. 29 survey invitations were issued to this group, 14 (48.3%) complete survey responses were received and used in the analysis. The survey results complemented findings from other lines of inquiry and were used to identify areas of strength and weaknesses with the design and administration of the Program and important opportunities for improvement. Comparative Analysis: The evaluation included a comparative analysis2 of the Program against other recruitment and development programs in the federal government. The intent of the exercise was to identify best practices and draw potential "lessons learned" to improve the efficiency and effectiveness of the Program. To this end, an analysis was conducted to identify and assess similarities and differences in the administration, governance and internal processes, including recruitment, evaluation and promotion. Findings from the study were incorporated into the overall evaluation findings. ### 3.2 Limitations The evaluation methodology had the following three limitations: • The participant survey was intended to reach all Program participants, past and present. Unfortunately many of individuals were unreachable, particularly those who had left the public service. This could introduce a bias in the participant survey results since the perspective of these individuals is not included. However, the evaluation mitigated this limitation by interviewing Program graduates who had left the public service in order to capture their perspectives. Thus, this limitation had minimal impact on the evaluation results. • Only 14 responses were obtained for the supervisor survey. This low number increases the possibility that supervisors’ viewpoints may not be adequately represented. This limitation was mitigated by in-depth interviews with supervising managers, and the fact that some of the Alumni Advisory Board members that were interviewed were also supervising managers. The answers to the questions asked in the interviews converged with the key findings, indicating that there was sufficient information to validate the findings and conclusions in this report. • The availability of financial information was affected by the decentralized nature of the Program. With no central office through which funds flow and costs are accounted for, information on actual program costs was assembled using information received from the program secretariat, as well as other sources, such as Steering Committee briefings, and pay rates in EC collective bargaining agreements. In addition, the program costs presented in this report do not capture costs associated with volunteer activities, which often took place during work hours, since these costs are not tracked and attributed to the Program. The findings presented in this report were reached while giving due consideration for the completeness of the available financial information. ## 4. Evaluation Findings The following section presents the main findings of the evaluation issue. Multiple lines of evidence were used to formulate the findings, conclusions and recommendations. ### 4.1 Relevance The origin of the APAP can be found in a junior analyst training program that was run by the former Ministry of State for Economic and Regional Development in the 1980s. Although the Program has since gone through changes in its administration, its main objective to prepare recruits with a high potential to excel in policy and leadership roles in the Government of Canada has remained the same. Therefore, for the Program to be relevant it must be established that there is a continued need in the federal government to recruit and develop high potential analysts; and, that the Program is consistent with the human resources priorities of the Government of Canada. Thus, answers to the following questions were specifically sought: To what extent is there still a continued need for the Program? In what way, and to what extent, does the Program meet the needs of the participating Departments? Finding: The Program addresses a continued need for knowledgeable policy analysts. It enables the government to recruit individuals with high-potential and prepares them with the experience and skills to excel in policy and leadership roles across the federal government. The continued need for recruiting and training policy analysts has been highlighted in recent years by public policy experts. In particular, it is noted that although policy advice to the government is gathered from a variety of sources, through non-governmental organizations, academic institutions, banks, business associations and think-tanks, the civil service serves the public by sifting through the policy research to weed out the weak and ineffectual ideas. To this end, the government needs to hire highly capable policy specialists that can analyze challenging economic and social issues while taking into account the complexity of the world today.3 Don Drummond, a former Assistant Deputy Minister at the Department of Finance Canada, as well as former Chief Economist at the Toronto Dominion Bank, provides compelling reasons in support of augmenting the federal government’s capacity for policy analysis, including that it mitigates the risk of losing inside government champions of good policy ideas, and that it is needed "for the sake of delivering solid policy."4 A lesson learned from previous rounds of expenditure management reviews was that the government cannot stop recruiting activities entirely even during times of spending reduction, as there is always a need to recruit and train new analysts to replace those who are retiring or leaving the Public Service. In 2009–10 the hiring rate for the Economics and Social Science Services occupational group (EC) was 7.4% of its total population. In 2012–13, this rate was reduced to 0.7% of the total EC population5. This represents a significant reduction in EC hiring activities. During the same period, the EC’s attrition rate increased from 3.9% to 7.6% of the total EC population. Thus, with more ECs exiting the core public service than being hired, it is important that new recruits are highly competent so that the federal government’s policy analysis capacity remains strong. The APAP is one of the only recruitment and development programs dedicated to contributing to maintaining a strong policy analysis capacity in the federal government. A large number of the interviewees and survey respondents agreed that the Program addresses a tangible need in the government. In the survey responses, 13 out of 14 supervisors and over 97% of Program participants felt that the need for the Program has either stayed the same or increased over time. Some of the reasons that were given include: • There is no other program that provides the same level of exposure to government operations and machinery—this depth of knowledge is recognized as extremely useful during the development of policy. • The Program played an important role in the participants’ decision to join the government. For example, 18.5% of participant survey respondents stated that they probably would not have considered the federal government as a career choice if they had not heard of the Program. • The government’s focus on employee performance issues makes the Program more relevant, since the Program has a track record for developing skilled and high performing analysts. • The number of deputy heads that have signed the latest memoranda of understanding (MOU) has increased, demonstrating increased demand for Program graduates, and increasing deputy head interest in the Program. Are the Program’s mandate, objectives and activities consistent with government’s roles, responsibilities and priorities? Finding: The Program’s mandate, objectives and activities are consistent with government priorities, roles and responsibilities. In particular, the Program is in line with the Public Service Renewal Action Plan and current human resources initiatives. A review of relevant literature, including the Public Service Renewal Action Plan, identified "targeted and coordinated recruitment" and "employee development through systematic and integrated approach" as two pillars of the public service renewal.6 The former Clerk of the Privy Council, on numerous occasions including in his Blueprint 2020 report, encouraged federal departments to find innovative ways "to ensure that the Public Service has the competencies and leadership skills needed to harness the best talent and the brightest ideas, wherever they may be found, to meet the evolving needs of Canadians."7 These innovative approaches to recruitment and development must be carried out in a manner that respects the government’s initiative on human resources management in the Public Service, which recognizes that deputy heads have the primary responsibility for human resources management and provides them with flexibility to respond to their specific business needs.8 The APAP recruitment process is a targeted and coordinated approach that aims to recruit capable, high-performing individuals who have exhibited leadership qualities and are equipped to address complex policy issues. The APAP is organized as a government-wide approach that is known to attract and retain high-caliber policy analysts. Each year, hundreds of qualified candidates apply for 8-12 positions. Roughly 70% of graduates of the Program stayed within the public service for the remainder of their career. The APAP employs a systematic and integrated approach to develop the competencies and leadership skills necessary for high quality policy analysis. Survey responses from supervisors of APAP graduates indicated that the experience gained during the Program was what provided the participants with the competencies and skills required for high quality policy analysis. As well, the current design of the Program is in line with the spirit of the government’s directive on human resources management. Deputy heads formally express their organization’s desire to participate in the Program by signing the Program MOU. The number of organizations willing to sign the MOU has increased, further demonstrating that the Program is responding to deputy heads’ specific business needs. ### 4.2 Performance #### 4.2.1 Effectiveness: Achievement of the Expected Outcomes To what extent has the Program been able to achieve its objectives and expected outcomes? Has the implementation of the Program resulted in any significant unintended outcome? Finding: The Program has been successful in achieving the two immediate (short-term) and the two intermediate (long-term) outcomes. A relatively high proportion of Program participants have advanced to management and leadership roles in the government. Short term outcomes: The Program has been successful in achieving the two immediate (short-term) outcomes: 1. Participants in the program gain the expected knowledge and skills. 2. Central agencies and federal organizations employ analysts who have the required knowledge for policy work. Almost all the interviewees were satisfied with the Program’s performance and believe that the Program has been successful in recruiting high potential individuals and training them for policy roles in the federal government. 88.1% of participant survey respondents felt they were well-equipped to conduct policy analysis. The survey results also clearly showed that supervisors were satisfied with participants’ attainment of knowledge and skills in the key aspects of the federal government. A number of participants stated in interviews that after graduating from the Program they were often singled out and requested to work on key files because of their understanding of central agencies’ roles and processes. This valuable understanding is usually obtained much later in a career, if at all. Supervisors of graduates stated in both interviews and in the survey that, when compared to other employees, APAP graduates exhibit greater knowledge of the overall functioning of government; the roles and responsibilities of central agencies; and the expenditure management system. Supervisors highlighted that it is the experience gained during the Program, particularly the assignments with the three central agencies that provided participants with the knowledge and skills conducive to high quality policy analysis. Interestingly, fewer supervisors felt that the participants were exposed to the expenditure management process as compared to other elements of the Program, such as the overall functioning of government. Similarly, participants indicated less satisfaction, relative to other elements, with their exposure to and understanding of the expenditure management process. The six-month assignments also offered opportunities to hone soft skills. Survey responses and interviews indicated that most of the supervisors felt that the assignments did help the candidates to improve soft skills such as analytical thinking and judgment, effective workload management, people skills and collaboration. Most surveyed APAP participants agreed that the assignments helped them to a great extent in this regard. However, from the supervisors’ perspectives, there was no noticeable difference between APAP participants and other employees in terms of soft skills. The alumni network is a strong feature of the Program. There is a central list of participants, both current and past that is updated on a regular basis. The network is fairly cohesive, in that the numerous social events held for APAP alumni are usually well-attended. For most of the participants in the survey, the alumni network is considered an excellent or good feature of the Program. For a little over half the survey participants, the network has been conducive to their career and improved their understanding and knowledge; but for most of the others, this has not been the case. Interviews with program managers and participants have noted that the network could be used more effectively. For example, in the past, the alumni had organized a large Policy workshop that was well-attended and appreciated. It was noted that more of these work-related gatherings could be arranged if members would be willing to take on the task of organizing them. Nonetheless, the balance of evidence indicates that program participants are graduating with the expected knowledge and skills; and that organizations are recognizing these abilities. Therefore, the evaluation concluded that the Program has been successful in achieving its immediate (short-term) outcomes. Longer term outcomes: The Program has also been found to be successful in achieving the two intermediate (longer-term) outcomes: 1. Highly trained and qualified analysts who excel in policy roles in the Government of Canada. 2. Increase in federal organizations’ access to highly trained and qualified policy analysts with the ability to lead. Program graduates were found to be excelling in their roles. Compared to those who did not participate in the program, a significantly greater proportion of APAP participants advanced to management and leadership roles in the government. 34.9% participant survey respondents reached the executive (EX-level); while 21.8% comparable non-participant survey respondents reached the executive level during the same period, a difference that is statistically significant at the 5% level. Amongst respondents in the EX-01 category, APAP participants took an average of 8.3 years to reach the EX-01 level from the EC-05 level, while it took comparable non-participants an average of 11.0 years to reach the same level. Leadership training is embedded in the various aspects of the Program. For example, participants must demonstrate leadership skills and experience during the recruitment process in order to be screened in. Additionally, the various assignments in the Program allow participants to be exposed to different leadership styles at senior levels of government, for example at Cabinet committees, and this also contributes to the participants’ on-the-job leadership training. Almost all of the interviewees and survey respondents expressed their satisfaction with the Program’s performance. 88.5% of APAP graduates stated that they were satisfied with their career advancement since graduation; and 93.0% felt that the Program facilitated their career. Meanwhile, 64.7% of non-participants were satisfied with their career. The networking opportunities furnished by the Program were described to be an important factor in the graduates’ career success and advancement. The Program and its graduates were also described as having a well-established reputation within the federal government. Almost all the supervisors (survey respondents and interviewees) indicated that they would recommend the Program to other managers. With this evidence, it is concluded that the Program has been successful in achieving the two intermediate (longer-term) outcomes. ### 4.2.2 Effectiveness: Program Design and Delivery • What are the key strengths and weaknesses of the Program? • To what extent do the existing structure, model and internal processes support the effective execution of the Program mandate and the achievement of its objectives? • Are there any changes to the program design that could help better achieve the objectives? Finding: The current structure and internal processes were found to be appropriate and contribute to the achievement of the Program objectives. Opportunities for improvement were found with four aspects of the Program: the complexity of the funding structure, the inconsistent quality of the six-month assignments, the obligation to return to the sponsoring organization, and the lack of succession planning for key positions within the administration. In 2006, the Public Service Modernization Act led to the devolution of certain key staffing responsibilities to deputy heads. Following this policy change and subsequent strategic review, the government eliminated all centrally funded development programs below the executive level. Without a home organization, the Program was slated to be phased out. In an effort to preserve the Program while complying with the government’s directive, a decision was made to decentralize the Program administration amongst participating organizations. Although this signified a profound change to the Program, many of its characteristics and internal processes, including recruitment, rotational assignments, evaluation and promotions, either remained unchanged or were slightly modified to reflect the new structure. In assessing the design and delivery of the Program, the evaluation examined the extent to which the current structure and internal processes are appropriate, and to what extent they have been contributing to the achievement of the Program’s objectives. Overall Program design and administration: The interview and survey results indicated that there is a high level of satisfaction with the current Program design and administration among all stakeholders. The transition to the decentralized model was found to have contributed to the Program’s preservation and continued success. The strengths of the centralized model that made the Program successful, in particular the rotational assignments, have been preserved under the new model. In addition, the establishment of a deputy head-level oversight committee and a greater level of alumni and participating organizations’ involvement in the administration of the Program are factors that have contributed to further strengthening the current model. The creation of a full-time program secretariat has improved the effectiveness of the Program, while reducing the workload of program advisors and other volunteers. Concerns had been raised that the decentralized model, with the heavy reliance on volunteers, may be unsustainable over the long term, and that a program of this nature and importance would be better run by full-time professionals. However, returning to a centralized model is not a viable option at this time, since it would be counter to current government human resources directives. Moreover, it is recognized that returning to a centralized model would make the Program more vulnerable to external influences. In its current form, the administrative burden is shared across many organizations, which helps isolate the Program from organization-specific pressures and changing priorities. The evaluation concluded that the current program design and administration are appropriate. Program funding structure: The funding structure was found to be functioning appropriately. However, there is an opportunity to streamline the administration of Program funds. Program funding is based on a 50/50 cost sharing ratio between sponsoring organizations and central agencies. Most interviewees indicated that this split and related administration is appropriate, albeit complex. The complexity of the administration is due to the lack of a central office through which funds flow and costs are accounted for. The current funding model requires a number of low-value transfers and cost recoveries between organizations. For example, funding a participant in the Program requires: a one-time transfer of funds from the sponsoring organization to the organization responsible for recruiting; three annual transfers of15K, one from each central agency, to the sponsoring organization; and, a year-end transfer from the sponsoring organization to the APAP Secretariat, to cover administration costs. This amounts to an average of nine transfers per participant over the two years they are in the Program.

A few sponsoring organizations raised concerns that the central agency funds arrived too late in the year to use, and therefore the funds were lapsed. Meanwhile, others did not appear to have this issue. The organizations that did raise the issue also acknowledged that it was not a significant enough problem to affect their participation in the Program.

The APAP Secretariat has taken steps to facilitate the planning and administration of the fund transfers. For example, the Secretariat sends out financial coding information to all participating organizations to initiate the central agency transfers. The correspondence is sent out in early July when participants are matched with their sponsoring organizations.

Nonetheless, there is an opportunity to facilitate the sponsoring organizations’ planning and administration by further streamlining the Program’s funding structure.

#### Recommendation 1

To facilitate the planning and administration of funds, Program administrators should explore options to improve the timing of fund transfers and cost recoveries, as well as to reduce the overall complexity of the funding structure.

Recruitment and screening processes: The recruitment process was found to be rigorous and efficient. Although recruitment activities have been conducted by different organizations, the process has remained relatively unchanged over the years. This has been identified as one of the important factors contributing to the Program’s positive and well-established reputation in universities across the country and among hiring managers in the federal government. The arrangement, under the current model, to transfer the responsibilities for recruitment and screening to different organizations every two years was described to be working well. The postmortem exercise that usually follows the completion of each recruitment campaign provides the opportunity to identify and share lessons learned and best practices, which has contributed to improving the efficiency of the recruitment processes over the years.

It should be noted that despite recent efforts to reach out to universities across the country, a large proportion of participants, 68.2%, are from universities in the Ontario and Quebec regions. Meanwhile, the proportion of survey participants who identified their first official language as English was approximately 75%, while 25% identified French as their first official language.

Prerequisites: The enrollment prerequisites, including the level of concentration on courses in economics, were found to be appropriate overall. Although the majority of the interviewees and survey respondents found the requirement to have a master’s degree to be appropriate, there were a few interviewees who indicated that, given the type of work and functions performed by Program graduates, a bachelor’s degree may be sufficient. The majority who commented on the economic course requirements stated that knowledge of economics was important for policy-making.

Assignments: The six-month assignment duration and the composition (i.e., three central agencies and one line department) were both found to be appropriate. The assignments at the three central agencies are considered the main strength of the Program. The comparative study found that this was a feature of the Program that other development programs would like to emulate.

However, the quality of the assignments was found to be inconsistent and therefore has been identified as an opportunity for improvement. Inconsistent assignment quality was raised in both interviews and in the survey results. This finding was also raised in previous assessments of the Program, for example, a survey conducted by the Privy Council Office in 2012.

Comments provided by participants indicated that at least one out of their four assignments was of low quality, which meant that either the work was not found to be sufficiently challenging, or did not meet the participants’ development expectations. 60.0% of the participants surveyed were somewhat satisfied or not satisfied with the tasks given during the placement. Assignment quality, particularly at the central agencies, was often related to the timing of the assignment. The cycle of organizational and government activities lends itself naturally to slower periods in the summer and more intense sessions during the fall or winter. In addition to the timing of the assignment, the degree of the assignment supervisors’ readiness and preparation to host a participant was an important factor in shaping the quality of the participant’s experience. With respect to this issue, organizations that had been involved with the Program longer and had established clear roles, responsibilities and assignment routines have provided participants with more satisfying experiences. Some of the supervisors in other organizations noted that if they were informed earlier about the new candidate, for example the name, education, or other background information, they would be better able to prepare appropriate tasks for the recruits.

The success of the Program depends heavily on the quality of the assignments, particularly since these assignments are often the first federal government work experiences for the new recruit. Low-quality assignments can affect participant retention and the reputation of the Program as well. In order to promote consistent high-quality assignments, the following recommendations are put forward:

#### Recommendation 2

To maintain high-quality assignments, Program administrators should consider:

1. Providing assignment supervisors with the background information of their assignees early in the process, allowing them to arrange for suitable and high-quality assignments.
2. Establishing a method of tracking assignment quality, such as brief post-assignment surveys to be completed by the participant, in order to identify lessons-learned and opportunities for improvement.

Evaluations and Promotions: Based on comments received from the majority of the interviewees and survey respondents, the participants’ performance evaluations and subsequent promotions were found to be carried out rigorously and consistently. Consistency has improved in recent years as a result of various measures taken, such as developing and implementing the use of common evaluation templates. To that end, some of the supervising managers described the evaluation templates to be too long and described the whole evaluation process as laborious. Similar practices in other development programs were examined in the comparative study, and in general, it was found that the performance evaluation practices in development programs, particularly for the purpose of a promotion, tend to be lengthy processes in order to comply with the human resources management requirements of the federal government.

With respect to promotions, the APAP was found to be the most accelerated program amongst the development and recruitment programs included in the study. First of all, the APAP recruits individuals at the EC-03 level, while most other EC development programs recruit at the EC-02 level.9 The APAP is also the only program that has a defined duration of two years. The other programs have no firm preset graduation timeframe, instead program participants’ progression is determined by their ability to develop the appropriate competencies and meet the graduation requirements at their own pace.

Returning to the sponsoring organization: The requirement for participants to return to their sponsoring organization upon graduation has been a source of contention.10 29 out of 49 participant survey respondents indicated that they were not satisfied or only somewhat satisfied with this requirement. Program documents indicate that since the introduction of the sponsoring organization model, at least one participant per cohort has not fulfilled their commitment, either by never joining the organization, or by leaving before they had fulfilled their one year commitment. Thus, opportunities for improvement have been identified for this aspect of the Program.

Since 2009 and under the current funding model, organizations pay recruitment costs and approximately 50% of salary and administration costs (net of fund transfers from the central agencies) for the two years their sponsored participant is in the Program. In return, after graduation, participants are expected to work at their sponsoring organization for at least one year. This arrangement was described as both a strength, since it provides participants with job security, and as a weakness, because it limits participants’ employment options.

While acknowledging the logic behind having a sponsoring organization, the majority of post-2009 participant interviewees indicated that the requirement to return to the sponsoring organization was not clearly communicated to them during their recruitment. Some expressed resistance to this aspect of the Program, since it was a relatively new requirement, having not existed prior to 2009. Participants added that at the time of recruitment, they had limited knowledge of the government operations. As a result, their sponsoring organization preferences changed while they were in the Program, particularly as the central agency assignments presented them with new ideas and opportunities.

Participants commented in the interviews and in the survey that sponsoring organizations have not always made the effort to establish on-going relationships with their participants, or make their future job relevant and attractive to them. In fact, some had very little contact with their sponsoring organizations during the rotational assignments. In some cases, participants apparently returned to their organizations to find that the job awaiting them was not commensurate with the knowledge and skills they had acquired while in the Program. On the other hand, other participants reported that their sponsoring organizations had kept in close contact with them, resulting in them being more engaged and willing to return upon graduation.

The premise of the Program is that through the rotational assignments, participants are exposed to the government machinery, gain experience in policy development/analysis and then take this experience with them to their sponsoring organizations. A breakdown of this model would undermine the Program’s value proposition, since returning to the sponsoring organization is necessary to justify the sponsoring organizations’ investment (dollars and time) in the Program. Since the Program’s continuation depends on finding a workable solution to this issue, the evaluation is recommending the following steps be taken to bolster the participants’ willingness to return to their home sponsoring organizations.

#### Recommendation 3

To help promote a positive transition back to the sponsoring organization, Program administrators should consider the following. Some of these steps have been outlined in the latest MOU (2014-19). However, the implementation of these should be monitored annually.

1. Clearly communicating to the participants, at the outset, the requirement to return to one’s sponsoring organization.
2. Encouraging the sponsoring organizations to foster a closer and more proactive relationship with the sponsored participants as outlined in section 3, Schedule B of the 2014-19 MOU.
3. Encouraging sponsoring organizations to take the participants’ interests more into consideration when preparing the three post-graduation job offers, as outlined in section 6 and 7, Schedule B of the 2014-19 MOU.

Level of oversight and governance: The level of oversight provided by the APAP Steering Committee was found to be appropriate. The members of the Steering Committee are deputy head-level or equivalent, providing an unusually high level of oversight for a program of this size. The evaluation examined whether this level of oversight was necessary, and whether this would impede the ability of the Program administrators to bring specific issues to the Committee’s attention, given the members’ positions and busy schedules. The majority of the interviewees believed that oversight at the deputy head level is important for establishing the Program’s relevance and necessary for the Program’s survival, since it underlines the strategic importance of the Program. Additionally, the actual time commitment required from the Committee members was described by them as ‘manageable’.

The development and approval processes for the Program’s memoranda of understanding (MOU) are complex but working well. The MOU defines and clarifies the roles, responsibilities and expectations of those involved in the Program. It also outlines the Program’s administration and funding structure. Deputy heads of organizations that wish to sponsor participants in the Program must sign the MOU. Although the five-year renewal cycle creates some uncertainty regarding the future continuation of the Program, it was found to be beneficial as it allows the participating organizations to assess if the Program continues to respond to their needs. It was noted that the MOU may be amended within the five-year cycle when changes to the Program’s administration or structure are needed. In such cases deputy heads sign the revised MOU to confirm their commitment to the Program, and their agreement with the changes.

Volunteers: The Program’s reliance on volunteers to administer its affairs was described both as a source of strength and weakness of the current model. There is an acute understanding among various stakeholders of the need for volunteers in the administration of the Program. Yet, some concerns were raised about overreliance, particularly in the Program’s core administration. Although there has never been a shortage of volunteers and it is widely believed that volunteerism brings a strong sense of ownership to the Program, reliance on volunteers could become a problem during periods of fiscal restraint when organizational resources are scarce.

Some interviewees raised concerns about the Program’s strong reliance on certain individuals rather than institutions. This situation, particularly in the case of sudden departure of these key individuals, was argued to impose a risk to the ongoing smooth functioning of the Program. The Program’s administrators are aware of these issues and have been progressively putting in place measures to mitigate the risk and to minimize its impact on the day to day functioning of the Program. The establishment of a full-time secretariat was described as one of these measures in institutionalizing the Program’s administration. Still, given its importance, further institutionalization of certain administrative practices, such as more systematic succession planning for key administrative positions would be beneficial. Hence, this presents the Program with an opportunity for improvement.

#### Recommendation 4

To promote seamless functioning of the Program in the future, Program administrators should consider putting in place a succession plan for key positions such as the Executive Advisor and the Organizational Coordinators.

#### 4.2.3 Efficiency and Economy

To what extent is the Program implemented and delivered both efficiently and economically?

Finding:

The Program was found to conduct its operations and deliver its activities efficiently and economically. Since 2009, the Program has reduced its administration costs, largely as a result of its decentralized structure.

Table 1
Program Expenditures and Number of Participants by Fiscal Year
FY 2010–11 FY 2011–12 FY 2012–13 FY 2013–14
Participant Salary $1,249,524$ 1,350,093 $1,174,543$ 1,159,973
Administration costs $117,132$ 164,580 $28,895$ 66,240
Recruitment costs $80,000$ 68,000 $55,728$ 67,000
Total program costs $1,446,656$ 1,582,673 $1,259,166$ 1,293,213
Number of participants in program (full time equivalents - FTE) 19 FTE 19.75 FTE 17 FTE 16 FTE
Average cost per participant $76,140$ 80,135 $74,069$ 80,826

Overall, the Program was found to conduct its operations and deliver its activities efficiently and economically. The table and graph above present the Program expenses from fiscal year 2010–11 to 2013–14. Over the past four years the average cost per participant has been relatively stable, ranging between $74–80K, while total program costs have trended downwards. Administration costs were reduced from$117K in 2010­–11 to $66K in 2013–14, this decrease reflects steps taken to lower program costs. For example, in 2011–12 the program stopped using a service provider for its administration; and orientation and graduation ceremonies began being delivered by a designated organization using existing departmental resources, such as boardrooms, rather than renting from external sources. In 2012–13 administration costs may have been higher than the figures presented since some costs associated with the newly formed APAP secretariat were not attributed to the APAP, and instead absorbed by its host department. In 2013–14, the APAP secretariat costs were recovered from participating departments. Recruitment costs over the past four years were reduced from$80K to \$67K. Electronic tools and technologies, such as video conferencing, are now being used and have helped reduce travel costs associated with recruitment and screening.

From 2010–11 to 2013–14, the full-time equivalents (FTE) in the two-year program has been reduced from 19 FTE to 16 FTE. With the current MOU signed by 16 departments, there may be a need to increase the number of participants in the program. Analysis suggests that a small increase in cohort size would not significantly impact administration and recruitment costs. Cohort size is more likely to be limited by the availability of quality assignments in the central agencies.

Finally, since decentralization, there has been an increased use of volunteers to support recruitment and other administrative aspects. 47.6% of participant survey respondents stated that they had been a volunteer at some point since graduating from the program. The reported time spent volunteering varied considerably. Some survey respondents volunteered only a few times during the year, while others volunteered for 2-3 hours a week. Some estimated they worked about 200-300 hours annually. Although volunteer work has helped keep program costs low, the work is often carried out during work hours (not on personal time), but without the associated costs being attributed to the APAP. The figures in the table above do not include these expenditures, since accurately tracking and assessing these costs was not feasible.

## Annex B: Comparative Study

The evaluation included a comparative analysis of five other EC recruitment/development programs in the federal government.11 The analysis focused primarily on identifying and assessing similarities and differences in the various aspects of the programs, covering administration, governance, and internal processes.

The following is a high level summary of the study’s findings:

Comparative Study
Key Aspects of the Program Similarities
Elements that the APAP has in common with other programs
Differences
Elements that distinguish the APAP from other EC recruitment/development programs
Other Key Observations
Program Characteristics
• All have the objective of attracting and retaining EC policy analysts.
• All require a minimum of a university degree and some courses in certain specialized areas, but the APAP and at least three other programs require a Master’s Degree.
• The APAP and four other programs recruit primarily through the University Recruitment Campaigns.
• All have performance-based evaluation and promotions in place.
• All have program oversight at senior management level.
• All have similar labour intensive HR services and needs.
• The APAP is interdepartmental in its scope.
• The APAP is the only program with a deputy head-level oversight committee.
• The APAP is the only program that offers rotational assignments in central agencies to its participants.
• Recruitment responsibilities for the APAP are rotated every 2 years.
• Despite the similarities, the APAP is not duplicating any of these programs.
• The APAP targets a talented and motivated group of university graduates with more selective recruitment and rigorous enrolment criteria.
• The rotational assignments component, particularly at the central agencies, is one aspect of the APAP that other programs would have liked to replicate in order to complement their programs.
• Transferring the responsibility of providing HR services to a different organization every two years allows different organizations to gain experience and develop expertise in APAP recruitment. There is a great level of interest and willingness on the part of these departments to assist each other by sharing experiences, tools and lessons learned.
Entry level and program duration
• None
• The APAP recruits at EC-03 level, while most other programs hire at EC-02 level (The Recruitment of Policy Leaders program hires at the EC-05 level).
• The APAP has a two-year defined duration, while in the other development programs recruits generally progress at their own pace.
• The APAP is the most accelerated program. It takes 2 years for the APAP participants to advance from EC-03 to EC-05. Meanwhile, in all other development programs it generally takes 5 years to advance from an entry level of EC-02 to EC-05 position.
Evaluation and Promotion
• All programs have well-structured processes in place for evaluation and promotion of their program participants.12
• All programs require that participants’ performance assessments and recommendations for promotions be assessed and endorsed by a second assessor (usually a management committee) to maintain rigour and consistency.
• Promotion is not guaranteed in any of the programs and most programs have a process in place to address under-performance on a case by case basis.
• APAP and at least three other programs have a recourse mechanism in place to address grievances and intractable issues.
• The APAP requires a board interview for each participant midway through the program.
• APAP candidates are evaluated on their performance after each assignment by a different manager. By the time they have completed the Program, their performance has been evaluated by four different supervisors.
• Given the accelerated nature of the APAP promotions, moderating the participants’ advancement expectations beyond their graduation appears to be a challenge.

## Annex C: Management Response & Action Plan

Management Response & Action Plan
Recommendation Management Response Planned Action Lead Target Date
Recommendation #1:
To facilitate planning and administration of funds, Program administrators should explore options to improve the timing of the fund transfers and cost recoveries, as well as reduce the overall complexity of the funding structure.
Management agrees with this recommendation.
1. Financial settlements with the Central Agencies will take place in Q2 of each fiscal year to ensure consistency for internal planning and financial forecasting.
2. The program will also explore options, within the framework of current government human resource directives, to streamline the program’s funding model.
2. APAP Steering Committee Chair
1. Q3 2014-15
2. Q4 2015-16
Recommendation # 2:
To maintain high-quality assignments, Program administrators should consider:
1. Providing assignment supervisors with the background information of their assignees early in the process, allowing them to arrange for suitable and high-quality assignments.
2. Establishing a method of tracking assignment quality, such as brief post-assignment surveys to be completed by the participant, in order to identify lessons-learned and opportunities for improvement.
Management agrees with this recommendation.
1. The APA Program will review its procedures with respect to establishing and communicating the assignment schedule, with a view to establishing best practice guidelines and associated time frames in sharing and finalizing information about assignments.
2. The APA Program will pilot and implement post-assignment surveying. Outcomes will be shared with member departments on an annual basis.
1. Q3 2015-16
2. Q3 2015-16
Recommendation # 3:
To help promote a positive transition back to the home department, Program administrators should consider the following. Some of these steps have been outlined in the latest MOU (2014-19) for the home department. However, the implementation of these should be monitored annually.
1. Clearly communicating to the participants, at the outset, the requirement to return to one’s home department.
2. Encouraging the home departments to foster a closer and more proactive relationship with the sponsored participants as outlined in section 3, Schedule B of the 2014-19 MOU.
3. Encouraging home departments to take the participants’ interests more into consideration when preparing the three post-graduation job offers, as outlined in section 6 and 7, Schedule B of the 2014-19 MOU.
Management agrees with this recommendation and confirms that (a) and (b) are already being done. The APA Program has taken steps to address the home department transition through changes to the 2014-2019 APAP MOU.

The program will continue to work with member organizations, trainees, and alumni to identify and implement further improvements to the home department-trainee relationship, including guidance materials, documenting and sharing best practices, and mentorship opportunities.
• Monitoring of progress will occur annually, at a minimum.
• Non-compliance by member departments with the terms of the MOU will continue to be reported to the DM Steering Committee.
APAP Alumni Advisory Board Q3 2015-16
Recommendation # 4:
To promote seamless functioning of the Program in the future, Program administrators should consider putting in place a succession plan for key positions such as the Executive Advisor and the Organizational Coordinators.
Management agrees with this recommendation. The APA Program Secretariat will explore options to ensure key program positions have a succession strategy in place. APAP Executive Advisor Q2 2015-16

1 This is in accordance with the 2014 MOU. Under the previous MOU, candidates were hired at the EC-02 level and advanced to EC-03 after six months.

2 Key findings from the Comparative Analysis can be found in Annex B.

5 Source: Information received from TBS on EC inflow and outflow activities in the Core Public Administration.2012-13 is the last year for which this information was available at the time of this evaluation.

6 2010-11 Public Service Renewal Action Plan, page 1.

7 Blueprint 2020 Report, page 3.

9 The exception being the Public Service Commission run recruitment program called the Recruitment of Policy Leaders Program, which recruits top level professionals at the EC-05 level.

10 The survey conducted by PCO in 2012 found similar problems with the obligation to return to the home department.

11 The five programs are: (1) Treasury Board Secretariat’s EC Analyst Development Program; (2) Public Service Commission’s Recruitment of Policy Leaders; (3) Finance Canada’s EC Analysts Program; (4) Agriculture and Agri-food Canada’s Policy Analyst and Economist Development Program; (5) Industry Canada’s Industry Sector EC Development Program.

12 This is not applicable to the Recruitment of Policy Leaders program.