ARCHIVED—Review of the Acquisition Process to Replace Canada's CF-18 Fleet

Final report

November 14, 2013

Table of Contents

• Introduction
  • Objective
  • Scope
  • Approach / Methodology
  • Review Limitations
• Major military acquisition
  • Overview of the Acquisition Process
  • Policy Framework
  • CF-18 Replacement Acquisition Project
• The joint strike fighter program
• Lessons learned and recommendations
  • Alignment of the cf-18 Replacement Project and the Joint Strike Fighter Program
    • Applicability of the Policy Framework
    • Recommendation #1
  • Project Management Capability
    • Recommendation #2
  • Statement of Requirement
    • Recommendation #3
  • Roles and Responsibilities
    • Recommendation #4
• Shortcomings identified by the auditor general of canada
• Conclusion
• Appendix A: Stakeholders Interviewed
• Appendix B: Overview of Process and Key Players
• Appendix C: Policy Framework
• Appendix D: Department of National Defence Organizational Structure
• Appendix E: Summary of Recommendations

Introduction

Objective

The objective of this engagement was to review the steps taken in the acquisition process for the replacement of the Department of National Defence (DND)'s CF-18 fleet up to June 2012; to provide lessons learned; and to propose recommendations for improvements to the current practices and policies to increase the level of due diligence for any future acquisitions of similar nature, risk, and/or complexity.

Furthermore, reviewers assessed whether the acquisition process shortcomings identified by the Auditor General of Canada have been addressed.

Scope

The review included an examination of the steps taken by Public Works and Government Services Canada (PWGSC) and DND in the acquisition process for the replacement of Canada's CF-18 fleet up to June 2012. We also examined the project activities undertaken by DND in the planning stages of the procurement life cycle and its participation in the Joint Strike Fighter (JSF) Program.

Our review specifically excluded revisiting the work conducted and the facts established by the Auditor General up to June 2010 and any work related to the assessment of the full life cycle costs of the F-35.

Approach / Methodology

Key stakeholders in DND and PWGSC involved in the CF-18 replacement program were interviewed. In addition, interviews were conducted with officials from the Treasury Board of Canada Secretariat (TBS), the Privy Council Office (PCO), and Industry Canada (IC). A list of interviewees is presented in Appendix A.

Available documentation was reviewed to understand the processes undertaken by DND and PWGSC, Industry Canada, as well as by the relevant central agencies. We examined what policies were in place at the time throughout the process as well as current Treasury Board policies relating to Crown procurement contracting, investment planning and project management. In addition, we examined the associated DND and PWGSC policies and practices for complex, high risk military acquisitions.

We also reviewed the 2012 Spring Report of the Auditor General of Canada: Chapter 2 – Replacing Canada's Fighter Jets to identify the underlying causes that lead to the Auditor General's findings.

The culmination of the above provided the basis for the development of the lessons learned for future similar acquisitions.

Review Limitations

The review team's ability to fully assess the actions undertaken by the government during the process was hampered by specific limitations:

• Cabinet Documents: Key documents, such as the 2010 Memorandum to Cabinet and relevant DND Treasury Board Submissions from 2001 and 2006, are Confidences of the Queen's Privy Council for Canada, and could not be provided. As such, reviewers were not able to examine the advice given to the applicable Ministers or the decisions emanating from Cabinet and Treasury Board deliberations
• Other documents: Documents such as DND's Strategic Capability Roadmap and a classified annex to the Statement of Operational Requirements were not provided due to their sensitive content
• Interviews: Interviews with key stakeholders involved in the process were not always possible as their employment with the federal government had ceased or they were otherwise unavailable. Individuals currently involved in the process did not necessarily have a complete knowledge or understanding of earlier activities that had been undertaken with respect to the CF-18 fleet replacement

Through our interviews we noted that various participating stakeholders (within and among departments) in the process have differing views about the steps that were undertaken and why they were undertaken.

Our lessons learned and recommendations are based upon the information provided and individuals interviewed throughout the review process.

Major military acquisition

Building a first-class military that is well-trained, well-equipped and ready to take on the challenges of the 21st century represents its own unique risks and opportunities.

Maintaining a military that is prepared to face uncertain future security challenges often requires the acquisition and procurement of new and technologically advanced equipment, which is a major expense.

In doing so, the Government of Canada's procurement system endeavours to balance the need for the Canadian Armed Forces to be equipped with a modern military that can carry out its missions while maximizing value for money and leveraging investments in military equipment to the benefit of the Canadian industry and workers.

Overview of the Acquisition Process

The federal government acquisition process is part of the broader asset management life cycle^{Footnote 1}, as depicted below:

This section addresses both the planning and the acquisition phases of the asset management life cycle.

Planning Phase

This initial phase of the asset management life cycle involves, among other things, establishing the operational need, producing the Statement of Operational Requirements, conducting an options analysis for the entire investment options over their whole life, establishing an appropriate management control framework, and conducting an initial assessment of project risk.^{Footnote 2}

The work undertaken during this phase is the responsibility of the department identifying the need.^{Footnote 3}

Acquisition Phase

This next phase involves all of the activities associated with planning for and acquiring goods and services in a manner that “… enhances access, competition, and fairness and results in best value or, if appropriate, the optimal balance of overall benefits to the Crown and Canadians.”^{Footnote 4} While respective departments are responsible to identify their requirements, the Minister of PWGSC has the exclusive authority to buy or otherwise acquire defence supplies and construct defence projects.

The acquisition phase is comprised of four sub-phases:

1. Pre-contractual: The activities associated with defining a requirement and planning for its procurement
2. Contract award: The activities associated with soliciting and evaluating bids and negotiating and awarding a contract
3. Contract administration: The activities associated with monitoring time, cost and contractor performance and administering a contract
4. Contract close-out: The activities associated with closing out the procurement, including final acceptance, final payment, and audits

DND has defined its standard project life cycle into five phases as part of the planning and the acquisition steps:

Appendix B presents a more detailed description of the first four phases.

Policy Framework

DND, like any other department is subject to a comprehensive set of laws, regulations and policies to be referred to in the conduct of its project planning and acquisitions activities. Treasury Board has instituted a number of key policies which departments must respect in conducting their necessary activities. Treasury Board also stipulates the requirement of appropriate governance and control processes to be adopted based on various risk considerations.

History of Treasury Board Policy Suite

In 2007, Treasury Board moved to less prescriptive, more principles-based policy direction that holds deputy heads accountable for achieving specific high-level management objectives. A summary table is presented in Appendix C with the applicable Treasury Board policy instruments in effect until 2009, and the policies in effect since that time. As it relates to the project management suite of policies, TBS adopted a phased approach to implementation of the Policy on the Management of Projects, starting in 2007 with a pilot with a group of departments^{Footnote 5} and culminating in full implementation on April 1, 2012. We note that DND signed the Production, Sustainment, and Follow-on Development Memorandum of Understanding (MOU) in 2006, at which point the old suite of Treasury Board policies would still have been in effect.

Public Works and Government Services Canada Policy Framework

The Department of Public Works and Government Services Act establishes PWGSC as a common service organization and gives the Minister of PWGSC the sole statutory authority for the procurement of goods. The Defence Production Act establishes the Minister of PWGSC as the exclusive authority to buy or otherwise acquire defence supplies and construct defence projects required by DND.

The PWGSC Supply Manual clarifies PWGSC's role and responsibilities in the acquisition process. At the definition of technical requirements and procurement planning stages, PWGSC is responsible for, among other things:

• Participating in the review and refinement of the Statement of Work and the development of the technical bid evaluation plan
• Assessing industrial capability
• Identifying the major contracting policies that must be considered in accomplishing the procurement; and
• Developing the contracting approach

Department of National Defence Policy Framework

Over the years, DND has developed policy guidance on the project life cycle from the identification of the need to the implementation of the procurement. Most of this guidance is described in the Project Approval Directive^{Footnote 6}. The Project Approval Directive is intended to reflect the standard practices resulting from changes to the Treasury Board project management policy suite and to reflect the Project Management Body of Knowledge^{Footnote 7}.

The Project Approval Directive assigns responsibility to the Project Sponsor for identifying the requirements and performing the option analysis.

CF-18 Fleet Replacement Acquisition Project

Canada currently operates a fleet of CF-18 Hornet fighter jets purchased in the 1980s with an original life expectancy until 2003. The fleet's life has since been extended to between 2017 and 2020 following a modernization program. National Defence has determined that a suitable replacement for the CF-18 is required, otherwise, it will lose its ability to carry out domestic and international missions as mandated by the Government of Canada.

In May 2008, through the Canada First Defence Strategy, the federal government announced its intent to replace the CF-18 fleet with 65 “next generation” fighter aircrafts. In July 2010, the federal government announced its decision to purchase the F-35 Lightning II (henceforth, the F-35), at a cost of approximately $9 billion. The purchase of the current fleet's replacement was to be done on a sole source basis, without following a competitive procurement process.

At the time of the announcement in 2010, the CF-18 replacement acquisition process had not reached the Implementation phase. The CF-18 replacement was designated as a project in January 2010 when it entered into the Options Analysis phase. In October 2010, a project management office was established within Assistant Deputy Minister (Materiel) and work was initiated to prepare a Treasury Board Submission for project approval, following the finalization of the Statement of Operational Requirements.

Governance Structure

A project is only designated as such when it reaches the Options Analysis phase.

From 1997, when the first MOU for the JSF Program was signed, up until January 2010, the project was in Identification phase. TBS was involved in JSF Program activities through their review of the Treasury Board submissions for the 2002 and 2006 MOU. PWGSC was not involved in the activities leading up to the signing of the MOUs since it was understood, within DND and TBS, that there was no commitment to purchase at that time.

During that period, Assistant Deputy Minister (Materiel) managed Canada's participation in the JSF Program.

In January 2010, the CF-18 replacement project entered the Options Analysis phase. In the policy context at that time, high risk projects such as the CF-18 replacement were required to have appropriate governance structure, project management capacity, controls, and reporting in place, as well as appropriate project approvals sought from Treasury Board. For the CF-18 replacement, the Department of National Defence Senior Review Board met in January 2010, the project went before the Major Crown Project Interdepartmental Oversight Committee for the first time in 2010 and a Senior Project Advisory Committee meeting was convened in October 2011.

Projects undertaken today would proceed similarly once entering the Options Analysis phase.

Project and Investment Planning

The 2001 CF-18 modernization program extended the estimated life expectancy of the fleet to 2017-2020. Ideally, a project to replace the modernized fleet should have been initiated in the years 2000-2002, given that projects of the magnitude of the CF-18 replacement typically take in excess of fifteen years to plan, implement and close out following the Identification phase. The CF-18 replacement project only entered the Options Analysis phase in January 2010 and it never reached the Implementation phase.

Interrelation of the Joint Strike Fighter Program and CF-18 Replacement Activities

Canadian participation in the JSF Program was an initial step taken by the DND in preparing for the eventual replacement of the CF-18. Participation in the JSF Program was intended to allow Canada to obtain industrial benefits, provide access to data about next generation fighter aircraft and cost avoidance should Canada chose to purchase the F-35.

Other than Canada's participation in the JSF Program, minimal steps were undertaken for the planning or acquisition to replace the CF-18 fleet before 2005, when an Operational Requirement and Concept Document was prepared. This work was initiated at the request of Assistant Deputy Minister (Materiel) to determine whether Canada should continue to participate in the JSF program. Following the signing of the 2006 MOU, a small project office was established within the Royal Canadian Air Force in 2007 to support the future replacement of the CF-18.

Current Investment Planning Process

The process for identification of capability deficiency has improved within DND over the recent years. The investment planning cycle is led by the Vice Chief of Defence Staff and Assistant Deputy Minister (Finance and Corporate Services). In the past, the identification of capability requirements was initiated by the various Sponsors (Air Force, Navy and Army) with little senior level strategic coordination, oversight or review. The creation of the Chief Force Development within Vice Chief of Defence Staff should enable the Canadian Armed Forces to better coordinate the needs, requirements and available funding. Furthermore, it allows better long term planning of investment through early identification of needs.

Additionally, when high risk and complex acquisition projects are identified during investment planning activities, departments are now required to provide the following information for Treasury Board approval with their resulting Investment Plan:

• Procurement strategy
• Estimated costs^{Footnote 8}
• Project Complexity and Risk Assessment –  with relevant and appropriate governance structure; and
• Related risk mitigating strategies

Following the identification of a project, DND is expected to involve necessary central agencies and PWGSC.

Recent changes in the process for identifying capability deficiencies and the new investment planning process should improve governance and oversight, including timely PWGSC participation. However, as indicated by the Auditor General and many stakeholders, several aspects surrounding the participation in the JSF Program through the use of MOU contributed to the misalignment of steps taken related to the JSF program with steps taken in the CF-18 fleet replacement project planning process.

Acquisition Project Management Process

Identification Phase

The Capability Based Planning exercise established in 2010 by Chief of Force Development is critical in establishing the best approach - including value for money – to meet the policy direction established by the Government, and informing the development of DND's Investment Plan. This new process has yet to complete its first three-year cycle. However, based on our review, it appears to be well structured, includes a challenge role and involves key stakeholders within the Canadian Armed Forces.

The Capability Based Planning exercise is led by Chief Force Development, with the involvement and cooperation of the Canadian Armed Forces (Royal Canadian Air Force, Royal Canadian Navy and Canadian Army) and other key players such as Chief of Programme. The output of this exercise is the Force Capability Plan, which is reviewed by senior leaders of the Canadian Armed Forces and submitted to the Chief of Defence Staff for approval. The Force Capability Plan details the military's future requirements and is a new deliverable produced by Chief Force Development for the Chief of Defence Staff. The first version of this plan, which is updated every three years, will be available in 2013. The Capability Based Planning exercise includes a review by senior military personnel to ensure the requirements presented by the Canadian Armed Forces are appropriately defined and supported.

Option Analysis Phase

Chief of Programme coordinates the development of the necessary Treasury Board submissions for project approval. Chief of Programme also initiates discussion with Assistant Deputy Minister (Materiel) on how to proceed with the procurement and to determine when the leadership of the project should be transferred from the Project Sponsor (Royal Canadian Air Force, Royal Canadian Navy, or Canadian Army) to a project manager in Assistant Deputy Minister (Materiel) and when a formal Project Management Office should be established within Assistant Deputy Minister (Materiel).

A Project Management Office is typically established following Cabinet approval of a Memorandum to Cabinet and when project approval has been received from Treasury Board. While a formal Project Management Office is typically established at the Definition phase of a project, the Project Sponsor usually has established a small project office at the time work is initiated at the Identification phase. For the CF-18 replacement, this was done in 2007, following the signing of the Production, Sustainment, and Follow-on Development 2006 MOU.

The work conducted in the Identification and Options Analysis phase is led by the Project Sponsor. For the CF-18 replacement, there was limited involvement of the project management personnel working under the auspices of Assistant Deputy Minister (Materiel). Although project management expertise from Assistant Deputy Minister (Materiel) is often involved in the earlier phases of a project with the Project Sponsors, it is not always the case. The Project Office that prepared the necessary supporting documentation for the 2010 Memorandum to Cabinet was relatively small for a project this size.

The joint strike fighter program

Canada's military enters into various international collaboration arrangements through the use of Memorandum of Understanding (MOU). In most cases, these MOU allow DND to participate in collaborative research and development projects and provide opportunities for Canadian industry participation.

However, the JSF Program was conceived to be a new model for development and acquisition of military equipment and considered unique at that time.

In 1997, Canada (through DND) signed an international MOU with the United States Government, along with eight other country partners, to undertake concurrent design, development and manufacture of the F-35 Lightning II aircraft via the JSF Program. For the Government of Canada, in particular Industry Canada and DND, the purpose of participating in this joint collaboration initiative was to have access to and transfer of classified information, to participate in an international program to design, develop and manufacture the aircraft, to benefit from the waiving of U.S. Foreign Military Sales surcharge, and to gain access to industrial opportunities for Canadian companies. Canada signed a second MOU in 2002 for the system development and demonstration of the F-35, and a third, in 2006, for the production, sustainment, and follow-on development of the F-35.

The third MOU signed in 2006 introduced the terms for the production of the aircraft and contained a chapter on procurement, which defined the process for country partners to acquire the F-35, if desired. The cost of participating in the 2006 MOU was based on the number of aircraft estimated to be purchased by each country. At the time the 2006 MOU was signed, Canada estimated it would purchase 80 aircraft; this number was later amended to 65.

Assistant Deputy Minister (Materiel) managed Canada's participation in the JSF Program, until a Project Office was created in October 2010. Prior to the signing of the 2006 MOU, Assistant Deputy Minister (Materiel), (DND) requested that the Royal Canadian Air Force complete a preliminary analysis of options by reviewing five candidate aircraft that could potentially replace the CF-18.

1997, 2002 and 2006 Memoranda of Understandings

The JSF Program was led by the United States Department of Defence. Canada was invited to participate in the Program through the signing of a series of MOUs. Three MOU were entered into for the JSF Program. They are:

• 1997 CD MOU: The Concept Demonstration (CD) MOU was signed in December 1997. With a US$10.6 million contribution, this allowed Canada to work with allies in developing a new fighter jet as well as providing opportunities for Canadian companies to be part of the design phase of the program. During the period covered by this MOU, the United States collaborated with the United Kingdom in establishing the requirements that led to a design competition between Boeing and Lockheed Martin for the selection of the prime contractor to build the F-35. Lockheed Martin was selected as a result of that competition
• 2002 SDD MOU: The System Development and Demonstration (SDD) MOU was signed in February 2002. Through this MOU, Canada committed to contribute US$150 million (US$100 million in cash and US$50 million through Industry Canada programs)
• 2006 PSFD MOU: The Production, Sustainment, and Follow-on Development (PSFD) MOU was signed in December 2006. Through this MOU, Canada committed US$551 million over 40 years

While the F-35 could become an alternative for the replacement of the CF-18, and the 2006 MOU had one chapter dedicated to the procurement process with an estimated number of planes Canada would acquire, there was no commitment to purchase for the signatories to these MOUs. However, the JSF Program established the acquisition process through these MOUs if Canada was to purchase the F-35.

TBS was involved with DND prior to signing the 2006 MOU in the review of the related Treasury Board submission. At that time, the MOU was not considered part of the CF-18 replacement project. PWGSC was not involved in any of the MOUs.

Prior to signing the 2006 MOU, although no CF-18 replacement project was in place, the Royal Canadian Air Force worked on a number of Options Analysis phase related activities. Although the documentation produced was limited, some of the activities conducted related to the following:

• Established operational requirements (Operational Requirement Concept Document and Statement of Operational Requirements)
• Conducted Preliminary Options Analysis
• Established Preliminary Project Costs
• Performed a Risk Assessment; and
• Prepared a Business Case

In 2010, to replace the CF-18 with what was considered the only valid Next Generation Fighter Aircraft and to ensure maximum Canadian industry participation in the supply chain supporting the JSF Program, a Memorandum to Cabinet (MC) was presented to Cabinet to obtain policy approval on the sole source purchase of the F-35.

Lessons learned and recommendations

Alignment of the CF-18 Replacement Project and the Joint Strike Fighter Program

Observation

The JSF Program was conceived to be a new model for development and acquisition of military equipment and considered unique. As described previously and presented in the figure below, Canada's participation in the JSF Program was under way before a decision was made on how to replace the CF-18 fleet. This reflects one of the unique aspects of the JSF program, which is a U.S.-led multinational effort.

JSF Program

CF-18 Replacement Project

The JSF Program requirement was that Canada had to sign an MOU in 2006 to support production related activities. However, the CF-18 replacement had not yet been defined as a project within DND and was still in Identification phase, and a thorough and documented options analysis had yet to be completed. PWGSC was not involved in the replacement of the CF-18 until January 2010.

It is our opinion that the parallel tracks of the JSF Program and CF-18 replacement project were misaligned. The options analysis for the CF-18 replacement project should have been completed to provide the necessary support for signing the Production, Sustainment, and Follow-on Development MOU in 2006.

Applicability of the Policy Framework

The current policy framework is applicable to all acquisitions. In our opinion, the current policy framework is adequate for all major military acquisitions that follow DND's standard project life cycle noted earlier. It provides for the early engagement of PWGSC in the planning of the acquisition and aims at ensuring that departments put the appropriate governance and project management processes and controls in place.

However, up until 2010, the JSF Program was not considered an acquisition process; the CF-18 replacement project was only entering in Options Analysis phase. There is little policy direction covering participation in an international initiative such as the JSF Program that is not yet part of an acquisition process, but could lead to a potential acquisition. As such, we found that the current policy framework does not provide sufficient direction to DND personnel regarding the necessary steps leading to participation in such a program.

Impact

PWGSC, as contracting authority, was not engaged in the participation to the JSF Program. These steps included the signing of the three MOUs but also the supporting analysis and activities such as the Operational Requirement and Concept Document. The signing of the production MOU for participation in the JSF Program was not synchronized with the approval process for the acquisition of a replacement for the CF-18. Canada was already involved with the F-35 aircraft (through the JSF Program) prior to conducting the CF-18 replacement Options Analysis phase in January 2010.

Lesson-learned

DND engages in a range of MOUs to support the development of new capabilities, but not all such arrangements lead to acquisitions. MOUs that can lead to potential acquisitions need to align with the underlying capital related project in order that program related decisions are adequately supported by well documented analysis. Key aspects of the procurement process should be completed and key stakeholders involved before entering a production MOU.

Recommendation #1

In order for the Government to make an early and informed decision on the available options and procurement strategy, we recommend that:

• The Department of National Defence develop necessary policy to ensure that the signing of production Memoranda of Understanding is done in conjunction with its project planning and acquisition cycle
• The Treasury Board Secretariat formally clarifies that Memoranda of Understanding which are contract-like should be treated as contracts and follow appropriate related processes
• Public Works and Government Services Canada always be involved in the development and signing of production related Memoranda of Understanding. Participation in the related program at the production phase should only be considered when a formal project has been designated, the Options Analysis phase has been completed and a procurement strategy has been defined

Project Management Capability

Observation

From the mid-late 1990's to January 2010, the Royal Canadian Air Force had been working on the CF-18 replacement project with limited involvement of Assistant Deputy Minister (Materiel).

From the mid-late 1990's to January 2010, the Royal Canadian Air Force had been working on the CF-18 replacement project in what is considered the Identification phase. Following the signing of the 2006 MOU, a small project office was established within the Royal Canadian Air Force in 2007 to support the future replacement of the CF-18. The same Project Office was involved in the preparation of the necessary supporting documentation for the 2010 Memorandum to Cabinet. It was recognized that this Project Office was relatively small for a project this size.

Although project planning is the foundation of effective project management and delivery, the Project Approval Directive does not require that any specific project management capability be established until the Definition Phase, when a Project Management Office within Assistant Deputy Minister (Materiel) takes the project lead. In the case of the F-35, this office was not established until October 2010.

While project management expertise from Assistant Deputy Minister (Materiel) is often involved in the earlier phases of a project with the Project Sponsors, it is not always the case.

Impact

Where there is limited professional project management capability assigned to an acquisition project there is increased risk that supporting documentation for key decisions is inadequate, lacks precision, cannot be produced in a timely manner and fails to consider all aspects of the eventual procurement requirements.

Lesson-learned

DND would benefit from early and appropriate project management capability involvement from Assistant Deputy Minister (Materiel) with the Project Sponsor's office on a consistent basis for its high risk projects.

Recommendation #2

The Department of National Defence needs to ensure that adequate Project management capability is assigned to the Project Sponsor project office as early as the Identification phase and that it is appropriately augmented until the project reaches Definition phase.

Statement of Requirement

Observation

In accordance with the National Defence Act, and as amplified in the PWGSC Supply Manual, DND is responsible for defining operational and technical requirements. This stated, under DND policy there has been no requirement to validate Statement of Operational Requirements prior to approval. As a result, Assistant Deputy Minister (Materiel) relied on the military, the Royal Canadian Air Force in the case of the CF-18 replacement, to define its requirements, and none of the parties believed the Statement of Operational Requirements had to be validated.

The justifications for the sole source procurement strategy relied on the assertion by the Royal Canadian Air Force that the Canadian requirements were such that only the F-35 could meet them. PWGSC agreed to the sole source procurement strategy primarily on the strength of a letter from the Assistant Deputy Minister (Materiel) to the Assistant Deputy Minister, Acquisitions Branch, PWGSC, in which DND asserted that only one firm was capable of supplying a 'fifth generation' fighter aircraft.

Public Works and Government Services Canada Specific Role and Responsibilities

The Department of Public Works and Government Services Act establishes PWGSC as a common service organization and gives the Minister of PWGSC the sole statutory authority for the procurement of goods. The Defence Production Act establishes the Minister of PWGSC as the exclusive authority to buy or otherwise acquire defence supplies and construct defence projects required by DND.

PWGSC is accountable to safeguard the integrity of the federal government's procurement process by ensuring that its procurement practices are fair, open and transparent. This involves numerous activities, including ensuring that clients have described their requirements in an objective manner and that these requirements have been justified. Inherent in this requirement, is the need to ensure that sole source justifications are reasonable and well supported.

Impact

Ineffective challenge or oversight over procurement strategies and the Statement of Operational Requirements may not result in obtaining the best value for money.

Lesson-learned

PWGSC should not rely exclusively on client departments in ensuring that statements of operational requirement are written in an objective manner. Consistent external independent oversight may help ensure an adequate challenge function for both the Statement of Operational Requirements and procurement method.

Recommendation #3

To ensure that Statement of Operational Requirements are objective and adequately supported for future high dollar value, complex projects, Public Works and Government Services Canada should:

• Seek the opinion of independent third party expertise where it feels that the department does not have the requisite knowledge or expertise to determine whether a client's description of its operational requirements is reasonable. This could be done through a senior panel of independent officials overseeing the Options Analysis phase results and the Statement of Operational Requirements (as it is currently being done through the National Fighter Procurement Secretariat) or through the hiring of independent third party experts
• Engage industry in the review of the requirements, through formal industry consultation mechanisms; and
• Ensure that the procurement decision (in particular sole source) is based on well-documented, objective and rigorous analysis with necessary expertise

Roles and Responsibilities

Observation

There is dispersed accountability and responsibility for certain steps of the acquisition process. This was more acute in the CF-18 replacement because of the unique context of the JSF Program and the misalignment of the project and program activities.

DND, PCO, Industry Canada, TBS and PWGSC were involved in the discussions that led to the signing of a Memorandum to Cabinet in 2010 while only DND, Industry Canada and TBS were involved with respect to the 2002 and 2006 MOU.

The National Defence Act gives the Minister of DND authority to define the necessary materiel required by the Canadian Armed Forces.

PCO provides the Prime Minister with public service support and is the Cabinet secretariat. PCO is responsible for consulting involved departments and agencies and providing analysis to the Prime Minister on proposed policies^{Footnote 9}.

TBS has a central oversight role to play in government-wide management practices and ensuring value for money^{Footnote 10}.

As indicated previously, The Defence Production Act establishes the Minister of PWGSC as the exclusive authority to buy or otherwise acquire defence supplies and construct defence projects required by the Department of National Defence.

For the CF-18 replacement, DND was the lead department and required a replacement for its aging fighter fleet. Industry Canada was responsible for facilitating industrial participation.

Through our discussion, we have noted that each of these stakeholders considered that they acted diligently throughout the MOUs and the Memorandum to Cabinet. However, key players, PWGSC in particular, were not sufficiently involved at critical stages of the process.

Impact

In the unique context of the JSF Program, the dispersed accountability and responsibility has contributed to weaknesses or gaps in the oversight, review and approval.

Lesson-learned

To ensure sufficient oversight and diligence in an environment of dispersed accountability and responsibility, there needs to be a clear understanding of the role and responsibilities of key stakeholders in the process.

Recommendation #4

The responsibilities, accountabilities and requirements for sponsoring departments and Public Works and Government Services Canada to consult and inform key stakeholders in the process, including the Treasury Board Secretariat and the Privy Council Office, should be included in one guidance document in order to provide increased clarity to officials managing future major military acquisitions.

Shortcomings identified by the auditor general of canada

As part of the review, one of the objectives was to determine whether the shortcomings identified by the Auditor General in the acquisition process have been effectively addressed.

The Auditor General's report presented a number of findings directly related to the planned procurement.

On April 3, 2012, the Government announced a comprehensive response to the Auditor General's report. The response indicated that it accepted the Auditor General's recommendation and conclusion and that, in response, it would be implementing a Seven-Point Plan (Refer to the National Fighter Procurement Secretariat web site: National Fighter Procurement Secretariat).

At the time of our review, the National Fighter Procurement Secretariat was putting in place many of the steps necessary for the implementation of the Seven-Point Plan. In fact, our review of the acquisition process is intended to support the sixth point of the Plan (with the exception of the sustainment costs), in providing lessons learned that are intended to help the government in improving similar acquisitions in the future. The National Fighter Procurement Secretariat provides regular updates on the status of each step of the plan on its website.

The shortcomings the Auditor General identified in the acquisition processes are currently being addressed through the implementation of the Seven-Point Plan.

Conclusion

Recent changes in the process for identifying capability deficiencies and the new investment planning process should improve governance and oversight, including timely PWGSC participation. In an effort to further improve the acquisition process, we believe that the implementation of our recommendations will improve the alignment of project and program activities and ensure that key stakeholders are informed and approvals are sought in a timely fashion. Additional independent oversight will contribute to decisions being based on objective and well documented options analysis.

This will ultimately increase the level of due diligence for any future acquisitions of a similar nature, risk, and/or complexity.

Appendix A: Stakeholders Interviewed

National Defence

Vice-Chief of Defence Staff

• Rear-Admiral M., Lloyd: Chief of Force Development
• Brigadier-General S.A., Brennan: Director General, Capability and structures integration, Chief Force Development
• Major-General Poulter I.C.: Chief of Programme

Assistant Deputy Minister (Finance and Corporate Services)

• Kevin Lindsey: Assistant Deputy Minister, Chief Financial Officer, Finance and Corporate Services
• Karen Cahill: Director General, Strategic Finance and Financial Arrangements
• Kirk King: Strategic Finance and Financial Arrangements

Assistant Deputy Minister (Materiel)

• Rear-Admiral P. Finn: Chief of Staff
• Troy Crosby: Director, Next Generation Fighter Capability, Project Management Office
• Corey Lee: Finance and Supply Team Manager Next Generation
  Fighter Capability - Project Management Office
• Michael Slack: Director Continental Materiel Cooperation Chief of Air

Staff – Royal Canadian Air Force

• Colonel J.P.A., Pelletier: Director Air Requirements
• Lieutenant Colonel Tim Shopa: Director Air Requirements 5 - Fighters and Trainers

Chief Review Services

• Chris Currie: Principal

Industry Canada

Aerospace, Defence and Marine Branch

• Jeffrey Waring: Director, F-35 Industrial Participation

Public Works and Government Services Canada

Acquisition Branch

• Tom Ring: Assistant Deputy Minister
• Joanne Provencher: Director General, Major Military Procurements

Next General Fighter Capability Project

• Frederick Theiner: Senior Manager

Office of Audit and Evaluation

• Linda Anglin: Chief Audit and Evaluation Executive

Treasury Board Secretariat

Investment, Project Management and Procurement Policy Directorate

• Elisa Mayhew: Executive Director
• Greg Kenney: Senior Director

Defence and Immigration Division

• Ian Matheson: Executive Director, International Affairs, Immigration, Defense
• Christopher Burt: Advisor/Economist

Privy Council Office

Economic and Regional Development Policy

• Rob Pinker: Senior Policy Analyst

Other

• Alan Williams: Author, Research Associate at Queen's University
  • Assistant Deputy Minister Supply Operations Services (retired), PWGSC (1995 - 1999)
  • Assistant Deputy Minister (Materiel) (retired), DND (1999 - 2005)
• Dan Ross: Assistant Deputy Minister (Materiel) (retired), DND (2005 - 2012)

Appendix B: Overview of Process and Key Players

Appendix C: Policy Framework

In considering possible lessons learned, it is important to consider the policies that are in place currently and those that were in effect at the time of the signing of the three MOUs.

History of Treasury Board Policy Suite

In 2007, Treasury Board moved to less prescriptive, more principles-based policy direction that holds deputy heads accountable for achieving specific high-level management objectives. As it relates to the project management suite of policies, TBS adopted a phased approach to implementation of the Policy on the Management of Projects, starting in 2007 with a pilot with a group of departments^{Footnote 11} and culminating in full implementation on April 1, 2012. We note that DND signed the last MOU in 2006, at which point the old suite of Treasury Board policies would still have been in effect.

The table below sets out the applicable Treasury Board policy instruments in effect until 2009, when DND's Investment Plan was approved by Treasury Board, and the policies in effect since that time.

Pre-2009 Treasury Board Policy Instruments

• Policy on Long-Term Capital Plans
• Project Management Policy
• Management of Major Crown Projects
• Procurement Review Policy
• Project Approval Policy
• Contracting Policy

Treasury Board Policy Instruments - 2009 to present

• Policy Framework for the Management of Assets and Acquired Services (2006)
• Policy on Investment Planning – Assets and Acquired Services
• Policy on the Management of Projects
• Standard for Project Complexity and Risk
• Standard for Organizational Project Management Capacity
• Project Complexity and Risk Assessment Tool
• Organizational Project Management Capacity Assessment Tool
• Procurement Review Policy
• Contracting Policy

Applicability of Treasury Board Policy to the Management of Projects

Project Governance

The objective of the Treasury Board Policy on the Management of Projects is to:

“… ensure that the appropriate systems, processes, and controls for managing projects are in place at a departmental, horizontal, or government-wide level and support the achievement of project and program outcomes while limiting the risk to stakeholders and taxpayers”.

The policy defines a 'project' as:

“…an activity or series of activities that has a beginning and an end. A project is required to produce defined outputs and realize specific outcomes in support of a public policy objective, within a clear schedule and resource plan. A project is undertaken within specific time, cost and performance parameters”.

For all projects, departments are expected to conduct a Project Complexity and Risk Assessment, using Treasury Board's Project Complexity and Risk Assessment Tool. Project complexity, as set out in the Treasury Board Standard for Project Complexity and Risk:

“Is based on the number of business rules, the technology employed and the project's size. It is a major component of project risk. Complexity should be determined at the start of all large projects, and when changes occur, so that appropriate action can be taken to minimize risk”.

Treasury Board has established four levels of project risk:

• Level 4 – Transformational
• Level 3 – Evolutionary
• Level 2 – Tactical
• Level 1 – Sustaining

In addition to the requirement to assess the complexity and risk of a project, the Treasury Board Standard for Organizational Project Management Capacity requires that departments assess their capacity, using the associated tool, to manage projects. The Standard aims to help departments determine the level of organizational capacity needed to manage projects and identify areas of capacity that should be improved or maintained. The Standard also requires that departments submit to TBS their investment plans along with their Organizational Project Management Capacity Assessment. Investment plans are discussed in the following pages.

The resultant level of project complexity and risk, coupled with an assessment of a department's project management capacity, determines the governance, controls and reporting required to appropriately manage a project. As shown in Appendix B, this determination of project complexity and risk is typically made during the Options Analysis phase, when a project's scope has been examined and costed.

Treasury Board Project Approval

Ultimately, the deputy head is responsible for ensuring that Treasury Board project approval is sought when the assessed risk and complexity of a project exceeds the assessed level of capacity that the sponsoring minister can approve^{Footnote 12}.

For large projects such as the CF-18 replacement, typically a Memorandum to Cabinet^{Footnote 13} precedes a submission to Treasury Board seeking project and expenditure authority. A Treasury Board submission is an official Cabinet document submitted by a sponsoring minister on behalf of a department seeking approval or authority from Treasury Board for an initiative that the department does not have the authority to undertake. Legislation, Treasury Board policies, or other Cabinet decisions usually establish the requirements for Treasury Board approval.

While a Memorandum to Cabinet focuses primarily on the policy rationale and overall funding for a new policy or program initiative at a high level, a Treasury Board submission fleshes out the details associated with delivering and implementing the initiative. According to A Guide to Preparing Treasury Board Submissions, the Treasury Board submission transforms policy rationale and objectives into a program that will achieve the approved policy objectives. The submission provides details on program design, specific costs, expected results and outcomes, and how the department will conduct monitoring and evaluation to ensure that the policy objectives are met. In terms of costing, the Treasury Board submission is expected to provide specific funding details by source of funds and fiscal year, and a clear rationale for what resources will be spent each year.

It is understood that while DND sought Treasury Board approval on two separate occasions for participation in the JSF Program, it has yet to seek Treasury Board project approval for the replacement of the CF-18 fleet.

Investment Planning

Departments like DND are required to submit a five-year Investment Plan to Treasury Board every three years for approval, and to include planned investments and their assessed Organizational Project Management Capacity. In addition to providing the deputy head with the information required to support strategic and operational decision, the plan is intended to inform the Treasury Board of key future investments.

We understand that DND's Investment Plan was approved by Treasury Board in 2009. While the 'Next Generation Fighter Capability' was included in the Investment Plan, it was not yet considered a project.

The Policy on Investment Planning – Assets and Acquired Services requires departments to inform key federal stakeholders such as common service providers, of planned investments.

Public Works and Government Services Canada Supply Manual

According to the PWGSC Supply Manual, PWGSC is responsible for participating in the following number of key activities:

• Delineating all feasible solutions, within government policy, to meet operational needs
• Initiating a Project Management Plan
• Establishing a Senior Procurement Advisory Committee to orient the project to achieve relevant national objectives and provide a forum for reviewing and discussing project objectives, requests for proposals and other key project instruments
• Obtaining approval of governance bodies such as DND's Senior Review Board and the Senior Procurement Advisory Committee; and
• Preparing Memoranda to Cabinet

At the definition of technical requirements and procurement planning stages, PWGSC is responsible for, among other things:

• Participating in the review and refinement of the Statement of Work and the development of the technical bid evaluation plan
• Assessing industrial capability
• Identifying the major contracting policies that must be considered in accomplishing the procurement; and
• Developing the contracting approach

Department of National Defence Project Management Policy

Up until 2012, DND's Project Approval Guide provided policy and procedural guidance on the approval process for projects, activities and initiatives.

In 2012, DND adopted the Project Approval Directive. The Project Approval Directive is policy-based direction and guidance and was meant to reflect the standard practices resulting from the recent Treasury Board framework and Project Management Body of Knowledge.

Appendix D: Department of National Defence Organizational Structure

The following organizational chart provides an overview of the key players involved in the planning and acquisition process within DND at the time of our review:

The following is an overview of their respective roles and involvement in the different project phases described in Appendix A:

Directorate of Air Requirements: Responsible for the development and advancement of Air Force projects through the Defence Management System, and for Air Force capital project advice to senior Air Force leadership. Directorate of Air Requirements works with Chief Force Development, Chief of Programme and Assistant Deputy Minister (Materiel) through all phases and is responsible, among other things, for preparing the Statement of Operational Requirements.

Chief Force Development: Harmonize, synchronize and integrate the force development activities of the Canadian Forces in order to develop the capabilities required. Chief Force Development leads the Identification phase through the Capability-Based Planning exercise.

Chief of Programme: Supports the Vice Chief of Defence Staff by providing analysis and advice on strategic planning options and resource allocations. Chief of Programme leads the Options Analysis Phase.

Assistant Deputy Minister (Materiel): Responsible for the procurement and management of materiel.

Directorate Continental Materiel Cooperation: Develops, promotes and executes bilateral cooperative activities with the U.S. Department of Defense. Directorate Continental Materiel Cooperation leads the signing of the MOUs with the U.S. Government.

Project Management Office - Next General Fighter Capability: Responsible for project management regarding the acquisition of Next Generation Fighter with necessary stakeholder's participation e.g. PWGSC.

Assistant Deputy Minister (Finances and Corporate Services): Provides comptrollership, financial management and corporate services to support DND/Canadian Armed Forces. Since 2011, they lead key activities in preparing the Investment Plan, together with Chief Force Development and Chief of Programme.

Appendix E: Summary of Recommendations

Recommendation #1

In order for the Government to make an early and informed decision on the available options and procurement strategy, we recommend that:

• The Department of National Defence develop necessary policy to ensure that the signing of production Memoranda of Understanding is done in conjunction with its project planning and acquisition cycle
• The Treasury Board Secretariat formally clarifies that Memoranda of Understanding which are contract-like should be treated as contracts and follow appropriate related processes
• Public Works and Government Services Canada always be involved in the development and signing of production related Memoranda of Understanding. Participation in the related program at the production phase should only be considered when a formal project has been designated, the Options Analysis phase has been completed and a procurement strategy has been defined

Recommendation #2

The Department of National Defence needs to ensure that adequate Project management capability is assigned to the Project Sponsor project office as early as the Identification phase and that it is appropriately augmented until the project reaches Definition phase.

Recommendation #3

To ensure that Statement of Operational Requirements are objective and adequately supported for future high dollar value, complex projects, Public Works and Government Services Canada should:

• Seek the opinion of independent third party expertise where it feels that the department does not have the requisite knowledge or expertise to determine whether a client's description of its operational requirements is reasonable. This could be done through a senior panel of independent officials overseeing the Options Analysis phase results and the Statement of Operational Requirements (as it is currently being done through the National Fighter Procurement Secretariat) or through the hiring of independent third party experts
• Engage industry in the review of the requirements, through formal industry consultation mechanisms; and
• Ensure that the procurement decision (in particular sole source) is based on well-documented, objective and rigorous analysis with necessary expertise

Recommendation #4

The responsibilities, accountabilities and requirements for sponsoring departments and Public Works and Government Services Canada to consult and inform key stakeholders in the process, including the Treasury Board Secretariat and the Privy Council Office, should be included in one guidance document in order to provide increased clarity to officials managing future major military acquisitions.