Evaluation of Immunization and Respiratory Infectious Disease Activities at the Public Health Agency of Canada 2011–2012 to 2015–2016

Prepared by Office of Audit and Evaluation
Health Canada and the Public Health Agency of Canada

September 2016

List of Acronyms

ADM 
Assistant Deputy Minister
aNICS
Adult National Immunization Coverage Survey
AEFI
Adverse Event Following Immunization
BPP
Bulk Purchasing Program
CAEFISS 
Canadian Adverse Events Following Immunization Surveillance System
CIRN
Canadian Immunization Research Network
CIC
Canadian Immunization Committee
CIG
Canadian Immunization Guide
CIRC
Canadian Immunizations and Coverage Network
CIRID
Centre for Immunization and Respiratory Infectious Diseases
cNICS
Childhood National Immunization Coverage Survey
CPIP
Canadian Pandemic Influenza Preparedness: Planning Guidance for the Health Sector
DG
Director General
DPR
Departmental Performance Report
F/P/T 
Federal/Provincial/Territorial
HPV
Human Papillomavirus
ICIS
Immunization Coverage and Information Systems Section
IDPCB
Infectious Disease Prevention and Control Branch
MAH
Market Authorization Holder
NACI
National Advisory Committee on Immunization
NAS
National Antiviral Stockpile
NESS 
National Emergency Strategic Stockpile
NGO
Non-Governmental Organizations
NIS
National Immunization Strategy
NML
National Microbiology Laboratory
NOC
Notice of Compliance
OGD
Other Government Department
O&M
Operations and Maintenance
PAA 
Program Alignment Architecture
PAHO
Pan American Health Organization
PHAC
Public Health Agency of Canada
P/T
Provincial/Territorial
RMAF
Results-based Management and Accountability Framework
RPP
Report on Plans and Priorities
SED
Surveillance and Epidemiology Division
TBS
Treasury Board of Canada Secretariat
VPD
Vaccine Preventable Disease
VSWG
Vaccine Supply Working Group
WHO
World Health Organization

Table of Contents

List of Tables

Executive Summary

This evaluation covered the Immunization and Respiratory Infectious Disease program for the period from 2011-2012 to 2015-2016. The evaluation was undertaken in fulfillment of the requirements of the Financial Administration Act and the Treasury Board of Canada's Policy on Evaluation (2009).

Evaluation Purpose and Scope

The purpose of the evaluation was to assess the relevance and performance of the Public Health Agency of Canada (PHAC)'s Immunization and Respiratory Infectious Disease Activities. The scope of the evaluation covered the period from April 2011 to March 2016 and included various PHAC activities such as National Immunization Strategy leadership and coordination; vaccine guidance; public and professional education and outreach to promote vaccine acceptance and uptake; security of vaccine supply; surveillance and epidemiology; coordinated immunization research, vaccine innovation and development; pandemic preparedness and coordinated outbreak response.

The evaluation did not cover emergency preparedness and response activities that fall outside of the responsibility area of the Centre for Immunization and Respiratory Infectious Diseases (CIRID) or those that are the responsibility of other jurisdictions (i.e., provincial and territorial responsibility for funding, program planning and the delivery of immunization programs within their own jurisdiction).

Program Description

The immune system is a special network in the body that protects people from bacteria and viruses that cause diseases. A vaccine preventable disease (VPD) is an infectious disease which can be prevented by a vaccine. Immunization is the process by which people become protected from a disease (such as through vaccination). The dead or weakened germs in vaccines help the body's immune system to develop both antibodies and immune memory which together will help to prevent the disease if the body is exposed in the future.

In Canada, all federal, provincial and territorial jurisdictions, as well as local authorities have substantial roles, authorities and activities in immunization. These involve not only all health departments, agencies and ministries, but also numerous other public authorities and public/private partnerships at various levels. These organizations are involved in diverse aspects of immunization program planning, implementation, evaluation and support, including vaccine research and development, assessment, regulation, guidance and promotion. While each jurisdiction and each respective agency/authority has a distinct mandate and a unique operating context, the activities of these diverse bodies are complementary and rely heavily on collaboration.

Conclusions - Relevance

Continued Need

There is a need for immunization activities as well as those to prevent and control respiratory infectious diseases such as influenza. In recent years, Canada has eliminated endemically-transmitted measles and rubella. With the exception of a few diseases that remain endemic and burdensome in Canada by having cyclical peaks or regional outbreaks (e.g., pertussis, invasive meningococcal disease, invasive pneumococcal disease), the current burden of VPDs has been at low levels or declining since the early 2000s. Seasonal influenza-related hospitalizations and deaths fluctuate depending on a number of factors including circulating strain and annual vaccine match, but continue to comprise an important annual burden of morbidity and mortality.

Highly infectious diseases require high immunization rates of the population to develop herd or community immunity (for example, 95% for measles). National vaccine coverage estimates derived from national immunization coverage surveys for children and adults indicate that all are below established targets; although this could be underestimated as it is based on parent-held immunization records that may be missing actual doses administered. Immunization, through vaccination, is considered to be one of the greatest public health achievements of the 20th century, providing a cost-effective tool to control and eliminate life-threatening diseases. It is estimated to have saved more lives in Canada over the past 50 years than any other single health intervention.

Alignment with Government Priorities

As outlined in a variety of strategic documents, preventing and controlling VPDs has been a priority for the Government of Canada for decades. Priorities to address these diseases were recently reiterated in the 2015 mandate letter to the Minister of Health as well as the 2016 Federal Budget, which specifically identified increasing vaccination rates as a top priority.

Alignment with Federal Roles and Responsibilities

There are multiple players (local, provincial/territorial, and federal) with responsibilities in this area. PHAC's immunization and respiratory infectious disease activities are aligned with the Minister's legislative authorities under the Department of Health Act (1996) and Public Health Agency of Canada Act (2006).

Cabinet authorities clearly articulate a public health leadership, coordination and collaboration role in various areas as well as for activities to strengthen national immunization infrastructure (vaccine safety monitoring; surveillance; immunization registry network; public and professional education; and approaches to enhance immunization coverage and acceptance in special populations). However, a number of program stakeholders expressed confusion about PHAC's role in research, surveillance and health promotion, as well as in immunization registries.

Conclusions - Performance

Achievement of Expected Outcomes (Effectiveness)

PHAC works with partners to have vaccine supply mechanisms in place to secure the supply of vaccines in Canada. PHAC and its partners have the capacity to address and minimize the impact of shortages and disruptions.

PHAC publishes and supports the development of critical knowledge products for the prevention and control of VPDs and respiratory diseases such as influenza; products such as the Canadian Immunization Guide, National Advisory Committee on Immunization (NACI) Statements and FluWatch reports are used by program stakeholders. Likewise, PHAC's public information tools such as the Immunization Schedule Tool and the Parent's Guide to Vaccination are used by Canadians.

While the vast majority of Canadians are immunized, pockets of under- or un-immunized Canadians exist across the country. Challenges to improving immunization coverage are varied, but include a lack of knowledge or awareness and an abundance of misinformation from a small but vocal anti-vaccination community. Furthermore, the availability of programmatic information was a factor in implementing vaccine recommendations across jurisdictions in a timely fashion.

Various PHAC activities contribute to Canadian capacity to deliver a coordinated and effective response to the risk of VPDs and infectious respiratory diseases. While comparable immunization registries across jurisdictions are still needed, many challenges exist in its establishment, and PHAC has taken steps to mitigate the challenges associated with this gap.

Demonstration of Economy and Efficiency

Immunization is a cost-effective intervention, with millions of dollars in direct and social costs saved through this activity. For example, each dollar invested in adult influenza immunization programs is estimated to save the health care system $45. Additionally, for children's vaccines, each dollar invested saves the health care system $16 related to measles, mumps and rubella, and $6 for diphtheria, pertussis and tetanus.Endnote 1 The cost per life year saved of various vaccinations compares well to other public health interventions, such as seat belt laws or chlorination of drinking water.

Generally, there is little variance between planned spending and expenditures, and any variances have tended to be linked to pandemic preparedness activities, which account for the majority of expenditures within the Centre. There may be efficiencies gained in examining governance structures and by clarifying the criteria used in conducting medical case reviews of adverse event reporting and/or by increasing the availability of training to enhance the medical skills and expertise for these reviews.

With respect to performance measurement, CIRID has implemented a performance measurement strategy and collected indicator data over the past two years which has recently helped identify areas for programmatic decision making. Areas of improvement could focus on timely indicator reporting, and the review of appropriateness of some indicators as valid measures of outcomes.

Recommendations

Recommendation 1

Strengthen evidence base to address current information gaps on the under- and un-immunized in Canada.

It is evident that there are pockets of under- or un-immunized individuals in Canada. PHAC's ability to identify these groups is limited, in part, by the absence of comparable immunization registries. The Program relies on surveys of the Canadian population. PHAC reports national immunization coverage estimates and measures factors influencing uptake by conducting national adult and childhood immunization coverage surveys. Given that the P/Ts have invested in various personalized approaches and IT solutions for their jurisdictions, an alternative method for obtaining a national immunization coverage picture (sourcing existing registry data) is needed.

P/T immunization registries remain an important surveillance tool in assessing immunization coverage and vaccine uptake, and there is a federal public health role in facilitating the development and P/T adoption of national standards for immunization registries. There is work already underway with the Canadian Immunization Registries and Coverage Network (CIRC) and partners such as Canada Health Infoway to revisit the functional and data element standards for immunization registries, and to explore innovative solutions for pulling the national immunization coverage picture from these existing and evolving systems.

Recommendation 2

Ensure timely availability of programmatic information regarding vaccinations for provincial/territorial use.

There is evidence that the provision of information (e.g., cost-effectiveness) that is used by jurisdictions to make vaccine programming decisions has not been timely or comprehensive enough. For example, provincial and territorial implementation of publicly-funded varicella immunization programs occurred gradually over 7 years. One of the factors in this gradual implementation may have been the timing of available programmatic guidance following the NACI publication of technical guidance for the varicella vaccine. Access to timely guidance on programmatic recommendations has been identified as one of the limiting factors to achieving uniform access to vaccinations across Canada. The program should continue its efforts to implement a mechanism to improve the timeliness and comprehensiveness of available technical and programmatic advice on vaccines in Canada.

Recommendation 3

Determine and implement strategies to influence Canadians' knowledge, attitudes and beliefs related to immunization and respiratory infectious diseases.

While the vast majority of Canadians are immunized, pockets of under- or un-immunized Canadians exist across the country. Vaccine hesitancy and anti-vaccination beliefs are barriers to improving immunization coverage in Canada. Knowledge, attitudes and beliefs about immunization are complex and varied, as are the reasons behind vaccine hesitancy and anti-vaccination beliefs which may be fueled by information globally available through social media, the internet and anecdotally. Recommendations from health care providers, a trusted source of information for Canadians, have been shown to be linked to greater vaccine uptake. While the program does produce tools and information for Canadians about immunization, program activities would be enhanced by the existence of a strategic framework and a social marketing strategy, including social media, to guide knowledge translation efforts intended to improve vaccine uptake.

Recommendation 4

Review governance to enhance efficiencies.

With 20 working groups that have varied governance structures, mandates and areas of focus addressing immunization and infectious respiratory diseases, there are indications that some working groups supported by CIRID could operate more efficiently. Some stakeholders have highlighted confusion surrounding roles and responsibilities in several working groups; other stakeholders have indicated that resourcing for some groups may not be adequate to the mandate assigned to them; and a review of working group records of decision has identified cases of working group focus extending beyond original mandates (e.g., discussing inter-jurisdictional data-sharing mechanisms). CIRID working group governance could be enhanced by a review targeted at clarifying roles and responsibilities and objectives of working groups supported by the Centre.

Recommendation 5

Enhance efficiencies in response activities (including medical case reviews).

PHAC has two separate standard operating procedures (SOPs) in adverse event reporting. First, the Centre meets data processing timelines for reporting on adverse events (generally meeting these procedures between 75% - 100% of the time). Second, PHAC conducts medical case reviews for adverse event reports, which are used to inform public health and regulatory decisions; additionally, serious cases are assigned to a medical safety expert for secondary case review. However, for 8 out of 24 months, report processing for medical case reviews fell below 70%, which would not have impacted immediate adverse event response activities. While this is partially due to one submission of backlogged cases, some key informants noted that the criteria for conducting a medical case review may not be clear and/or the specific medical skills, expertise and training required for this activity were not always available.

Furthermore, there are strong indications that CIRID engages in significant levels of outbreak response activities, including, but not limited to, risk assessment, issues management, strategic communications, and the development of guidance (for example, in the case of measles or pertussis outbreaks). However, it remains difficult to accurately quantify the level of CIRID effort associated with these activities. Given that key informant interviews indicated that outbreak response activities may impact staff capacity to complete regular activities in a timely manner, a clearer understanding of what outbreak response consists of, in the form of regular activity and output tracking would be useful for upstream planning within the Centre. A clear articulation of CIRID outbreak response activities, outputs and capacity would also contribute to overall PHAC outbreak surge capacity assessments.

Management Response and Action Plan

Evaluation of Immunization and Respiratory Infectious Disease Activities at the Public Health Agency of Canada 2011-2012 to 2015-2016

Management Response and Action Plan Table
Recommendations Response Action Plan Deliverables Expected Completion Date Accountability Resources
Recommendation as stated in the evaluation report Identify whether program management agrees, agrees with conditions, or disagrees with the recommendation, and why Identify what action(s) program management will take to address the recommendation Identify key deliverables Identify timeline for implementation of each deliverable Identify Senior Management and Executive (DG and ADM level) accountable for the implementation of each deliverable Describe the human and/or financial resources required to complete recommendation, including the source of resources (additional vs. existing budget)
Strengthen evidence base to address current information gaps on the under and un-immunized in Canada. Management agrees with this recommendation. Explore and implement enhanced methodologies for estimating immunization coverage, including innovative approaches to leverage Provincial/Territorial (PT) registry data, and understanding the motives underlying immunization decisions. Content related to immunization knowledge, attitudes and behaviours in the National Immunization Coverage Survey is enhanced. April 2018 CIRID DG
IDPCB ADM
Existing resources will be applied to support this work with additional resources as outlined in the Budget 2016 (approx. $4.3M).
Grant opportunity through CIHR is launched to support research projects that leverage Canadian expertise to identify under/unimmunized populations and explore the factors influencing immunization status. September 2016 CIRID DG
IDPCB ADM
Grant/contribution opportunity through the Immunization Partnership Fund is launched to determine a methodology to pull together the national immunization coverage picture using the information available across the diverse PT registry systems. June 2017 CIRID DG
IDPCB ADM
Ensure timely availability of programmatic information regarding immunization for provincial/territorial use. Management agrees with this recommendation. Expand the mandate of the National Advisory Committee on Immunization (NACI) to include timely and comprehensive programmatic considerations. Revised terms of reference reflecting the new mandate of NACI is completed (economic and modelling work). March 2017 CIRID DG
IDPCB ADM
Additional resources will be applied to this work as outlined in the Budget 2016 (approx. $1.9M).
Standard operating procedures for inclusion of additional programmatic information into NACI recommendations are completed. November 2017 CIRID DG
IDPCB ADM
Determine and implement strategies to influence Canadians' knowledge, attitudes and beliefs related to immunization and respiratory infectious diseases. Management agrees with this recommendation. Develop a program to support increasing immunization uptake and equitable access to immunization. New Immunization Partnership Fund program is launched to support initiatives focussed on increasing immunization, including projects focussed on aspects of immunization knowledge, attitudes and beliefs. June 2016 CIRID DG
IDPCB ADM
Existing resources will be applied to support this work with additional resources as outlined in the Budget 2016 (approx. $18.8M).
Develop a multi-year social marketing strategy, including social media to help achieve greater immunization uptake and reduction in the incidence of vaccine preventable diseases (VPD). Social marketing strategy, including social media is developed (led by the Communications and Public Affairs Branch, Health Canada). March 2017 PHSCD DG
CPAB ADM
CIRID DG
IDPCB ADM
Review governance to enhance efficiencies. Management agrees with this recommendation. Review PHN and PHAC-led immunization and VPD related working groups to clarify roles and structures, including the Canadian Immunization Committee (CIC) and its subgroups. Recommendations to improve clarity and alignment of working groups is completed (e.g. revised ToRs + membership lists). December 2016 CIRID DG
IDPCB ADM
Existing resources will be applied to support this work.
Recommendations from review are implemented. December 2017 CIRID DG
IDPCB ADM
Review governance to enhance efficiencies. Management agrees with this recommendation. Review PHN and PHAC-led immunization and VPD related working groups to clarify roles and structures, including the Canadian Immunization Committee (CIC) and its subgroups. Recommendations to improve clarity and alignment of working groups is completed (e.g. revised ToRs + membership lists). December 2016 CIRID DG
IDPCB ADM
Existing resources will be applied to support this work.
Recommendations from review are implemented. December 2017 CIRID DG
IDPCB ADM
Enhance efficiencies in response activities (including medical case reviews). Management agrees with this recommendation. Review the program objectives for medical case reviews for serious cases of adverse events following immunization, to ensure program design and process alignment to objectives. Report on program review is completed. October 2016 CIRID DG
IDPCB ADM
Existing resources will be applied to support this work.
Action plan to address report recommendations is completed. February 2017 CIRID DG
IDPCB ADM
Articulate CIRID issues management and outbreak response activities, outputs and capacity. Tracking system/tool (that includes identification of activities / resources needed to be reassigned to the event) on issues management and outbreak response is developed and implemented. January 2018 CIRID DG
IDPCB ADM

1.0 Evaluation Purpose

The purpose of the evaluation was to assess the relevance and performance of the Immunization and Respiratory Infectious Disease program for the period of 2011-2012 to 2015-2016.

2.0 Program Description

2.1 Program Context

The immune system is a special network in the body that protects people from bacteria and viruses that cause diseases. A vaccine preventable disease (VPD) is an infectious disease which can be prevented by a vaccine. Immunization is the process by which people become protected from a disease (such as through vaccination). The dead or weakened germs in vaccines help the body's immune system to develop both antibodies and immune memory which together will help to prevent the disease if the body is exposed in the future.

In Canada, all federal, provincial and territorial jurisdictions, as well as local authorities have substantial roles, authorities and activities in immunization. These involve not only all health departments, agencies and ministries, but also numerous other public authorities and public/private partnerships at various levels. These organizations are involved in diverse aspects of Immunization Program planning, implementation, evaluation and support, including vaccine development, assessment, regulation, guidance and promotion. While each jurisdiction and each respective agency/authority has a distinct mandate and a unique operating context, the activities of these diverse bodies rely heavily on complementarity and collaboration.

2.2 Program Profile

The Centre for Immunization and Respiratory Infectious Diseases (CIRID) resides within the Public Health Agency of Canada (PHAC)'s Infectious Disease Prevention and Control Branch (IDPCB). CIRID has two divisions with the following areas of focus:

The Surveillance and Epidemiology Division (SED) undertakes activities to strengthen national surveillance and response capacity through:

  • National coordination of surveillance and epidemiologic analysis dealing with the incidence and prevalence of disease in large populations to assist both the development of evidence-based immunization recommendations and the investigation and management of VPD outbreaks by governments and health care providers, and to facilitate decision making and policy development in support of national and international prevention and control of influenza and other respiratory infectious diseases;
  • Facilitating F/P/T engagement (e.g., CIRC) for the development, adoption and implementation of national data and functional standards, policies and best practices to achieve comparable immunization registries across jurisdictions to support immunization delivery, immunization coverage monitoring, adverse events following immunization (AEFI) and supply management;
  • Conducting National Immunization Coverage Surveys and complementary studies that enable Canada to (1) track progress toward achieving the National Immunization Goals and Targets; (2) meets its international immunization coverage and disease elimination reporting obligations; and (3) report against PHAC performance measurement indicators;
  • Post-market surveillance of AEFI and vaccine safety issues, including the coordination of the Canadian Adverse Events Following Immunization Surveillance System (CAEFISS) and maintenance of a national database for the purpose of vaccine safety signal detection and stakeholder feedback on the safety profiles of all vaccines marketed in Canada;
  • Enhancement of national vaccine vigilance through a variety of mechanisms, such as harmonized and routine F/P/T reporting mechanisms, safety alert networks and committees, and improved communications; and
  • Aligning Centre surveillance activities with PHAC surveillance objectives and building new immunization vaccine safety and infectious diseases surveillance capabilities and capacities as needed to fulfil PHAC's mandate.

The Immunization Programs and Pandemic Preparedness Division supports vaccine guidance, supply, acceptance and uptake, research and innovation by:

  • Providing leadership, advice, coordination and support towards the development of immunization guidance for the use of vaccines and immunization programs for Canadians, delivered with key stakeholders and experts;
  • Developing strategies for (and the coordination of) vaccine supply activities to proactively prevent supply shortages;
  • Coordinating the development, implementation, evaluation and ongoing improvement of various knowledge translation and exchange (KTE) activities and products, in collaboration with internal and external stakeholders, to promote heath professionals' and the Canadian public's acceptance and uptake of recommended vaccines; and
  • Facilitating targeted collaboration with OGDs, P/T governments, industry, academia, NGOs, and public health experts to strengthen research networks and enhance immunization research and innovation.

Additionally, the Division undertakes activities under PAA sub-program 1.3.1 – Emergency Preparedness and Response to strengthen national pandemic preparedness and response capacity by:

  • Leading, coordinating and supporting seasonal and pandemic influenza activities of P/Ts and national stakeholders;
  • Supporting improvements to Canada's pandemic response capacity;
  • The development, maintenance and renewal of Canadian Pandemic Influenza Preparedness: Planning Guidance for the Health Sector (CPIP);
  • Providing oversight and management of pandemic influenza vaccine contracts to ensure adequate access to domestically produced pandemic vaccine; and
  • Providing support for a nationally coordinated approach to the stockpiling of antiviral medications to mitigate the risk of pandemic influenza, including the oversight and management of F/P/T governments' bulk procurement contracts primarily for the National Antiviral Stockpile (NAS) and the antiviral component of the National Emergency Strategic Stockpile (NESS).

The Director General's Office is accountable for the identification, research and analysis of issues and the provision of strategic input and policy advice to support the Centre in the delivery of national/international public health initiatives, policies and programs, including a review of the NIS framework to identify opportunities for strengthened F/P/T collaboration on Immunization Program issues and initiatives.

2.3 Program NarrativeFootnote i

Responsibility for public health is shared across jurisdictions, with the P/Ts having primary responsibility for immunization programs and delivery (a stakeholder summary is provided in Appendix 1). The federal role, as reflected in the activities and outputs of the Program, is considered to be one of leadership and coordination. In this role, the Program helps to protect Canadians from public health threats posed by VPDs by attempting to influence and support P/T and other public health stakeholders in their actions. This is done in order to appropriately manage risk, and to provide timely access to immunizations that are safe and administered securely, effectively and efficiently.

The activities carried out by the Program can be categorized into four broad groups:

  • surveillance;
  • leadership, consensus-building and coordination;
  • knowledge development, translation and exchange; and
  • public health event management (these activities and their associated outputs are described in Appendix 2).

The assumptions and expectations associated with the levels of outcomes contained in the Immunization Program Logic Model and its associated Theory of Change are presented in the following table:

Table 1: Theory of Change
Immunization Program Theory of Change Assumptions and Expectations
Outcome Stream 1 Outcome Stream 2
Immediate Outcomes
The program assumption is that the information and resources, supplemented by mechanisms that facilitate collaboration among all of the program stakeholders (including F/P/T organizations) that are delivered through the program, will equip these stakeholders with the means to collaboratively prevent and control health risks associated with VPDs and vaccine safety issues. The program assumption is that if science, public health, and other health professionals and organizations (including F/P/T organizations) have the capacity to deliver coordinated responses to health risks associated with VPDs, then Canadians and others living in Canada, including targeted and at-risk populations, will have access to the vaccines and information and tools they need to protect themselves from the health risks associated with VPDs.
Intermediate Outcomes

If the mechanisms that are put in place through the Program's activities and outputs are appropriate and functioning well, and if the program stakeholders are provided with the information and resources they need to prevent and control health risks associated with VPDs and vaccine safety issues, then there is a reasonable expectation that program stakeholders will have the capacity to deliver coordinated and effective responses to health risks associated with VPDs and vaccine safety issues.

At this outcome level, the program assumption and expectation is that stakeholders within the Canadian PH system are applying a risk-management approach to the prevention and control of vaccine preventable diseases, which includes demonstrating the consistent and sufficient capacity to reach and immunize target groups, as well as taking other appropriate actions to reduce the impact of VPDs. It is also recognized that the Program's role is to assist in the identification of at-risk groups and the development of information to support public health/immunization stakeholders in order to better reach populations vulnerable to VPDs.

If Canadians and others living in Canada, including targeted and at-risk populations, are provided with access to vaccines and the information and tools they need to protect themselves from VPDs, then there is a reasonable expectation that many will take positive actions to protect themselves from health risks associated with VPDs.

At this outcome level, the program assumption and expectation is that the constructive engagement and combined influences of (P/T and other) program stakeholders directly reaching, immunizing and providing other related services (including information), along with information generated by the Program, results in the appropriate awareness and basic understanding of disease risks and related immunization benefits by targeted at-risk groups and their surrounding communities. This information is used to assist target groups and their support communities with making informed decisions about immunization (i.e., to appropriately use immunization services and to take other related preventative actions).

Ultimate Outcome
The final link in the program theory of change is based upon the assumption that if program stakeholders have the capacity to deliver coordinated and effective responses to health risks associated with VPDs and vaccine safety issues, and Canadians and others living in Canada are taking positive actions to protect themselves from the health risks associated with VPDs, then a reasonable contribution will have been made to the high-level performance expectation that Canadians and others living in Canada are protected from health risks associated with vaccine preventable diseases.

The connection between these activity areas and the expected outcomes is depicted in the logic model (see Appendix 3).

2.4 Program Alignment and Resources

The program's financial data for the years 2011-2012 through 2015-2016 are presented below (Table 1). Overall, the program had a budget of $181.8 million over 5 years.

Table 2: Program Resources ($ in Millions)*
Year Gs&Cs SPAs O&M Salary Total
2011-2012 $0.2 $27.1 $20.8 $7.3 $55.2
2012-2013 $0 $5.5 $18.7 $6.5 $30.7
2013-2014 $0 $18.5 $18.2 $6.0 $42.7
2014-2015 $0 $3.8 $17.9 $6.8 $28.5
2015-2016 $0 $0 $17.9 $6.8 $24.7
Total $0.2 $54.9 $93.5 $33.4 $181.8
* Data verified by the Office of the Chief Financial Officer

3.0 Evaluation Description

3.1 Evaluation Scope, Approach and Design

The scope of the evaluation covered the period from April 2011 to March 2016 and included various activities such as National Immunization Strategy leadership and coordination; vaccine guidance; public and professional education and outreach to promote vaccine acceptance and uptake; security of vaccine supply; surveillance and epidemiology; coordinated immunization research, vaccine innovation and development; pandemic preparedness and coordinated outbreak response.

The evaluation did not cover emergency response activities that fall outside of the responsibility area of CIRID or those that are the responsibility of other jurisdictions (i.e., provincial and territorial responsibility for funding, program planning and the delivery of immunization programs within their own jurisdiction).

The evaluation issues were aligned with the Treasury Board of Canada's Policy on Evaluation (2009) and considered the five core issues under the two themes of relevance and performance, as shown in Appendix 4. Corresponding to each of the core issues, specific questions were developed based on program considerations and these guided the evaluation process.

An outcome-based evaluation approach was used for the conduct of the evaluation to assess the progress made towards the achievement of the expected outcomes, whether there were any unintended consequences and what lessons were learned. Based on the objectives, a non-experimental design was deemed appropriate for this evaluation.

Data for the evaluation was collected using various methods, which were document (including performance measurement information), media and literature reviews, an international review, key informant interviews and a survey of stakeholders. More specific detail on the data collection and analysis methods are detailed in Appendix 4. In addition, data were analyzed by triangulating information gathered from the different methods listed above. The use of multiple lines of evidence and triangulation were intended to increase the reliability and credibility of the evaluation findings and conclusions.

3.2 Limitations and Mitigation Strategies

Most evaluations face constraints that may have implications for the validity and reliability of evaluation findings and conclusions. The following table outlines the limitations encountered during the implementation of the selected methods for this evaluation. Also noted are the mitigation strategies put in place to ensure that the evaluation findings can be used with confidence to guide program planning and decision making.

Table 3: Limitations and Mitigation Strategies
Limitation Impact Mitigation Strategy
Key informant interviews are retrospective in nature As interviews are retrospective in nature, this may lead to the provision of recent perspectives on past events. This can impact the validity of assessing activities or results relating to improvements in the program area. Triangulation of other lines of evidence to substantiate or provide further information on data received in interviews.
Low response rate on stakeholder survey (26%) Lack of reliable evidence information on certain knowledge products, such as the Parent's Guide to Vaccination Triangulated patterns of results with previous studies conducted by the program in 2015. While questions were different and trend data was not available, the patterns of results remained similar.
Low take-up of Local Public Health Unit survey Lack on information on certain knowledge products, such as the

Parent's Guide to Vaccination

As responses were minimal, this survey was not used. Instead previous surveys gauging from this stakeholder group provided insights on various products and services.
Performance measurement information limited in certain areas Lack of trend information availability for many indicators Expert interviews and document reviews were used to assess progress over time, as well as supplemented by strong trend information available in some areas (such as disease incidence).

4.0 Findings

4.1 Relevance: Issue #1 – Continued Need for the Program

SUMMARY: Immunization affects all Canadians, with infectious diseases impacting across national and international jurisdictions. There is a continued need for PHAC's immunization activities as vaccine preventable diseases remain persistent in Canada. Challenges and barriers to immunization remain, such as vaccine hesitancy and uptake, surveillance, and access to vaccines.

Immunization, through vaccination, is considered to be one of the greatest public health achievements of the 20th century, providing a cost-effective tool to control and eliminate life-threatening diseases.Endnote 2 It is estimated to have saved more lives in Canada over the past 50 years than any other single health intervention.Endnote 3

Burden of vaccine preventable diseases

In recent years, Canada has eliminated endemically-transmitted measles and rubella. With the exception of a few diseases that remain endemic and burdensome in Canada by having cyclical peaks or regional outbreaks (e.g., pertussis,Endnote 4 invasive meningococcal disease,Endnote 5 invasive pneumococcal disease), the current burden of VPDs has been at low levels or declining since the early 2000s.

VPDs in Canada

While VPD incidence is low in the population, the impact of these diseases can be severe, leading to lifelong morbidity or death. VPDs in Canada include:

Measles: Measles is one of the most highly communicable infectious diseases. In Canada, the last endemic measles case occurred in 1998. While immunization coverage for measles remains high in Canada, under- and un-immunized communities are particularly vulnerable to imported cases. Because measles activity remains endemic in other parts of the world, there have been small numbers of imported cases in Canada that on occasion have resulted in outbreaks, which have been successfully contained and have not led to the reintroduction of endemically transmitted measles in Canada. In 2015, the incidence of measles in Canada was 5.5 cases per 1,000,000 population, with 196 cases across four provinces. The majority of cases (87.2%, n=171) were not immunized and both age-specific incidence rates and case counts were highest among those aged 10 to 14 years (29.5 cases per 1,000,000 population, n=55). This was due in large part to a sizeable outbreak in a non-immunizing religious community.Endnote 6

Invasive Pneumococcal Disease (IPD): Invasive pneumococcal disease is an acute and serious illness caused by the bacterium Streptococcus pneumoniae. Among vaccine preventable diseases, IPD incidence is high, with approximately 3,000 cases of IPD reported in Canada each year (3,178 reported cases in 2014).Endnote 7 IPD is generally found among the very young (those under the age of 5) or the elderly (those 65 and older).

Pertussis: Pertussis, also known as whooping cough, is a contagious infection caused by bacteria called bordetella pertussis. Pertussis is a vaccine preventable disease of high incidence worldwide and is endemic in Canada, with 1,000 to 3,000 domestic cases (1,529 reported cases in 2014)Endnote 8 caused annually by cyclical outbreaks, and is most dangerous for children under 1 year old, especially if they are un-vaccinated or under-vaccinated.

Influenza: Influenza (the flu) infects an estimated ten to twenty percent of the population each year.Endnote 9 It is caused mainly by two types of viruses (influenza A, influenza B), and spreads very easily from person to person. Different influenza strains circulate in the population during flu season and can vary from year to year. Influenza accounts for a high annual burden of morbidity and mortality; influenza rates are the highest in children aged five to nine years old, with serious illness and death highest in children aged less than two years old and adults aged 65 years or older.Endnote 10 Seasonal influenza-related hospitalizations and deaths fluctuate depending on a number of factors, including circulating strain and annual vaccine match. Vaccination is the most effective way to prevent influenza and its complications. Annual vaccination is required because the body's immune response from vaccination diminishes within a year. Also, because influenza viruses change often, the vaccine is reviewed each year and updated as necessary to better match the changing viruses.

The ease of global travel also has an impact; if the disease exists in other parts of the world, there is the potential for that disease to be brought to Canada, along with the consequences of transmitting these diseases as mentioned above.Endnote 11 An example is measles, which is still a leading cause of death around the world – in 2014, there were 114,900 measles deaths globally.Endnote 12 Un-immunized travellers are a susceptible population for the disease in Canada.

Additionally, it may be difficult for health professionals to recognize the symptoms of the disease as they may not have seen them previously,Endnote 13 or for the population at large to recall the consequences of having certain diseases.Endnote 14

Herd/community immunity

High immunization rates are required for populations to develop herd immunity to highly infectious diseases (for example, 95% for measles).Endnote 15,Endnote 16 Herd immunity is the term given to the concept that, when a certain threshold of the population has developed immunity to a disease, it helps protect the population at large by limiting or eliminating the possibility of its transmission.

This then helps protect more susceptible segments of the population, such as:

  • infants who are too young to be vaccinated;
  • people who cannot be vaccinated for medical reasons (e.g., certain immunosuppressed people who cannot receive live vaccines); and
  • people who may not adequately respond to immunization (e.g., the elderly).Endnote 17

Based on coverage surveys, national vaccination coverage estimates for children and adults are somewhat below current targets.Endnote 18,Endnote 19 There is a need to increase vaccine uptake and coverage among groups that are below recommended rates, who are at increased risk of infection, or have limited access to vaccination.Endnote 20,Endnote 21,Endnote 22,Endnote 23 Such groups are adults and seniors,Endnote 24 populations in rural areas,Endnote 25 homeless youth,Endnote 26 new immigrants and refugees,Endnote 27 health care workers,Endnote 28 and pregnant women.Endnote 29

Factors and challenges to vaccine uptake

According to the literature, improved vaccine uptake faces several challenges, including the cost of vaccines that are not publically available; lack of access to vaccines due to living in a rural areas or not having a health care provider; nonmedical immunization exemptions (e.g., religious, philosophical or personal beliefs);Endnote 30 fear of needles;Endnote 31 and the usage of alternative medicine to vaccines (nosodes or natural health products).Endnote 32 A media review that was conducted on immunization in Canada outlined health practitioner concerns over the availability and regulation of natural health products and alternatives to vaccines in Canada.Endnote 33

Vaccine-hesitant individuals are those in between vaccine acceptance and refusal.Endnote 34 There is a need to influence the knowledge, attitudes and beliefs of vaccine-hesitant individuals, who may believe there is a link between autism and vaccination,Endnote 35 can favour alternative medicine over vaccination,Endnote 36 and refuse vaccination to build natural immunity to VPDs.Endnote 37 Further information can be found in section 4.4.2.

Increasing financial burden of vaccination programs

The overall public health spending on childhood immunization per child has increased from $35 in 1986 to over $450 for boys and $800 for girls in 2009 due to higher costs and more products available to protect Canadians from diseases that are now vaccine preventable.Endnote 38 Furthermore, the projected financial burden of vaccination will increase in the future as vaccination becomes more complex. There are increasing numbers of vaccines for the same infectious agents,Endnote 39 new vaccine technologies such as new adjuvants,Endnote 40 higher dosage vaccines,Endnote 41 new biomarkers for safety,Endnote 42 and increasing spectrum of vaccine recipients (e.g. niche vaccines for certain groups) and therapeutic vaccines.Endnote 43

Adult vaccination may also increase in the future, adding to the cost and accessibility challenges, which may be partially due to the development of new vaccines (e.g., zoster), waning immunity over time,Endnote 44 risk of professional exposure, travel, and immigration.Endnote 45

4.2 Relevance: Issue #2 – Alignment with Government Priorities

Immunization is a priority of the Government of Canada, as evidenced by the inclusion of immunization in recent federal agenda-setting documents.

Immunization and vaccine preventable diseases have been a longstanding priority for the Government of Canada. In 2003, the Government of Canada recognized the importance of immunizations by providing $45 million over five years in the federal budget to establish the National Immunization Strategy. This strategy is evergreen and ongoing, and receives $5.6 million annually. At the same time, the Government of Canada articulated the health of Aboriginal and Northern populations as a priority by providing $32 million over five years for a national on-reserve immunization strategy.

More recently, since 2010, the Government of Canada has indicated through the Speech from the Throne and federal budget plans, that the safety and security of Canadians and their families is a priority.

  • In 2015, the mandate letter to the Minister of Health specifically identified increasing vaccination rates as a top priority. The mandate letter also made reference to the need to adopt new digital health technology to improve access, increase efficiency and improve outcomes for patients.Endnote 46

One of PHAC's key priorities is the prevention and control of infectious diseases, including vaccine preventable diseases. This priority has been highlighted in PHAC's strategic plans, Corporate Risk Profiles, and Reports on Plans and Priorities, since at least 2010. Moreover, the Corporate Risk Profiles from 2010 to 2013-2015 highlight the importance of infectious disease as a PHAC priority by indicating the risks that may arise if the appropriate actions are not taken. For example, without appropriate action, there is a risk that PHAC will not be able to monitor, detect and coordinate a response to an infectious disease outbreak and that effective medical countermeasures, including vaccines, will not be available.

PHAC's recent Reports on Plans and Priorities highlight enhancing surveillance, ensuring a safe, affordable and accessible supply of vaccines, and collaboration with other federal, provincial and territorial governments, technical experts and other health professionals, on vaccine preventable diseases.

4.3 Relevance: Issue #3 – Alignment with Federal Roles and Responsibilities

Immunization in Canada is an area of shared jurisdiction. Provincial and territorial governments are responsible for funding, and immunization program planning and delivery, including decisions surrounding inclusion of vaccines in publicly-funded schedules. At the federal level, approval of vaccines is done by Health Canada, and PHAC has a role in activities such as health promotion, research and surveillance, guidance development and public leadership and coordination.

PHAC is mandated to have a clear public health leadership and coordination role in the area of immunization and respiratory infectious disease activities.

Legislative, program and policy authorities establish a clear role for PHAC's immunization and respiratory infectious disease activities. PHAC's activities in the area are aligned with the Minister's legislative authorities under the Department of Health Act (1996), from which the public health mandate of PHAC is derived, and the Public Health Agency of Canada Act (2006), which established PHAC for the purpose of assisting the Minister of Health in exercising or performing the Minister's powers, duties and functions in relation to public health. PHAC's avian and pandemic influenza preparedness activities are further aligned with the Minister's legislative authorities under the Emergency Management Act (2007).

PHAC policy and program authorities outline a public health leadership, coordination and collaboration role in the following areas: pursuit of a national immunization strategy; provision of a forum for collaborative F/P/T work; immunization research priority-setting; and collaborative activities to address vaccine supply issues. PHAC's immunization activities are also aligned with policy and program authorities that outline a federal public health role in strengthening national immunization infrastructure in the following areas: vaccine safety monitoring; surveillance; immunization registry network; public and professional education; and approaches to enhancing immunization coverage and acceptance in special populations.

Program and policy authorities outline a role for PHAC in ensuring federal capacity to respond to avian and pandemic influenza, including vaccine readiness and clinical trials; contribute to national antiviral stockpile; surveillance; emergency preparedness; and influenza research network.

While the PHAC role is generally clear to stakeholders, clarification is needed regarding certain activities, such as the federal public health role in immunization registries, research and surveillance, and health promotion activities.

Consistent with program authorities, a majority of key stakeholder respondents indicated that PHAC has a clear role in the following areas: vaccine supply (76%); guidance development (67%); vaccine safety (65%); and pandemic preparedness (64%).

However, stakeholder survey respondents indicated a lack of clarity in their understanding of PHAC's role in immunization registries. Less than a third of survey respondents (29%) agreed that PHAC's role is clear in the area of immunization registries; and less than a quarter of survey respondents (19.3%) agreed that PHAC's role is appropriate in the area.

Stakeholder survey respondents had mixed perceptions of the PHAC's role in research, surveillance and health promotion. For example, while just over a third of respondents (38%) agreed that the difference between PHAC's role and that of partner organizations was clear in the area of research and surveillance, roughly a third (28%) noted that the difference in roles is not clear, and another third (30%) neither agreed or disagreed that the difference in roles is clear.

With regards to health promotion, while almost half of respondents (47%) indicated their agreement that PHAC's role and PT government's roles are clear in the area, more than a third (36.1%) of respondents disagreed.

Commentary from this group on PHAC's role in these areas provided some clarity about their confusion. A few concentrated on the Agency's leadership role versus what they perceived as coordination in immunization programming. A few also noted their confusion regarding what the role actually was and how far it extended into other jurisdictions. Furthermore, more expressed confusion regarding the differences between PHAC's role and other health portfolio entities (such as CIHR and Health Canada), specifically in the areas of health promotion, guidance development, research and surveillance, and national registries.

International obligations

PHAC's immunization and respiratory infectious disease activities contribute to Canada's compliance with the International Health Regulations (IHR) (2007). Under the IHR, Canada is required to establish and maintain public health core capacity requirements, including surveillance capacity, and has an obligation to report certain diseases immediately (such as a SARS-like influenza). Other international commitments to which PHAC's pandemic influenza activities contribute include the WHO Pandemic Influenza Preparedness Framework and the North American Plan for Animal and Pandemic Influenza; polio eradication with the WHO and maintaining measles, rubella and congenital rubella syndrome elimination with PAHO.

4.4 Performance: Issue #4 – Achievement of Expected Outcomes (Effectiveness)

4.4.1 To what extent have the immediate outcomes been achieved?

Immediate outcome #1: Canada has mechanisms such as bulk vaccine purchasing and registry standards in place to prevent and control health risks.

Canada has effective mechanisms to prevent and control health risks, such as the Bulk Purchasing Program in partnership with PWGSC. While comparable immunization registries are still needed to capture a picture of immunization coverage in Canada, as well as providing needed information for immunization delivery, vaccine supply management, and adverse event reporting, many challenges still exist in establishing such a system. PHAC, in collaboration with partners, has taken steps to mitigate the challenges associated with this gap.

Vaccine supply in Canada

All 14 F/P/T jurisdictions purchase and use vaccines through the Bulk Purchasing Program (BPP) for their publicly funded immunization programs. Each authority determines which vaccines are to be included, the schedules that will be followed, and the design and implementation of immunization programs in their respective jurisdiction. The planning, procurement and use of vaccines including for routine public immunization programs and for special program purposes (e.g., tropical disease vaccines for military and foreign service personnel serving abroad) is handled by the appropriate immunization authority within each jurisdiction, often in collaboration with agencies that support the procurement and supply management processes. Vaccine supply also includes a range of vaccines used for privately funded and delivered immunization programs and services (e.g., people travelling to a country with diseases not found in Canada), which also play a vital role in health protection.Endnote 47

Managed by Public Services and Procurement Canada (PSPC), the BPP provides a common service function in the procurement process for participating jurisdictions. The BPP provides a single "window" for better negotiated prices, contract administration allowing for early identification of potential supply issues, and serves as a mechanism for a coordinated response and integration of contracting strategies that enhance the security of vaccine supply.Endnote 48 It covers many, but not all, of the vaccines obtained and used by F/P/T immunization authorities and their client agencies.

A majority of P/Ts purchase vaccines through the BPP. Roughly three quarters of stakeholder survey respondents (74%) indicated that between 1-10% of vaccines for their jurisdictions were purchased outside of the BPP. Quebec accounts for most of the vaccines purchased outside the BPP; Quebec purchases only three vaccines through the BPP (flu, rotavirus and HPV). Reasons cited by surveyed stakeholders and key informants for purchasing outside the BPP vary from price, service, flexibility, transparency of process, product availability through the BPP to program preference for purchasing outside the BPP.

Vaccine supply governance also appears to work well through the Vaccine Supply Working Group (VSWG). The VSWG is co-chaired by representatives from PHAC and a province or territory, and reports to the Pan-Canadian Public Health Network's Canadian Immunization Committee. In a recent stakeholder survey conducted for this evaluation, 85% of respondents who were members of the VSWG assessed the working group to be very effective in areas such as appropriate membership, sharing of information and best practices, adequate connection to a government structure and decisions made in a timely manner. Some external key informants expressed the desire for enhanced transparency of the decision-making process, specifically regarding blended prices for the influenza, which was also noted in the 2011 VSWG survey of members. In addition, these key informants noted that timing of P/T budget allocation with contract negotiations may not be well aligned with their own immunization programming.

Immunization registries in Canada

Immunization registries are an important surveillance tool in assessing immunization coverage and vaccine uptake, and there is a federal public health role in facilitating the development of an immunization registry network. Immunization registries are gradually being implemented by provinces and territories, but evidence shows that registries across jurisdictions differ in content and design. Some P/Ts also have multiple registries within their own province. This work is the responsibility of the Canadian Immunization Registries and Coverage (CIRC) Network.Footnote ii The CIRC Network is co-chaired by representatives from PHAC and a province or territory, and reports to the Pan-Canadian Public Health Network's Canadian Immunization Committee. PHAC, together with the leadership and direction of the CIRC Network, has also developed national standards for immunization coverage and has conducted National Immunization Coverage Surveys every two years for children and adults to estimate coverage. Coverage surveys are addressed further in section 4.4.2.

Immunization registries can be useful tools to help determine the level of protection a population has against vaccine preventable diseases, and while immunization registries are part of the P/T responsibilities for health care delivery, the federal government has funded other organizations such as Canada Health Infoway to help support this work. Since 2001, Infoway has received $2.1 billion to fund five activity streams, one of which is for the development and implementation of electronic health records, including immunization registries.Endnote 49,Endnote 50 As of November 2015, CIRID's Immunization Coverage and Information Systems Section (ICIS) has been involved in several projects and working groups with Infoway as part of ICIS' immunization registries portfolio. Infoway maintains membership on CIRC Network Task Groups, and ICIS is also involved in collaborative communities led by Infoway. These cross-memberships are an effort to enhance collaboration and avoid duplication of effort in the work towards common goals.

Immediate outcome #2: Program stakeholders have required information and resources

In partnership with others, PHAC develops critical knowledge products for the prevention and control of vaccine preventable diseases and respiratory diseases such as influenza. While challenges have been noted regarding the timeliness of some of this information, PHAC and its partners are currently taking steps to ensure comprehensive information is available to program stakeholders in a timely fashion following the introduction of new products, or changing indications or evidence.

PHAC supports the development and publication of the following immunization knowledge products: the Canadian Immunization Guide (CIG), National Advisory Committee on Immunization (NACI) Statements and FluWatch reports. These products are targeted to health care providers (i.e., front-line clinicians, public health practitioners and physicians) and P/T immunization programs. In addition, PHAC has knowledge products such as weekly measles reports that are posted on the website.

The Canadian Immunization Guide

The CIG, first published in 1979, is developed based on NACI Statements and recommendations and the work of the Committee to Advise on Tropical Medicine and Travel, and it is reviewed and updated as new evidence (e.g., recommendations) becomes available.

Between 2013 and 2015, the CIG homepage was the second most frequently visited PHAC webpage, second only to the PHAC homepage in two out of these three years. As with web knowledge products for Canadians, web views tend to increase cyclically (flu season) or during events such as measles outbreaks. Between 2013 and 2015, the average number of daily visits increased with each consecutive year (33,872 in 2013; 43,932 in 2014; and 52,568 in 2015) with each year seeing an increase in visits between September and November, most likely the result of the rollout of provincial and territorial seasonal influenza programs. CIG Chapter webpages also saw a trend in increased web traffic during this period. Of note, in February 2015, there was a sharp peak in the average number of daily webpage visits which coincided with a measles outbreak across Canada.

External key informants noted that the CIG is a highly respected, trusted guide in the field, and that every provider has a copy in either book form or electronically. The guide is used to communicate with frontline providers. Recent stakeholder surveys conducted by the program confirmed that the CIG is a key knowledge product that respondents are aware of and use to inform their work (82.1% and 87% respectively). The guide is most often used to inform the development of policies or to make decisions; to provide information to patients or clients, and for teaching purposes. Furthermore, respondents strongly agreed or agreed that the content of the guide is both accurate (88.8%) and up-to-date (78.2%).

National Advisory Committee on Immunization (NACI) Statements

NACI is an external advisory body that makes recommendations for the use of vaccines currently or newly approved for use in humans in Canada, including the identification of groups at risk for vaccine preventable diseases for whom vaccination should be targeted. NACI knowledge syntheses, analyses and recommendations on vaccine use in Canada are included in published literature reviews, statements and updates. The NACI Statements are detailed, evidence-based knowledge products intended for P/T immunization programs and health care providers.Endnote 51

Various studies over the years have indicated the importance and usefulness of NACI Statements. Recently, 81% of stakeholders surveyed were aware of and used NACI Statements – of those, 90% reported that the content was accurate. Almost three in four (72%) reported that they used NACI Statements to inform development of policies or to make decisions, for teaching purposes, and to develop programs. This was underlined by external key informants who referred to NACI Statements as 'the bible' and 'essential in public health'. A recent stakeholder survey (April 2016) noted that 81.3% of respondents were aware of and used the NACI Statements, with respondents strongly agreeing or agreeing that the content is both accurate and timely (90% and 67.1%, respectively).

Similarly, a 2015 survey of NACI members and stakeholders noted that three-quarters (76%) of respondents referred to NACI Statements with nine in ten expressing satisfaction with NACI Statements' trustworthiness and quality of information. Furthermore, 88% of respondents agree that NACI recommendations represent sound advice based on thorough analysis of the best available evidence.

NACI provides medical and scientific guidance on immunization products; policy and program immunization guidance was developed by the Canadian Immunization Committee that publishes a variety of statements, including those on rotavirus, varicella and human papillomavirus that were published in 2014. Summaries of these statements were published in the Canada Communicable Disease Report with the full statements made available on the Government of Canada publications website. An integration of scientific and programmatic advice would result in comprehensive guidance on immunization in Canada, providing all jurisdictions with evidence-based information to help make immunization decisions, ultimately leading to greater efficiencies in guidance. Internationally, most advisory groups provide both forms of guidance; most National Immunization Technical Advisory Groups (NITAGs) provide scientific and programmatic advice, enabling national policy makers and program managers to make evidence-based immunization-related policy and programmatic decisions.Endnote 52 This is discussed further in Outcome #4.

FluWatch Reports

FluWatch is a PHAC national surveillance system that monitors the spread of flu and flu-like illnesses on an on-going basis. Weekly FluWatch reports contain specific information for health professionals on flu viruses in Canada. The FluWatch program consists of a network of labs, hospitals, doctor's offices and provincial and territorial ministries of health and aims to:

  • Provide timely and up-to-date information on flu activity in Canada and abroad to health professionals as well as Canadians;
  • Monitor circulating strains of the flu virus (e.g., H1N1) and changes in antiviral resistance; and
  • Provide surveillance information that the World Health Organization can use to assist in decision making for the following season's flu vaccines.Endnote 53

While fewer stakeholders reported using FluWatch (58.7%) than other PHAC products, of those that did report using Flu Watch, over three in four found it useful and up-to-date. Several external key informants also noted the benefits of this product for determining their own surveillance needs, demonstrating trends and providing information for patients. A recent stakeholder survey found that 58.7% of respondents were aware of and used the weekly FluWatch reports and 48% were aware of and used the annual FluWatch reports. Respondents noted that FluWatch weekly content is accurate (77.5% strongly agreed or agreed), that it was up-to-date (74.2% strongly agreed or agreed), and that the content is timely (78.7% strongly agreed or agreed).

Educational Opportunities for Health Care Providers

PHAC conducts other activities to translate knowledge such as working with partners to coordinate the biennial Canadian Immunization Conference which brings together a broad range of stakeholders to share leading knowledge and advance the science in the field of immunization. The Conference provides a national forum for knowledge exchange and networking with an approximate attendance at the 2014 conference of 880 participants. The conference was organized by the Canadian Public Health Association (CPHA) in collaboration with PHAC, the Canadian Association for Immunization Research and Evaluation (CAIRE), and the Canadian Paediatric Society (CPS).

Canadian Immunization Conference participants rated the quality of their Conference experience, with a majority of participants rating it very highly (85% in 2012 and 92.3% in 2014), stating that it exceeded expectations. Further, 92% of conference participants intended to share knowledge with colleagues, with 54% intending to use the content for program planning and 51% for patient/client education.

PHAC also supports the Canadian Paediatric Society's online Education Program for Immunization CompetenciesFootnote iii for health professionals. This program is now a mandatory part of the College of Pharmacist's curriculum in provinces where pharmacists provide immunizations. In September 2011, Australia requested PHAC's authorization to use the Immunization Competencies as a guide to development of a framework for vaccination competency standards. In 2012-2013, a contract was put in place with the Canadian Paediatric Society to update modules and make the online program easily available and accessible. Moreover, it is offered for free to any publicly funded health professional.

Even though PHAC develops critical knowledge products, there have been challenges with providing information to program stakeholders. External key informants and stakeholders surveyed for this evaluation noted that the website is difficult to navigate and that certain pages, such as those for the NACI Statements, case definitions and certain influenza pages have been archived even though they are current documents.

Immediate outcome #3: Canadians have information and tools to protect themselves

There are indications that Canadians access PHAC's information tools to protect themselves from vaccine preventable diseases. However, there are knowledge gaps, specifically related to vaccination status, and areas for improvement, including access to personal vaccination information and enhancing knowledge about vaccine preventable diseases in a variety of areas.

PHAC, in collaboration with its partners, develops or makes available key information tools and products for Canadians including the immunization schedules, the immunization schedule tool, the Parent's Guide to Vaccination (previously known as the Parent's Guide to Immunization) and the ImmunizeCA mobile app. Additionally, PHAC produces other knowledge products; for example: infographics, posters, videos, and the Guidelines for the Prevention and Control of outbreaks such as measles, mumps and invasive meningococcal disease.

Immunization Schedule and the Immunization Schedule Tool

As previously mentioned, NACI makes recommendations for the use of vaccines currently or newly approved for use in humans in Canada. Following an assessment of recommendations, evidence and local information and data, provinces and territories determine the best schedule for their region, leading to immunization schedules that vary between P/Ts.

The provincial/territorial immunization schedule tool is specific to the age of the child/youth based on where the child lives in Canada (e.g., Schedule for Children under 6 years of age and Schedule for children and youth in grades 1 through 12 or Grades 1 to Secondary V in Quebec). The top two Immunization Coverage and Immunization Systems' products in 2015 were the Immunization Schedule Tool and Immunization Schedules.Endnote 54 Page views for the Immunization Schedules increased from 2013 to 2014, and also saw an increase in early 2015 (aligned with the measles outbreak, which began the end of January into February 2015). Page views have decreased throughout 2015; however, the decrease may be attributable to a large migration of websites to healthycanadians.gc.ca. Whereas previous content was spread over multiple pages, content on the new site is contained on a single page which may mean that overall uptake did not decrease as much as it appeared. This trend was similar for the immunization schedule tool. Canadians and program stakeholders turn to PHAC's webpages for information during events, such as the 2015 measles outbreak, as demonstrated by a spike in website hits. These webpages can be considered as a knowledge product in and of itself that helps enable Canadians to protect themselves from VPDs.

Parent's Guide to Vaccination

The Parent's Guide is the flagship deliverable of PHAC's public education campaign aimed at parents of children ages 0-2, It's Time to Immunize. It was the most regularly ordered publication within PHAC in 2014, with over 110,000 orders for printed copies of the 2014 updated guide but there were fewer than 30,000 page views in 2015, compared to over 100,000 page views for 2013-2014 which may again be due to the migration to the healthy Canadians website and a reduction of the number of pages available on this subject. While there is data available on the various outreach activities and the numbers of the Guide ordered or viewed online, there is no information available regarding Canadians' perceptions of the Guide (e.g., whether it is easy to understand and up-to-date).

Over the period covered by the evaluation, a variety of activities took place to support the Parent's Guide to Vaccination's public education campaign, It's Time to Immunize. For example, service level agreements were established with Service Canada to promote and distribute copies of the Guide (in June 2011, a total of 36,723 copies of the Guide were sent to 330 Service Canada Centres for distribution); collaboration with Canada Revenue Agency resulted in promotional campaign inserts being mailed along with the Universal Child Care Benefit Statements (in July 2011, inserts were mailed to 1.7 million households and to 3.8 million households in July 2013); and several outreach activities targeted to public health professionals at a number of conferences.

ImmunizeCA mobile app

With PHAC funding, the Canadian Public Health Association (CPHA), Immunize Canada and the Ottawa Hospital Research Institute have collaborated to develop a free app for smart phones and other mobile devices that will help Canadians keep accurate immunization records and make informed decisions about their health. The app allows Canadians to:

  • Easily record and store vaccine information;
  • Access vaccination schedules;
  • Manage vaccination appointments for the entire family;
  • Access evidence-based and expert-reviewed information about recommended and routine vaccinations for children, adults and travelers; and
  • Receive alerts about disease outbreaks in their area.Endnote 55

Since its launch in 2014, there have been 143,000 downloads of the ImmunizeCA app. External key informants identified several challenges with the app, for example it is not easy to use, the information is not transferable from one computer to another, from one device to another and between provinces and territories. As a result of the overall uptake of Phase I of the app, in April 2016 PHAC committed $3.5 million over three years to support the app into Phase 2, enhancing its feasibility and usefulness for Canadians in the areas for improvement that have been identified.

Knowledge, attitudes and beliefs

In addition to data on coverage estimates for vaccines, national immunization coverage surveys collect data on Canadians' knowledge, attitudes and beliefs about vaccines. According to the 2011 cNICS childhood survey,Endnote 56 doctors remain a trusted source of information when it comes to immunization, and most parents follow the advice of a doctor or nurse; among parents/guardians of a two-year old child who had visited a health care provider in the past year, 74% reported that a doctor or nurse had not discussed immunizations with them, which may have resulted in missed opportunities to discuss concerns regarding immunizations.

Furthermore, although 95% of parents/guardians believe that vaccines are safe, more than two thirds are concerned about vaccine side effects and more than one third believe that vaccines can cause the diseases they are meant to prevent. This shows that there is still an opportunity to increase public knowledge about immunization. These views are similar to those expressed in the Survey on Vaccine Hesitancy with Health Professionals that measured, among others, patients' most common statements regarding vaccines including: vaccines cause autism (79%) and vaccines might cause the disease it's supposed to prevent (44%).

Even though Canadians access PHAC's information tools, challenges remain in terms of knowledge gaps about personal vaccination information. The aNICS (2014) found that approximately three of four adults reported that they knew enough about the benefits of immunization, but of particular concern is that while 80% believed they had received all the vaccines required for someone of their age, after a validation exercise it was found that only 6% had the recommended number of pertussis and tetanus vaccines doses in adulthood. Without accurate, easily accessible records and the ability to pull the national picture out of the various jurisdictional registry solutions, Canada will continue to rely on telephone surveys to estimate immunization coverage.

Although not a federal public health responsibility, there are implications for missing immunization records, even for adults. For example, in March 2016, after a measles case presented at a hospital in Edmonton, over a hundred people were quarantined to prevent an outbreak because they were unable to provide written documentation of their vaccination records. This had repercussions on their ability to work (financial hardships), travel or be with their broader community.Endnote 57

In recognition of the importance of accurate immunization records for children that are easily accessible when needed, Canada Health Infoway conducted a survey and focus groups to examine parents' experiences managing their children's immunization records. While the survey found that a large majority of parents (70%) were confident that they could provide complete and accurate immunization information, 38% of parents felt that it would take a lot of time and effort to compile their children's immunization information. Focus group sessions revealed that most parents relied on family physicians, paediatricians or nurses to keep up-to-date immunization records and to inform them of upcoming vaccinations, while some parents felt that the school system played a larger role in ensuring all records were current, considering many had been notified by the school board about upcoming immunizations or about records that were not up-to-date. Parents did not have a clear idea of which jurisdiction/organization was responsible for immunization records. Moreover, new immigrant parents believed that the federal government had a record-keeping system given that many provided medical records to Immigration, Refugees and Citizenship Canada (although this is not the case).Endnote 58

Public education around the benefits of immunization is an ongoing challenge. Most internal and external key informants noted some challenges in this area (supported by the media and literature review) including a lack of knowledge regarding vaccine preventable diseases as people no longer see their impact, as well as competing information channels such as doctors, pharmacists, and the Internet. The Internet is used to disseminate anti-vaccination information to vaccine-hesitant people. There are many reasons for vaccine hesitancy such as concerns about the safety or efficacy of vaccines, and the belief that vaccine preventable diseases do not pose a serious health risk, and while the Internet has a lot of information about vaccines and vaccine preventable diseases, there is also misinformation, some of which can be harmful if used to make decisions about children's health. These challenges are not unique to Canada, according to key informants in other jurisdictions such as the United Kingdom, Australia and the United States.

Although CIRID has conducted a variety of social media activities such as numerous facebook and twitter messages, in addition to YouTube videos to promote the activities of the National Immunization Awareness Week and It's Time to Immunize Campaign, there were no data collected on subject-specific social media analytics for the Centre, making it difficult to fully analyse social media reach and engagement.

Immediate outcome #4: Canadians have access, through CIRID's stakeholders, to vaccines

While vaccine disruptions and shortages occur, PHAC and its stakeholders have the capacity and tools to address and minimize the impact. Availability of programmatic information was a factor in implementing vaccine recommendations across jurisdictions in a timely fashion.

Vaccine disruptions and shortages in Canada

Vaccine supply disruption is the result of unforeseeable, isolated incidents that can occur during manufacturing (e.g., loss of product or product/facility quality issue), during transport or storage (e.g., broken cold chain, expiry or other damage), or as a result of an increase in demand (e.g., regional or global disease outbreaks). This type of shortage may be limited in duration and scope, but can have longer lasting, broader potential to interrupt/disrupt immunization programs.

While vaccine disruptions and shortages occur in Canada, PHAC and its stakeholders have the capacity and tools to address and minimize the impact. For example, following the flu vaccine shortages of 2012 that mostly affected Saskatchewan and Prince Edward Island, the VSWG conducted an assessment of supply issues with recommendations to prevent future disruptions.

PHAC has started to track the number and duration of vaccine supply disruptions as a key indicator for their activities in this area. In fiscal year 2015-2016, there were 16 vaccine supply disruptions affecting 14 different vaccine brands. Of these, four disruptions began in a previous fiscal year and carried over into the current reporting period. Seven disruptions were resolved during the reporting period and only these disruptions have been included in the calculation of the median vaccine supply disruption duration. It should be noted that, for two of the resolved issues (for BCG and Quadracel), "resolution" involved formal notification by the supplier that the vaccine would no longer be available on the Canadian market.

Four disruptions were classified as "red" (of highest priority) with no vaccine available from the supplier, with the rest classified as "orange" (high priority) meaning that some vaccine was available, but access was limited. The median duration of a disruption was 152 days and disruptions ranged from a low of 30 days to a high of 1,270 days.

While analysis is limited given the lack of data from prior reporting periods, CIRID has determined that the identification of a single supplier as the predominant source of supply issues has indicated the need for follow-up discussions with that supplier and an action plan for better monitoring supply.

Access to vaccines in Canada

Provinces and territories make vaccine programming decisions for their own jurisdictions using various information sources and assessments. This includes guidance published by PHAC and provided by NACI. This guidance provides recommendations for vaccine use based on an assessment of available data and evidence focussed on disease burden, as well as immunogenicity, efficacy, safety and effectiveness of vaccines under consideration. Other analyses and programmatic information are also used by P/Ts, including economic considerations (cost-effectiveness, budget impact, and other economic analyses), feasibility, acceptability, ethical and equity considerations, and local public health system capacity to implement vaccine programs. This programmatic information was previously produced by the federal, provincial and territorial Canadian Immunization Committee, but is not currently available. Therefore, equitable access to vaccination programs in Canada may be impacted by jurisdictional decisions about whether to adopt NACI recommendations in their programs and the timeliness with which information is available to support this decision-making process.

A large proportion of NACI recommendations have been adopted by all provinces and territories, and of the 13 programs to address specific VPDs, only four are not equitably available in Canada (varicella, influenza, rotavirus, HPV for males). For example, the rotavirus vaccine is available to infants and children in all but three provincial and territorial public health programs. Similarly for adults, while all provinces and territories have the influenza vaccine as part of their public health program, three provinces have a targeted versus a universal program for influenza.

Providing equitable access to programmatic information in a timely fashion is an element of the National Immunization Strategy. A substantial number of stakeholders (both internal and external) stated that the provision of programmatic information, which aids in decision making, has not been timely. For example, in 1998, Canada issued the notification of compliance for the varicella vaccine. The following year, Health Canada published technical guidance provided by NACI for the vaccine (prior to the creation of PHAC), but programmatic information was not available. As individual provinces and territories made programmatic decisions on the vaccine, implementation of the publicly funded varicella immunization programs occurred gradually over seven years.

There was a national decline in serious varicella (chicken pox) infections, expectedly, following the implementation of varicella immunization programs over this seven-year period. There have been no varicella-related deaths since full provincial and territorial implementation and morbidity has also decreased. This is significant considering that, while the population may perceive varicella to be a mild disease, it can result in serious health outcomes. In the United States, there were approximately 100 deaths per year prior to introducing the vaccine. In Canada, there were 59 varicella-related deaths between 1987 and 1997, just before the vaccine was authorized for use in Canada and Health Canada issues technical guidance (provided by NACI) on the vaccine.Endnote 59

PHAC is aware of these issues and has examined mechanisms to improve the comprehensiveness and timeliness of available technical and programmatic advice through expanding the mandate of and by revisiting the roles and responsibilities of NACI and the Canadian Immunization Committee (CIC). Between 2013 and 2015, the Public Health Network (PHN) was extensively consulted on a federally-led model with an expanded NACI mandate to include socioeconomic and programmatic guidance; the new model, with implementation planning and pilots, is proceeding in 2016.

4.4.2 To what extent have the intermediate outcomes been achieved?

Intermediate outcome #1: Canada has the capacity to deliver a coordinated and effective response

PHAC activities, including surveillance and vaccine safety reporting, guidance development and research coordination, contribute to a coordinated and effective response. Lessons learned from H1N1 have been implemented and regular seasonal flu activities present opportunities to exercise the avian and pandemic influenza preparedness system.

Surveillance-informed guidance

A total of ten PHAC surveillance systemsFootnote iv contribute data to VPD, infectious bacterial disease and viral respiratory disease surveillance at PHAC. PHAC surveillance data contributes to a coordinated and effective response by generating critical data for determining disease rates and the development of PHAC guidance, among other activities. For example, in 2015-2016, FluWatch reported the identification of H1N1 as the predominant circulating strain, which was well-matched to the vaccine-recommended strain. In addition, there was a late start to the influenza season. This allowed PHAC to tailor public health messaging around expectations that younger populations would be at risk, the value in vaccination, and maintaining vigilance later in the season about identification, treatment and containing the spread of influenza.

Vaccine safety reporting

PHAC coordination, collation and sharing of adverse events following immunization (AEFI) reports contribute to national assurance of vaccine safety. PHAC receives AEFI reporting from P/T public health authorities and IMPACT data (an active paediatric hospital-based surveillance network for selected serious AEFIs) and maintains CAEFISS, the Canadian Adverse Events Following Immunization Surveillance System, a federal/provincial/territorial public health post-market vaccine safety monitoring system. Over 90% of CAEFISS reports are submitted by public health authorities in the P/Ts, who received them from local public health units.Endnote 60

Additionally, PHAC's collation of AEFI data in CAEFISS allows for signal detection, analysis and reporting. PHAC collates national-level AEFI reporting in the AEFI Quarterly and Annual Reports, which were generally reported by survey respondents to be accurate, up-to-date and timely.

PHAC and Health Canada share AEFI data to enable regulatory action related to vaccines marketed in Canada. For example, in 2016, PHAC detected a higher rate of adverse reactions to a vaccine, and worked with Health Canada towards a precautionary batch lot recall by the vaccine manufacturer to prevent any further possible adverse events. Health Canada also receives AEFI reports from Market Authorization Holders (MAHs), such as the pharmaceutical industry.

National-level guidance

PHAC guidance development activities, which are often informed by surveillance data (see above), contribute a coordinated response by providing national-level information and frameworks for health sector responses to VPDs and respiratory infectious diseases. PHAC produces a variety of different types of guidance, for example, the PHAC 2012 User Guide to Completion and Submission of AEFI reports was developed by a multi-jurisdictional collaborative working group (the Vaccine Vigilance Working Group) to provide technical assistance to jurisdictions when reporting AEFIs, to ensure consistent and harmonized reporting.

In another example, the Canadian Pandemic Influenza Preparedness: Planning Guidance for the Health Sector (CPIP) was updated based on H1N1 lessons learned and published online in 2016. The CPIP outlines a framework for pandemic preparedness and response for the Canadian health sector. Approximately 60% of stakeholders surveyed indicated that they are likely to use the updated CPIP to inform future changes to their approach to pandemic preparedness. See immediate outcome #2 for additional examples of the use and impact of PHAC national guidance.

Research coordination and acceleration

Key stakeholders reported that PHAC has contributed to coordination in the area of research priorities, primarily through the PHAC and National Research Council-led Action Plan on Vaccine Research, Innovation and Development. Stakeholders also highlighted that as part of the Canadian Immunization Research Network (CIRN), PHAC and CIHR collaborated to implement a Rapid Research Response Mechanism in July 2015, in order to expedite research in the case of a public health event of significance. This mechanism has been used for influenza and Ebola virus disease-related research, and was developed as a result of lessons learned from the H1N1 influenza pandemic of 2009.

Vaccine Supply Coordination

As outlined in section 4.4.1, PHAC-supported activities have contributed to ensuring a cohesive national response to vaccine supply issues and to economies of scale and better prices at the national level.

Seasonal flu activities contribute to pandemic preparedness

Key informants indicated that many of PHAC's seasonal flu activities, including FluWatch surveillance, vaccine supply coordination, AEFI surveillance collection and reporting, and NACI statements on influenza are crucial components of the PHAC preparedness activities for avian and pandemic influenza. A few key informants noted that PHAC's yearly seasonal flu activities contribute to regularly exercising PHAC's avian and pandemic influenza preparedness capacity because activities would generally be the same.

Tracking outbreak response

In addition to the above coordination activities, key informant interviews have indicated that PHAC has exercised its capacity to respond to immunization and infectious respiratory disease events during a number of events and outbreaks since 2011, including the PHAC response to the emergence of novel virus strains, (such as H5N1, H7N9 and MERS-CoV) and PHAC response activities during multiple P/T pertussis and measles outbreaks.

In the case of health events that are not escalated to the Health Portfolio Operations Centre, response activities to immunization events, infectious respiratory diseases and novel viruses are conducted by CIRID staff; the Centre does not currently have a system for documenting the level of effort and nature of these activities. While PHAC has developed a template for conducting rapid risk assessment in the case of an emerging event (the Rapid Public Health Risk Assessment Framework), PHAC did not routinely track or report on its ad hoc response activities, such as monitoring and guidance development for emergence of novel virus strains or issues management/strategic communications to coordinate and support a national and/or international response.

Intermediate outcome #2: Canadians take positive actions to protect themselves

While the vast majority of Canadians are immunized, there remain pockets of under- or un-immunized Canadians across the country. Challenges to improving immunization coverage include a lack of vaccine awareness or knowledge, as well as attitudes and beliefs. Health care workers, a trusted source of information for Canadians, have been linked to increased vaccine uptake.

How immunization coverage is estimated

It is difficult to provide an accurate assessment of the state of vaccine coverage in Canada because each P/T compiles coverage data differently (e.g., registries, surveys, paper-based, electronic) and has varied immunization schedules for different age groups at different times, making data comparability challenging. In an effort to address these difficulties, PHAC estimates national immunization uptake (coverage) through the childhood National Immunization Coverage Survey (cNICS) and the adult National Immunization Coverage Survey (aNICS). Coverage estimates inform immunization campaigns and programs, track Canada's progress toward achieving the National Immunization Goals and Targets for immunization coverage.

Both the childhood (cNICS) and adult (aNICS) coverage surveys are commissioned by PHAC every two years, providing the only national estimate of vaccine coverage. The cNICS is conducted by Statistics Canada and the aNICS is conducted by a private polling firm. The surveys do have the same limitations one expects to find in any national survey based on self-reporting, including: potential inaccuracy of self/parent reporting; low response rates; frequent methodological changes that make multi-year comparisons of coverage challenging; and, under-representation of special populations (e.g. First Nations people living on reserves, individuals whose first language is neither English nor French).Endnote 61

The vast majority of Canadian children are immunized

The World Health Organization (WHO) considers measles immunization coverage to be a reliable indication of the strength of childhood immunization programs.Endnote 62 In Canada, this is measured through cNICS coverage surveys. The cNICS measles immunization coverage rates provide a useful estimated national uptake of routine childhood immunizations recommended by NACI.

Measles vaccine coverage at two years of age in Canadian children has remained stable over the last two cNICS cycles in 2011 and 2013. In 2011, measles coverage for the age group was 88.4% and in 2013 it was 89.6%; coverage has remained below the 97% target set in 1993 at the Consensus Conference on Measles.Endnote 63 The cNICS estimates likely represent an underestimate of coverage rates, because cNICS methods are based on data collected from immunization records held by parents. As a result, some doses actually received by children may be missing from records, leading to underestimates of coverage.

Overall, recent cNICS surveys tell us that most children have received at least one immunization (only 1.5% of parents surveyed reported never having immunized their children).Endnote 64 As children get older, there is an increased risk of incomplete immunization records, particularly because children may receive vaccines from a variety of sources (e.g., school, pharmacy).Endnote 65

External key informants and document review data indicated that the most commonly refused vaccines are the seasonal influenza vaccine and the varicella (chicken pox) vaccine, because parents lacked confidence in the influenza vaccine, and some do not consider the chicken pox to be a serious disease.Endnote 66

Adult coverage

The pneumococcal polysaccharide vaccine, which the P/Ts publically fund, is recommended by NACI to be administered to seniors at the age of 65 and above. Pneumococcal vaccine coverage estimates are used by CIRID as an indicator of progress made in adult vaccination coverage.

As reported by aNICS, between 2006 and 2014, pneumococcal vaccine coverage in adults over 65 years of age remained stable between 34% and 38%. These coverage estimates are well below the target of 80% (agreed upon at the National Goals and Objectives Conference in 2005) and, according to aNICS analysis, cannot be explained by methodological problems.Endnote 67

Other vaccinations have similarly low coverage in the adult population in Canada. For example, although NACI recommends that all healthy adults receive a seasonal influenza vaccination, according to aNICS, in 2013-14, less than half of adults (40%) had received a dose of influenza vaccine.Endnote 68 Similarly, NACI recommends that adults under the age of 50 and serologically determined to be susceptible to varicella (adults who never had the chickenpox) should receive two doses of univalent varicella vaccine. However, aNICS found in 2013-14 that among adults without a self-reported history of varicella, only 35.1% had received at least one dose of the varicella vaccine.Endnote 69

The aNICS identifies multiple factors that contribute to non-immunization, including the role of health care providers, knowledge gaps, and attitudes, values and beliefs.Endnote 70 See below for how these factors, as well as varied access between P/Ts, contribute to non-immunization and lower than targeted coverage.

Improving immunization coverage

The role of health care providers

The role of health care providers is crucial. Based on the literature review and key informant interviews with public health representatives, health care workers are a trusted source of information for Canadians and have been linked to increased vaccine uptake. EKOS Research conducted a survey in 2011 which found 89% of Canadian parents followed their health care provider's advice regarding immunizations. The 2012 and 2014 aNICS surveys found that a recommendation from a health professional was a strong prediction of vaccine uptake for both the seasonal influenza and pneumococcal vaccines.

Knowledge gaps

Additionally, there continue to be knowledge gaps in adults related to immunization. As outlined in section 4.4.1, many Canadian adults are not aware of their actual vaccination status despite believing that they are up to-date with recommended vaccinations. These knowledge gaps may contribute to under-vaccination amongst adults.

Attitudes and beliefs

Further, attitudes and beliefs are complex and vary. In a recent PHAC Survey on Vaccine Hesitancy with Health Professionals, nine in ten health professionals (91%) reported that they believe vaccine hesitancy is an issue today, either somewhat (59%) or to a great extent (32%). Endnote 71 Reasons for vaccine hesitancy and non-vaccination vary, and range from non-belief in the efficacy, safety or necessity of vaccination, to passive omission (non-vaccination) due to perception that risks are lower in a specific geographic location.Endnote 72 Internal and external key informants highlighted that knowledge, attitudes and beliefs contribute to vaccine hesitancy, and vary amongst vaccine-hesitant individuals and the unvaccinated.

Factors that impact these reasons for non-vaccination extend beyond vaccine hesitancy, and, according to findings from a recent report from the PHN Vaccine Acceptance and Uptake Task Group include: barriers to access (e.g., distance from clinics, scheduling conflicts, language or financial barriers); a lack of general exposure to disease impacts (which could affect attitudes towards vaccination).Endnote 73 These challenges are not unique to Canada, according to key informants in other jurisdictions such as the United Kingdom, Australia and the United States.

Access in P/Ts varies

Finally, provincial and territorial immunization programs, vaccine supply, and access to clinics vary by jurisdiction. These factors may also contribute to confusion surrounding immunization schedules.

Performance: Issue #5 – Demonstration of Economy and Efficiency

Immunization is a cost-effective public health intervention, as millions of dollars in direct and social costs are saved through this activity. Within PHAC, efficiencies have been realized through the effective use of some performance measurement information. There may be gained efficiencies to examining governance structures and adverse event case reviews.

Observations on Economy

Vaccines are efficient as Canada's investments in immunization have yielded major health system and societal benefits. A 2015 NACI report indicated that the return on investment for each dollar invested in adult influenza immunization programs saved the health care system $45. Additionally, for children's vaccines, a dollar invested saved $16 for measles, mumps and rubella and $6 for diphtheria, pertussis and tetanus for children.Endnote 74

Furthermore, the Canadian Immunization Guide outlined the cost per life year saved of various vaccinations versus other public health interventions. In their analysis, cost per life saved for vaccination programs range from $164 for the hepatitis B screening in pregnancy and immunization of children of carriers to $125,000 for the pneumococcal conjugate vaccine for children. This compared reasonably to other public health interventions such as $69 for the mandatory seat belt law, $3,100 for the chlorination of drinking water to $210,000 for smoke detectors in homes and $100,000,000 for radiation emission standard for nuclear power plants.Endnote 75

Internal Expenditures

Recognizing that immunization is a cost-effective public health intervention, PHAC internal expenditures would include various activities outlined in this report such as surveillance, vaccine research coordination and health promotion, with the intended goal of enhancing immunization in Canada writ-large. A summary of budgets and expenditures in this area is presented in the table below.

Table 4: Variance Between Planned Spending vs Expenditures*
2011-2012 and 2015-2016 ($M)
Year Planned Spending ($) Expenditures ($) Variance ($) % planned budget spent
Gs&Cs SPAs O&M Salary Total Gs&Cs SPAs O&M Salary Total
2011-2012 $0.2 $27.1 $20.8 $7.3 $55.2 $0.5 $0.4 $17.2 $8.4 $26.5 -$28.7 48%
2012-2013 $0 $5.5 $18.7 $6.5 $30.7 $0 $5.3 $18.3 $6.7 $30.3 -$0.4 99%
2013-2014 $0 $18.5 $18.2 $6.0 $42.7 $0.2 $16.5 $20.3 $6.7 $43.7 +$1 102%
2014-2015 $0 $3.8 $17.9 $6.8 $28.5 $0.1 $3.8 $17.6 $7.5 $29 -$0.5 102%
2015-2016 $0 $0 $17.9 $6.8 $24.7 $0.33 $0 $14.6 $7.5 $22.4 -$2.3 90%
* Data Source: (1) CIRID administrative data for planned spending; and 2) Office of Audit and Evaluation (DAE Service), using the SAP transactional database for expenditures using CIRID cost center information. Special Purpose Allotments (SPAs) include planned spending and expenditures for Fill-Line and National Antiviral Stockpile (NAS) when applicable.

Generally, there is little variance between planned spending and expenditures, and it is worthwhile noting that a majority of CIRID spending goes to pandemic preparedness (58%), which could be partially explainable by activities arising after the H1N1 outbreak in 2009.

Upon examination, variance in spending was also related to pandemic preparedness activities. In 2011-2012, variance was due to the reprofiling of the Fill-LineFootnote v contract due to contract amendments. Variance in 2013-2014 O&M amounts were due to a banking day funding request for the cNICS that was not included in planned spending. Variance in 2015-2016 was based on an allocation of $9.8 million for the primary pandemic readiness feeFootnote vi.

Observations on Efficiency

Adverse Event Standard Operating Procedures

PHAC has recently developed standard operating procedures to track processing times for reviews of cases received, which in turn supports, amongst other activities, ongoing and timely identification of any vaccine safety issues that require further attention. The process of public health reporting for adverse events following immunization can be found on PHAC's website. Standard operating procedures have been determined based on the severity of cases, with the most serious to be processed within two business days of arrival at PHAC (priority I).

Generally, PHAC was able to meet standard operating procedures for reviewing priority I cases (90%), priority II cases (94.5%) and priority III cases (99%) in 2015 (median times). Deviation from the goal to report within timelines in August and September of 2015 was the direct result of receipt of a backlog of more than 2,000 cases spanning more than a decade from one jurisdiction, which would not have impacted immediate adverse event response activities, but could have if these cases were more recent, as surge capacity is lacking.

PHAC also conducts more detailed case reviews (including a more substantial secondary review of more serious cases conducted by a medical safety expert). PHAC's ability to meet standard operating procedures in this area varied more (54% (median) of timelines met in 2015, based on monthly reporting). While this could be partially attributed to the backlog of cases mentioned previously, PHAC did not meet commitments over a number of months during a 24-month period. This deviation from standard operating procedure targets may be partially due to the SOP criteria for when medical case reviews are required (i.e., serious cases vs. non serious cases). Key internal informants noted that specific medical skills, expertise and training was needed for this exercise, as one needed to make judgements based on the medical information available in the case file, which was not always available in a timely manner for this process.

PHAC is looking at various ways to address capacity, and has recently increased the number of its medical professionals to address these types of issues. Furthermore, PHAC responds regularly to adverse events related inquiries received from F/P/T health authorities and other stakeholders (e.g., the public). CIRID is developing a solution to enable F/P/T stakeholders to access a subset of their own data as well as national data for comparison purposes, and at their own convenience. This should improve service delivery, enhance transparency as well as data reporting capabilities.

Governance

There are currently 20 working or task groups in place to address aspects of immunization and infectious respiratory diseases with CIRID participation and/or support. There are indications from the stakeholder survey that some working groups could operate more efficiently. Part of this may be attributed to membership confusion of roles and responsibilities, as well as a lack of clarity of the purpose and goals of the working group and how it fits into the broader F/P/T governance structure. For example, the Vaccine Vigilance Working Group (VVWG) consistently discussed issues relating to inter-jurisdictional data sharing mechanisms, including barriers and agreements. While important to their work, these activities are beyond the scope of their core mandate, which, according to the VVWG terms of reference, is to focus on vaccine safety issues of greatest relevance to F/P/T jurisdictions in routine or mass immunization programs, as well as to assist the Canadian Immunization Committee in improving vaccine safety as recommended in the National Immunization Strategy.

Immunization activities are complicated and multifaceted. For some working groups, this complexity may impact their mandate. For example, there is an extensive list of activity areas for the Canadian Immunization Committee, and a key stakeholder noted that the resources required to address its mandate may not be adequate. Furthermore, reporting relationships may be confounded by the necessity for technical advice through the Chief Medical Officers of Health Committee, rather than the policy focussed Communicable Infectious Disease Steering Group. As noted previously, there have been issues relating to the provision of programmatic information to support provincial and territorial decisions relating to public health vaccination programs. Governance may be an additional element to consider for addressing those challenges.

Observations on the Adequacy and Use of Performance Measurement Data

CIRID is a learning organization. Over the past few years, there have been a variety of studies conducted which have been used to advance work within the Centre. This is most evident by planned activities by the Centre, which are in line with findings of performance measurement (formal and informal) collected over the past two years.

The Performance Measurement Strategy developed by CIRID proved to be very helpful for this evaluation in:

  • explaining the purpose and objectives of the program;
  • describing key partners (critical considering the multijurisdictional nature of the area);
  • providing historical context; and
  • outlining indicators used for measuring progress.

While not all data were available, the majority of indicator reporting provided key lines of evidence for the evaluation but also directed the evaluation team in primary data collection activities. While some trend data is still absent, others will now be considered baseline data in moving forward.

Areas of improvement could focus on timely indicator reporting, reviewing the appropriateness of some indicators as valid measures of outcomes, whether other information available to the program could be outlined within the strategy. This will help in future program decision making. Finally, consistently updating stakeholder lists to ensure information is available and accurate in a timely fashion. This will also help in any future health promotion activities with partners and stakeholders.

5.0 Conclusions

5.1 Relevance Conclusions

5.1.1 Continued Need

There is a need for immunization activities as well as those to prevent and control respiratory infectious diseases such as influenza. In recent years, Canada has eliminated endemically-transmitted measles and rubella. With the exception of a few diseases that remain endemic and burdensome in Canada by having cyclical peaks or regional outbreaks (e.g. pertussis, invasive meningococcal disease, invasive pneumococcal disease), the current burden of VPDs has been at low levels or declining since the early 2000s. Seasonal influenza-related hospitalizations and deaths fluctuate depending on a number of factors including circulating strain and annual vaccine match, but continue to comprise an important annual burden of morbidity and mortality.

Highly infectious diseases require high immunization rates of the population to develop herd immunity (for example, 95% for measles). National vaccine coverage estimates for children and adults indicate that they are below established targets; although this could be underestimated as it is based on parent-held immunization records that may be missing actual doses administered. Immunization, through vaccination, is considered to be one of the greatest public health achievements of the 20th century, providing a cost-effective tool to control and eliminate life-threatening diseases. It is estimated to have saved more lives in Canada over the past 50 years than any other single health intervention.

5.1.2 Alignment with Government Priorities

As outlined in a variety of strategic documents, preventing and controlling VPDs has been a priority for the Government of Canada for decades. Priorities to address these diseases was recently reiterated in the 2015 mandate letter to the Minister of Health as well as the 2016 Federal Budget, which specifically identified increasing vaccination rates as a top priority.

5.1.3 Alignment with Federal Roles and Responsibilities

There are multiple players (local, provincial/territorial, and federal) with responsibilities in this area. PHAC's immunization and respiratory infectious disease activities are aligned with the Minister's legislative authorities under the Department of Health Act (1996) and Public Health Agency of Canada Act (2006).

Cabinet authorities clearly articulate a public health leadership, coordination and collaboration role in various areas as well as for activities to strengthen national immunization infrastructure (vaccine safety monitoring; surveillance; immunization registry network; public and professional education; and approaches to enhance immunization coverage and acceptance in special populations). However, some program stakeholders expressed confusion about PHAC's role in research, surveillance and health promotion, as well as in immunization registries.

5.2 Performance Conclusions

5.2.1 Achievement of Expected Outcomes (Effectiveness)

PHAC works with partners to have vaccine supply mechanisms in place to secure the supply of vaccines in Canada and affordable prices. PHAC and its partners have the capacity to address and minimize the impact of shortages and disruptions and negotiate lower prices for vaccines.

PHAC publishes and supports the development of critical knowledge products for the prevention and control of VPDs and respiratory diseases such as influenza; products such as the Canadian Immunization Guide, NACI statements and FluWatch reports are used by program stakeholders. Likewise, PHAC's public information tools such as the Immunization Schedule Tool and the Parent's Guide to Vaccination are used by Canadians.

While the vast majority of Canadians are immunized, pockets of under- or un-immunized Canadians exist across the country. Challenges to improving immunization coverage are varied, but include a lack of knowledge or awareness and an abundance of misinformation from a small but vocal anti-vaccination community. Furthermore, the availability of programmatic information was a factor in implementing vaccine recommendations across jurisdictions in a timely fashion.

Various PHAC activities contribute to Canadian capacity to deliver a coordinated and effective response to the risk of VPDs and infectious respiratory diseases. While comparable immunization registries across jurisdictions are still needed, many challenges exist in its establishment, and PHAC has taken steps to mitigate the challenges associated with this gap.

5.2.2 Demonstration of Economy and Efficiency

Immunization is a cost-effective intervention, with millions of dollars in direct and social costs saved through this activity. For example, each dollar invested in adult influenza immunization programs is estimated to save the health care system $45. Additionally, for children's vaccines, each dollar invested saves the health care system $16 related to measles, mumps and rubella, and $6 for diphtheria, pertussis and tetanus. The cost per life year saved of various vaccinations compares well to other public health interventions, such as seat belt laws or chlorination of drinking water.

Generally, there is little variance between planned spending and expenditures, and any variances have tended to be linked to pandemic preparedness activities, which account for the majority of expenditures within the Centre. There may be efficiencies gained in examining governance structures and by clarifying the criteria used in conducting medical case reviews of adverse event reporting and/or by increasing the availability of training to enhance the medical skills and expertise for these reviews.

With respect to performance measurement, CIRID has implemented a performance measurement strategy and collected indicator data over the past two years which has recently helped identify areas for programmatic decision making. Areas of improvement could focus on timely indicator reporting, and the review of appropriateness of some indicators as valid measures of outcomes.

6.0 Recommendations

Recommendation 1

Strengthen evidence base to address current information gaps on the under- and un-immunized in Canada.

It is evident that there are pockets of under- or un-immunized individuals in Canada. PHAC's ability to identify these groups is limited, in part, by the absence of comparable immunization registries. The Program relies on surveys of the Canadian population. PHAC reports national immunization coverage estimates and measures factors influencing uptake by conducting national adult and childhood immunization coverage surveys. Given that the P/Ts have invested in various, personalized approaches and IT solutions for their jurisdictions, an alternative method for obtaining a national immunization coverage picture (sourcing existing registry data) is needed.

P/T immunization registries remain an important surveillance tool in assessing immunization coverage and vaccine uptake, and there is a federal public health role in facilitating the development and P/T adoption of national standards for immunization registries. There is work already underway with the Canadian Immunization Registries and Coverage Network (CIRC) and partners such as Canada Health Infoway to revisit the functional and data element standards for immunization registries, and to explore innovative solutions for pulling the national immunization coverage picture from these existing and evolving systems.

Recommendation 2

Ensure timely availability of programmatic information regarding vaccinations for provincial/territorial use.

There is evidence that the provision of information (e.g., cost-effectiveness) that is used by jurisdictions to make vaccine programming decisions has not been timely or comprehensive enough. For example, provincial and territorial implementation of publicly-funded varicella immunization programs occurred gradually over 7 years. One of the factors in this gradual implementation may have been the timing of available programmatic guidance following the NACI publication of technical guidance for the varicella vaccine. Access to timely guidance on programmatic recommendations has been identified as one of the limiting factors to achieving uniform access to vaccinations across Canada. The program should continue its efforts to implement a mechanism to improve the timeliness and comprehensiveness of available technical and programmatic advice on vaccines in Canada.

Recommendation 3

Determine and implement strategies to influence Canadians' knowledge, attitudes and beliefs related to immunization and respiratory infectious diseases.

While the vast majority of Canadians are immunized, pockets of under- or un-immunized Canadians exist across the country. Vaccine hesitancy and anti-vaccination beliefs are barriers to improving immunization coverage in Canada. Knowledge, attitudes and beliefs about immunization are complex and varied, as are the reasons behind vaccine hesitancy and anti-vaccination beliefs which may be fueled by information globally available through social media, the internet and anecdotally. Recommendations from health care providers, a trusted source of information for Canadians, have been shown to be linked to greater vaccine uptake. While the program does produce tools and information for Canadians about immunization, program activities would be enhanced by the existence of a strategic framework and a social marketing strategy, including social media, to guide knowledge translation efforts intended to improve vaccine uptake.

Recommendation 4

Review governance to enhance efficiencies.

With 20 working groups that have varied governance structures, mandates and areas of focus addressing immunization and infectious respiratory diseases, there are indications that some working groups supported by CIRID could operate more efficiently. Some stakeholders have highlighted confusion surrounding roles and responsibilities in several working groups; other stakeholders have indicated that resourcing for some groups may not be adequate to the mandate assigned to them; and a review of working group records of decision has identified cases of working group focus extending beyond original mandates. CIRID working group governance would be enhanced by a review targeted at clarifying roles and responsibilities and objectives of working groups supported by the Centre.

Recommendation 5

Enhance efficiencies in response activities (including medical case reviews).

PHAC has two separate standard operating procedures (SOPs) in adverse event reporting. First, the Centre meets data processing timelines for reporting on adverse events (generally meeting these procedures between 75% - 100% of the time). Second, PHAC conducts medical case reviews for adverse event reports, which are used to inform public health and regulatory decisions; additionally, serious cases are assigned to a medical safety expert for secondary case review. However, for 8 out of 24 months, report processing for medical case reviews fell below 70%, which would not have impacted immediate adverse event response activities. While this is partially due to one submission of backlogged cases, some key informants noted that the criteria for conducting a medical case review may not be clear and/or the specific medical skills, expertise and training required for this activity were not always available.

Furthermore, there are strong indications that CIRID engages in significant levels of outbreak response activities, including, but not limited to, risk assessment, issues management, strategic communications, and the development of guidance (for example, in the case of measles or pertussis outbreaks). However, it remains difficult to accurately quantify the level of CIRID effort associated with these activities. Given that key informant interviews indicated that outbreak response activities may impact staff capacity to complete regular activities in a timely manner, a clearer understanding of what outbreak response consists of, in the form of regular activity and output tracking would be useful for upstream planning within the Centre. A clear articulation of CIRID outbreak response activities, outputs and capacity would also contribute to overall PHAC outbreak surge capacity assessments.

Appendix 1: Stakeholders

Stakeholder Roles and Responsibilities
Canada's Health Jurisdictions

All 14 F/P/T jurisdictions have substantial roles, authorities and activities in immunization. P/Ts are responsible for funding, program planning, and the delivery of immunization programs in their respective jurisdictions. Ongoing P/T immunization programs account for the majority of immunization activities and investments in Canada, including bulk purchase of vaccines (plus storage and handling), vaccine administration, development and maintenance of immunization registries, participation in vaccine surveillance systems, conduct of program evaluation and research, and design and delivery of public and professional education, awareness and engagement initiatives.

The Government of Canada is the fifth-largest immunization jurisdiction in Canada, with immunization responsibilities and interests for seventeen federal agencies, covering First Nations, Inuit, and federal populations, such as military, federal inmates, refugees, the Royal Canadian Mounted Police, veterans, and foreign services. Within the federal government, Public Services and Procurement Canada (PSPC) administers the F/P/T Bulk Purchasing Program for most of the publicly-funded vaccines used in Canada. PHAC, other government departments and the Canadian Institutes for Health Research (CIHR), in collaboration with the pharmaceutical industry, support the national coordination of immunization research and innovation.

Health Canada Health Canada is responsible for the scientific review and testing process for the quality, safety and efficacy of vaccines before vaccines are approved for the Canadian market. Following vaccine approval, Health Canada and PHAC monitor vaccine safety on an ongoing basis.
National Microbiology Laboratory Working with the National Microbiology Laboratory (NML), CIRID is contributing to a Branch-wide "Science-Policy Integration" initiative to bring NML's specialized expertise and technology closer to the policy function that provides program analysis, research and development advice, and guidance to PHAC management. Specifically, CIRID is working with its counterparts at NML to explore further development of infectious disease surveillance pilot projects to foster strengthened epidemiology-laboratory integration. CIRID/NML collaboration objectives include: Strengthening national joint measles/rubella surveillance to support timely integration of epidemiological and laboratory data and to enable indicator-based monitoring and evaluation of Canada's measles elimination status; Maintain and expand laboratory capacity to support timely and effective public health action, and to support the Pan American Health Organization (PAHO) measles elimination goals and objectives; Strengthen current surveillance systems to support the integration of epidemiological and laboratory data for the monitoring and evaluation of Canada's current national immunization recommendations and programs; Contribute to the Federal Framework on Antimicrobial Resistance and Use in Canada by monitoring antimicrobial susceptibility of select pathogens at a national level; and Strengthen national surveillance systems by supporting the integration of epidemiological and laboratory data to provide baseline data for future national immunization recommendations and programs.
Canadian Public Health Association (CPHA) In 2009, the CPHA issued a report (Setting the Stage for Advancements in Immunization in Canada) that identified several areas of priority concern to address in order to enhance Canada's readiness for new developments in immunization. The CPHA, in collaboration with PHAC, also organizes the biennial Canadian Immunization Conference, and (with Immunize Canada and the Ottawa Hospital Research Institute) co-developed the ImmunizeCA Mobile App, which provides Canadians with the ability to store and manage vaccine and vaccination information, schedules, appointments, access evidence-based and expert-reviewed information about recommended and routine vaccinations for children, adults and travellers, and receive alerts about disease outbreaks in their surrounding area.
Canadian Paediatric Society (CPS) A report issued by the CPS in 2009 (Are We Doing Enough: A status report on Canadian public policy and child and youth health) highlighted the importance of the early years in child development on longer-term health, emotional well-being and life success. Among several key factors, the report focussed specifically on publicly-funded immunization programs. In addition to the slate of vaccines that have been a part of the routine immunization schedule for several years, the report recommended immunization against other diseases, including rotavirus, varicella, pertussis, influenza and certain forms of meningitis, and that human papillomavirus (HPV) vaccinations should be provided at no charge. The CPS is also a partner with the Immunization Program in the delivery of an immunization competencies program. It has also issued an immunization report card/status report.
Canada's Vaccine Industry Along with researchers and biotechnology companies, the industry is interested in ensuring Canada has a secure supply of vaccines and that the country meets evolving health needs for innovative vaccines. BIOTECanada is the national industry association, with about 250 members; its Vaccine Industry Committee is composed of the leading vaccine manufacturers. The companies that provide vaccines under the PSPC bulk purchase vaccine program include Merck Frosst, GlaxoSmithKline, Pfizer, Crucell and Sanofi Pasteur.
Academia Immunization experts from academia undertake research and/or provide advice to governments on the use of vaccines. The Canadian Immunization Research Network (CIRN) engages these experts to address key areas of immunization research, such as the rapid evaluation of candidate vaccines for safety and immunogenicity, evaluating vaccine effectiveness and safety following release, vaccine hesitancy, vaccine coverage (including isolated communities and cohorts of concern), vaccine safety and VPD surveillance. In addition, PHAC has provided a five-year research grant to VIDO-InterVac, a leading non-profit vaccine and immunotherapeutic-technologies research and development centre at the University of Saskatchewan. VIDO-InterVac has an array of capabilities related to research and vaccine testing, vaccine and technology improvements, large-scale science, and intramural research to benefit the public health of Canadians.
International Organizations As a member state of Pan American Health Organization, Canada is a partner along with other countries of the Americas in supporting the implementation of PAHO's Regional Immunization Strategy 2007-2015. Canada's participation in PAHO takes place under the broader context of the country's commitments and activities as a member state of the World Health Organization, a UN agency. Canada is subject to the International Health Regulations (IHR), a binding international legal instrument aimed at helping the international community prevent and respond to acute public health risks that have the potential to cross borders and threaten people worldwide. The IHR, which came into effect in 2007, require countries to report certain disease and public health events of international concern to the WHO. The IHR also require countries to strengthen their existing capacities for public health surveillance and response.

Appendix 2: Activities and Outputs

1. Surveillance

National surveillance of vaccine preventable diseases is essential to providing up-to-date information to various levels of government, public health officials, and advisory bodies (such as NACI) in order to assist with the development of evidence-based guidelines. It is also required to assist jurisdictions across Canada, including health care providers, in the investigation, containment and management of vaccine preventable disease outbreaks. National surveillance of vaccine preventable diseases helps to identify and quantify risk factors, enabling appropriate control of communicable diseases. It also enables health professionals to respond to the general public and media, both in terms of a real or perceived crisis and routine daily affairs. Effective and timely surveillance is critical to the ability of the GC and P/Ts to accurately track health information, and plan for (and respond to) public health events.

PHAC administers the Canadian Notifiable Disease Surveillance System, which receives reports of national notifiable diseases from the P/Ts. The Canadian Network of Public Health Intelligence is a key enabler of surveillance and is a secure, web-based collection of applications designed to facilitate real-time collection and dissemination of laboratory and epidemiological surveillance data and coordination of public health response.

Using these PHAC instruments, the Program provides financial and technical support for case-based, active surveillance of vaccine preventable diseases such as invasive pneumococcal disease, varicella, pertussis, Haemophilus influenzae, and conditions such as acute flaccid paralysis through a paediatric hospital-based surveillance network (the Immunization Monitoring Program ACTive (IMPACT)) and the Canadian Paediatric Surveillance Program. Additionally, the Program supports the International Circumpolar Surveillance (ICS) system for particular bacterial vaccine preventable diseases that occur in specific at-risk populations in the North. These surveillance systems provide important data that contribute to stakeholder understanding of certain diseases and conditions, including disease severity, hospitalization, sequelae and death. IMPACT and ICS also provide an ability to evaluate the effectiveness of vaccines currently marketed in Canada, but are limited to targeted populations, namely children and populations in the North respectively. There are also several sentinel disease surveillance systems, including FluWatch Surveillance, which is designed to provide a national picture of influenza activities and trends in Canada.

The program carries out post-market vaccine surveillance through CAEFISS. This is a voluntary reporting system where local and P/T public health officials can report adverse events following immunization. Additionally, IMPACT nurse monitors identify select AEFI events involving children, sending AEFI reports to both the appropriate P/T and to PHAC.

Population surveys provide estimates of immunization coverage, identify any patterns of under-coverage, and obtain information on public knowledge, attitudes and behaviours, which can be used to design immunization strategies and approaches. Until Canada is able to pull the national picture out of the diverse registry solutions used across the jurisdictions, the Program relies on surveys of the Canadian population. The Childhood National Immunization Coverage Survey and the Adult National Immunization Coverage Survey (aNICS) are each conducted every two years to assess routine childhood immunizations and selected adult vaccines, respectively.

2. Leadership, Consensus-building and Coordination

Under this activity group, the Program establishes suitable mechanisms and is responsible for the direction, oversight and coordination of F/P/T initiatives under the aegis of the NIS; this includes the articulation of immunization goals to focus and inspire F/P/T collaboration and constructive engagement in areas of mutual interest.

Vaccine guidance is an important leadership role within this activity stream. The regulatory authorization of a new vaccine for use in Canada is the responsibility of Health Canada's Biologics and Genetic Therapeutics Directorate. Once authorized via a Notice of Compliance (NOC), a vaccine undergoes review by two national expert committees. The program provides secretariat and technical support to NACI, which provides immunization guidance to enable policymakers and health professionals to make immunization-related evidence-based decisions. The program also provides secretariat supports to the CIC, an F/P/T committee that reports to the Public Health Network on a variety of Immunization Program planning issues. Subsequent to NACI recommendations, the CIC conducts reviews and develops recommendations on Program planning issues based upon analytical techniques, such as the cost-benefit analysis of the use of vaccines. All NACI recommendations on vaccine use are published in the Canadian Immunization Guide, the second-most viewed page on the PHAC website (only the PHAC website homepage receives more views).

Common vaccine guidance is an element of the NIS; its objective is to establish an integrated scientific/technical and socio-economic/programmatic guidance with more timely guidance delivery following an NOC. In 2014, PHAC, via NACI, piloted common vaccine recommendations for Meningococcal B (Men B). Currently, PHAC is developing options for an integrated process that would meet F/P/T needs, informed by lessons learned from the Men B pilot project.

Immunization registries are population-based applications that have the capacity to perform the scheduling of immunization appointments, the management and recording of immunization events, and the notification of when immunizations are due. Fully functional registries will only be as good as the data entered into them – they will be more reliable the day that we can ensure that electronic health records and immunization systems capture all vaccine doses administered and include entries for all children (including those who were never vaccinated) in order to produce real-time immunization coverage monitoring and reporting as well as provide needed information for immunization delivery, vaccine supply management and adverse event reporting. Many challenges still exist in establishing such a network assessing immunization coverage as compared to population surveys. The Program, therefore, monitors the development and implementation of registries by the P/Ts as part of an overarching NIS objective to develop a national network of immunization registries.

The Program also assists with the coordination and oversight of public market vaccine supply and distribution in Canada. Through the F/P/T Vaccine Supply Working Group, the Program advises on supply issues across Canada and works with PSPC, which administers the bulk F/P/T vaccine procurement program and manages vaccine tenders and contracts on behalf of all participating jurisdictions.

3. Knowledge Development, Translation and Exchange

Within this activity group, the Program undertakes cooperative, coordinated and specific collaborative efforts to generate information that can be shared by all key Canadian public health stakeholders to help improve the consistency and effectiveness of their immunization planning, management and delivery.

It works with key science and public health intermediaries that engage with and subsequently support the Program in the conduct of targeted research and surveillance related to specific vaccine preventable diseases. These intermediaries also assist the Program with data collection, compilation and validation, followed by analysis, interpretation and integration, which in turn supports knowledge production, dissemination and advisory functions.

In collaboration with Immunize Canada, the Program supports National Immunization Awareness Week, an outreach initiative to build capacity and support public awareness campaigns across all jurisdictions held in conjunction with Vaccination Week in the Americas. Additionally, the Program runs the annual National Influenza Immunization Campaign to support and complement P/T efforts in raising awareness on appropriate infection prevention behaviours to help prevent the spread of influenza. The Program has also developed a multi-component public education campaign targeting parents of children 0-2 years of age entitled, It's Time to Immunize and issues A Parent's Guide to Vaccination. The ImmunizeCA Mobile App was also developed in collaboration with Immunize Canada, the CPHA and the Ottawa Hospital Research Institute.

The Immunization Program also sponsors the biennial Canadian Immunization Conference, which brings health professionals from across the country together to expand their knowledge regarding the latest developments in immunization research, policies, programs and practice. Outreach to mobilize P/T and local public health units, community health centres and individual health professionals is greatly supported through collaboration with the major national professional associations. The Program led the development of the Immunization Competencies Education Program (ICEP) and partnered with the Canadian Paediatric Society to offer an online learning program on these competencies, which is free to publicly funded health professionals. The ICEP offers the latest immunization information and builds public health capacity through its certification aspect, and is recognized nationally by health professional associations and immunization programs to train and certify new immunization providers.

4. Public Health Event Management

Under this fourth and final activity group, CIRID undertakes cooperative, coordinated and specific collaborative efforts to manage vaccine preventable and respiratory infectious disease public health events. The public health event management framework rests upon four linked pillars: prevention, preparedness, response and recovery. CIRID's work in this activity area supports the work of the Centre for Emergency Preparedness and Response (CEPR), within the Emergency Preparedness and Response Program (PAA 1.3.1).

The activity groups described above generate outputs that contribute to two outcome streams. The first stream of program outcomes is influenced by the program outputs that support and constructively engage PH stakeholders and intermediaries so that they can make effective use of the information, resources and mechanisms to build consensus and collaboration on the delivery of appropriate risk-based immunization actions across Canada. In turn, this stream of outcomes amplifies a second stream of outputs and outcomes; in this stream, actions are taken by both CIRID and intermediaries to influence the attitudes and behaviours of Canadians (including target populations and at-risk groups) towards immunization, such as communicating and amplifying key messages being delivered by the Program and actions taken by F/P/T organizations to deliver immunization uptake services.

Appendix 3: Logic Model

Text Equivalent - Logic Model

Evaluation of Immunization and Respiratory Infectious Disease Activities at the Public Health Agency of Canada 2011-2012 to 2015-2016

Appendix 3 - Logic Model

Responsibility for public health is shared across jurisdictions, with the provinces and territories having primary responsibility for immunization programs and delivery. The federal role, as reflected in the activities and outputs of the Agency's Immunization and Respiratory Infectious Disease Program, is considered to be one of leadership and coordination. In this role, the Program helps to protect Canadians from public health threats posed by vaccine-preventable diseases by attempting to influence and support provinces and territories and other public health stakeholders in their actions. This is done in order to appropriately manage risk, and to provide timely access to immunizations that are safe and administered securely, effectively and efficiently.

The Agency provides inputs in order to implement activities. The inputs consist of human resources; financial resources (operations and management, grants, contributions); policy and legislative authorities; IT technology; surveillance data; Government of Canada global agreements; corporate support (internal services); Federal / Provincial / Territorial and other agreements; laboratory-based surveillance, research and reference services.

The activities carried out by the Program can be categorized into four broad groups:
surveillance (data collation, integration, analysis and interpretation);
leadership, consensus-building and coordination;
knowledge development, translation and exchange; and
public health event management (Prevention, preparedness, response and recovery.

The activities are expected to reach the following program stakeholders: Federal / Provincial / Territorial organizations, other government departments, academia, industry, public health and health professionals, international public health collaborators and emergency management stakeholders. In addition, the activities are expected to benefit Canadians and others living in Canada, including targeted and at-risk populations (with uptake services delivered via Federal / Provincial / Territorial organizations).

As a result of each activity, the Program aims to generate a number of outputs, namely:
Surveillance and related data products;
Institutional arrangements and frameworks (data-sharing; funding; vaccine supply management; vaccinovigilance); strategies; standards and protocols;
Guidance, advice and recommendations; public and professional education opportunities; knowledge products and tools; and
Public health event strategies, protocols, guidance, advice and recommendations.

Through these outputs, program activities correspond with specific immediate, intermediate and long-term outcomes.

In the immediate term, program activities are expected to lead to:
Program stakeholders have information and resources to prevent and control health risks associated with vaccine preventable and respiratory infectious diseases and vaccine safety issues;
Canada has mechanisms in place to collaboratively prevent and control health risks associated with vaccine preventable and respiratory infectious diseases;
Canadians and others living in Canada have access to vaccines to protect themselves from vaccine preventable and respiratory infectious diseases; and
Canadians and others living in Canada have information and tools to protect themselves from vaccine preventable and respiratory infectious diseases.

The achievement of these immediate outcomes is expected to lead to the following intermediate outcomes:
Canada has the capacity to deliver coordinated and effective responses to health risks associated with vaccine preventable and respiratory infectious diseases and vaccine safety issues; and
Canadians and others living in Canada take positive actions to protect themselves from health risks associated with vaccine preventable and respiratory infectious diseases.

In the longer-term, the Program's activities are expected to lead to Canadians and others living in Canada being protected from health risks associated with vaccine preventable and respiratory infectious diseases. This contributes to the Agency's Strategic Outcome of protecting Canadians and empowering them to improve their health.

Enlarge Image - Logic Model

Appendix 4: Evaluation Description

Evaluation Scope

The scope of the evaluation covered the period from April 1, 2011 to March 31, 2016, and included most immunization activities. It did not cover emergency response activities that fall outside of CIRID's responsibility or those that are the responsibility of other jurisdictions.

The evaluation issues were aligned with the Treasury Board of Canada's Policy on Evaluation (2009) and considered the five core issues under the two themes of relevance and performance, as shown in table 1. Corresponding to each of the core issues, specific questions were developed based on program considerations and these guided the evaluation process.

Table 1: Core Evaluation Issues and Questions
Core Issues Evaluation Questions
Relevance
Issue #1: Continued Need for Program
  • What is the current and projected burden of vaccine preventable diseases in Canada? How has the environment changed over the past few years?
  • What is the impact of these changes on need for the Agency's activities in this area?
  • Are program activities responsive to current needs?
Issue #2: Alignment with Government Priorities
  • What are the federal priorities related to immunization? Are current activities aligned with federal priorities?
  • What are the PHAC priorities related to immunization? Are current activities aligned with these priorities?
Issue #3: Alignment with Federal Roles and Responsibilities
  • What is the federal public health role related to immunization?
  • Are current activities aligned with the federal public health role?
  • Do the federal public health role and current activities duplicate the role of partners and stakeholders? Are there any gaps or overlaps?
Performance (effectiveness, economy and efficiency)
Issue #4: Achievement of Expected Outcomes (Effectiveness)
  • Does Canada have the mechanisms in place to prevent and control health risks?
  • Do program stakeholders have the required information and resources?
  • Do Canadians have the information and tools to protect themselves?
  • Do Canadians have access, through CIRID's stakeholders, to vaccines?
  • Does Canada have the capacity to deliver a coordinated and effective response?
  • Do Canadians take positive actions to protect themselves?
Issue #5: Demonstration of Economy and Efficiency
  • What is the impact of the program?
  • Has the program undertaken its activities in the most efficient manner?
  • Has PHAC produced its outputs and achieved its outcomes in the most economical manner? Has the program budget been spent as planned? Provide an explanation for any observed variances. How could economy be improved? Are the costs of inputs similar to other related programs?
  • Is there appropriate performance measurement in place? If so, is the information being used to inform senior management decision making?

An outcome-based evaluation approach was used for this evaluation to assess the progress made towards the achievement of the expected outcomes, whether there were any unintended consequences and what lessons were learned.

Data Collection and Analysis Methods

Evaluators collected and analyzed data from multiple sources. Data collection started in April 2016 and ended in June 2016. Data for the evaluation was collected using the following methods:

  • Literature review – a search of Canadian literature from the past two years using search terms such as "immunization", "vaccine preventable diseases", "influenza", "anti-vaccine", "vaccine hesitant", "vaccine awareness" and "outbreak". After examining documents to ensure relevance, 50 articles were reviewed.
  • Document review – approximately 38 documents pertinent to immunization were reviewed for information regarding the relevance (priorities, roles and responsibilities) of the activities.
  • File/performance data review – approximately 190 documents, held by the divisions responsible for the immunization and respiratory infectious disease activities, were reviewed to obtain information regarding all aspects of the activities related to immunization and in particular the performance (achievement of outcomes, economy and efficiency) of activities.
  • Surveys – an online survey of program stakeholders was conducted between March 31st and April 13th; it was sent to 319 stakeholders, 81 completed the survey for a response rate of 26%. In addition to the stakeholder survey, this evaluation consulted the following surveys: a survey of NACI members, a Survey on Vaccine Hesitancy with Health Professionals, adult and childhood National Immunization Coverage Surveys, a Canada Health Infoway survey on parent's perspectives on immunization record keeping and EKOS research.
  • Financial data review – a review of financial data from 2011-2012 to 2015-2016 was conducted, including budgeted and actual expenditures.
  • Key informant interviews – interviews were conducted with 25 stakeholders; internal staff and management (n=6); external stakeholders and other organizations (n=19) representing: other government departments; academia/experts/non-governmental organizations; provincial and territorial representatives and national committee chairs. Key informants were selected for their knowledge of and experience with immunization activities at PHAC and/or issues related to immunization.
  • International analysis – a review of immunization activities was conducted for Australia, France, Germany, Italy, Japan, the United Kingdom and the United States.

Data were analyzed by triangulating information gathered from the different methods listed above. The use of multiple lines of evidence and triangulation were intended to increase the reliability and credibility of the evaluation findings and conclusions.

Appendix 5 – Summary of Findings

Rating of Findings

Ratings have been provided to indicate the degree to which each evaluation issue and question have been addressed.

Relevance Rating Symbols and Significance:

A summary of Relevance ratings is presented in Table 1 below. A description of the Relevance Ratings Symbols and Significance can be found in the Legend.

Table 1: Relevance Rating Symbols and Significance
Evaluation Issue Indicators Overall Rating Summary
Continued Need for the Program

What is the current and projected burden of vaccine preventable diseases in Canada? How has the environment changed over the past few years?

What is the impact of these changes on need for PHAC's activities in this area?

  • Vaccination rates, coverage and trends
  • Evidence of current and projected burden:
  • Rates per 100,000 of key infectious diseases (Invasive pneumococcal disease in adults, 60 years and older)
  • Number of pertussis deaths in the target population of less than or equal to 3 months of age
  • Rates per 100,000 of key infectious diseases (5-year median incidence of non-imported cases of measles, aged 7 years or older)
  • Rates per 100,000 of key infectious diseases (Varicella-related hospitalizations)
High

There is a continued need for PHAC's immunization activities as vaccine-preventable diseases remain persistent in Canada. Challenges and barriers to immunization such as vaccine hesitancy and vaccine uptake, surveillance, projected burden of disease, and access to vaccines remain.

Immunization, through vaccination, is considered to be one of the greatest public health achievements of the 20th century, providing a cost-effective tool to control and eliminate life-threatening diseases. It is estimated to have saved more lives in Canada over the past 50 years than any other single health intervention.

The current burden of vaccine-preventable diseases (VPDs) has been on the decline since the early 2000s; however, some VPDs continue to remain endemic and burdensome in Canada by having cyclical peaks or regional outbreaks, such as pertussis (whooping cough), invasive meningococcal disease, invasive pneumococcal disease, and rotavirus. Seasonal influenza affects an estimated 10-20% of the population each year and over the past four seasons; related hospitalizations and deaths for adults have increased.

Highly infectious diseases require high immunization rates of the population to develop herd immunity (for example, 95% for measles).

The projected burden of disease will increase in the future as vaccination becomes more complex.

Improving vaccine uptake faces several challenges, such as non-medical immunization exemptions, fear of needles, usage of alternative medicine to vaccines (nosodes or natural health products), and vaccine hesitancy.

Alignment with Government Priorities
What are the federal priorities related to immunization? Are current activities aligned with federal priorities?
  • Evidence that the CIRID activities and objectives align with, and contribute towards, federal priorities
High As outlined in a variety of strategic documents, preventing and controlling vaccine preventable diseases has been a priority for the Government of Canada for decades.
What are the PHAC priorities related to immunization? Are current activities aligned with these priorities?
  • Evidence that activities and objectives align with, and contribute towards, PHAC priorities
High Priorities to address vaccine preventable diseases was recently reiterated in the 2015 mandate letter to the Minister of Health, which specifically identified increasing vaccination rates as a top priority.
Alignment with Federal Roles and Responsibilities
What is the federal public health role related to immunization?
  • Evidence that roles and responsibilities are defined, implemented, and are aligned with the federal PH role
  • Evidence of extent to which PHAC is meeting international commitments
High

PHAC has a clear public health leadership and coordination role in the area of immunization and respiratory infectious disease activities. PHAC's immunization and respiratory infectious disease activities contribute to Canada's compliance with the International Health Regulations (IHR) (2007).

Are current activities aligned with the federal public health role?
  • Program objectives align with federal jurisdiction
High Cabinet authorities clearly articulate a public health leadership, coordination and collaboration role in various areas as well as for activities to strengthen national immunization infrastructure (vaccine safety monitoring; surveillance; immunization registry network; public and professional education; and approaches to enhance immunization coverage and acceptance in special populations). However, a number of stakeholders expressed confusion about PHAC's role in research, surveillance and health promotion, as well as in immunization registries.

Legend – Relevance Rating Symbols and Significance:

There is a demonstrable need for program activities; there is a demonstrated link between program objectives and (i) federal government priorities and (ii) departmental strategic outcomes; role and responsibilities for the federal government in delivering the program are clear.

There is a partial need for program activities; there is some direct or indirect link between program objectives and (i) federal government priorities and (ii) departmental strategic outcomes; role and responsibilities for the federal government in delivering the program are partially clear.

There is no demonstrable need for program activities; there is no clear link between program objectives and (i) federal government priorities and (ii) departmental strategic outcomes; role and responsibilities for the federal government in delivering the program have not clearly been articulated.

Performance Rating Symbols and Significance:

A summary of Performance Ratings is presented in Table 2 below. A description of the Performance Ratings Symbols and Significance can be found in the Legend.

Table 2: Performance Rating Symbols and Significance
Issues Indicators Overall Rating Summary
Achievement of Expected Outcomes (Effectiveness)
Does Canada have the mechanisms in place to prevent and control health risks?
  • Percentage of the vaccines purchased for publicly funded immunization programs in Canada that were purchased through the FPT Bulk Purchasing Program (BPP)
  • Percentage of PTs that have committed to adopting the national functional immunization registry standards.
Some minor, but important, issues to address

Canada has effective mechanisms to prevent and control health risks, such as the Bulk Purchasing Program in partnership with PWGSC. While comparable immunization registries are still needed to capture a picture of immunization coverage in Canada, as well as providing needed information for immunization delivery; vaccine supply management; and adverse event reporting, many challenges still exist in establishing such a system. PHAC, in collaboration with partners, has taken steps to mitigate the challenges associated with this gap.

All 14 F/P/T jurisdictions purchase and use vaccines for their publicly funded immunization programs. Managed by Public Works and Government Services Canada, the BPP provides a common service function in the procurement process for participating jurisdictions.

Immunization registries are an important surveillance tool in assessing immunization coverage and vaccine uptake. These registries are gradually being implemented by provinces and territories, but evidence shows that registries across jurisdictions differ in content and design.

Due to a variety of challenges, Canada does not have a national system for immunization registration and the standards for such a system are under development. In lieu of this type of system, PHAC and the Canadian Immunization Registry Coverage Network have developed national standards for immunization coverage and have conducted a National Immunization Coverage Surveys that are administered every two years. Even though there is no national system in Canada, the federal government has funded other organizations such as Canada Health Infoway to help support this work. Since 2001, Infoway has received $2.1 billion to fund five streams, one of which is for the development and implementation of electronic health records, including immunization registries.

Do program stakeholders have required information and resources?
  • % of stakeholders who feel they have the information and resources from PHAC to inform policies, programs or practices
  • Average number of visits to the Canadian Immunization Guide website per month
  • Average number of visits to webpages in the three months following publication of new NACI Statements and Literature Reviews
  • Quality of the Canadian Immunization Conference attendee experience
Some minor, but important, issues to address

In partnership with others, PHAC develops critical knowledge products for the prevention and control of vaccine preventable diseases and respiratory diseases such as influenza. While challenges have been noted regarding the timeliness of some of this information, PHAC is currently taking steps to ensure comprehensive information is available to program stakeholders in a timely fashion.

Various studies over the years have indicated the importance and usefulness of these knowledge products, in particular the NACI statements. In addition to these knowledge products, CIRID supports the Canadian Immunization Conference and the CPS' online Immunization Competencies Education Program for health professionals.

There have been challenges with providing information to program stakeholders, including a website that is difficult to navigate and pages that have been archived even though they are current documents. In addition, technical information for vaccinations (i.e., cost-benefit analysis) has not been available in a timely fashion, which has likely contributed to differences in implementation of NACI recommendations across the country. However, a recent pilot of combined technical and programmatic guidance for the MenB vaccination was available within 60 days of the Notice of Compliance.

Do Canadians have information and tools to protect themselves?
  • Uptake of select immunization assessment and information online knowledge translation products:
  • Immunization promotion and education social media reach and engagement:
  • Number of downloads and number of printed copies ordered of A Parent's Guide to Vaccination
Some minor, but important, issues to address

There are indications that Canadians access PHAC's information tools to protect themselves from vaccine preventable diseases. However, there are knowledge gaps, specifically related to vaccination status, and areas for improvement, including enhancing knowledge about vaccine preventable diseases in a variety of areas.

Page views for the Immunization Schedules increased from 2013 to the early part of 2015 when they saw a decrease that is most likely attributable to a migration to the healthy canadians website.

The Parents Guide to Vaccination was the most regularly ordered publication within PHAC (over 110,000 orders for printed copies of the 2014 updated guide) with just under 30,000 page views in 2015 (although there were over a 100,000 page views the two years previously). This decrease in page views may be due to the migration to the healthy Canadians website. The Parent's Guide is part of a public education campaign, It's Time to Immunize, and while there is data available on the various outreach activities and the numbers of the guide ordered or viewed online, there is no information available regarding Canadians' perceptions of the guide (easy to understand and up-to-date).

Since its launch in 2014, there have been 143,000 downloads of ImmunizeCA app, and while external key informants indicate that challenges remain with the app (e.g., not easy to use, information is not transferable between P/Ts), PHAC has just committed $3.5 million over three years to enhance its feasibility and usefulness for Canadians.

Health professionals are critical in this process - recent analysis also shows a strong linkage to increased vaccine uptake when it is recommended by a health care worker. PHAC also supports the CPS' online Immunization Competencies Education Program for health professionals which have even been adopted and adapted by international partners (e.g., Australia).

Even though Canadians appear to access PHAC's information tools, challenges remain in terms of knowledge gaps about personal vaccination information; therefore, Canada estimates immunization uptake (coverage) through the childhood National Immunization Coverage Survey (cNICS) and the adult National Immunization Coverage Survey (aNICS). Recent coverage surveys tell us that most children have received at least one immunization (only 1.5% of parents surveyed reported never having immunized their children); however, immunization coverage in Canadian children remains below national immunization targets. According the aNICS (2014), approximately three of four adults reported that they knew enough about the benefits of immunization, but of particular concern is that while 80% believed that they received all the vaccines required for someone of their age, only 6%, through a validation exercise, had the recommended number of pertussis and tetanus vaccines doses in adulthood. Both the cNICS and the aNICS found that Canadians prefer to seek immunization advice from doctors and nurses.

Public education around the benefits of immunization is an ongoing challenge as the Internet plays a large role in disseminating anti-vaccination information.

Do Canadians have access, through CIRID's stakeholders, to vaccines?
  • % of program stakeholders who believe that the supply chain for vaccines is adequate and Canadians have access to vaccines
  • Number and duration of vaccine supply disruptions
  • % of high-priority/critical vaccines for which a supply risk management approach/plan exists
Some minor, but important, issues to address

While vaccine disruptions and shortages occur, PHAC and its stakeholders have the capacity and tools to address and minimize the impact. The availability of programmatic information was a factor in implementing vaccine recommendations across jurisdictions in a timely fashion.

In fiscal year 2015-2016, there were 16 vaccine supply disruptions affecting 14 different vaccine brands. Of these, 4 disruptions began in a previous fiscal year and carried over into the current reporting period. Seven disruptions were resolved during the reporting period. The median duration of a disruption was 152 days and disruptions ranged from a low of 30 days to a high of 1,270 days.

Equitable access to vaccination programs in Canada may be impacted by jurisdictional decisions on whether to adopt NACI recommendations in their programs and the timeliness with which information is available to support this decision-making process. A substantial number of stakeholders (both internal and external) stated that the provision of programmatic information, which aids in decision making, has not been timely. P/T implementation of the publicly funded varicella immunization programs occurred gradually over 7 years, as programmatic guidance was not available. Access to timely guidance on programmatic recommendations (e.g., cost-effectiveness) was identified by some key informants as one of the limiting factors creating this gap in access amongst Canadians.

PHAC is aware of these issues and is currently examining mechanisms to improve the timeliness of available technical and programmatic advice through expanding the mandate of and by revisiting the roles and responsibilities of NACI and CIC.

Does Canada have the capacity to deliver a coordinated and effective response?
  • % of program stakeholders (P/T reps, frontline users) who feel that guidance from PHAC improves their decision making or practice
Some minor, but important, issues to address

PHAC activities such as guidance development, surveillance reporting, vaccine safety reporting and research coordination, contribute to a coordinated and effective response. Lessons learned from the H1N1 pandemic of 2009 have been implemented and regular seasonal flu activities present opportunities to exercise the avian and pandemic influenza preparedness system.

Key informant interviews have indicated that PHAC has demonstrated its capacity to deliver a coordinated and effective response to immunization and infectious respiratory disease events through its response to a number of events since 2011, including response to the emergence of novel virus strains, (such as H5N1, H7N9 and MERS -CoV), multiple provincial and territorial pertussis and measles outbreaks. While PHAC has developed a template for conducting rapid risk assessment in the case of an emerging event (the Rapid Public Health Risk Assessment Framework), PHAC does not routinely track or report on its ad hoc response activities.

Do Canadians take positive actions to protect themselves?
  • % of 2-year old Canadian children having received at least one dose of measles-containing vaccine by their second birthday
  • % of adults aged 65 and over that have received pneumococcal vaccine (as per NACI recommendations)
  • % of national Immunization coverage targets met
Some minor, but important, issues to address

The vast majority of Canadian children are immunized; there remain pockets of under- or un-immunized in the country.

In 2011, measles coverage for the age group was 88.4% and in 2013 it was 89.6%; coverage has remained below the 97% target set in 1993 at the Consensus Conference on Measles. The cNICS estimates likely represent an underestimate of coverage, because cNICS methods are based on data collected from immunization records held by parents. Pockets of under- or un-immunized Canadians remain: aNICS data help monitor national coverage levels for the vaccines recommended by NACI, and coverage levels for specific target groups. The pneumococcal polysaccharide vaccine is recommended by NACI to be administered to seniors at the age of 65 and above. As reported by aNICS, between 2006 and 2014, pneumococcal vaccine coverage in adults over 65 years of age remained stable between 34% and 38%. These coverages are well below the target of 80%.

Challenges to improving immunization coverage include a lack of vaccine awareness or knowledge. Health care workers, a trusted source of information for Canadians, have been linked to increased vaccine uptake.

Demonstration of Economy and Efficiency
Has the program undertaken its activities in the most efficient manner? Are there efficiencies to be gained in the infrastructure of surveillance systems or in coordination and collaboration activities when working with PHAC's partners in this area?
  • Ability of the IT systems and services to properly support the needs, priorities and timelines of the surveillance program
  • Level of coordination and collaboration with PHAC's partners (e.g. agreement for research products with CIHR)
  • Efficiency of the governance model of the Immunization Program (maybe, to consider)
  • Mean number of days following a Notice of Compliance to release NACI Statements/statement updates for newly authorized products in Canada
  • Variance between planned and actual expenditures, and implications
  • Appropriateness of administrative overhead % (to consider - 1.4M$ in G&Cs to CIHR for research)
  • Degree to which CAEFISS SOP data processing timelines are met (surveillance)
  • Degree to which Medical Case Reviews (MCR) are completed according to MCR SOPS (surveillance)
  • Timeliness of the surveillance data
  • Efficiency of the immunization procurement process and comparative analysis with other models (e.g. pharmaceuticals)
Some minor, but important, issues to address

Immunization is a cost-effective public health intervention, as millions of dollars in direct and social costs are saved through this activity. Within PHAC, efficiencies have been realized through the effective use of some performance measurement information. There may be gained efficiencies of examining governance structures and adverse event case reviews.

PHAC has recently developed standard operating procedures to track timely serious case review for vaccine adverse event reporting. Generally, PHAC was able to meet standard operating procedures for reviewing of priority 1 (90%), priority II cases (94.5%) and priority III (99%) cases in 2015 (median times). Deviation from the goal to report within timelines in August and September of 2015 was the direct result of receipt of a backlog of more than 2,000 cases spanning more than a decade from one jurisdiction, which would not have impacted immediate adverse event response activity.

PHAC also undertakes medical case reviews (including a more substantial secondary review of more serious cases conducted by a medical staff expert). PHAC's ability to meet standard operating procedures in this area varied more (54% (median) of timelines met in 2015, based on monthly reporting). While this could be partially attributed to the backlog of cases mentioned previously, PHAC did not meet commitments over a number of months during a 24 month period. PHAC is looking at various ways to address capacity, and has recently increased the number of medical professionals to address these types of issues.

There are indications from the stakeholder survey that some working groups could operate more efficiently. Part of this may be attributed to membership confusion of roles and responsibilities, as well as a lack of clarity of the purpose and goals of the working group and how it fits into the broader F/P/T governance structure.

Has PHAC produced its outputs and achieved its outcomes in the most economical manner?
  • Views on whether costs of producing outputs is as low as possible and value is being obtained
  • Views on if funds are appropriately targeted
Some minor, but important, issues to address Internally, PHAC has expended approximately $151.9 million over a five-year period. Generally, there is little variance between planned spending and expenditures, and it is worthwhile noting that a majority of CIRID spending goes to pandemic preparedness (58%). Upon examination, overspending has been for pandemic planning purposes: in 2012-2013 operating expenditures were higher than expected for vaccine readiness and clinical trials, contributions to the National Antiviral Stockpile (NAS) and other minor activities to advance pandemic preparedness. These types of activities would have been required after the response to H1N1 in 2009 and 2010.
Is there appropriate performance measurement in place? If so, is the information being used to inform senior management decision-makers?
  • Existence of performance measurement framework or strategy
  • Adequate collection of performance information
  • Use of performance information in decision making
Some minor, but important, issues to address

Over the past few years, there have been a variety of studies conducted which have been used to advance work within the Centre. This is most evident by planned activities by the Centre, which are in line with findings of performance measurement (formal and informal) collected over the past two years.

While not all data were available, the majority of indicator reporting provided key lines of evidence for the evaluation but also directed the evaluation team in primary data collection activities. While some trend data is still absent, others will now be considered baseline data in moving forward.

Areas of improvement could focus on timely indicator reporting, reviewing the appropriateness of some indicators as valid measures of outcomes, whether other information available to the program could be outlined within the strategy, which could help in future program decision making.

Legend – Performance Rating Symbols and Significance:

The intended outcomes or goals have been achieved or met.

Considerable progress has been made to meet the intended outcomes or goals, but attention is still needed.

Little progress has been made to meet the intended outcomes or goals and attention is needed on a priority basis.

Endnotes

Page details

Date modified: