ARCHIVED - Evaluation of the C-EnterNet Program

 

Final Report

Prepared for:
Public Health Agency of Canada

Project Number: 570-2791
Program Evaluation & Performance Measurement Services
November 2009

Table of Contents

Executive Summary

List of Acronyms

1. Introduction

2. Methodology

3. Evaluation Findings

4. Conclusions and Recommendations

Annex A: Evaluation Matrix

Annex B: Matrix of Interview Questions

Annex C: List of Documents Reviewed

Annex D: Focus Group Moderator Guide

List of Tables

Table 1. Overview of C-EnterNet Surveillance Components

Table 2. Financial Resources ($’000) Committed to C-EnterNet by Source and Year

Table 3. C-EnterNet Program Timeline

Table 4. Summary of Evaluation Issues and Questions

Table 5. List of Interviews by Interview Group

Table 6. Perceived Usefulness of Different Program Components

Table 7. Articles and Reports Produced by C-EnterNet Program

Table 8. C-EnterNet Information Recipients – By Stakeholder Group

Table 9. Proportion of Survey Respondents Indicating that Different Types of Information had been used by them or their Organization

Table 10. Frequency of Different Usages of C-EnterNet Information, Among Respondents Stating They/Their Organization Had Made Use of C-EnterNet Information.

Table 11. Direct Costs of C-EnterNet Program

Table 12. Indirect Costs of C-EnterNet Program

Table 13. Costs per Program Element

List of Figures

Figure 1. Trends in Selected Enteric Illness, Canada, 1994-2004

Figure 2. Occupational Areas of C-EnterNet Information Recipients

Figure 3. C-EnterNet Information Recipients’ Views on Information

 

Executive Summary

This report presents the findings of the evaluation of the National Integrated Enteric Pathogen Surveillance (C-EnterNet) Program.  The evaluation was undertaken by Government Consulting Services (GCS) for the Public Health Agency of Canada (PHAC) between May and October 2009.  The purpose of the evaluation was to inform senior management resource allocation decisions and future decisions related to the program’s design and delivery.  The evaluation followed the scope and methodology set out in an evaluation plan completed prior to the commencement of the evaluation, and is aligned with the Treasury Board Policy on Evaluation.

The C-EnterNet Program, facilitated by PHAC, is designed to support activities to reduce the burden of enteric disease in Canada through sentinel site surveillance.  The core objectives of the program are:

  • to detect changes in trends in human enteric disease and in levels of pathogen exposure from food, animal and water sources in a defined population;
  • to generate human illness attribution values (i.e., the proportion of human cases due to exposure via water, food and animals); and
  • to improve the analysis, interpretation and reporting of laboratory and epidemiological data for public health, water and agri-food purposes.

C-EnterNet is delivered through the assistance of partnerships with the Region of Waterloo Public Health (ROWPH) and other organizations.  Partners within the sentinel site provide information on infectious enteric disease cases and coordinate with C-EnterNet’s team of scientists.  Information is also collected over time on rates of infection, transmission pathways, and pathogens.

The program was funded primarily through the Agricultural Policy Framework (APF) of Agriculture and Agri-food Canada (AAFC) between 2003-04 and 2007-08.  Since 2008-09, the program has been funded federally through the Food Safety Action Plan as well as through PHAC.

While the program design called for five sentinel sites across Canada to meet its pan-Canadian objectives, C-EnterNet implemented one pilot sentinel site in June 2005, in the Region of Waterloo, to assess the feasibility and usefulness of the program’s design.

Methodology

The evaluation made use of multiple lines of evidence to ensure the reliability of results reported.  The following research methods were used to gather data for the evaluation:

  • 25 key stakeholder interviews, including C-EnterNet team members, PHAC senior management, program partners and advisors, and experts external to the program;
  • a document review;
  • a survey of 102 recipients of C-EnterNet information products;
  • a review of administrative and financial data;
  • a survey of partners to determine the value of in-kind support leveraged by the program; and
  • a focus group with staff members of the ROWPH contributing to the program.

During the evaluation, the impact of the C-EnterNet Program was considered within the context of the pilot nature of the program and the recent implementation of many program elements.  In addition, the evaluation was descriptive in nature, and did not involve extensive comparative analysis.  Finally, the extent to which there may have been response bias in the survey of C-EnterNet information recipients is not known.

The evaluation concentrated on four evaluation issues: relevance; design and delivery; success; and cost-effectiveness/alternatives.

Relevance

There is a continued need for the C-EnterNet Program, due to the prevalence of enteric illness and its associated costs to Canadian society.  Research suggests that demographic changes and increasing globalization may result in increasing numbers of future cases of enteric illness in Canada.  Food-borne and waterborne illness, which make up a significant proportion of enteric illness, is a strong concern of the Canadian public.  The program is aligned with national priorities related to food and water safety and with international surveillance activities.  Support is strong among stakeholders for the continuation of all components of the program, and for expansion of the program through additional sentinel sites.

C-EnterNet is aligned with Government of Canada (GoC) objectives and priorities with respect to working towards the objective of healthy Canadians and the priority of ensuring food safety.  Within PHAC, the program is aligned with the program activities of infectious disease prevention and control and strengthening public health capacity.  The program is also aligned with the objectives of the ROWPH, and has contributed to the public health department priority of establishing a culture of excellence through collaboration and innovation.

The current role of the federal government as lead in the program is appropriate due to the intended national scope of the program and its relevance to public health and food safety.  Given the multi-disciplinary nature of the program, and its potential benefits for other organizations, future funding from other federal and provincial organizations and programs would be appropriate.

While the C-EnterNet Program as a whole does not duplicate the work of other programs or organizations, program complementarity would likely be improved through closer collaboration with the Canadian Integrated Program for Antimicrobial Resistance Surveillance (CIPARS), also located within PHAC and, potentially, with other organizations.  While the program has developed a variety of partnerships to assist in sampling, testing, and analysis, a comprehensive assessment of potential future partners would help to support planning, particularly as the program is working to implement a second sentinel site in 2009-2010.

Recommendation #1: The potential for increased collaboration with other federal and provincial organizations and programs should be assessed through a formal and systematic review.  The review should examine whether activities can be achieved through pre-existing processes in use by other organizations, and whether increased complementarity can be pursued with CIPARS and others.  This review should also seek to identify any potential future funding partners. 

Design and Delivery

The skills, knowledge and composition of the C-EnterNet team were identified by stakeholders as key strengths of the program.  The pilot nature of the program, with only one sentinel site in operation, is widely seen as the biggest impediment to the program fully achieving expected results.  Without a network of sentinel sites as envisioned in the original program design, data are not as statistically powerful, cannot be used for comparisons across regions, and cannot generate national estimates.

The C-EnterNet design and delivery structure is, overall, flexible enough to be adapted to different sentinel sites across Canada.  The distance and time required to establish and maintain additional sentinel sites will be a challenge, particularly given the limited size of the C-EnterNet team and its heavy involvement in operational matters in the pilot site.  Expansion to additional sentinel sites will not be possible without additional resources or a re-conceptualization of C-EnterNet team roles and responsibilities.

Recommendation #2: A review of the structure, roles and responsibilities of the C-EnterNet team should be undertaken, particularly in light of future program expansion.  In the absence of additional funding, the program should examine ways to reduce the scope of activities undertaken by the C-EnterNet team through more extensive operational involvement by partners.

Currently, senior management responsibility for C-EnterNet is located within two centres of PHAC.  Day-to-day management has been undertaken at the level of the Program Lead, and management processes have been largely ad hoc and informal.  While the joint management structure and the lack of formal management processes have not been an impediment to success in program implementation, they do present a risk to continued success, particularly as the program expands.  Effective management processes and a more clear management structure will be critical in moving forward.

Recommendation #3: Senior management responsibility for C-EnterNet should be placed solely within one centre at PHAC

The program has established an Advisory Committee made up of scientists and stakeholders from different organizations and with different areas of expertise.  While effective as an advisory body, the Advisory Committee has not taken a strong role in strategic planning.

Recommendation #4: C-EnterNet should undertake a formal, annual program planning exercise.  This should involve input from Advisory Committee members to help identify future directions and opportunities for program marketing and collaboration.

A Steering Committee has also been established at the ROWPH, which has provided an effective venue for communication with ROWPH senior management.

Program Success

The C-EnterNet Program has developed and distributed a series of information products, including annual reports and newsletters.  The program team has also authored or co-authored several peer-reviewed articles on source attribution and other aspects of the program’s research.  Program members have also organized and attended numerous presentations and other venues for knowledge dissemination. 

Recipients of C-EnterNet information reported a high degree of satisfaction with the data and information produced by the program.  Timeliness of the information appears to be somewhat of an issue, however. 

Recommendation #5:  The C-EnterNet team should develop a strategy for improving the timeliness of its information products.

Most recipients of C-EnterNet information recipients are making use of the information provided.  Stakeholders and recipients of information products felt the major gap in the data was that it was from only one sentinel site, and therefore not nationally representative or able to be used for regional comparisons.

Recommendation #6:  Additional sentinel sites should be pursued, as envisioned in the original program design.  However, establishing additional sites should be contingent on identifying additional funding partners or arrangements, as expansion should not risk having a negative impact on the pilot site.

C-EnterNet has successfully established and developed collaborative networks related to enteric disease with both Canadian and international experts.  These networks have extended across multiple disciplines and included all levels of government.

With respect to the source attribution component of the program, C-EnterNet has brought together scientists to advance source attribution through a working group, and has contributed to a review of source attribution methodologies developed outside of Canada.  The program has begun to adapt approaches used elsewhere to Canadian data.  The lack of nationally representative data from C-EnterNet has meant that other data sources have been used for estimates undertaken to-date.  Preliminary source attribution estimates have been undertaken, but, relative to the integrated surveillance component, progress appears to be somewhat limited in this area.

C-EnterNet information and data appear to be most often feeding into the work of researchers and those involved in services/program delivery, and less often senior management or policy analysts.  As additional years of data become available, and trends emerge from the data analysis, it is expected that the implications of the data in shaping policy and practices will become more apparent.  To date, C-EnterNet information products have not addressed potential implications for policy/practice of the data presented.  This has been exacerbated by the heavy workload of the core team during implementation of the pilot.  It is also a result of a lack of a clear marketing strategy for the program.

Furthermore, based on the survey results, a proportion of C-EnterNet information recipients lack awareness of the program, is not reviewing information products they are sent, and is not making use of the data.  Additional marketing of the program would raise awareness of its applicability to stakeholders and their work.

Recommendation #7:  C-EnterNet should develop a marketing and knowledge dissemination strategy.  This strategy should work to identify: 1) key messages derived from program findings that could contribute to informing programs and policies; 2) the appropriate audiences or stakeholders with whom these messages should be shared to have maximum impact; and 3) venues/products most effective for reaching these audiences/stakeholders.  Again, the Advisory Committee should be solicited for input in this activity.  The program should examine the possibility for collaboration with CIPARS in this area.  A clear conceptual framework document for the program should also be developed, which will help with marketing and knowledge dissemination.

C-EnterNet has had a positive impact on the level of knowledge of enteric illness and exposures among information recipients and staff at the ROWPH.  To date, information disseminated by the program has been largely used to support other research activities, but other uses have also been made at the level of informing programs and practices.

The C-EnterNet Program has increased public health capacity through its partnership with the ROWPH.  This has included developing a new case follow-up questionnaire and factsheets on enteric illness, and undertaking training and workshops with public health inspectors and other staff in the health department.  There is minimal evidence that public health capacity has been strengthened significantly outside of the sentinel site at this time.

Evidence from across the evaluation suggests there is potential for the program to help reduce the burden of human enteric illness in the future.  Additional time and a network of sentinel sites are, however, critical to achieving this long-term goal.

Cost-effectiveness and Alternatives

The program design and delivery approach appears to have been developed to ensure cost-efficiency within the methodology selected for the program.  As discussed, the feasibility of other cost-sharing arrangements or cost-efficiencies from partnerships should continue to be pursued.  To date, the program has benefitted from substantial in-kind support, particularly in relation to its water component, and appears to provide value relative to its costs.  However, the full value of the program will not be fully realized until C-EnterNet becomes pan-Canadian in scope.

Page details

Date modified: