ARCHIVED - Evaluation of the C-EnterNet Program Final Report

 

Methodology

The methodology for the C-EnterNet Program evaluation was developed during an evaluation planning phase completed before the evaluation.  Evaluation planning was undertaken in consultation with the C-EnterNet team, and with input from the C-EnterNet Advisory Committee.  The following section outlines the evaluation issues and questions, data collection methods, and methodological considerations for the evaluation.

2.1 Evaluation Issues and Questions

The C-EnterNet Program evaluation examined issues relating to program relevance, design and delivery, success, and cost-effectiveness/alternatives.  Table 4 provides a summary of the evaluation issues and questions addressed in the evaluation.  Please see Annex A for the complete evaluation matrix, which also includes specific indicators and methodologies for each evaluation question.

Table 4. Summary of Evaluation Issues and Questions
Evaluation Issue Evaluation Question
Relevance
  • Is there a continued need for C-EnterNet?
  • Does C-EnterNet align with Government of Canada, PHAC objectives and priorities, and local public health priorities?
  • Is the current role of the federal government appropriate?
  • Does C-EnterNet complement, duplicate or overlap the work of other programs or organizations?
Design and Delivery
  • What factors in the C-EnterNet design and delivery facilitate or impede the success of the program?
  • What does the implementation of C-EnterNet in the pilot site suggest in terms of the potential for having a network of sentinel sites across Canada?
  • To what extent is the C-EnterNet program well-managed?
Program Success Outputs
  • Did the C-EnterNet pilot generate the intended information (outputs)?
Immediate Outcomes
  • Has up-to-date and relevant information on enteric diseases and exposures been received by the right stakeholders in a timely manner?
  • Has C-EnterNet resulted in the establishment and development of collaborative networks related to enteric disease?
  • What contribution has C-EnterNet made to the advancement of source methodologies for Canada
Intermediate Outcomes
  • Has C-EnterNet resulted in increased knowledge of enteric diseases and exposures among stakeholders?
  • To what extent has C-EnterNet strengthened public health capacity?
  • Did the information produced by C-EnterNet inform, create or help to evaluate programs, policies, practices (e.g., risk assessments) or health messages related to infectious diseases and exposures?
  • Has C-EnterNet established any Canadian estimates on source attribution?
Ultimate Outcomes
  • Is C-EnterNet making progress in achieving its ultimate goal of contributing to reducing the burden of enteric illness in Canada?
  • Has the program resulted in any unintended impacts?
Cost-effectiveness and Alternatives
  • Are there alternative design/delivery approaches/models or modifications that might be more cost-effective and still achieve similar or better outcomes?
  • Is C-EnterNet providing value for its cost?

2.2 Data Collection Methods

The C-EnterNet evaluation included the use of multiple lines of evidence and complementary research methods to help ensure the reliability of information and data collected.  The following data collection methods were used to gather data for the evaluation:

  • key stakeholder interviews;
  • a document review;
  • a survey of C-EnterNet information recipients;
  • a review of administrative and financial data;
  • a survey of partners to examine costs of the program; and
  • a focus group with staff of the ROWPH.

Each of these methods is described in more detail as follows.

2.2.1 Key Stakeholder Interviews

A total of 25 interviews were completed for the evaluation.  Interviews were undertaken with members of the C-EnterNet team, PHAC senior management, members of the ROWPH Steering Committee, members of the C-EnterNet Advisory Committee, and relevant experts.  The number of interviews completed for each interview group is illustrated in Table 5.

Table 5. List of Interviews by Interview Group
Interview Group Number of Interviews Conducted
C-EnterNet team members 3
PHAC senior management 3
ROWPH Steering Committee members 4
C-EnterNet Advisory Committee members 12
Experts 3
Total 25

Different interview guides were used for each of the interview groups (see Annex B for a matrix of the interview questions used).  Interviews were conducted both in-person (during a site visit to Guelph and Waterloo, as well as in the National Capital Region) and by telephone.  Interviews were entered into a database for subsequent analysis.

2.2.2 Document Review

The following types of documentation were reviewed during the evaluation:

  • Corporate, accountability and political documents: including PHAC’s Performance Measurement Framework and Program Activity Architecture, Departmental Performance Reports and Reports on Plans and Priorities. Other documents included Speeches from the Throne and the Government of Canada’s Science & Technology Strategy, among others.
  • Operational documents: including the C-EnterNet business case, business plan, and previous reviews of the program.
  • Program outputs: including annual and accomplishment reports, newsletters, and information on activities related to source attribution completed by the program.
  • Academic and “grey” literature: including information on the costs borne by Canadian society from enteric disease, trends in pathogens, and other background documentation identified during the planning phase of the evaluation.

Annex C contains a full list of documents reviewed.

During the review, findings from documents were entered into a matrix, organized by evaluation issue.  The document review was primarily used to inform an assessment of the relevance and the design and delivery of the program.  It also provided information related to program success and cost-effectiveness/alternatives. 

2.2.3 Survey of Information Recipients

The C-EnterNet Program maintains a distribution list of individuals who are sent C-EnterNet documents/outputs, including annual reports and newsletters.  Using this distribution list, the evaluation team conducted an online survey of C-EnterNet information recipients. 

A survey questionnaire was developed to address the evaluation questions identified in the evaluation matrix.  GCS undertook a pre-test of the survey questionnaire, in both English and French, with seven respondents from a pre-determined group of individuals identified by the C-EnterNet team.  Minor revisions to the survey questionnaire wording were undertaken based on the results of the pre-test.

Potential respondents were sent an e-mail invitation inviting them to participate in the online survey.  An e-mail invitation was sent to 284 potential respondents.   The online survey form remained active from June 29th to August 14th, 2009.  During this time, GCS also sent out reminder e-mails at two intervals to those who had not completed the survey.

In total, 102 respondents completed the survey, representing a response rate of 35.9%.  Based on these numbers, the overall margin of error for the survey is ±7.8%, 19 times out of 20.  It should be noted that the number of survey responses did not allow for extensive analysis of the survey results by particular respondent groups (e.g., results from federal government respondents compared to provincial government respondents).

2.2.4 Administrative and Financial Data Review

Administrative Data

Two major types of administrative data were analyzed during the evaluation: program data related to program activities/outputs and program financial data.

Program activity/outputs data analysis included: an examination of the number and type of presentations, training and workshops completed by project team members; the number and nature of partnerships; and the types of information inquiries received by C-EnterNet. 

Financial Data

Financial information from the C-EnterNet Program was analyzed to assess the total costs of the program, as well as the specific costs of different areas of the program, including administrative versus other operating costs, and costs related to each of the program components.  Costing was undertaken according to the general approach outlined in the Treasury Board Guide to Costing.

Financial data was obtained from C-EnterNet management.  As available data from fiscal year 2006-2007 was the most detailed, and little change in program budget was noted in recent years, 2006-2007 became the focus of the costing analysis. 

2.2.5 Costs Survey

A short survey of partners and other agencies with whom C-EnterNet collaborates was undertaken to collect data to estimate the value of leveraged and in-kind contributions of other agencies to the C-EnterNet Program. These data helped address the value for money provided by C-EnterNet through establishing an estimate of cost-savings realized by C-EnterNet through collaborative activities.

The evaluation team worked with C-EnterNet to finalize a list of 19 partners and other agencies that have provided in-kind support to the program for which they were not provided financial compensation.  In-kind support was often provided by partners in return for access to C-EnterNet Program data.

Each of these organizations was contacted by email to ask for their participation in a short e-mail survey.  To obtain maximum response to the survey, telephone follow-up was undertaken by the evaluation team with those who did not complete the survey.  In total, GCS received nine responses.  These responses do, however, represent those partners identified by the C-EnterNet team as providing the most significant portion of in-kind support to the program.

2.2.6 Public Health Department Staff Focus Group

GCS moderated a focus group with seven ROWPH staff members involved in activities in support of the C-EnterNet Program.  Participants included an epidemiologist, a planner and five public health inspectors.  The focus group was used to examine whether, and in what ways, C-EnterNet has increased public health capacity, including what types of impacts the program has had on ROWPH staff and the region.  A moderator’s guide was developed in advance of the focus group to ensure that the session addressed all relevant evaluation questions.  The moderator’s guide can be found in Annex D.

2.3 Limitations and Considerations

As in all evaluations, there are methodological limitations and considerations that should be noted.

The impact of the C-EnterNet Program must be considered within the context of the pilot nature of the program and its recent implementation.

C-EnterNet’s program design envisions a series of surveillance sites located across Canada.  Multiple sentinel sites are intended to allow for results to be extrapolated across all regions of Canada.  At the time of the evaluation, the program included only one sentinel site.  Consequently, the program could not reasonably be expected to fully achieve its expected outcomes on a pan-Canadian level.  However, the need for additional sentinel sites was examined as part of the evaluation.

Similarly, an important purpose of the data collected by the C-EnterNet Program is to examine trends over time in both human cases of enteric disease, and in pathogens from retail food, animal and surface water sources.  The program only released its first results of its surveillance activities in 2006, and has only produced annual reports for three years.  In addition, some surveillance activities only commenced more recently, such as produce sampling.  Trends over time are only now beginning to be identified.

Evaluation is descriptive, rather than comparative, in nature.

It was beyond the scope of the evaluation to empirically assess the program’s design, delivery, success, and cost-effectiveness in comparison to similar programs, in Canada or internationally.  The evaluation approach is, therefore, descriptive, rather than comparative, in nature.

The extent of response bias in the survey of information recipients is not known.

It was not possible to undertake non-response bias analysis for the survey of C-EnterNet information recipients.  As a result, the degree to which the responses of the survey represent those of the entire population of information recipients is not known. 

Despite these considerations, GCS has designed the evaluation to use multiple lines of evidence (e.g., interviews, surveys, administrative data, documentation, and a focus group), thus strengthening the reliability and validity of the evaluation results.

Page details

Date modified: