ARCHIVED - Formative Evaluation of the National Collaborating Centres for Public Health Program (NCCPH)

 

Appendix E: Detailed Results by Evaluation Issue

Issue 2:  Achievements and Successes

Issue #2 relates to the evaluation question – what is the status of the NCCPH Program’s progress toward achieving its immediate goals and objectives?  Findings are presented under the following section headings:

  • Description of  NCCPH Program Secretariat activities;
  • Overview of NCC activities
  • Immediate outcome: Collaboration;
  • Immediate outcome: Knowledge translation;
  • Immediate outcome: Knowledge gap identification;
  • Immediate outcome: Networking; and
  • Intermediate outcomes.

In the feedback components within the sections that follow, unless identified as individual comments, themes are presented in general descending order of the frequency with which they were mentioned across 10 informant groups.  Each of the six NCCs comprise an informant group, as do Advisory Council (AC) members, NCC Advisory Board members, PHAC or national representatives, and interested or involved parties. 

NCCPH Program Secretariat Activities and Products/Outputs

The activities of the NCCPH Program Secretariat are reported below based on available documents, interviews conducted during a site visit to Ottawa, and ongoing communications with staff members.  The reported activities relate to expected activities listed in the RMAF and/or Progress Evaluation Framework.

Establish and provide secretariat support to the Advisory Council

The NCCPH Program Secretariat was required to establish an Advisory Council (AC) comprised of diverse public health experts.  According to available documents, the initial development of the AC occurred as early as December 2004.  The AC Terms of Reference were established in early 2005 and are reviewed annually in February.  The first in-person AC meeting was held in July 2005.  PHAC provides secretariat support to the AC, including preparing meeting agendas and materials and drafting meeting summaries and reports for AC approval.  The Terms of Reference specify meetings will be held up to four times a year. 

Preparation of documents for funding approval

Following approval of the NCCPH at the policy level of government, the NCCPH Program Secretariat was required to provide documents to secure funding such as a program design, RMAF and risk assessment plan, and to complete a Funding Approval Form for each NCC.  

Establish NCCs

The NCCPH Program Secretariat solicited and reviewed proposals for the six NCCs.   The NCCPH Program Secretariat began negotiating NCC Contribution Agreements with NCCs.  Evidence of the establishment of the six NCCs is the signed Contribution Agreements for each centre, the dates of which are listed in chronological order:

  • Healthy Public Policy: December 20, 2005
  • Aboriginal Health: December 22, 2005
  • Infectious Diseases: December 23, 2005
  • Environmental Health: July 14, 2006
  • Determinants of Health: August 18, 2006
  • Methods and Tools: December 1, 2006
Monitor program

As part of their Program monitoring function, the NCCPH Program Secretariat is required to review and approve NCC deliverables, including annual work plans, budget review, and semi-annual progress reports and to monitor activities related to environmental scans, strategic plans, and evaluation. They accomplish this with input from the AC in relation to content and technical expertise.  The submission, review and approval process of NCC deliverables follows a typical pattern.  An overview of this pattern is provided in Table 3. 

Table 3: Pattern of the submission, review and approval process of NCC deliverables by calendar year
Month Description

January

NCCs submit work plans and projected cash flow for fiscal year

NCC Scientific Directors present work plans to AC and NCCPH Program Secretariat in-person

NCCs submit annual update on projected spending

February

NCCPH Program Secretariat provides written feedback on work plans to NCC Scientific Directors

April

NCCs submit revised work plans, if required

May

NCCs submit progress report including year end record of expenditures (actual spending) for the previous fiscal year

June/July

NCCPH Program Secretariat provides written feedback on progress report (NCC specific and general) to NCC Scientific Directors

October

NCC submit progress report, includes cash flow forecast (projected spending) and record of expenditures (actual spending)

November/December

NCCPH Program Secretariat provides written feedback on progress report (NCC specific and general) to NCC Scientific Directors


Generally, in January, the Scientific Directors (SDs) of the NCCs submit to the NCCPH Program Secretariat their annual work plans and projected cash flow forecast.  These are shared with the AC.  Two members of the AC are assigned per NCC to conduct an in-depth review of the work plans and cash flow forecast.  The NCCPH Program Secretariat organizes an in-person meeting of the SDs, AC, and the NCCPH Program Secretariat.  At the meeting, the SDs are given approximately 15 minutes each to present their NCCs’ work plans.  Next, the AC members responsible for the in-depth review per NCC provide feedback to the SD on the work plans and open the floor to other AC members.  The NCCPH Program Secretariat documents the feedback on the work plans elicited at the in-person meeting.  Written feedback is provided to the SDs of each NCC via the NCCPH Program Secretariat within one month of the in-person meeting.  The work plans and projected cash flow for each NCC are either approved or require revisions.  If revisions are required, the revised work plans and/or projected cash flow forecast is due as soon as possible, typically in April. 

Also in January, the SDs submit to the NCCPH Program Secretariat and the AC for review their annual update on projected spending.

In May and October, the SDs of the NCCs submit to the NCCPH Program Secretariat and the AC for review their semi-annual progress reports as related to their annual work plans.  In the May progress report, NCCs provide their year end record of expenditures (actual spending) for the previous fiscal year. 

This progress report may also include an annual evaluation and/or annual audit report.  The October progress report includes the cash flow forecast (projected spending) and record of expenditures (actual spending).  In addition to providing the SDs of each NCC with feedback specific to their progress related to their Centre’s work plans, the NCCPH Program Secretariat and AC provide general feedback related to the progress of all six NCCs.   

Based on review of the documents available to the evaluators, the NCCPH Program Secretariat and AC members undertake this review activity on a routine bases and provide timely feedback in relation to the NCC work plans and semi-annual progress reports.  Based on review of the written feedback provided to NCCs, and evidenced by examples of work plans requiring revisions, it appears the review process is a substantive one, taken seriously by all parties involved including the NCCs.  The activities of the NCCs in relation to their reporting requirements are provided in the section Description of NCC Activities. 

Risk assessment of the NCCPH Program and each NCC

As described in the RMAF, the NCCPH Program is contained under the umbrella of the Promotion and Population Health’s (PPH) Terms and Conditions for Contributions and, therefore, the PPH – RBAF Key Risk Areas were used as the framework for risk assessment.  Under this framework, the nature of potential risks that were identified were:

  • Reporting on results;
  • Internal communications;
  • Changing environment;
  • Capacity to manage and monitor;
  • Focus on particular clients, projects, delivery lines;
  • Bilingual service;
  • Addressing short term need;
  • Potential for surprise; and
  • Inappropriate use of funds.

Management strategies for each were developed, all of which are outlined in the RMAF document (July 2005). 

According to the Contribution Agreement, the NCCPH Program Secretariat is required to conduct annual risk assessments for each NCC.  The project risk assessment questionnaire used by the Program Secretariat consists of 20 questions that address the following general topics:

  • Reporting on results (e.g., timeliness and quality of progress reports and work plans);
  • Capacity to manage and monitor (e.g., ability to manage activities and overall project);
  • Focus on particular clients (e.g., ability to engage public/community involvement);
  • Use of funds (e.g., amount of Contribution Agreement, amendments, completion and quality of financial reports, number and results of audits, additional public funding received); and
  • Human resources (e.g., staff stability).

Each question has a rating on a three-point scale, with one being the most desirable score and three being the least desirable score.  Scores are entered into the departmental Grants and Contributions database, where the final score is calculated and assignment of a risk level of low, medium, or high is determined.     

The NCCPH Program Secretariat has conducted risk assessments for all six NCCs for the 2006-2007 and 2007-2008 periods.  Four NCCs received a risk level rating of low in both 2006-2007 and 2007-2008. 

One NCC received a medium risk level rating in 2006-2007 and a low risk level rating in 2007-2008.  The risk scores for this NCC improved from 2006-2007 to 2007-2008 across nine items, including completeness of narrative reports, compliance with program guidelines, meeting requirements for project evaluation, ability to manage activities and the project, and staff stability.  

One NCC received a low risk level rating in 2006-2007 and a medium risk level rating in 2007-2008.  The risk scores for this NCC deteriorated between the assessment periods across eleven items, including completeness of narrative reports, working relationship with recipient, ability to engage public/community involvement, compliance with program guidelines, meeting requirements for project evaluation, completeness of financial records, ability to manage activities and the project, and staff stability.  

Financial review

According to the NCCPH Program Secretariat, a financial audit or review is conducted periodically or based on the results of the annual NCC risk assessment, the NCC semi-annual progress reports, and/or as requested by the Management and Program Services Directorate.  In April 2008, PHAC contracted BMCI Consulting Inc. to conduct a financial review of the NCCPH program.  The objective of the review was to reconcile PHAC financial records (e.g., PHAC Grants and Contribution database) with financial records from the six NCCs (e.g., Statement of Receipts and Disbursements).  All six NCCs were reviewed for the funding years of 2005-2006, 2006-2007, and 2007-2008, where applicable (i.e., upon entering into a Contribution Agreement with PHAC).Footnote 34

NCC site visits

As a requirement of the Contribution Agreement program, the NCCPH Program Secretariat must conduct NCC site visits.  According to the NCCPH Program Secretariat, site visits are conducted periodically or based on the results of the annual risk assessment/NCC, NCC progress reports, and/or as requested by the Management and Program Services Directorate.  To date, the NCCPH Program Secretariat has conducted three site visits to three NCCs between April 2007 and June 2008.  The Program Secretariat anticipates conducting another four site visits by March 2009.

In general, the purpose of the site visits completed to date were to gain a better understanding of the issues and concerns related to establishing a NCC from a user perspective and to assess NCC management and governance, financial management, and compliance with Contribution Agreement clauses.  The NCCPH Program Secretariat provided written feedback including recommendations to the SDs of the three NCCs within two months of the visits. 

Development of program materials

As specified in the 2005 RMAF document,Footnote 35 the NCCPH Program Secretariat was responsible to develop Program materials for the NCCs (e.g., newsletters, website, and media kits).  In late 2005, the NCCPH Program Secretariat informed the NCC leads (e.g., Scientific Directors) that they could not provide common services across the NCCs due to the arm’s length nature of the Contribution Agreements.  The NCC Leads subsequently established a Leads Secretariat, and responsibility for undertaking the development of program materials appears to have shifted to the NCC Leads, as evidenced by the development of a communication plan as one of their primary collaborative activities.

Planning and priority setting

The intention behind this logic model item was not clear to the evaluators, and limited evidence of NCCPH Program planning and priority setting activities or documents were evident until recent months.  These are described below.

At the NCCPH Program level, two major planning and priority setting activities were planned and executed: Strategic Directions Meeting at the Summer Institute in August 2008; and NCCPH Strategic Planning Retreat in November 2008. 

At the 2008 Summer Institute, the leads of the six NCCs met with PHAC staff, Advisory Board members, and Advisory Council members to discuss strategic directions for the Program over the next six to 12 months.  The outlined objectives of the meeting were to:

  • Ensure national scope;
  • Develop a joint strategy to identify key vulnerable populations;
  • Identify and amass what is happening and working well across the NCCs; and
  • Address current political environment and implications for the direction of NCCs.

Meeting activities focused on identifying new projects, developing consistent processes, and building networks across the NCCs.

In addition to the Strategic Directions Meeting at the Summer Institute in August 2008, the previous two Summer Institutes (2006 and 2007) provided the NCCs with a forum to meet and plan. 

At the NCCPH Strategic Planning Retreat, the leads of the six NCCs met with PHAC staff, Advisory Board members, and Advisory Council members with the purpose of drawing from the experience and perspectives of the involved parties to identify strategic directions and actions to strengthen the contribution of the NCCs to public health infrastructure and public health practices in Canada.  The intended outcomes of the session were:

  • A high level Strategic Directions Plan with supporting action steps tool to help guide the development and activities of the six NCCs over the next five years;
  • Points of focus and proposals that will be advanced through the renewal of the NCCPH Program; and
  • Steps to strengthen collaboration and cooperative efforts among the NCCs.

Based on the summary report of the Strategic Directions MeetingFootnote 36, the parties worked together to:

  • Develop a shared understanding of the NCCPH accomplishments, challenges and opportunities;
  • Establish a set of strategic directions, priorities and practical actions for guiding NCC activities and moving the NCCPH forward over the next 2 to 5 years; and
  • Identify points of focus and potential proposal to be advanced through the NCCPH Program Renewal Process.   
Evaluation and quality improvement activities

Programs need to comply with RMAF requirements which place a strong emphasis on evaluation activity.  An evaluation framework was required for the NCCPH (completed in 2005).  This initial evaluation framework was subsequently refined and is presented in the Progress Evaluation Framework document (February 2008).   The evaluation framework includes a program logic model, evaluation questions, indicators and potential data sources and methods.  This report represents the completion of a formative evaluation and will provide baseline information for the future summative evaluation, planned for completion before March 2010.   

Reporting

A Summary Report of NCC Program activity was produced in May 2006 by the NCCPH Program Secretariat.  In it, the author provided an overview of the NCC approach, including early lessons learned; program progress to date, brief descriptions of each NCC, program linkages, and 2005-2006 fiscal activity; objectives for 2006-2007, and issues and recommendations.  This document was submitted to the Deputy Chief Public Health Officer.

Overview of NCC Activities

The six NCCs have followed a general pattern of planning, implementation, identifying KSTE processes, and producing products.  The planning stage consisted of developing initial work plans for the NCC.  For some NCCs, this stage included engaging interested and involved parties to prioritize work plan areas.  Generally, implementation activities focused on infrastructure development and accessing human resources.  Next, the staff of the NCCs began determining internal processes to address their KSTE mandate.  The final stage of development was producing KSTE products (e.g., fact sheets, synthesize reports on best practices).  

All six NCCs have touched on all four stages of development.  However, as evident in Figure 3 of the main document, the six NCCs are at different stages of development based on when each entered into a Contribution Agreement.  

The NCC primary activities, based on available documents, are reported below.  It is important to note this section describes the submission of documents and reports only and does not address the completeness or quality of the reports. 

All host organizations submitted a proposal in response to the RFP and all Contribution Agreements were signed by December 2006.  All six NCCs have established Advisory Boards with a mix of experts.

Four NCCs submitted their annual work plans in January 2006.  The NCCDH submitted three work plans in 2006, two of those prior to the signing of their Contribution Agreement.  All six NCCs submitted work plans for 2007 and 2008.

It appears staff of the NCCs had difficulty submitting their semi-annual progress/narrative reports early in their development but have gained consistency over the past three years.  Two out of the three established NCCs and one pre-established NCC submitted May 2006 progress reports.  One of the three established NCCs and one pre-established NCC submitted their October 2006 progress reports.  In 2007, five out of six NCCs submitted their May progress reports and all six submitted their October progress reports.  In 2008, all six NCCs submitted their May progress reports.  Reporting on the NCC submission of October 2008 progress reports was outside the timeline of this evaluation.

It appears that there is some uncertainty as to the NCC requirements in relation to evaluation activities. While it is stated in the Contribution Agreement that annual evaluation and audit reporting is to be submitted in the May progress reports, the NCCPH Program Secretariat and staff of one NCC stated that annual evaluation and audit reporting by the NCCs is not a requirement.  Based on the available NCC documents, none of the three established or three pre-established NCCs completed an annual evaluation or audit in 2006.  In 2007, three out of six NCCs reported conducting an audit and one out of six reported conducting an evaluation.  The same three of six NCCs that conducted an audit in 2007 did so again in 2008.  Three out of six NCCs reported evaluations (either complete or in progress) in 2008.  

According to the Contribution Agreement, NCCs are also required to conduct an evaluation by January 2009.  Three NCCs have included evaluation as part of their annual work plans.  To date, five out of six NCCs report having either an evaluation plan (1) or completing an evaluation (4).   Delay with this activity is not surprising for those NCCs whose implementation was delayed.

NCCs are also required to participate in any regional, provincial, territorial and/or national evaluation.  All six NCCs willingly and openly participated in this formative evaluation.

At the NCC level, Managing Directors reported quality improvement and evaluation activities for the six NCCs.  Five out of six NCCs report having conducted a program evaluation or having an evaluation plan.  Four NCCs reported meetings as a mechanism of quality improvement, including staff meetings, regular multidisciplinary meetings and debriefings (formal and informal).  Two out of six reported quality assessment reviews or quality assurance plans.  Two NCCs identified user or expert feedback as a mechanism for quality improvement.  Quality improvement or evaluation activities reported by individual NCCs include: Advisory Broad, reports, environmental scan, and work plan development.   

Further detail about NCC KSTE activities and products/outputs are described in the sections that follow. 

Immediate Outcome: Collaboration

The program logic model identifies as an immediate outcome increased opportunities for collaboration with (a) the health portfolio and (b) NCCs.  Descriptions of collaboration with the health portfolio and the NCCs are provided in turn.   

Description of NCC Collaboration with the Health Portfolio

According to the RMAF, the ‘Health Portfolio’ refers to all programs carried out with the authority of the Minister of Health, e.g., PHAC, Health Canada, CIHR, etc”.Footnote 37   In keeping with this definition, including a focus at the federal level, other examples of health portfolio agencies could include First Nations and Inuit Health (FNIH) and the Public Network Health (PHN).

Several examples of collaboration of the NCCs with the health portfolio were provided by informant groups:

  • HIV/AIDS Policy Coordination and Programs Division of PHAC collaborated with the NCCID and co-founded a National HIV Prevention Forum held in Ottawa in April 2007.  The objective of the forum was to (a) build bridges/relationships between traditional public health and front line practitioners and (b) identify gaps and priorities for the future.  A broad invitation was sent out; approximately 100 people participated including clinicians, researchers, organizations, as well as public health nurses, doctors and medical officers.
  • Centre for Chronic Diseases of PHAC contracted the NCCMT to develop the Canadian Best Practices Portal for Health Promotion and Chronic Disease Prevention tool.
  • Community Acquired Infections Division of PHAC collaborated with the NCCID related to HPV vaccination.
  • NCC Leads Secretariat provided a presentation to the PHN in September 2008.
  • NCC Leads Secretariat conducted a survey of the Chief Medical Officers of Health.
Description of NCC Collaboration

Increased opportunities for collaboration with NCCs can be divided into two categories: the NCC Collaborative; and NCC to NCC initiatives.

NCC Collaborative

The purpose of the NCC Collaborative is to provide a forum for collective action and address administrative issues across the six NCCs and the NCCPH Program.  Membership as reported in meeting minutes (Dec. 22, 2005) includes one representative from each NCC and two representatives from the NCCPH Program Secretariat.  The meeting schedule consists of one meeting prior to the NCC Directors meeting (up to four times per year) and two monthly teleconferences.  The next section provides a brief historical description of the establishment of the NCC Leads Secretariat.

The first documented meeting of the NCC leads occurred on September 22, 2005.  According to available documentationFootnote 38, staff at the NCCID proposed and offered to carry out a transitional set of common services, originally called the NCC Common Services, in January 2006.  This was in response to the:

  • Changing role of the NCCPH Program Secretariat in providing organizational and administrative support to the NCCs due to the nature of the arm’s length relationship;
  • AC recommendation that the NCCs work together more thoroughly to develop common approaches, share information, expertise, capacities and resources; and
  • Similar needs of all NCCs for organizational and administrative support. 

At this stage, the purpose of the NCC Common Services was limited to: organizing meetings/ teleconferences; developing collaborative project matrix; facilitating strategic planning sessions regarding a common NCC vision; and facilitating NCC presentations at events.  While this group was viewed as valuable, several challenges were identified (e.g., website development progress) as a result of the informality of the arrangement and confusion around decision-making authority.  In November 2006, the NCCID proposed to serve as the provider of a more formalized Common Services Network.  Shortly after, the NCCID encountered a shift in management and staff felt they could not act as an effective host for the Common Services Networks during this transitional period. 

In April 2007, the NCC Common Services was relocated to the NCCDH and is currently referred to as the NCC Leads Secretariat.  The position of Project Officer of the Leads Secretariat was established.  The responsibilities of the Lead Secretariat for the time period of April 2007 to March 2009 are:

  • Organize meetings/teleconferences;
  • Facilitate plans and arrangements for NCC presentations;
  • Develop Collaborative Project Matrix; and
  • Facilitate development of an NCCPH website.

A total of 41 NCC leads meetings have been held between September 2005 and May 2008.  In addition, the NCC leads have held separate meetings related to communications and the Summer Institute.  The NCC leads meetings provide a formal opportunity to share identified best practices among the NCCs.  

To date, the primary achievements of the NCC Collaborative are:

  • Establishment of the NCC Leads Secretariat and Project Officer (as described above);
  • Initiation, organization and evaluation of three Summer Institutes;
  • Development of a common web-portal; and
  • Finalization of a communication strategy.

The Summer Institute represents a formal opportunity for the sharing of best practices among the NCCs. The inaugural Summer Institute was held in Toronto, Ontario in July 2006 with a focused agenda to build inner capacity in KSTE within the Program.  Participation was limited to NCC personnel, PHAC staff, and invited presenters.  Approximately 50 individuals attended.  The second Summer Institute was held in Baddeck, Nova Scotia, in August 2007 with an agenda focused on KSTE definitions, understanding what is evidence, and engagement of communities.  Again, participation was limited to NCC personnel, PHAC staff, and invited speakers.  The number of participants increased from 50 to 100.  The third Summer Institute was held in Kelowna, British Columbia in August 2008 with the objectives to explore what works in KSTE, learn from NCCs and other organizations involved in KSTE, network, and identify opportunities related to KSTE.  This most recent Summer Institute represents the first to be open to the public with over 200 individuals participating.  

The NCC Collaborative’s common web-portal provides information on: the mission, vision, principles, and key activities of the Program; the NCCs; NCC collaborative projects; events; Summer Institute; KSTE; and links to national and provincial/territorial institutes and associations. 

The finalization of a communication strategy, including branding a common look, was identified in the interviews as a collaborative achievement.

Looking forward, one informant group cited the following examples as potential future NCC Collaborative activities:

  • Collective abstracts and joint presentations at conferences (e.g., CPHA, JASP);
  • Collective work plan across the NCCs;
  • Identification of one topic area across the NCCs;
  • Presentation of information about the NCCs and their value to universities with masters of public health programs; and
  • Initiation of additional collaborative projects, such as policy fact sheets.

NCC to NCC initiatives/activities

The NCCs have collaborated on several initiatives.  NCC to NCC initiatives represent another opportunity for NCCs to share information and best practices among the Centres.  Three NCCs identified all other NCCs as their partners and/or collaborators.  The other three NCCs identified four NCCs as their partners and/or collaborators. 

The following list of illustrative examples of NCC to NCC initiatives was compiled using the following data sources: NCCPH website; interview/focus groups; and completed NCC document review instruments.  Whenever possible, the participating NCCs are identified at the end of each example.

  • Conducting a year-long study focusing on drinking water including investigation of factors associated with water-related illness in Canada that is expected to result in evidence-based changes to practice and policy (NCCEH and NCCAH);
  • Developing a web-based table mapping the responsibilities for public health functions in Canada that can be accessed by public health policy-makers, frontline practitioners, researchers, and stakeholders (NCCHPP, as lead, and NCC Collaborative);
  • Understanding what counts as evidence, including ongoing dialogues (all six NCCs/Summer Institute);
  • Examining reviews of the effectiveness of community interventions in reducing symptoms of Attention Deficit and Hyperactive Disorder among children and youth to identify effective interventions that could be modified to be culturally sensitive for Native Canadians (NCCAH and NCCDH);
  • Examining the governance context for local environmental decision-making that affects public health (NCCEH and NCCHPP);
  • Establishing the First Nations Environmental Health Innovation Network to facilitate exchange of environmental health knowledge concerning First Nations communities across Canada and produce a series of state-of-the-knowledge papers on environmental health, addressing such issues as indoor air quality, food security, and housing (NCCAH and NCCEH);
  • Developed Health Impact Assessment (HIA) tool that is expected to contribute to the formulation and evaluation of public policy (NCCHPP and NCCDH);
  • Participation in the Summer Institute 2008 Steering Committee (NCCHPP, NCCDH, NCCAH, NCCEH, and NCCID); 
  • Presented at Rural Centre ’06 Meeting (NCCEH, NCCAH and NCCDH);
  • Co-led third Summer Institute (NCCAH and NCCEH);
  • Organized training course on conducting systematic reviews (NCCAH and NCCEH);
  • Staffed booth at Canadian Institute of Public Health Inspectors 2007 conference (NCCAH and NCCEH);
  • Organized roundtable for western provinces on promoting healthy public policy (NCCHPP and NCCEH);
  • Participated in knowledge exchange meeting on antimicrobial resistance (NCCID and NCCEH);
  • Developing tools for gathering evidence (NCCMT and NCCAH);
  • Partnering with an NGO to compile repositories for information (e.g., systematic reviews and synthesis of best evidence) (NCCEH and NCCMT); and
  • Facilitated early childhood development workshop in Saskatoon (NCCAH, NCCDH and NCCHPP).

Feedback on Collaboration

Collaboration with health portfolio

Feedback related to collaboration with the health portfolio was limited.  In general, only national and PHAC representatives spoke to this issue.  Little feedback was received at the NCC level.  However a few NCC informant groups indicated they did not understand the term ‘health portfolio’.  

PHAC and other NCCPH representatives at the national level identified two challenges or barriers in relation to NCC collaboration with the health portfolio:

  • Lack of connection between PHAC programs and the NCCs - PHAC is not engaged with the NCCs and PHAC influence over the NCCs has been limited due to the nature of Contribution Agreement funding. 
  • Lack of connection between the NCCPH program and the PHN - activities of the NCCs need to be matched to the work of the PHN.  PHN is not aware of the NCCs and the NCCs are not engaging the health portfolio.  For example, the NCC Leads’ presentation to the PHN was the first time the NCCs presented to this body.  In addition, the AC is not engaging the health portfolio (e.g., PHN) and it is not clear how they should engage.  PHN members do not feel the AC is connected to them. 
Feedback on NCC collaboration

In general, informant groups did not make a distinction between the NCC Leads Secretariat and NCC to NCC initiatives when providing feedback on NCC collaboration.  

When asked about the benefitsof NCC collaboration, the major themes were:

  • Increased awareness of other NCC projects;
  • Avoid duplication of efforts;
  • Exchange information and expertise;
  • Promote greater visibility of NCCs; and
  • NCC collaboration is reflective of the nature of public health.

Informant groups were asked what is working well in relation to NCC collaboration.  Representatives from three NCCs identified communications, including email exchange, the leads meetings, and that people are open, available, and respond quickly.  The Summer Institute was cited as a successful forum for collaboration, including engagement of a broader audience and as a place to share best practices.  One NCC Managing Director noted being in almost daily communication with counterparts in other NCCs.

Informant groups were asked to identify barriers or challenges to NCC collaboration.  More than half of the NCC informant groups cited amount of time and resources required.  Other major theme was diversity between the NCCs, including different mandates, topic areas, cultures, approaches of the SDs, models of organization, host organization restrictions around web-based information sharing and presentation), and stages of development.  Finally, the process of collaboration was noted to be slow (e.g., reaching consensus, discussion of minute details at leads meetings).

Suggestions to Strengthen Collaboration

The following recommendations were offered by informant groups regarding NCC collaboration with the health portfolio:

  • Involve more PHAC program representatives in the NCCPH Program to ensure they are up to date on NCCPH happenings in their content areas, (e.g., include PHAC content experts on  Advisory Boards of NCCs as relevant to their topic of expertise);
  • NCCs need to connect with related PHAC program areas;
  • NCCs need to become integrated with the PHN;
  • Increase awareness of the NCCPH among the health portfolio (e.g., monthly update in PHN newsletters; push website); and
  • Continue providing presentations to the PHN.

The following suggestions were offered by informants regarding NCC collaboration.  More than half of the NCCs informant groups recommended provision of adequate resources (human and infrastructure) for collaboration.  In relation to human resources, specific examples included hiring a communications officer, collaborative input into the Project Officer’s job description and evaluation of this role, and capacity building.  Two informant groups suggested resources and a common infrastructure to support the systematic exchange of information. 

Four informant groups recommended standardization across the six NCCs including developing and communicating a common vision, providing common KT perspectives and parameters, developing a common image for the NCCPH Program, and clarifying mandates.  Three informant groups recommended reducing the time needed to collaborate by assigning leadership for the collaborative, discussing what is crucial versus every detail, and prioritization.  Two informant groups recommended considering the role of NCCMT to support NCC collaboration (i.e., as an underpinning rather than parallel structure).

Immediate Outcomes: Knowledge Translation and Gap Identification

In the NCCPH logic model, the immediate outcome of knowledge translation is defined as the exchange, synthesis and application of scan and research findings disseminated among researchers and knowledge users.  Knowledge gaps, once identified, act as catalysts for applied or new research.  

Description of Knowledge Translation Products and Processes

As a component of the document review template, NCC staff were asked to list their site-specific KSTE products/services and to provide a description of the type of product/service for each item (e.g., fact sheet, workshop, video).  A total of 188 KSTE products were identified across the NCCs with a range of one to 66 products per NCC.

All KSTE products/services across the NCCs were compiled and grouped into five categories: knowledge products, networking/consultations; ongoing communications (e.g., website, newsletters), presentations/ conferences, and language translation activities.  The most common KSTE product developed across NCCs was knowledge products (143), followed by presentations/conferences (19).  A breakdown of product type by NCC is provided in Table 4.

Table 4. KSTE product type by NCC
*The product type ‘presentations/conferences’ includes workshops and sessions.
Product Type NCCs Total
NCCAH NCCDH NCCEH NCCHPP NCCID NCCMT
Knowledge products 64 1 39 10 19 10 143
Networking/consultation 1 0 2 8 0 1 12
Ongoing communications 0 0 0 4 0 5 9
Presentations/conferencesTable 1 - Footnote 1 * 1 0 3 13 0 2 19
Language translation activities 0 0 0 5 0 0 5
Totals 66 1 44 40 19 18 188

A detailed list of KSTE products/services for each NCC is provided in the summary of achievements and activities in Appendix F.  In this appendix, the type of product/service is self-defined by NCC staff.  A compiled list of products/services by NCC is provided in Appendix G.

Description of Knowledge Gap Identification Processes and Products

Managing Directors or NCC staff were asked to list and describe their Centre’s knowledge gap identification processes and products.  All six NCCs identified environmental scans and two NCCs listed consultations (e.g., key informant interviews or working group session).  The other types of processes/products identified by NCCs include: literature review; survey; paper; upcoming strategic plan; and networking at conferences.  A detailed list of knowledge gap identification products and process by NCC are listed in the summary of achievements and activities in Appendix F.

Little information was provided by NCC informant groups regarding the identification of research or knowledge gaps through their synthesis and translation activities; nor how these gaps are subsequently communicated with the research community.  However, the need to strengthen this aspect of the NCCPH work was highlighted at the November 2008 Strategic Planning Retreat.  Specifically, the need to engage researchers in a two-way dialogue to stimulate relevant research, based on identified gaps, was discussed and considered in the development of the strategic plan.

Feedback on Knowledge Translation and Gap Identification Processes

NCC representatives were asked about the processes used to achieve the immediate outcomes.  Feedback related specifically to KSTE processes is presented below.

The predominant themes were:

  • There was a lack of leadership/direction to direct this kind of work;
  • This kind of work requires processes and systems to be established (e.g., peer review and project management processes); and
  • NCCs are achieving individually but not at a collaborative level.

Informant groups offered the following suggestions for enhancement of the processes:

  • Determine the basic processes behind KT and what it includes (e.g., evidence reviews or translation of reviews into practice);
  • Articulate an explicit strategy, focus, and/or deliverables to achieve outcomes;
  • Identify and work towards defined public health goals (e.g., Aboriginal Health) across the NCCs;
  • Identify processes that all NCCs have in common;
  • Develop a mechanism to identify gaps in tools for KT (e.g., registry);
  • Develop a collaborative tool across NCCs to improve networking;
  • Develop consistent language and terms; and
  • Enhance NCCPH Program Secretariat role (e.g., translate knowledge products; assist in developing common language and terms, and determine KT processes).

Immediate Outcome: Networking

The fourth immediate outcome identified in the NCCPH logic model is networking, defined as increased collaboration of the NCCs among and across public health at all levels. 

Description of Networking Activities

Managing Directors or NCC staff were asked to describe mechanisms their Centre employs in relation to networking.  All six NCCs listed the following networking mechanisms:

  • Website;
  • Communications plan; and
  • Conferences/workshops/forums.

Three NCCs also identified as networking mechanisms the use of email or contact lists, networks (e.g., knowledge exchange networks, network of ‘multipliers’), and products (e.g., list of KSTE resources, evidence reviews, registry of methods and tools).  Additional networking mechanisms identified by individual NCCs include: learning site; collaborative portal project for the NCCs; Advisory Committee and working groups; and satellite for the Cochrane Health Promotion and Public Health Field.

As one component of the document review template, staff of the NCCs were asked to list their partners, collaborators, and/or key connections.  The total number of distinct partners, collaborators, and/or key connections identified across the NCCs is 270, ranging from 11 to 182 by NCC (Appendix H). 

The partners, collaborators, and/or key connections were compiled across the six NCCs and categorized by type and, for some, by location of organization or connection.  The most common types of partner, collaborator, and/or key connection were provincial or territorial organizations (101) followed by national or regional organizations (82).  The least common type of partner, collaborator, and/or key connection identified was private business (5).  The number of connections by type of partner, collaborator, and/or key connection across NCCs is provided in Table 5.

Table 5: Reported number and type of partners/collaborators/connections across NCCs
Type of organizations/connections represented # of organizations/ connections reported

International (total)

29

Based outside of Canada

21

Based in Canada

8

National or regional (e.g., Atlantic Provinces, Territories)

82

Provincial/territorial governments and organizations(total)

101

British Columbia

34

Quebec

15

Ontario

14

Alberta

7

Manitoba

6

Saskatchewan

6

New Brunswick

4

Northwest Territories

4

Newfoundland

3

Nova Scotia

3

Yukon

2

Nunavut

1

Prince Edward Island

1

Universities or colleges, includes Canada and the USA (total)

39

Ontario

16

Quebec

7

British Columbia

6

Alberta

3

Nova Scotia

2

Manitoba

1

Newfoundland

1

Saskatchewan

1

United States of America

2

National Collaborating Centres (including the NCCPH Secretariat and the Summer Institute 2008 Steering Committee

8

Private business (e.g., consultants)

5

Other, including local/municipal organizations or unknown types of organizations

6

TOTAL

270

NCCs were noted to define partners and collaborators somewhat differently.  For example, one NCC said they were uncomfortable using the terms “partners” or “collaborators” in case these partners and collaborators did not consider this NCC as a partner or collaborator in return.  They preferred to use the term “key connections” instead.  Based on this distinction, another NCC may have been more liberal in their definition of what constitutes a partner or collaborator.  For example, individuals of organizations who subscribe to the NCC electronic bulletin are listed as established target audiences. 

Feedback on Partnerships and Collaborations

The informant group representing interested and involved parties was asked what is working well in their partnerships or collaborations with any of the NCCs.  They identified:

  • A good working relationship with the NCC (e.g., with the SD);
  • Affiliation or partnership of the NCC with the host organization;
  • Positive feedback on products developed as a result of the partnership/collaboration;
  • Invitation of interested/involved party to Summer Institute offered opportunity to learn about other NCCs;
  • Funding received from NCC to develop KT products; and
  • NCC is connected with well-established organizations (e.g., SEARCH, Health Evidence.ca).

Interested and involved parties were also asked to identify current challenges or barriers to partnerships or collaborations.  Barriers or challenges identified included:

  • Confusion regarding mandates of NCCs (e.g., research, training, education, and/or advocacy);
  • Diversity of mandates across the NCCs;
  • Staff turnover at NCCs;
  • Limited NCC push of product to additional audiences;
  • Limited NCC networking (e.g., only one NCC has physically visited the Yukon). 

The following recommendations were offered regarding partnerships and collaborations:

  • Continue to nurture networks; and
  • Promote the NCCPH Program, including the NCCs, and what the program has to offer (e.g., in the form of newsletters or bulletins). 

Other individual suggestions included:

  • Provide clarification of mandate by offering guidelines around research/applied research;
  • Use the Summer Institute as a venue to further discuss this topic;
  • Provide leadership to ensure cross pollination;
  • Develop products that are bilateral, trilateral, and multiparty among the NCCs; and
  • Specifically partner/collaborate with: Health Sciences Librarians, Ontario Agency for Health Protection and Promotion, and Cochrane.

Intermediate Outcomes / Impact

Intermediate outcomes of the NCCPH Program are outlined in the program logic model and include:

  • Increased availability of knowledge for evidence-based decision making in public health;
  • Increased use of evidence to inform public health programs, policies, and practices;
  • Partnerships developed with external organizations; and
  • Mechanisms and processes to access knowledge.

The anticipated results of these intermediate outcomes are research/knowledge, capacity building, and inter-sectoral collaboration.

The NCCPH Program and the six NCCs are still in their infancy and therefore, the intermediate to long-term outcomes could not be assessed in this formative evaluation.  However, while the six NCCs are at different stages in development, all have produced KSTE products.  It was appropriate at this time to examine the precursors to impact (i.e., the degree to which the NCCs are set to meet the intermediate and long-term outcomes outlined in the logic model).  Precursors include awareness and perceived credibility of the NCCPH and NCCs, and awareness and perceived credibility of products.  The evaluation team conducted targeted interviews with those informant groups who were most likely to have used any of the NCCs products, including interested and involved parties, AB members, AC members, and national representatives. 

The reader is reminded that one important informant selection criterion was extent of involvement with the NCCPH; thus, the feedback received does not reflect the broader population of public health practitioners and decision makers.

Perceived Awareness and Credibility of the NCCPH Program and NCCs

Informant groups were asked how aware they thought public health communities across Canada are of the NCCPH Program overall or of individual NCCs.  Several informant groups said there is some awareness or “pocketed” awareness of the Program or the NCCs.  In relation to pocketed awareness, many informant groups said there is awareness at higher levels (e.g., federal government, Senate, senior managers, policy programmers, decision makers) and less at the provincial or regional level as well as at the grassroots or smaller organization level.  A few informant groups noted lack of awareness about the Program or the NCCs within the PHAC and the PHN.  

A few informant groups indicated there was room for improvement regarding awareness of the Program or the NCCs among public health communities; however, they recognized the Program is still in the early stages of development.

Recommendations to improve awareness of the NCCPH Program or the NCCs included:

  • Get products into the hands of the public health communities;
  • Create a centralized contact database;
  • Identify and capitalize on advertising opportunities with professional organizations and associations;
  • Highlight added value of products; and
  • Develop a process by which individuals can request information from the NCCs. 

Informants were asked to describe their perception of how acceptable and credible the NCCPH Program and NCCs are in public health communities.  The most prevalent theme was “they’re/we’re getting there” or it takes time to develop acceptance and credibility.  Respondents in several informant groups said the development of quality products will determine the acceptability and credibility of the Program and NCCs.  Generally, respondents thought that public health communities perceive the Program and the NCCs as credible, as demonstrated by people using and/or requesting products.  A theme in a few informant groups was that awareness among the public health communities is still too low for the Program and NCCs to be considered the place to go for information/products. 

Awareness of NCC Products

Informant groups were asked to what extent they perceived increased availability of knowledge for evidence based decision making in public health.  The vast majority of informant groups provided examples of knowledge products available to inform decision making, including:

  • NCCHPP health impact assessment;
  • NCC websites;
  • NCC presentations/events to target audience;
  • NCCAH papers on social determinants of indigenous peoples’ health;
  • NCCAH material on ethical space for researchers;
  • NCCAH environmental scan of Aboriginal Health in Canada;
  • NCCAH excel database on ‘who’s who’;
  • NCCEH materials related to Listeria;
  • NCCEH pamphlet on cleaning up mercury spills;
  • NCCEH paper on cleaning up methamphetamine labs;
  • NCCEH directory of legislation for all of Canada;
  • NCCHPP directory of policy options with respect to tobacco;
  • NCCID materials on best practice on HIV prevention and transmission of sexually transmitted disease;
  • NCCID materials related to HPV;
  • NCCMT quality filter for public health articles;
  • NCCMT public health plus product;
  • NCCMT health evidence.ca product;
  • NCC quarterly updates; and
  • Monographs are useful KT tools.

Respondents from over half of the informant groups indicated that increased availability of knowledge to inform decision making is a longer term goal and that it is too early to determine impact in this area.  Several informant groups said availability of knowledge is getting better or is growing.

Reported Use of NCC Products

Informant groups were asked if they have used any of the NCC products.  Specific examples of products mentioned include:

  • NCCHPP weekly emails;
  • NCCEH document on training for small water system operators;
  • NCCID analysis of HIV blood borne infection programs;
  • NCCAH fact sheets and reports;
  • NCCHPP website tool on smoking;
  • NCCMT compendium and applicability tool;
  • NCCHPP summaries of French-language initiatives translated into English;
  • NCCEH guides on cleaning up mercury spills and grow operations; and
  • NCCEH background documents and position papers.

Generic examples of products mentioned were:

  • Materials from the websites;
  • Monthly newsletter;
  • Research article;
  • Materials available at the Summer Institute; and
  • Workshops.

A few informants provided examples of how they use NCC products, specifically:

  • NCCEH products to inform policy; and
  • NCCHPP tobacco timelines as a teaching tool.

A number of informants stated they had not used any of the products and, of these, most said they had referred NCC products to others. 

A couple of informant groups said there is a need for greater awareness of the Program and the NCCs or that there is a need to develop capacity (e.g., skills to appraise and apply evidence to decision making). 


Page details

Date modified: