ARCHIVED - Formative Evaluation of the National Collaborating Centres for Public Health Program (NCCPH)

Chapter 3:  Summary of Findings

In this chapter, a summary of the evaluation findings is presented for each evaluation question (as per the data matrix in Appendix A), and organized under each of the four fundamental evaluation questions.  Summary statements are based on the descriptive information provided in available documents and on the themes that emerged from interviews and focus groups with key informants.  A detailed report of the evidence to support the summary statements and evaluation conclusions (Chapter 4) may be found in Appendix E.   

Design and Delivery

The overarching question of interest related to design and delivery is – to what extent is the design and delivery of the NCCPH Program appropriate?  Nine sub-questions were posed in the evaluation framework to contribute to answering this fundamental question.

How are the NCCPH Program design and delivery mechanisms contributing to the achievement of the program’s goals?

Based on the logic model and other NCCPH documentation, the evaluators determined that the NCCPH Program is defined by the following features that collectively represent a unique niche within Canada’s health system:

• Geographically dispersed Centres of Excellence across Canada;
• Located within the umbrella of a host organization that is research based or has strong linkages with a research centre;
• Representing independent legal entities operating at arm’s length from government;
• Funded and managed through the federal government’s Contribution Agreement mechanism;
• Mandated to provide KSTE across Canada;
• Focused on a specific, relevant, priority public health topic area; and
• Expected to function collaboratively.

No fundamental changes to these design elements are suggested by the evaluation findings.

The governance structure for the NCCPH is presented in Figure 2.  The six NCCs operate at arm’s length from PHAC, supported by Advisory Boards or Committees that provide guidance and support specific to their respective NCC’s topic area.  Each NCC is located within a host organization that has strong linkages to the research community.  Collective effort on the part of all six NCCs is undertaken through an NCC Collaborative which is supported by a Leads Secretariat. 

The NCCPH Program Secretariat resides within PHAC and works closely with the Advisory Council, whose members provide technical expertise to the NCCPH Program. 

Specific roles and responsibilities for PHAC, the Advisory Council, the NCCPH Program Secretariat and the NCCs are described in Appendix E.

Figure 2.  NCCPH governance structure

Notes: This figure is adapted from the February 2008 Progress Evaluation FrameworkFootnote 19.  The term “Advisory Board” is used in this diagram; however, the term is used variably across the NCCs.  For example, the terms “National Advisory Committee”, “Advisory Committee” and “Advisory Board” are used. 

Evaluation findings suggested the following aspects of design and delivery as the most critical to the success of the NCCPH:

• The Advisory Council was seen as instrumental in guiding the vision and design of this Program and for transforming a relatively vague and untested concept into reality.  The members’ role in providing technical guidance during implementation has been critical.
• The NCCPH Program Secretariat has successfully fulfilled its role in program management as evidenced by achievement of planned activities and outputs.  This has occurred despite human resource challenges.
• For the most part, NCC affiliation with host organizations has resulted in benefits in both directions, including knowledge and capacity building, networking opportunities, enhanced credibility, and recruitment of qualified staff. 

What are the challenges, constraints and/or systemic barriers in the implementation of the NCCPH Program? 

An overview of the key implementation activities and milestones at the government and NCC levels is provided in Figure 3.  In this figure, the primary timeline shows the main activities and milestones in relation to the NCCPH Program.  Below this main timeline are six timelines, one for each NCC, in order of when the Contribution Agreement was signed.  The symbols on the individual NCC timelines indicate the major activities and achievements for each, including Contribution Agreements, work plans, progress reports, and key planning documents such as environmental scans, needs assessments, and/or strategic/evaluation plans.

Figure 3.  NCCPH Implementation Milestones

Evaluation findings identified a number of themes related to design and implementation challenges, constraints and barriers:

• The unique concept represented by the NCCPH – theNCC concept was breaking new ground and presented challenges in terms of developing a common understanding of knowledge translation, clarifying the mandate, finding and keeping people with knowledge translation expertise and reconciling differences in NCC styles and approaches.  There are lingering questions related to the NCCPH mandate, including suggestions on the part of some that the mandate be expanded to include knowledge generation, capacity building in the field and advocacy for specific evidence-based policy. 
• Government of Canada and PHAC’s accountability structures and processes – some perceived government requirements to present as barriers to what they would have considered the normal way of doing business.
• Heterogeneity across NCCs – differences among NCCs and host organizations were noted to present as challenges, and resulted in variations in approaches used and ability to progress with the mandate.  Informants discussed the different stages of individual NCC development and the impact this has on the ability to demonstrate overall success.  Other issues included a lack of common KSTE strategy and tools, and a tendency of NCCs to operate in silos rather than as a collective. 
• Staff recruitment and retention – difficulties in finding staff with the right expertise in both KSTE and public health were noted, particularly in the early stages of implementation and with finding and keeping Scientific Directors.  Staff turnover within the NCCPH Program Secretariat was identified as an ongoing issue needing to be addressed.
• Funding cutbacks – NCCs are acutely aware of cutbacks experienced in their annual funding over the last two years.  Three year funding cycles present as challenges to long term planning, security and staff recruitment.
• Host organizations’ role in the accountability structure – issues were raised regarding the dual reporting relationship of NCCs to PHAC and the host organization, the struggle to balance the mandates between the NCCPH and the host organization, differences among the NCCs related to the type of host, and a disconnection between PHAC and the host organization.

Are different NCCs facing different challenges? How are these challenges being addressed?

Variation is evident across NCCs and some of these variations presented as challenges:

• NCCs are at different stages of development, attributed in large part to delays resulting from decision processes related to their siting.
• They are located in sites with varying (pre-existing) levels of capacity and support.
• Variations in the type of host organization may be related to different emphases and approaches, i.e., more or less research versus practice oriented.
• Their topics range widely from broad, overarching topics to those more readily contained in  terms of scope and target audience.
• They operate in different contextual environments.
• The NCCs have varied ability to attract and retain qualified staff, with NCCs located in rural areas appearing to experience greater difficulty.

The NCCPH Program Secretariat and National Advisory Council have played a considerable role in ensuring the evolution of the NCCs remains true to the intended mandate.  In addition, many challenges experienced by the NCCs are addressed through the NCC collaborative efforts.  An open and trusting relationship among NCCs was evident, and there appears to be mutual support, mentorship and discourse at the NCC Leads level which serves, to some extent, the achievement of a level of program cohesion.

What is working well in the NCCPH Program?

Informants identified the following aspects of the NCCPH Program to be working well: 

• Collaboration among NCCs – informants perceived this as a strength and provided numerous examples of collaborative activities including networking, information and resource sharing, identifying and collaborating on joint projects, mentorship across NCCs, cross-assignment of NCC representatives on Advisory Boards, and the ability to leverage from each other.  The Summer Institute was frequently mentioned as a particular success story of NCC collaboration.
• Advisory Council composition, role and approach – specifically in terms of the composition of the Council, members’ characteristics and credibility, their role in providing technical feedback to NCCs, and their approach and style.
• NCCPH Program Secretariat – is perceived to provide leadership and commitment, and offer support to the NCCs; and Program Secretariat staff is described as being trustworthy, open and transparent.
• The NCCPH mandate is perceived to be unique, appropriate and necessary.  There is a sense of an emerging KSTE capacity in staff and infrastructure. 
• Affiliation/relationships of specific NCCs with their host organization – includes the ability to leverage infrastructure through this affiliation.

Does the NCCPH Program have appropriate performance measurement and reporting strategies?

The NCCPH Program Secretariat has developed and implemented a performance measurement plan consistent with Government of Canada requirements, including but not limited to:

• An annual work plan and semi-annual progress report review process;
• Risk assessment process and follow up action when risk is identified; and
• Commissioning of the current formative evaluation.

The accountability and reporting mechanisms are described in detail in Appendix E.

The role of the Advisory Council in providing technical guidance to the NCCPH Program, including feedback on NCC work plans, was generally perceived to be working well.

NCCs, in general, perceived the reporting requirements as excessively burdensome.  They recommended streamlining the reporting process and possibly reducing the frequency to annual reporting, once NCCs are well established.  They also recommended a shift in focus from activity based to results based reporting.

Are NCCs reporting on planned results? 

The NCCs have submitted annual work plans at the start of each calendar year, and completed semi-annual progress/narrative reports on their activities in relation to the work plans.  In addition, NCCs are required to provide updates on projected spending each January.   

The Contribution Agreement specifies the requirement of NCCs to conduct an annual evaluation and an audit each May; however, there seems to be lack of clarity as to whether or not this is truly a requirement. All NCCs have undertaken some evaluation activity, ranging from engagement of external evaluators for development and implementation of NCC logic models and evaluation plans, to targeted reviews of specific products. 

To what extent are NCCs meeting the requirements of the Contribution Agreements?

Some NCCs struggled to meet the requirements and expectations of the Contribution Agreement during the start up phase.  The NCCPH Program Secretariat, with the technical guidance and support of the Advisory Council, addressed these issues.  Currently, the NCCs are meeting the requirements of their Contribution Agreements.

Are the NCCs collecting data on immediate outcomes? 

The four immediate outcomes are: collaboration, knowledge translation, knowledge gap identification and networking.  The NCCs appear to be at different stages of development in their ability to systematically document their activities and achievements with respect to each outcome.  Some NCCs have established evaluation frameworks and processes; thus, are further ahead than others in collecting data on immediate outcomes.  The evaluators were able to obtain information to report progress on immediate outcomes for this formative evaluation.  Based on this and the information generally provided in their progress reports, NCCs appear to be collecting data, formally or informally, on immediate outcomes. 

Are options/alternatives to improve program design/delivery being considered or recommended?

NCCs have established mechanisms for sharing information among themselves on service design/delivery options that they have used.  Examples include NCC Leads meetings, Summer Institute and informal networking.

At the NCCPH Program level, strategic planning initiatives have been undertaken, resulting in a general consensus among the NCCPH Program and NCCs on strategic directions and priorities. 

Achievements and Successes

The fundamental evaluation question is – what is the status of the NCCPH Program’s progress toward achieving its immediate goals and objectives?  A summary of the evaluation findings is presented under each of nine sub questions.

In what manner and to what extent have the NCCPH Program’s planned activities been implemented?

Those individuals involved in the design and early implementation stages of NCCPH development reported that the NCCPH Program was implemented according to what was originally envisioned. 

The NCCPH Program Secretariat established and implemented processes as proposed in the original RMAF document.  The noted exception was the lack of development of “Program materials for NCCs, which may include materials for public consumption such as newsletters, web site content, media kits, etc.”  Minimal activity in this area may be attributable to enhanced understanding, over time, of the limitations imposed by the arm’s length requirements for Contribution Agreements. 

Are the planned activities producing the expected (a) management level outputs; and (b) program (NCC) level outputs?

The NCCPH Program Secretariat implemented the planned activities and has produced the expected management level outputs, under recruitment and retention constraints.  Highlights of achievements were:

• Established an Advisory Council (AC);
• Established NCCs;
• Organized meetings involving joint forums between the AC and NCC, supporting organization of Summer Institutes/associated meetings, and organizing NCCPH Program level forums;
• Reviewed and approved NCC deliverables, including work plans, progress reports, and projected spending reports;
• Conducted risk assessments for the NCCPH Program and annually for each NCC;
• Conducted a formal financial review of NCCs (provided under contract to PHAC);
• Conducted NCC site visits, as required; and
• Developed an evaluation framework, commissioned a formative evaluation, and have started planning for a summative evaluation.

The NCCs have implemented the planned activities and are producing the expected outputs.  All six NCCs submitted a proposal, entered into a Contribution Agreement by December 2006, established their Advisory Boards, and have demonstrated increased consistency in submitting annual work plans and semi-annual progress reports.   Details of each of the NCC’s activities and achievements may be found in Appendix F.

To what extent is the NCCPH Program on target toward achieving its immediate outcomes related to increased opportunity for collaboration?

Findings from the formative evaluation revealed substantial evidence of collaboration among NCCs.  Achievements of the NCC Leads as a Collaborative included:

• Conducting regularmeetings;
• Establishing the NCC Leads Secretariat to support the activities of the NCC Collaborative;
• Organizing and holding three Summer Institutes;
• Developing a common web-portal; and
• Finalizing a communication strategy.

Seventeen examples of NCC to NCC initiatives were identified through the evaluation.  Most NCCs list all other NCCs as partners or collaborators.

Key issues identified as barriers to NCC collaboration included the time and resources required; NCC diversity (e.g., mandates, topic areas, stages of development); and a perception that NCC Leads’ collaborative decision making process is too slow.  Suggestions offered by informants to promote NCC collaboration included strengthening the infrastructure for collaboration with the provision of adequate resources.   

Some evidence of NCC collaboration with the health portfolio was identified, although this was limited in comparison with the collaboration evident among NCCs.  Examples of collaborative effort with the health portfolio included:

• Collaborative presentation to the Public Health Network (September 2008);
• Survey of Chief Medical Officers of Health;
• PHAC/NCC joint initiative co-founded a National HIV Prevention Forum (Ottawa, April 2007);
• PHAC/NCC joint initiative in the development of the Canadian Best Practices Portal for Health Promotion and Chronic Disease Prevention tool; and
• PHAC/NCC joint initiative related to HPV vaccination.

Suggestions offered by informants to increase connectivity between the NCCs and the health portfolio included having PHAC program representation on relevant NCC Advisory Boards (i.e., as relevant to content area), greater alignment of work between the Public Health Network (PHN) and the NCCs, and increased promotion/awareness of the NCCPH Program to the health portfolio (e.g., contributions to the PHN newsletter, enhancing the collective website).

To what extent is the NCCPH Program on target toward achieving its immediate outcome related to knowledge translation?

The NCCPH Program is on target to achieving this immediate outcome, as evidenced by the number of knowledge synthesis and knowledge translation documents and activities reported.  A total of 188 products have been developed across the NCCs including the following as reported by the NCCs:

• 143 knowledge products;
• 19 presentations/conferences;
• 12 networking/consultation sessions;
• 9 ongoing communications activities; and
• Language translation activities.

A list of all knowledge translation products, as reported by the NCCs, is presented in Appendix G. 

To what extent is the NCCPH Program on target toward achieving its immediate outcome related to identifying knowledge gaps?

NCCs have utilized varying methods to determine what front line practitioners and policy makers perceive as their knowledge gaps (i.e., to inform topic areas for NCC focus), with some being challenged by the breadth of their target audience.  All six NCCs identified environmental scans and two NCCs listed consultations as methods for identifying gaps.  A detailed list of knowledge gap identification products and processes are listed, by NCC, in Appendix F.

Little information was provided by NCC informant groups regarding the identification of research or knowledge gaps through their synthesis and translation activities; nor how these gaps are subsequently communicated with the research community.  However, the need to strengthen this aspect of the NCCPH work was highlighted at the November 2008 Strategic Planning Retreat. 

How are those knowledge gaps being addressed?  Are lessons learned being identified and applied to ongoing activities?

The NCCs are at various stages in formalizing a solid KSTE process.  Some NCC staff described general processes for incorporating learnings into ongoing activities within a quality assurance or quality improvement framework.

The majority of NCCs expressed a need for more direction and systems to support their KSTE processes, with some noting that they are achieving results individually but not collectively in this regard.  They suggested a need for more clarity of KSTE processes and enhanced collaborative effort towards KSTE strategies, processes and tools. 

To what extent have important partnerships and linkages been established?

Most NCCs have been successful in establishing partnerships and linkages as evidenced by a reported total of 270 partners, collaborators and key connections.  These partners and collaborators span organizations operating at the local, provincial, national and international level, and include academic, policy and practice settings, and professional organizations.

A list of partners, collaborators and key connections, as reported by the NCCs, is presented in Appendix H.

Are best practices being identified and shared among the NCCs?

NCCs reported sharing best practices as part of their regular activities.  NCCs have developed and use formal and informal mechanisms for information sharing.  Formal mechanisms include their regular Leads meetings, the Summer Institute, common web portal, and the initiation of NCC to NCC collaborative initiatives/projects.  Informal mechanisms of information sharing among the NCCs include regular (e.g., daily, weekly, monthly) communications via electronic mail or telephone.

Determination of the proportion of information shared that was specific to “best practices” was beyond the scope of this evaluation, as was any determination of the quality of the best practice information shared. 

Is the program on target toward achieving its (a) intermediate outcomes; and (b) longer-term outcomes?

The NCCPH Program appears to be on target towards achieving the stated intermediate and longer-term outcomes, although it is too early to provide evidence of impact.

There are promising indications of the “precursors” to achieving impact:

• There is a perception of growing awareness of the NCCPH Program or of individual NCCs among Canada’s public health community, although this is reported to be true more for national organizations and at higher levels of management than for front line practitioners.
• Substantial exposure to the Program appears to have been achieved, based on the number of individuals who have been engaged directly in the NCCPH Program and number of organizations reported in the NCC’s network of partners, collaborators and connections.
• Interested and involved parties interviewed were generally positive about products they had seen.  Many of those who were aware of the products indicated they had referred, recommended and distributed the documents to others.  A few provided examples of the use of the products for teaching purposes or for influencing/adopting policy.

Resource Adequacy

The fundamental evaluation question is – to what extent is the right level of resources available to effectively deliver the NCCPH Program? Two sub-questions were posed at the outset of the evaluation.

Are the allocated resources (i.e., human, financial) sufficient and/or available to achieve the NCCPH Program’s intended immediate outcomes given the planned reach?

Staff recruitment issues represented a predominant theme, particularly in the early stages of implementation and specifically in relation to finding appropriate Scientific Directors for some topic areas.  Recruitment issues were attributed to factors such as difficulty finding staff with the necessary content and KSTE expertise, location of the NCC, funding uncertainties, and inability to attract front line workers because of term positions.  This situation appears to have improved, as most NCCs reported an appropriate mix of people to be available at the NCC level at the present time.

According to a financial review in which cumulative budgets (projected expenses) were compared with actual expenses from 2005/06 to 2008/09, five out of six NCCs were noted to have under spent their available dollars, and one NCC was reported to have exceeded their budget.  Some representatives indicated that, despite initial under spending during the start up years, they anticipate the need for additional funding to accomplish the activities they envision in the future.  Other NCCs noted they have adjusted their activities to align with the available dollars and did not express a need for more funding.  Only one NCC expressed an immediate need for additional dollars.  

Based on Government of Canada spending reductions, an amendment was made to each NCC’s Contribution Agreement in each of the last two years, resulting in a reduction in funding of $110,000 per NCC.  There was a commonly voiced concern with these cutbacks and a related issue, the sustainability of government funding over the long term, was raised.  

Three out of six NCCs have received additional dollars to support their activities from provincial governments, the federal government (e.g., FNIH, PHAC), and/or provincial, national, or international organizations.

How could the efficiency and effectiveness of the NCCPH Program be improved?

The evaluation findings suggested several options, most notably:

• There is room to review and reconcile the organizing framework for the NCC topics (i.e., number and topic area) with a view to increasing efficiencies and effectiveness;  
• NCCs expressed concerns about the amount of individual NCC resources (time and dollars) spent on activities that could be accomplished more effectively and efficiently by a central body.  One example is the organization of the Summer Institute.  These activities at the collective level, while deemed important, are perceived to detract from what NCCs believe are their core activities; and 
• Many NCCs appeared to be working independently on designing their KSTE processes (although they may share information across NCCs).  There is an opportunity to develop a single cohesive KSTE framework and associated processes.   It was suggested that this could be assigned to the NCCMT.

Relevance

This final section addresses the question – to what extent does the NCCPH Program continue to be consistent with federal government and PHAC priorities?  A summary of the findings is presented under each of seven sub questions.

To what extent does the NCCPH Program continue to be consistent with (a) Government of Canada and (b) PHAC priorities?

The need for KSTE support in public health is documented in the literature.  The NCCPH Program continues to be relevant and is aligned with Government of Canada, PHAC and public health priorities. 

Is the NCCPH Program still consistent with the program’s original logic model?

The general program logic remains consistent with the concepts included in the original logic model; however, NCC representatives and the evaluators identified a number of issues with the original logic model mostly in terms of: organization, logic, understanding/definition, and content/wording.  Some aspects of the logic model represent implementation rather than ongoing activities.  It may be timely to revisit the model in view of the current status of the NCCPH Program and NCCs.

Highlights of suggestions suggested by informants were to:

• Distinguish the NCCPH Program (and Program Secretariat level) logic model from the NCC logic model (i.e., in the current model, the activities and outputs relate primarily to PHAC activities whereas the outcomes relate to NCC activities);
• Include the NCC Collaborative in the logic model; and
• Generally strengthening the logic and definitions (e.g., is collaboration an activity or an outcome? Is there too much of a jump in logic between outputs and immediate outcomes?).

Is the NCCPH Program meeting the broader public health goals?

It is too early for long-term outcomes to be demonstrated; however, the NCCPH is well positioned to support public health practitioners and policy makers in applying the best available evidence towards achieving the broader public health goals.

Is this program model still relevant?

The program model is still relevant, based on the following considerations:

• There is an increasing emphasis on KSTE in health care generally, and there is no other agency or organization currently fulfilling this need in the area of public health; and
• Informants reinforced the need for such a model in their interviews and focus groups.  There was no suggestion that the overall model was inappropriate, although some refinements were proposed.

Are the public health priorities still the same?

Based on informant interviews and available PHAC documents, there is no indication that public health priorities have changed since the establishment of the NCCs.

Should there be other National Collaborating Centres (NCCs)? 

Generally, informants did not perceive that adding NCCs was warranted at this time.  They suggested that:

• There is a need to review and reconcile areas of focus among the existing NCCs within a more logical organizing framework;
• PHAC should allow the existing NCCs to become fully stable and demonstrate success before considering expansion; and
• Any future expansions should be carefully planned with respect to topic area and location.

What changes, if any, would make the NCCPH Program more relevant?

No major changes were suggested by informants to increase the relevancy of the NCCPH Program.  There was evidence of interest in strengthening the NCCPH ability and capacity to identify and address common priority areas – to strengthen collaborative effort toward common priorities.  For example, focusing efforts on northern residents was one of a number of suggested topic areas to which all NCCs could relate.