Formative Evaluation of the National Collaborating Centres for Public Health Program (NCCPH)

[Previous] [Table of Contents] [Next]

Chapter 4:  Conclusions and Recommendations

In this chapter, the evaluators’ conclusions are presented under each of the four overarching evaluation questions.  Many observations and suggestions were made, as reported in the previous chapter and in Appendix E.  The recommendations contained in this chapter are those the evaluators believe are the most critical to position the NCCPH Program for continued success into the future.   

Issue 1:  To what extent is the design and delivery of the NCCPH Program appropriate?

Conclusions

The fundamental design of the NCCPH model is appropriate and no major changes are needed to the following defining NCCPH design elements:

• Geographically dispersed Centres of Excellence across Canada;
• Located within the umbrella of a host organization that is research based or has a strong linkage with a research centre;
• Representing independent legal entities operating at arm’s length from government;
• Funded and managed through the federal government’s Contribution Agreement mechanism;
• Mandated to provide KSTE across Canada;
• Focused on a specific, relevant, priority public health topic area; and
• Expected to function collaboratively.

Governance and accountability structures have been established and implemented, and proven effective in identifying and addressing risks with specific NCCs.  In particular, the technical advice and guidance of the National Advisory Council has been critical to the success of the NCCPH Program through the design and implementation phases, and to maintaining the integrity of the model.  As well, the NCCPH Program Secretariat, despite human resource challenges, was effective in establishing and implementing the administrative processes to support the governance structures and accountability requirements.  Strong leadership on both parts was essential as the model as a whole was without precedent in Canada. 

Placing NCCs in host organizations that are well positioned to bridge the gap between the research and policy/practice communities is another critically important design consideration.  The extent to which NCCs are placed in an environment where KSTE is understood appears to have had a positive impact on the implementation of the mandate.

The KSTE mandate is appropriate, necessary and fulfills a unique niche within Canada’s public health context.  The emerging KSTE capacity in NCCs, made possible through dedicated and nationally distributed funding, is a promising indicator of the potential for significant impact. 

The Contribution Agreement mechanism has proven to be challenging in the implementation of the NCCPH Program.  However, key players appear to have adjusted to its constraints and made this mechanism work.  No other suitable mechanism is evident.  The arm’s length relationship from government is viewed by all to be appropriate.

Despite operating as independent entities, the NCCs appear committed to collaborative effort and report this as one of the Program’s greatest successes.  A collegial, open and respectful relationship is evident across NCCs and, in general, among the NCCs, NCCPH Program Secretariat and Advisory Council.  Transparency and information sharing is also evident; NCCs appear to support each other in their activities as much as they receive support from the NCCPH Program Secretariat and Advisory Council.  They are justifiably proud of their greatest collective achievement – the successful organization and delivery of three Summer Institutes, where there has been increasing involvement by public health representatives from across the country.

Looking forward, a number of challenges and vulnerabilities related to the overall design and delivery need to be addressed.  These relate to the overall organizing framework, mandate, centralized function, role of host organizations, and long-term sustainability.

• Organizing framework

  While findings did not suggest the need for changes to the location of the existing six centres, the organizing framework for the NCC topic areas is questioned and should be revisited.  In particular, some of the content areas overlap, with NCCs topics representing a mix of determinants, population, methods, and issues.  These differences have resulted in different approaches and abilities to define target audiences.  In addition, some informants perceived an implied rigidity in these topics and expressed a desire for more fluidity to identify and address priority public health issues across NCCs.  There was a sense that it is possible to reconcile these issues within the existing Program structure and, possibly, within the existing budget.

• Mandate

  While the public health KSTE mandate was endorsed as being important and necessary, there continues to be some confusion about the KSTE concept and how it should be implemented, especially concerning the NCC role vis-à-vis knowledge generation and how far one reaches into policy and practice – i.e., defining the arrows in Figure 4 that represent the functions that intersect with academia, policy makers and practitioners (Figure 4). 

Figure 4.  The KSTE mandate within the broader research to practice context

The confusion represents issues of understanding, interpretation and questions about the appropriate scope of NCC activity.  While the NCCPH Program Secretariat, Advisory Council and NCC Scientific Directors have a clear understanding of their KSTE mandate, there is evidence that different frameworks, terminology and definitions are applied across Centres. The broader community of interested and involved parties is still attempting to gain clarity about the role of the NCCs and the KSTE mandate. 

In addition, questions are emerging among informants with greater KSTE experience regarding the appropriate future role of the NCCs in addressing identified gaps in the broader system represented by the three mandates identified in Figure 4.  For example,

• What should NCCs do when there is no research evidence available on a topic area raised as a priority knowledge gap by practitioners – how can one translate and exchange information on knowledge that does not exist?
• Who is building KSTE capacity in public health, including preparing public health professionals with sufficient KSTE knowledge and skills (who may potentially be employed by NCCs), and ensuring decision makers and practitioners have sufficient knowledge and capacity to incorporate evidence into policy and practice?
• What is the NCC role in advocacy for evidence-based policy?
• Given that NCCs are encouraged to obtain direction from practitioners in the public health field, is there a potential for NCCs to offer much needed professional support in addition to KSTE?

• Centralized NCCPH Program function

  It appears that, at the time the NCCPH Program was established, there may have been an expectation for the NCCPH Program Secretariat to provide more central direction and coordination of NCC activity than was possible through the arm’s length arrangement allowed under a Contribution Agreement funding model.  This is evidenced by the roles and responsibilities outlined for the Secretariat in some program documents, for example, the development of NCC promotional materials, including a program website.  The NCCPH Program Secretariat made it clear they could not directly support NCC activities, and as a result, the NCC Leads Secretariat was established to fulfill this function. The issue for NCCs, clearly expressed in this evaluation, is that while these collaborative efforts are critically important and should be strengthened, there are insufficient resources available for this collaborative effort.  Coordination activities detract from their core NCC functions and draw heavily on limited resources.

  A related issue is a desire for a common vision and strategy, increased presence for the NCCPH Program as a collective voice at the national level, and strengthened linkages with common partners/collaborators, such as with the Public Health Network and with national research, policy and practice organizations.

  In addition, a number of related vulnerabilities are apparent.  While the Advisory Council and the NCCPH Program Secretariat have provided the necessary technical advice, support and leadership to guide this Program in its implementation phase, there are concerns about the sustainability of this effort over the long term.  The Chair of the Advisory Council has been successful in accessing highly credible and well respected advisors to serve on the Council; however, the ability to sustain their interest and involvement over the long term is questioned, especially once all NCCs are fully implemented and stable.  Additionally, the NCCPH Program Secretariat has experienced staff recruitment and retention issues resulting in under staffing, and there is no indication that this will change in the future. 

  
  Top of Page

• Host organizations

  While not an issue for all NCCs, a few host organizations stated discomfort with their limited role in the accountability relationship with PHAC.  While they are pleased and proud to host the NCCs within their organizations, they feel overlooked in the NCCPH accountability structure and processes.  From the hosts’ perspective, the accountability relationship, as per the signatures on the Contribution Agreement, is between the host organization and PHAC, not the NCC and PHAC.  Those voicing concern perceived their involvement to be limited to rubber stamping the budget which does not sit well with them.  Some informants from host organizations stated they have more capacity to support and advance the NCCPH agenda than they are currently drawn upon to provide.

• Long term sustainability

  Finally, NCCs are acutely aware of the budget cutbacks they have experienced.  Over the long term, if funding from PHAC does not keep pace with cost increases, the NCCs foresee a curtailment of activities and limitations to achieving their potential.  They are also unclear whether actively seeking alternate funding sources is endorsed or discouraged.

Recommendations

It is recommended that:

1. PHAC ensures the NCCPH Program governance structures remain sound and effective. Specifically:
   • 1.1 The Advisory Council and NCC Advisory Board structure be maintained.
   • 1.2 The NCCPH Program Secretariat and Advisory Council review the role of the Advisory Council in view of the support needed by NCCs as they move beyond the implementation phase into stable organizations.
   • 1.3 PHAC Senior Management investigates and addresses underlying NCCPH Program Secretariat recruitment and retention issues.
2. PHAC, in collaboration with the Advisory Council and NCCs, review the existing NCCPH mandate, specifically to:
   • 2.1 Review and clarify the parameters for NCC involvement in knowledge generation, capacity building, advocacy and professional supports. 
   • 2.2 Review the organizing framework for the NCC areas of focus with a view to enhancing the logic, reconciling redundancies, clarifying target audiences, and increasing the flexibility to address new and emerging priority areas.
   • 2.3 Revisit NCC funding allocations in view of the decisions arising from the reviews proposed in 2.1 and 2.2 above.
3. PHAC, in consultation with the Advisory Council and NCCs, establish an independent mechanism to strengthen leadership, support and coordination.
   • 3.1 The mechanism be designed to strengthen the following essential functions:
     • Strategic direction and leadership for the NCCPH Program, including enhanced presence and linkages with common partners, collaborators, and interested parties;
     • Technical KSTE support to NCCs, including further development of a common KSTE framework, enhancing collaborative effort towards KSTE strategies, processes and tools, reducing duplication of effort, and continuing to build KSTE knowledge and capacity in NCC staff; and

     • NCC coordination activities, including but not limited to the existing Leads Secretariat functions, hosting and maintaining the common web portal, organizing the Summer Institute and other common forums, and further developing centralized systems that support all NCCs.

       While the last of the three functions may be accommodated through assignment to one or more of the existing NCCs (as per the Leads Secretariat), the nature of the first two functions suggest a distinct overarching rather than peer-level structure and mandate. The potential for accomplishing this through an additional centre should be explored.

     • 3.2 PHAC provide sufficient additional resources for the implementation of Recommendation  
       • 3.1.  Existing NCC and NCCPH Program Secretariat budgets would not cover the range of functions and activities envisioned
4. The NCCPH Program Secretariat meets with host organization representative(s) to determine and resolve any concerns identified.
5. The NCCPH Program Secretariat, in consultation with the Advisory Council and NCCs, determines a long term strategy for sustainability of the NCCs.  This strategy should include considering  the appropriate approach to and parameters associated with accessing funding from other sources.

Issue 2:  What is the status of the NCCPH Program’s progress toward achieving its immediate goals and objectives?

Conclusions

Given the short, four-year history of this initiative, a tremendous amount has been achieved in taking a relatively vague and untried concept through design, implementation and output phases.  The NCCPH Program Secretariat, with technical and guidance of the Advisory Council, has achieved its planned activities and outputs, including establishment of six NCCs, and developing and managing the associated Contribution Agreements. 

Accountability structures and processes are in place and NCCs are reporting to the Program Secretariat on their planned activities.  NCCs perceive the accountability requirements and reporting processes to be burdensome and inappropriate, given the nature of the host organizations and qualifications of NCC Scientific Directors.  

The NCCs are at different stages of development and progress toward achieving their immediate outcomes.  The reasons for the delay and slower progress for some centres stem from initial decisions made related to the siting of the centres, and not from lack of action on the part of current personnel in the NCCPH Program Secretariat or NCCs.  There is a general perception among NCC representatives that, despite initial hiccups, the NCCs are now on track toward achieving their objectives. 

There is clear evidence of extensive collaboration among NCCs, and interaction between the NCCPH Program Secretariat and the NCCs.  There is less evidence of collaboration with the health portfolio (as was expected based on the first immediate outcome in the logic model), and there appears to be confusion and differing expectations as to the appropriate relationship between NCCs and PHAC’s programs and services.   PHAC representatives interviewed expressed a desire for greater understanding of and involvement with the NCCPH as they see a need to strengthen KT internally and do not wish to duplicate efforts with the NCCs.  A few PHAC programs have received services from NCCs under specific contract arrangements.  Generally, however, NCC Leads and staff do not appear to perceive PHAC programs and service providers as target audiences, partners or collaborators, and demonstrate little interest in including federal government personnel in the NCCPH, other than respecting the relationship with the PHAC Secretariat.  While the need for an arm’s length relationship between PHAC and the NCCs is understood with respect to accountability, one questions whether, due to lack of any linkages with the NCCs, PHAC will establish parallel internal public health KSTE capacities and resources for their own program staff, ultimately leading to duplication of effort.

Some, but not all NCCs, have clearly demonstrated achievement in knowledge translation and gap identification, dependent largely on their stage of implementation and trajectory on the KSTE learning curve.  There is room to strengthen further development, understanding, and consistent application of the KSTE framework, possibly informed by approaches and models used by other KSTE organizations, such as the NICE and the Canadian Agency for Drugs and Technologies in Health (CADTH).  For example, over time health technology assessment agencies and others have evolved clearly distinguishable processes and sub-processes that span the spectrum of activities within KSTE, from horizon scanning (needs identification/gap identification), to topic selection and prioritization, knowledge synthesis, appraisal, knowledge translation, guidance/ recommendation, knowledge exchange (engagement/ mobilization), and including a feedback loop to researchers.  The last component can include mechanisms for stimulating appropriate primary research in areas where there are identified gaps. 

Collectively, NCCs have demonstrated an impressive connection with the broader public health community at provincial, national and, even, international levels.  This suggests the potential for broad-based influence and impact, once more NCC products and resources become available.

Although it is too early to expect achievement of the NCCPH’s intermediate and long term outcomes, in general, the NCCs appear to be on target to achieving these outcomes.  There are early signs of awareness and impact of NCC products on the part of interested and involved parties interviewed. 

Recommendations

It is recommended that:

6. PHAC review the NCC reporting requirements and processes with a view to reducing the paper work burden.

• 6.1 The amount of reporting required of NCCs should be in proportion to assessed risk.
• 6.2 For well established NCCs with low assessed risk, reporting should be reduced to once annually and be results rather than activity based.

7. PHAC clarify and communicate to PHAC programs and NCCs the expectations of and appropriate mechanisms for the NCCs to collaborate and link with, and provide KSTE products and services to PHAC programs and services.

Issue 3:  To what extent is the right level of resources (human and financial) available to effectively deliver the program?

Conclusions

The NCCs and the NCCPH Program Secretariat have experienced human resource challenges related to finding and attracting suitably qualified personnel, although most NCCs indicate this is improving.  The underlying issue is the availability of sufficiently educated public health professionals with education and/or experience in KSTE.  The issue is not helped by short term funding cycles and funding cutbacks.

There is little evidence to support an increase to the original $9.15M annual base funding envelope for the existing six NCCs at the present time; however, it may be appropriate to revisit this issue as part of the summative evaluation.  Most informants perceived a need for the NCCs to stabilize and demonstrate achievement before additional dollars are allocated.  Most NCCs under spent their available dollars during their implementation phase, although about half foresee the need for additional dollars to achieve the potential they envision.   Having said this, it is expected that additional dollars will be required to implement Recommendation 3, the formation of a separate, centralized leadership, support and coordination mechanism.  In addition, PHAC should reconsider the funding cutbacks of the last two years and ensure funding into the future includes adjustment for cost increases. 

There is an opportunity for potential efficiencies through a reconciliation of mandates across NCCs (see Recommendation 2.2), and the development of integrated and coordinated mechanisms, including a common KSTE framework, and common tools and process templates (see Recommendation 3.1). 

Given the diversity of mandates and potential target audiences across NCCs, a one-size-fits-all budget for each NCC may not be appropriate.  Full investigation of this issue was beyond the scope of this evaluation but should be considered in light of Recommendations 2.2 , a review and potential reconciliation of NCC topic areas. 

Issue 4:  To what extent does the NCCPH Program continue to be consistent with federal government and PHAC priorities?

Conclusions

The NCCPH Program continues to be consistent with Government of Canada and PHAC priorities.  The need for KSTE support in public health, well documented in the literature, was validated in this evaluation.  While additional topics were suggested for consideration as priority areas, it is premature to add new NCCs.  Further expansion in the future should be carefully planned and considered only after a review and reconciliation of the existing NCC mandates (see Recommendation 2). 

There is room to enhance the presentation of the NCCPH Program in the logic model.  For example,  the NCCs, as independent organizations, should be considered separately from the NCC Program Secretariat; their activities, outputs and outcomes are different.  Some aspects are presented as an implementation logic model; many of the activities and outputs are no longer relevant.  Informant feedback suggested other issues related to logic and organization, understanding and definition, content/wording, attribution and presentation.  These issues should be addressed before additional evaluation is undertaken.

The evaluators offer several references that may prove useful in revising the logic model.  The Guide to Monitoring and Evaluating Health Information Products and Services²⁰ produced by USAID, suggests a detailed conceptual framework, including logic model and indicators that are directly relevant to KSTE.  It includes indicators related to reach, usefulness, use, and collaboration and capacity building. 

The evaluators have used the conceptual framework suggested by Lafortune et al. (2008)²¹ to develop an evaluation framework for a national level KSTE agency.  Indicators associated with internal capacity and structure, external awareness (e.g., of NCCPH and NCCs), credibility, visibility/awareness, partnership engagement, and organizational responsiveness can be viewed as  critical precursors to success.  Immediate outcome or impact indicators may be segmented into a logical hierarchy, from target audience awareness of products/services, to acceptance, satisfaction (e.g., with quality, perceived usefulness), utilization/use, and finally through action (change in practice or policy). 

Recommendation

It is recommended that:

8. The NCCPH Program logic model be reviewed and revised prior to initiating a summative evaluation.

---

[Previous] [Table of Contents] [Next]