ARCHIVED - Integrated Strategy on Healthy Living and Chronic Diseases - Community Based Programming Functional Component

 

Executive Summary

Background

Diabetes Community-Based Programming and Community Capacity Building (referred to as the Diabetes CBP), is part of the renewed Canadian Diabetes Strategy under the Integrated Strategy on Healthy Living and Chronic Disease (ISHLCD), which was launched in October 2005.  The ISHLCD consists of integrated as well as disease-specific strategies, and is administered through a matrix structure composed of six Functional Components and 14 Program Components. It is also one of the six elements of the renewed Canadian Diabetes Strategy (CDS) Program Component that also include: National Diabetes Surveillance System; Knowledge Development and Exchange for Diabetes Prevention and Management; Diabetes Public Information; Diabetes Coordination; and Monitoring and Evaluation.

Diabetes CBP is designed to benefit populations who are at higher risk for developing type 2 diabetes as well as those living with diabetes.  The program provides funding for project Grants and Contributions (Gs&Cs) to develop, implement and evaluate community-based initiatives at both the national and regional levels.  The aim is to promote a positive shift in health status in high-risk populations and those living with diabetes through supporting community-based programs that are tailored to the particular needs of communities.  Key activities under Diabetes CBP also include partnership development and collaboration, as well as knowledge dissemination by the Public Health Agency of Canada (PHAC) staff.

Context

Diabetes CBP formative evaluation serves the reporting requirements under the ISHLCD’s fully-elaborated Results-Based Management Accountability Framework (RMAF) developed in February 2007.  It is also designed to produce evidence that will feed into the overall assessment of the ISHLCD in relation to its relevance, program delivery and design, and success/progress.  This evaluation focuses primarily on Diabetes CBP, while including limited assessment of the coordination and integration between CBP and other Functional Components, as well as among elements within CBP.  This evaluation covers fiscal years 2005-06, 2006-07 and 2007-08 (until November 2008).

The objectives of the Diabetes CBP formative evaluation are to examine issues related to relevance and design and delivery, determine the success/progress of the CBP Functional Component, and propose improvements to Diabetes CBP, as well as to the ISHLCD as a whole.

Methodology

Four lines of evidence were used in this evaluation: a review of performance monitoring data collected by 2006-07 funded projects using pilot Program Evaluation Reporting Tool (PERT) data; a document review; a review of the Early Outcome Evaluation of the Diabetes Community-Based Projects Report; and interviews with three program representatives.

The limitations of these lines of evidence are outlined in Section 2.2 of the report.

Findings

Relevance

The findings of this evaluation confirm general support for PHAC’s role in the prevention of chronic disease, including diabetes.  The latest surveillance data indicate a rapidly increasing prevalence of diabetes in Canada with approximately 1.9 million Canadians, or about 5.9% of the population, having been diagnosed with the disease as of 2005-06.  There was general recognition among stakeholders consulted through the 2007-08 Diabetes Policy Review of the importance of supporting community-based programs in promoting healthy living and preventing diabetes and its complications.

At the same time, interviews with program representatives found evidence of potential overlap or duplication between CBP and the Knowledge Development, Exchange & Dissemination (KDED) Functional Component, largely as a result of a lack of clarity with regards to how the two components fit under the ISHLCD.  Also, the information collected and reviewed as part of this evaluation suggests some overlap between Diabetes CBP and the Healthy Living Fund in supporting communities in promoting healthy living (healthy eating and physical activity) to prevent chronic disease, including diabetes.  Representatives of non-governmental organizations (NGOs) and provincial/territorial (P/T) governments, consulted through the Diabetes Policy Review, cited the lack of clarity between different funding programs under the ISHLCD as a major challenge to their respective work.

In addition, feedback received from the regional program representative and the analysis of information from the program document/file review point to the need to define PHAC’s role vis-à-vis that of P/T governments in addressing the management of diabetes, including support for community-based programs in this area.

Design and Delivery

Feedback from program representatives and a review of program data suggested there was an overall lack of clarity with regards to roles and responsibilities, as well as processes, for program delivery.  There was also evidence of a lack of consistency in the delivery of Diabetes CBP across regions, which was primarily attributed to the lack of human and financial resources in most Regional Offices, as well as the program’s inability to obtain timely funding approval for the solicitation process for Gs&Cs.

The information collected and reviewed as part of this evaluation pointed to a number of challenges and gaps in relation to the design and delivery of Diabetes CBP.  Most notably, both internal and external stakeholders expressed concern over the inability to provide multi-year funding under the program.  The use of multi-year funding was considered critical in supporting the achievement of more meaningful and longer-term results.

Among the key gaps and challenges identified were knowledge transfer through dissemination and evaluation.  The expected program activities in the area of dissemination have not have been clearly defined.  A review of program information indicated a lack of ongoing monitoring of the administrative data of expected results and outcomes from the program activities.  While the intent was to include questions on the outcomes achieved from project activities in the performance monitoring data, this was not done for the 2006-07 PERT pilot.  Largely as a result of the delay in initiating the Early Outcome Evaluation project, only limited data on project outcomes was collected. In addition, the collection of program-level data undertaken as part of this evaluation revealed a lack of systematic tracking of program administrative documents at both the national and regional levels.

Success/Progress

This evaluation found the implementation of the ISHLCD, including CBP, has been significantly challenged by a number of factors within PHAC as well as in the broader environment in which it operates.  Most notably, delays in obtaining funding approval resulted in a limited solicitation process for Gs&Cs in many regions, as well as at the national level.

Data reported by community-based projects funded in 2006-07 show these funded projects generally succeeded in reaching their target populations, while numbers of individuals reached varied greatly from one project to another depending on the type of project.  However, the program’s reach to persons living with diabetes was limited, as the majority of the funded projects were aimed at addressing prevention among high-risk groups.  Data also suggest partnerships were a key feature of the funded projects, with most projects (85%) reporting having formed new partnerships as a result of the projects.

There was insufficient data to assess some of the outcomes from funded projects examined through this evaluation.  Through PERT and the Early Outcome Evaluation, most programs reported some success in achieving a number of outcomes.  These included increasing awareness and knowledge among high-risk populations and contributing to organizational or system-wide policy changes.  However, these reports of success were often not substantiated with systematically collected evidence.

At the program level, only a small number of activities were undertaken in the areas of partnership development/collaboration, and dissemination to support the transfer of knowledge generated from community-based programs.

Recommendations

While this evaluation has found that Diabetes CBP has made promising progress in achieving some of its expected results, it has also found a number of challenges in relation to its design and delivery approaches.  The following are recommendations for improving Diabetes CBP, including its evaluation function, and the broader ISHLCD Program and Evaluation functions:

Diabetes CBP:

  • 1. Identify, assess and communicate risks to senior management associated with one-year funding agreements in community-based diabetes initiatives and within the broader context of the PHAC Grants and Contributions (Gs&Cs) Realignment Initiative.
  • 2. Enhance transparency and consistency in Diabetes CBP project solicitation and decision-making processes.
  • 3. Increase stakeholder awareness of Diabetes CBP priorities, progress, roles, responsibilities and vision.
  • 4. Identify and communicate internal and external linkages between Diabetes CBP, Knowledge Development, Exchange & Dissemination (KDED), and the Healthy Living Program.
  • 5. Disseminate evaluation findings and lessons learned from funded projects.

Diabetes CBP Evaluation:

  • 6. Establish a performance measurement framework and monitoring system to track activities, outputs and immediate outcomes at the program-level of the Diabetes CBP Functional Component.
  • 7. Improve project-level performance reporting by enhancing the diabetes-specific PERT questions.
  • 8. Develop approaches and instruments to measure intermediate outcomes of Diabetes CBP to facilitate ongoing decision-making and future evaluation designs.
  • 9. Strengthen the program and evaluation linkages within Diabetes CBP (between the Community-based Network and the Diabetes and Healthy Living Evaluation Working Group).

Overall ISHLCD Program and Evaluation

  • 10. Ensure all ISHLCD Functional Component Gs&Cs adhere to the forthcoming findings and recommendations from the PHAC Gs&Cs Realignment Initiative.
  • 11. Develop an approach and instruments to systematically assess the common immediate outcome of “engagement” which crosses all ISHLCD Functional Components.
  • 12. Establish a performance measurement framework and monitoring system for the ISHLCD as a whole to track key outputs and immediate outcomes.
  • 13. Identify and communicate the Knowledge Development & Exchange (KD&E) linkages across Functional and Program Components of the ISHLCD matrix.
  • 14. Identify, assess and communicate internal and external ISHLCD coordination mechanisms reflecting best practices for matrix management and an integrated approach to healthy living chronic disease prevention and control.

Page details

Date modified: