ARCHIVED - Promotion of Population Health Grant and Contribution Programs: Summary of Program Evaluations, 2004-2009

9. Impact and Effect

The evaluation question on “impact and effect” in the PPHGC evaluation strategy is:

To what extent have the programs contributed to the achievement of outcomes identified in the logic model?

The associated indicators are:

• improved capacities for influencing action on the determinants of health;
• improved access to health and social services for target populations;
• strengthened health promotion policies and actions within the health system; and
• integrated evidence base to shape population health policy and practice.

This section summarizes the evaluation findings for achievement of the outcomes identified in the logic model and performance measurement strategy in the umbrella PPHGC RMAF.

The umbrella PPHGC logic model (Annex 1) is a generic, high-level representation of key outcome areas for promotion of population health programs. It depicts the presumed effects of PHAC’s PPHGC interventions on program participants, from the individual and community up to the target-population or program-objective levels.

Not all the evaluations directly addressed all the outcome areas. In reviewing the objectives, activities, outcomes and performance indicators for each of the programs, however, it is evident that all programs contribute to all of them. The four formative/implementation evaluations, though, focused mostly on implementation issues and generally did not examine impact and effect.

As noted in the FI formative evaluation:

The AHSUNC evaluation looked at the programmatic impact on participating children, families, and communities and not other outcomes (that are in the PPHGC logic model).

There are two broad categories of outcomes identified in the logic model. The immediate and intermediate outcomes represent, in essence, the systemic changes that flow from the putting in place of the key elements that define a promotion of population health approach – i.e., an increased capacity on the “supply” side to bring about behavioural change in the target population. They are focused mainly on:

• the creation, dissemination and up-take of promotion of population health knowledge; and
• creating the environment and means to influence the development, up-take and application of that knowledge.

The longer-term (labelled as “long-term” and “ultimate”) outcomes, relate to the results arising from behavioural change in the target population.

This review and the evaluations cannot determine the extent to which the outcomes articulated in the PPHGC logic model and performance measurement strategy were met because standards and criteria to assess “success” were not well-developed – and there were no baseline data or benchmarks used in the evaluations to help interpret the findings.

9.1. Immediate Outcome Achievement: Improved Capacities for Influencing Actions on the Determinants of Health

Capacity building is focused on developing and enhancing the physical and intellectual capability, skills, resources, reach, structures, processes, and commitment of upstream and downstream contributors, delivery partners and the target population to effect change. Outputs include transferable models and approaches, professional training and public education provided, and human and financial resources leveraged. The expected outcomes are increasing program participants access to resources, increased reach, improved processes, and a broader and deeper capacity and capability to influence or carry out policy development and program delivery. Some capacities might need to be sustained over long periods of time while others might only be needed for the duration of a task or project.

The performance indicators noted in the umbrella RMAF focus on the human, financial and other resources (such as networks and knowledge) that other parties provide that allow communities to exercise greater control over the factors that influence their health, and the participation of target groups and communities in the governance projects.

The nature of the population health approach and its emphasis on networking, collaboration, and empowerment means that all programs can claim success in achieving this outcome – and almost all of them do. In addition to PHAC’s direct funding to community organizations, often noted examples of capacity building were:

• the large number of volunteers (including members of the target population) in projects funded by the program, who not only contribute resources but also become resources themselves by bringing their own ideas and expertise, and participating in organizational governance, administration and service delivery; and
• enlargement of the range of resources made possible through collaborative arrangements with other orders of government and private-sector and community organizations, resulting in monetary and in-kind contributions, mutual support and sharing of expertise across jurisdictions, reduction of overlap and duplication, and sharing (giving and receiving) training and development.

In addition, programs such as PHF have the development and sharing of knowledge and program models as a major objective. PHF and all programs reported the development and sharing of capacity-enhancing tools such as research reports, literature reviews, policy analyses, program manuals, and public and professional awareness materials.

9.2. Immediate Outcome Achievement: Improved Coordination for Intersectoral Cooperation Among Governments, Sectors and Stakeholders

Intersectoral collaboration is the developing, advancing and sustaining of engagement, collaboration and networks between sectors (private-publicvoluntary, health-non-health), systems and levels of government so as to improve the coordination, integration, comprehensiveness and effectiveness of promotion of population health efforts. Primary objectives are knowledge development, knowledge transfer, service delivery, or the strengthening of community infrastructure. Outputs include partnership and collaboration agreements and other mechanisms, and networks, committees and other working groups or teams established. The desired outcomes are sustained collaboration, coordination and relationships with and among contributors and delivery partners a, to improve service delivery and policy development and implementation.

Intersectoral collaboration is a cornerstone of the promotion of population health approach so it is present in all programs and comes in many different forms and with a variety of parties. For some programs, such as NCCPH and the Centres of Excellence for Children’s Well-being, collaboration and establishing/maintaining collaborative networks are especially important to their fulfilling their core mandate. Immediate outcome performance indicators in the umbrella RMAF are:

• examples of cooperation, collaboration and integration within and among organizations and sectors (e.g., formal protocols, agreements and planning tables with stakeholders); and
• improved relationships with stakeholders.

Every evaluation, with the possible exception of the Integrated Strategy on Healthy Living and Chronic Disease implementation review, reports wellfunctioning collaborative efforts with parties ranging from other government departments, other orders of government, businesses, non-profit and professional organizations, and schools and other institutions, to individuals and, in many cases, internationally. Typical of achievements in this area are the observations in the CBCI-CCB evaluation:

The ISHLCD implementation review noted that although progress has been made in delivering activities and implementing the coordination mechanisms for the ISHLCD, the move from a traditional, hierarchically-structured bureaucracy to a matrix model has resulted in some confusion and frustration at least in the shortterm and “it is not clear to all staff that the coordinating structures and its accompanying tools have actually improved coordination.” (p. 7)

Other evaluations noted some areas for improvement:

• CPNP noted that although collaborative efforts might be going well within a region, the collaboration across regions might not be as effective. In addition, collaboration with and learning from other initiatives within the region could be improved. These observations probably apply, in different degrees, to all the other programs.
• Several evaluations noted that the inadequacy of or competition for resources has hindered effective collaboration, with the CBCI-CCB evaluation reporting significant barriers to accessing the benefits of the program in Saskatchewan, Quebec, Nova Scotia, and Newfoundland and Labrador as a result. The CBCI-CCB evaluation also noted the dangers to the sustainability of other provincial and territorial partners because of the lack of funding.

On the type of collaboration, the Hepatitis C evaluation reported that “successful partnerships / collaboration were achieved on many fronts” but those “built on funding alone did not necessarily survive the extension years” (p. 62). This illustrates a program’s need to distinguish between where sustained and sustainable or transitory collaboration is needed or possible.

Collaboration requires time, effort and dedication. Some of the challenges noted by the evaluations include identifying and connecting with appropriate partner organizations, working with partners with different mandates and perspectives, getting the relationships established, overcoming geographic and regional dispersion, and turnover of contacts.

9.3. Immediate Outcome Achievement: Implementation of Strategies and Policies to Support Health Promotion

This outcome relates to PHAC’s working with other federal departments, other orders of government and other sectors to develop, advocate for and enhance national consensus and a responsive and enabling policy base and strategies for the promotion of population health. Outputs are the nature of PHAC-initiated policy (advocacy or options) papers or proposals on population health issues; population health strategies, guidelines, practices and models; and consensus on (specific) population health issues or approaches. The intended outcomes are:

• strengthened political commitment, policy support, social acceptance and systems support for population health in general or for a particular health issue or program, thereby fostering or creating a more seamless, transparent, harmonious and integrated policy base and strategic directions; and
• more coherent, cohesive, effective population health policies and practices across sectors, settings, and disciplines.

Indicators of success in achieving the desired immediate outcomes in the PPHGC RMAF, include:

• the number of policy documents presented, accepted and implemented; and
• population health principles being reflected in policy shifts or new policies and programs at the community level, other orders of government and other sectors.

Most of the evaluations did not address this outcome area to any significant degree probably because many programs have service delivery as their primary goal rather than being charged specifically with developing or influencing policy. In addition, policy and policy-related issues – particularly those further removed from program delivery – could be the responsibility of or have been assumed by organizations other than the individual PPHGC programs.

For example, a quick Internet tour reveals that a lot of provincial and local health authorities and volunteer/non-profit organizations have developed guides on the implementation of a promotion of population health approach that reference PHAC regional or national input.

All PPHGC programs follow the promotion of population health model in implementation and, therefore, can be expected to have influenced the adoption of population health principles in the strategies and practices of fund-recipient and other collaborating organizations. This was reflected in some evaluations, for example, in stakeholder comments on the useful guidance from PPHGC program consultants and specific references to the programs in local implementation guides, manuals and toolkits. Other evaluations, such as that for the Health Canada-Veterans Affairs Canada (VAC) Falls Prevention Initiative (FPI), reported on PHAC’s working successfully with VAC colleagues (who are more accustomed to using other delivery models) to embed the promotion of population health approach in delivering that program. In addition, most programs reported working through a variety of committees and working groups consisting of federal, provincial, territorial, community and other key stakeholders to address planning, priorities, resourcing, coordination, and other key issues; all of which would not only contribute to but also establish a stronger and broader foundation for achievement of this outcome.

Of the evaluations that directly addressed this outcome area, the Centres of Excellence for Children’s Well-being Program, which has policy advice and strategy development as a specific responsibility, was assessed as having evidence that showed some success in engaging and influencing policy makers at the provincial and territorial level. Similarly, the NCCPH formative evaluation reported that the National Collaborating Centres are well positioned to support public health practitioners and policy makers in applying best available evidence because they are placed in host organizations that, in the normal course of their on-going operations, facilitate bridging the gap between research and policy/practice. Both evaluations observed that it was too early in the life-cycle of those programs to expect evidence of actual changes to policies or programs, an opinion that was shared by the CDS evaluation^{Footnote 9}. The CPNP evaluation noted that after ten years of program delivery experience (including the maintenance of a comprehensive database on participants and projects) and evaluation activity, the program has increased its capacity to inform policies and practice. It also acknowledged some need for additional training and gaps in national guidelines and practices. The PHF evaluation reported that policy changes resulted from about 45% of their projects – and provided quantitative evidence on other outcomes.

The FI evaluation reported success in an aspect that is particularly important to that initiative – horizontal management across the four participating federal government organizations and the need for a strong vision of shared purpose and the identification of shared priorities, interdependencies, and collaboration.

9.4. Immediate Outcome Achievement: Increased Awareness and Use of Reliable Health-Related Evidence

As defined in the Ottawa Charter for Health Promotion, the promotion of population health seeks to increase the options available to people to exercise more control over their own health and their environments. This can be achieved in part by creating and continually improving a reliable knowledge and evidence base, and encouraging its increased use by individuals, health providers, and groups and organizations involved in the promotion of population health or in activities on the determinants of health.

PPHGC knowledge outputs are in the nature of research or analytical material, for example, on population health practices and the supporting empirical outcome evidence, including the relationships among health, health status, determinants of health and health practices.

Integrated communication and dissemination strategies aim to increase public and policy-maker awareness, understanding and use of the knowledge and evidence base. Outputs are in the nature of dissemination strategies, tools and messages that are appropriate for intended audiences. The use of the health knowledge is intended to bring about behaviour change in program participants that is then reflected in improved policies and practices and, ultimately, improved population health and reduced health disparities.

PPHGC RMAF immediate outcome performance indicators for this outcome area include:

• increased targeted population, public and media awareness of information and knowledge being available as indicated by such data as the number of requests for information (including web-hits); and
• the perception of key stakeholders regarding the quality, availability and accessibility of information.

All eleven evaluations that assessed performance in research and knowledge development reported contributing new knowledge to their area of responsibility, ranging from on-going project and program evaluations (including CPNP’s extensive database) to directed and solicited research such as those done through the Centres of Excellence for Children’s Well-being and the Population Health Fund and shared learning from hands-on program delivery and surveillance activities such as in the Federal Initiative to Address HIV/AIDS in Canada.

Programs used a variety of means to disseminate the materials they had produced, either directly or through partners and collaborators: web-sites; the regular mail system; print and broadcast media; audio, video and CD products; local, regional or national presentations to stakeholder groups and at conferences; workshops; and, probably most importantly, in face-to-face dealings with members of the target population and those working with them. All programs used multiple channels.

Most evaluation findings are based on opinions but some, such as the CPNP evaluation, had access to significant quantitative evidence and were successful in providing analysis of the extent to which the program is reaching its targeted audience and its major gaps. The CPNP evaluation reported:

The Falls Prevention Initiative evaluation provided concrete examples of behaviour and attitude changes, ranging from measures taken by seniors themselves to prevent falls to changes in the practices of local housing and homemaker associations and authorities that were influenced by Initiative output. The PHF evaluation noted that one-quarter of the 116 projects completed between October 2005 and October 2008 reported behaviour change. The CCB, although not explicitly claiming credit, also provided anecdotal evidence of behaviour change – examples:

• “In some cases, the empowerment of women through breast cancer information was said to have impacted on physicians’ practices, as the women were well-informed – sometimes more than the physicians – and eager to assume a partnership role in their care…”(p. 32)
• Screening program managers in a province affirmed that the CCB network had been influential in changing age eligibility practices for screening mammography in the provincial program. (p. 33)

The CPNP evaluation substantively discussed the qualitative and quantitative evidence on the program outcomes of healthy birth weight, infant health, maternal health, and breastfeeding. The quantitative evidence demonstrated an impact on breastfeeding initiation but was considered exploratory and not sufficiently robust to support conclusions on impact for the other outcome areas. The qualitative evidence from a survey of program participants, however, concluded that they were “overwhelmingly pleased” with the services, all major aspects of the program were important and valuable and the program produced a range of outcomes that were consistent with the program objectives. (p. 76)

9.5. Intermediate Outcome Achievement: Evidence Base to Shape Promotion of Population Health Policy and Practice

This outcome is the result of the continued expansion and improvement of the evidence base and dissemination/exchange of this knowledge to inform policy development and action. The output is an increasingly more integrated, comprehensive, reliable and accessible evidence base that, through effective dissemination strategies and approaches, would result in its effective and efficient use by individuals and policy and other decision makers, and other PPHGC program participants to inform policy and practice.

The PPHGC RMAF performance indicator for this “intermediate” outcome area is the frequency of references to the evidence base in policy and program decisions in a variety of jurisdictions and sectors. While none of the evaluations provided evidence along the lines suggested by the PPHGC RMAF performance indicator, information in the evaluations indicates that the evidence base on the promotion of population health has improved through the programs’ research and other program delivery work.

The evaluation of the Hepatitis C Prevention, Support and Research Program, for example, quoted key informant and other stakeholder opinions that there has been a “dramatic” increase in Hepatitis C research since the inception of the Program. Collaborative efforts with research organizations such as the Canadian Institutes of Health Research and the National Canadian Research Program in Hepatitis C are expanding the evidence base. The CDS evaluation reported that the National Diabetes Surveillance System, although not yet able to provide incidence data at the time of the evaluation, has already significantly improved the information base on diabetes. Research-focused programs such as PHF and the Centres of Excellence for Children’s Well-being are further strengthening the evidence base through their solicited and directed research efforts, as are other PPHGC programs through their ongoing formative and summative evaluation and other research and program delivery activities.

Further, programs such as the National Collaborating Centres for Public Health, which are explicitly charged with enriching the evidence base for decisionmaking in public health, reported an extensive and varied output in their areas of responsibility. In addition, they conduct environmental scans and consultations with key stakeholders (such as policy makers) to identify knowledge gaps.

Policy and practice changes of the types noted above for the Falls Prevention Initiative and CBCI-CCB and the large number of guides and toolkits on implementing the promotion of population health approach at the provincial and local levels, particularly those acknowledging PHAC contributions, indicate that this evidence is being used.

The above examples are from the individual program perspective. The programs also contribute to “corporate” resources or initiatives such as the Canadian Best Practices System and, through those, influence policy and practice.

9.6. Intermediate Outcome Achievement: Strengthened Health Promotion Policies and Actions Within the Health System

This outcome area relates to continually improving the positioning, coordination and integration of population health within the broader social, economic, and health development agendas at all levels of government, communities and nongovernment sectors. Key activities include improving the link between national population health leadership and community needs through partnerships, enhancing population health human resources, and improving the implementation of policies aimed at improving the health of populations. Indicators of performance in the PPHGC RMAF for this intermediate outcome area are:

• evidence of working with partners in translating and adapting best practices to specific settings;
• stakeholder perception of:
  • reduced duplication; and
  • increased linkages of federal objectives and provincial strategies; and
• evidence of movement toward integrated partnerships, policies, information, legislation, regulation, programs
  • within the health sector (F/P/T/, municipal); and
  • with and within other sectors (F/P/T/, municipal) that are consistent with the promotion of population health.

None of the evaluations systematically assessed their program against this outcome along the lines of the PPHGC RMAF performance indicators but that is not to say that it is not being accomplished. Work to bring about this outcome is conducted at all levels – from community project working groups to privatevoluntary- public sector consultations, and federal-provincial-territorial committees, to ministerial and international conferences and meetings. PPHGC program collaboration, partnering, sharing of best practices and other contributions to health system improvement are being done and have been noted in this report.

9.7. Intermediate Outcome Achievement: Improved Access to Health and Social Services for Target Populations

An important role in the promotion of population health is facilitating that access through policies and other action that remove barriers to accessing services. Many of these barriers are reflected in the determinants of health – e.g., income (e.g., cannot afford access to the Internet or to improve living conditions), education and literacy (written information cannot be well-understood), physical environment (geographic isolation faced by rural and remote populations), and culture (need culturally appropriate approaches and services).

Indicators of performance in the PPHGC RMAF for this intermediate outcome area are:

• number of visits to website/frequency of use;
• number of referrals to other services; and
• replication of promotion of population health best practices in provincial and territorial systems.

In general, the evaluations report that PPHGC programs not only provided information and tools to help members of targeted populations change attitudes and behaviour – and sustain this change – they also facilitated access to and, in the case of programs such as AHSUNC, CAPC and CPNP, provided services directly. Advocacy (such as CBCI-CCB) and research-focused programs also identified and worked with policy or delivery organizations to address gaps in service.

Programs such as CPNP make significant effort in identifying their target populations and locating/funding projects in close proximity to them so as to increase accessibility. The evaluations also reported that PPHGC programs developed culturally appropriate approaches and content for such populations as immigrants and Aboriginal Peoples. The CPNP evaluation further noted that projects also tailored their delivery methods to suit their clientele, such as some projects using informal, drop-in formats and others, more structured approaches.

PPHGC programs expand their reach through their partners, collaborators and networks. The evaluations for CHN, CAPC and CPNP were also successful in providing quantitative indicators of reach.

As indicated by the CBCI-CCB evaluation’s finding on changes in physicians’ practices, PPHGC programs are not only helping in improving access, they are also improving the health literacy of their target population so that the quality of that access is also enhanced.

9.8. Long-Term and Ultimate Outcomes

The immediate and intermediate outcomes facilitate progress toward improved health outcomes and reduced health disparities. They, in essence, create the environment and provide the tools, such as:

• coordinated, integrated, coherent policies, strategies, work plans and activities that focus on and advocate for improving the health of populations;
• continually improved knowledge base and skills that individuals need to make healthy choices on factors that are within their control;
• dissemination strategies and measures to inform at-risk populations of health risks and the support that is available to them;
• the policy, regulatory and legislative environment to eliminate/reduce, avoid or mitigate risks and other factors that individuals, by themselves, cannot control; and
• services (or access to them) that individuals need to carry through on or support their healthy choices.

The PPHGC logic model identified the following long-term outcomes and associated performance indicators:

• Improved health outcomes, as indicated by
  • increased life expectancy across populations;
  • healthy birth weight; and
  • disability-free life-years;
• Improved personal health practices and skills, as indicated by
  • physical activity, distribution by age, income, ethnicity and gender; and
  • percentage of Canadians engaging in behaviours that decrease risks of disease by type.

It also identified the ultimate outcome for the PPHGC Program as:

• Improved population health and reduced health disparities, as indicated by
  • changes in the health determinants; and
  • quality-adjusted life-years across populations.

These are population-level outcomes and were not covered by the evaluations of the individual programs.