Page 6: Evaluation of the Aboriginal Head Start in Urban and Northern Communities Program at the Public Health Agency of Canada

3. Findings

This section provides a summary of the findings organized under two broad headings:

• the relevance of the Public Health Agency’s involvement in early childhood development programming for Aboriginal children off-reserve (section 3.1)
• the performance of the AHSUNC program (section 3.2).

3.1 Relevance

This section examines the overall relevance of the program in terms of need, alignment of the AHSUNC program with the role of the federal government and the Public Health Agency and alignment of the program with the mandate and strategic priorities of the Public Health Agency. The section on federal role will also briefly examine current federal and provincial/territorial responsibilities with respect to early childhood development in Canada.

Continued need

Finding 1 There is a continued and growing need for the Aboriginal Head Start in Urban and Northern Communities program

There are approximately 47,900 Aboriginal children aged three to five years in Canada^Footnote18 (living off-reserve) and AHSUNC reaches approximately 4,640 children aged zero to six (86 per cent of them are between the ages of three to five) each year. Population growth rates are higher for Aboriginal people than for the general Canadian population, and a shift towards urban settings has increased the need for early childhood development programming off-reserve. Seventy per cent of young Aboriginal children live off-reserve and of those, 80 per cent live in urban centres.^Footnote19 A young and growing population is also contributing to a strong need for the program in northern Inuit communities.

Aboriginal children are at higher risk for poor developmental and health outcomes due to several factors. In Canada, Aboriginal children:

• are over-represented in the child welfare system
• experience higher levels of moderate and severe food insecurity (33 per cent) than non-Aboriginal populations (nine per cent)
• are twice as likely to experience poverty as the general Canadian population
• are two to three times more likely than non-Aboriginal Canadians to be raised by young, single parents.

These risk factors correlate with inequalities in early childhood development indicators (e.g. scores on motor, social and emotional development). Considerable evidence supports the mitigating role of community-based early childhood development programs in the lives of children facing similar risks, which underscores the need for early childhood development programming for Aboriginal children off-reserve.^{Footnote 2020} Furthermore, solid evidence is emerging of the key influence of cultural identity in mental health promotion and suicide prevention of First Nations youth.^Footnote21

Continued need for the program is also evidenced by the long wait lists for participation. Over the last four years, there was an average of 1,300 children per year waiting for a spot in the AHSUNC program. This high level demand likely reflects the limited number of free and culturally sensitive programs in Canada.

Northern communities have specific needs such as limited access to early childhood educator training programs and thus, a shortage of qualified staff and high material costs. In addition, some northern communities have long wait lists, while other communities do not have an AHSUNC site at all. Qualified staff who run sites are reportedly overburdened by requests to serve as advisors and proposal developers since there are few who can represent their communities’ needs. Each of these factors underscores a continued and strong need for this program in the north.

Alignment — role

Finding 2. Delivering the AHSUNC program is an appropriate role for the federal government and the Public Health Agency

The federal role

According to the Department of Health Act,^Footnote22 the Minister of Health’s powers are to protect and preserve the health of the people of Canada. The Public Health Agency was created to assist the Minister in fulfilling her powers and duties in relation to public health. In Canada, public health is a shared responsibility between federal, provincial and territorial and municipal governments, the private sector, non-government organizations, health professionals and the public.

The Government of Canada is not prohibited from investing in areas that are primarily within provincial and territorial jurisdiction and thus, often makes policy decisions to do so. For example, while provinces and territories have the primary responsibility for early childhood development, the Government of Canada plays a supporting role.

Typically, when investing in an area of shared jurisdiction with the provinces and territories, the federal government’s role is to:

• address an issue of national scope that is beyond the capacity of any particular province or territory to address by itself
• fill gaps for a vulnerable population
• collaborate with provinces and territories to complement provincial or territorial directions.

The AHSUNC program meets all of these criteria.

Address an issue of national significance

There are well-documented inequalities between Aboriginal people and the general population of Canada, and action is required to close the gap and reduce disparities in well-being. Factors that have been implicated in the specific inequalities for Canada’s Aboriginal peoples include the historical colonial relations and the legacy of residential schools, which have had inter-generational negative effects on Aboriginal societies. Research demonstrates the importance of using a holistic approach to address the well-being of Aboriginal people (physical, emotional, cultural and spiritual) and emphasizes the importance of culture and language in positive well-being.

Fill gaps for a vulnerable population

The AHSUNC program is a targeted initiative for a vulnerable population and was recognized as such by the Chief Public Health Officer’s Report on the State of Public Health in Canada (2009).^Footnote23 Head Start programs differ from other early childhood development programs such as kindergarten and child care in that they target low-income children and families living in high-risk circumstances and provide comprehensive programming in the areas of social, emotional, physical and cognitive preparedness required for school. Parent involvement is a central component of the Head Start philosophy; in addition to helping plan and run the program, parents also receive services for their own social, emotional and vocational needs. Head Start projects generally include a community component, aiming to build supportive connections between communities and families. The AHSUNC program is unique in that it has an integrated Aboriginal language and culture component in its design. While there are several cities in Canada with other Head Start programs, these are not necessarily designed specifically for Aboriginal clients, and are not available consistently across the country.

The AHSUNC program supplements provincial and territorial investments in the area of early childhood development programming. Despite increased provincial and territorial investments that have resulted in enhanced availability and accessibility of early childhood development programming in Canada, there is still more demand for these programs than can be met at this time. For example, in every province other than Quebec, the number of children between the ages of two and four exceeds the number of licensed child care spaces available. Factors such as location, hours of operation, affordability as well as language and cultural irrelevance can serve as barriers to accessing early childhood development programs. Research shows a direct correlation between family income and access to child care, such that over 65 per cent of children under five years of age in Canada have no involvement in out-of-home care, compared to only 30 per cent of children in the most affluent families. Information collected from case studies indicates that Aboriginal parents may choose not to access provincial early childhood development programs available to them due to cultural irrelevance. Within the AHSUNC program, 52 per cent of sites have a waiting list for a total of almost 1,300 children in line to be enrolled. This issue of waitlists is linked to the need to review site locations and is addressed further in this report (see section on efficiency and economy).

Given the current trend within provinces and territories to recognize the importance of the early years in the public education system, the early childhood development landscape in Canada will continue to evolve with respect to provincial and territorial reach to young children and accessibility of publicly funded early childhood programs. The role of the federal government in delivering the AHSUNC program should evolve with these changes and should be aligned and integrated with provincial and territorial directions to ensure the most effective use of funds. For example, this might include the establishment of consistent messaging, seamless access and knowledge development and exchange related to promising practices in early childhood education.

Collaborate with provinces and territories to complement provincial and territorial programs

AHSUNC projects must follow applicable child care or preschool legislation for daycare licensing regulations in their respective province or territory.^Footnote24 At the site level, AHSUNC project staff partner with provincial and territorial staff in public health, social services and education to ensure the delivery of complementary, efficient and effective programming. Provincial and territorial representatives often participate in AHSUNC advisory committees to ensure alignment with provincial and territorial priorities.

The Public Health Agency’s role

The Public Health Agency’s mission is to promote and protect the health of all Canadians, including Aboriginal peoples, through leadership, partnership, innovation and action on public health. The Public Health Agency fulfills this mission in a number of different ways, one of which is the funding of community-based public health programs through grant and contribution transfer payments. Given the alignment of the AHSUNC program with the federal role described above, delivering this community-based program is a legitimate and appropriate role for the Public Health Agency.

However, funding community-based programs is not the Public Health Agency’s only role, as the Public Health Agency is also expected to provide leadership, collaboration and coordination on matters of national public health significance. With the exception of a few program areas (e.g. Aboriginal Head Start in Urban and Northern Communities), the Public Health Agency’s role in Aboriginal public health is to foster collaboration with other federal government departments, provincial and territorial governments, Aboriginal organizations and other key stakeholders on Aboriginal public health priorities.Whether the Public Health Agency leads, co-leads, coordinates, or otherwise supports specific action to address Aboriginal or other public health depends on:

• whether the issue falls under the Department of Health Act,^Footnote25 the Public Health Agency of Canada Act^Footnote26 and/or is national in scope
• the level of priority accorded to a particular issue by the Public Health Agency or the broader Government of Canada
• federal Health Portfolio policy management and federal/provincial/territorial and other considerations.

Currently, the role of the Public Health Agency concerning the AHSUNC is limited to funding of the program. There is limited national coordination with provinces and territories, National Aboriginal Organizations and other stakeholders (to be discussed further under Finding #10).

Alignment — priorities

Finding 3. The AHSUNC program aligns with the Public Health Agency’s mandate and strategic priorities , particularly with respect to Northern and Aboriginal public health

The Public Health Agency’s mandate includes health promotion. As well, the Public Health Agency has an established role to strengthen intergovernmental collaboration on public health and facilitate national approaches to public health policy and planning. This speaks to partnering with stakeholders and leveraging resources across all levels of government, to address Aboriginal health inequalities. AHSUNC aligns with the Public Health Agency’s strategic priorities and mandate, by directly addressing social determinants of health in a known vulnerable population. Northern and Aboriginal public health was determined to be one of the Public Health Agency's top six strategic priorities in an Executive Committee retreat discussion on policy priorities in January 2011. Aboriginal public health was subsequently emphasized as a top priority in a Public Health Agency Resources, Planning and Management committee meeting. All executives at the Public Health Agency are now asked to support the Agency’s program and policy results by translating their branch's vision into concrete direction and directorate plans that support the strategic priorities, including Aboriginal public health, as per their annual performance agreements.

AHSUNC is a national public health program, providing accessible and targeted services across numerous Canadian communities with under-served and vulnerable Aboriginal populations, directly addressing the priority of Aboriginal public health. AHSUNC also addresses other Public Health Agency priorities including obesity prevention, injury prevention and mental health promotion. Early childhood is a crucial developmental stage during which the impacts of the determinants of health become magnified. This was further emphasized in the Chief Public Health Officer's Report on the State of Public Health in Canada (2011), where he highlighted Aboriginal youth and the health inequalities they face.^Footnote27 With its broad public health focus delivered to a vulnerable population, AHSUNC helps the Public Health Agency directly improve the health outcomes of Aboriginal children.

3.2 Performance

This section examines the AHSUNC program's performance in terms of outcomes for children, parents and families as well as communities.

Effectiveness

Finding 4. The program has had a positive effect on school readiness, specifically in improving children’s language, social, motor and academic skills. Performance results have also demonstrated effectiveness in improving cultural literacy and enhanced exposure to Aboriginal languages and cultures. There are also positive effects on health promoting behaviours such as children's access to daily physical activity as well as determinants of health, such as access to health and dental care. The program has not systematically assessed results related to knowledge development and exchange, collaboration with Aboriginal early childhood development programs or support to parents and families

The table below provides a high-level review of the outcomes described in the program logic model^Footnote28 along with an assessment of the extent to which these have been measured through performance measurement activities, the strength of the available data, and the evidence that results have been achieved. (See Appendix E to view the program logic model.) From the figure below (Figure 6), it can be observed that certain elements of performance received higher priority for measurement, particularly outcomes related to school readiness. Overall, there is strong evidence that the program has an immediate impact on children’s school readiness and cultural literacy. There is limited evidence of the program’s impact on parents and the broader community, and of increased collaboration with other Aboriginal early childhood development programs. There is no evidence of the program’s impact on dissemination and uptake of research.

Outcomes for children

School readiness

Carefully designed program performance studies demonstrate that AHSUNC has had a measurable, positive effect on participants’ language, social, motor and academic skills. Length of time in the program correlates to higher school readiness scores: children with prior participation in the program had significantly higher school readiness scores at the beginning of the school year than new registrants of the same age. In addition, although AHSUNC participants begin the program with lower school readiness scores than the normative sample (i.e. age-matched peers), at the end of the year AHSUNC participants have scores that are comparable to or above those of the normative sample (Figure 7). These findings were also supported during the case studies, where parents and teachers described a variety of impacts on children ranging from improved confidence as well as social and verbal skills to greater cultural awareness and comfort with routines.

Figure 7: A comparison of school readiness scores between children enrolled in AHSUNC (n=1,310) and their normative sample (i.e. age-matched peers)

Source: Public Health Agency of Canada 2011a

This study used the Brigance Screens, which were developed for screening and monitoring child development. The Screens measure several dimensions of school readiness (language, motor and academic skills) and allow participants to be compared to the normative sample (i.e. age-matched peers). AHSUNC participants had a statistically significant increase in school readiness skills. This figure illustrates the children’s results in the fall (Phase 1) and late spring (Phase 2), as compared to the normative sample.

The following figure (Figure 8) presents the AHSUNC participants’ weighted mean scores (mean per cent of correct answers across the three age groups). It should be noted that participants of different ages answered different questions (age-specific measures).

Figure 8: Progress in scores by skill area, all ages combined

Source: Public Health Agency of Canada 2011a

Increased cultural literacy and enhanced Aboriginal language and cultural exposure

Case study observations, parent feedback and performance data indicate that the language and cultural component of the AHSUNC program is highly valued. According to these same sources, it appears that this component has led to increased exposure to Aboriginal culture and language skills. Parents and staff view the culture and language component as key to fostering a sense of pride and identity for parents and children.

A performance study using a quasi-experimental design compared Aboriginal children who have participated in AHSUNC to Aboriginal children who have not. This study used data from the Statistics Canada Aboriginal Children Survey, which collected data about First Nations, Inuit and Métis children under six years of age living off-reserve in urban, rural and northern communities. This analysis revealed that there is a small statistically significant increase in AHSUNC participants’ feeling that Aboriginal culture is important to them, which speaks to cultural literacy. As well, the analysis revealed a statistically significant increase in exposure to cultural activities that was associated with hours per week of AHSUNC participation. In other words, AHSUNC participation was positively associated with an increase in participation in cultural activities such as telling stories, singing songs and participation in traditional or seasonal activities.

Health promoting knowledge, skills and behaviours

Performance information and the case studies revealed that the program has positive effects on children’s access to daily physical activity as well as health and dental care. There is some evidence that AHSUNC participation may increase the frequency with which children consume healthy foods and consistent evidence that AHSUNC participation encourages consumption of traditional foods.

Outcomes for parents and families

At many sites, parents appear to have benefited from the opportunity to volunteer, attend training events and gain increased social support. However the outcomes of parental involvement have not been systematically tracked. Furthermore parental involvement is the program component with the greatest degree of variability. Regional evaluation and case study findings indicate that the role or level of involvement by parents in AHSUNC is not consistent across project sites.

Through interviews, performance information and the case studies, many stories were shared about parents and families whose lives have been transformed by their involvement in the program, such as parents who became motivated to go on to achieve higher levels of education and come back as early childhood educator in the program or elsewhere. The following stories illustrate this theme.

I was new to the city when my daughter was in the program. I remember coming to parent meetings and craft days but feeling very shy and unsure of myself. I didn’t know where in the world I fit in. I got my first job outside the home during that time. Eventually with improved self-esteem I went back to school to get my Grade 12 and got my driver’s license. My reason for going back to school was to show my daughter that I could do it. My work placement was at the [local] Head Start program. Through my work placement I developed relationships with staff and began to feel like I fit in. By the time I enrolled my son, I was very engaged with the program and served as a parent representative on the board and filled in as a substitute teacher to cover when people were sick. Eventually I filled in for someone on maternity leave. I received my certificate in Early Childhood through online course work and passed with honours. I am now completing the last year of my diploma. AHS can change peoples’ lives. It can make a big difference. When you start seeing things in a different way from your little bubble, it makes a difference… I learned so much. (paraphrase from a current teacher)

I myself am a parent, my four kids all graduated from here. When I came in I had no experience on councils or anything and then, “here, you are in charge,” and I thought “me!?” So I was on the parent’s council, went to all the training and just kept going. I got a part time job here and went from part time to a full time job. (paraphrase from a current employee whose children are now in high school)

Outcomes for communities

Building a sense of community through community hubs

Although the program’s impact on communities has not been consistently tracked, there is evidence that some of the AHSUNC project sites have become known as the community ‘hub’, creating a sense of community for Aboriginal children and their families.

Alignment with municipal, provincial or territorial services and early child development systems

Although there are a few regional examples of alignment with provincially or territorially funded early child development systems or the school system, overall there was limited evidence of this outcome. Through the case studies, staff relayed stories of changes in community structures and supports to better serve the Aboriginal population as a result of partnerships and connections of the projects with the broader community. Some sites are advocating for changes to systemic barriers that they have observed families facing. For example, in one province work is underway to support AHSUNC children’s transition into mainstream schools. Another example is a site that is using education and advocacy to influence changes in the Children’s Aid Society, namely the formation of a team devoted to this Aboriginal community. The Children’s Aid Society team members received specialized training that sensitizes them to the specific challenges faced by this population and they were made aware of culturally relevant supports that are available to support families during challenging times. This same site gives presentations to schools to raise awareness of the Inuit culture and the challenges they face in this community “to dispel myths and talk with pride about the Inuit culture” and with other agencies to encourage availability of “culturally safe” services.

Knowledge development and exchange

While knowledge development and exchange is underway at the community level, this has not been measured. For example, knowledge development and exchange for continuous improvement is encouraged at the community level through regional training events and networking activities. However, the impact of knowledge development on policy and practice is unclear. There was limited evidence of the dissemination of knowledge to influence policy or practice within the Public Health Agency or among other federal, provincial and territorial government departments, or national Aboriginal organizations. Some notable exceptions were identified; for example, one case study site described how the program is influencing provincial policy such that the provincial Ministry of Education used components from AHSUNC (e.g. parental and family involvement and transportation) as an example to provide a preschool program for children within the school system.

There was limited evidence of integration with provincially run early child development programming and Health Canada’s Aboriginal Head Start On-Reserve program and links to training and research institutions, thus limiting the potential reach and influence of AHSUNC and potential enrichment of AHSUNC sites (e.g. through access to funding, curriculum supports and knowledge exchange). However, the lack of formal networking between most AHSUNC sites and provincially funded early child development programs and the Aboriginal Head Start On-Reserve program suggests that there is potential for greater synergy among federally, provincially, territorially and municipally funded programs. From case studies at the early childhood development programs that are not funded by the Public Health Agency it observed that there is potential for AHSUNC sites to influence other programs that might reach Aboriginal children. Conversely, it was also observed there is potential for AHSUNC sites to learn from non-AHSUNC sites, for example about trends in program delivery, curriculum development, governance and fundraising.

Efficiency and economy

This section examines the overall approach to program delivery, including a review of program efficiency and associated challenges. Although aspects of the program delivery were found to be efficient, a number of program inefficiencies were also noted. A comparison of the AHSUNC program design to similar early childhood development programs in other countries is provided in Appendix F and discussed in section 3.2.3.

Program design and delivery

A description of how the AHSUNC program is designed and delivered is provided in the Background section of the report. The following section discusses strengths associated with the current program design and also examines the various models currently used to support program delivery.

A number of program strengths were revealed through the document review, regional evaluations, interviews and case studies.

• Program components: Implementation of the six program components and guiding principles works well and provides some national consistency in program delivery.
• Community-based projects delivered by Aboriginal organizations: Feedback collected from multiple sources including the National Aboriginal Head Start Council group interview revealed that because the program is community-based and delivered by Aboriginal organizations, it fosters a sense of empowerment, ownership and community pride.
• Alignment with schools: An interesting observation from the case study analysis is that five of the case study project sites were either located in a school, next to a school or had developed a strong partnership with a school. These links brought many benefits to the projects, the children and their families. Projects located in or near a school gained access to kitchen facilities, playground equipment and a gym that the project was not required to maintain. It is interesting to note, in the Aboriginal Head Start on-reserve situation, it is not uncommon for there to be physical co-location of early childhood development programming in schools or with community daycare centres. In rural areas, some of the AHSUNC programs are delivered in school buildings that also have before and after school child care programs, so their models are more integrated.
• Responsiveness to local needs: While most sites run half-day programming three to four times per week, this is not a universal practice. Evaluation findings indicate that a number of sites have modified their programming schedule to facilitate improved accessibility to the program. For instance, in Ontario, parents believed Aboriginal Head Start programming should be run five days per week in order to mirror how kindergarten programs run in their community. As a result, the AHSUNC sites in Ontario now mirror local kindergarten schedules.
• Provision of transportation: Though not all sites provide transportation, parents at sites with transportation cited this as a key factor that supports their involvement in the AHSUNC program.

Finding 5. There are a variety of models across the country for delivering the AHSUNC program

While some sites operate as a project by themselves, without support from other programs, other sites operate within (co-located with) an early childhood education activity hub (Figure 9). In this situation, AHSUNC activities are still distinct, but complementary to a number of programs offered at the site. Some AHSUNC funding is used to enhance other programs in ways that complement the AHSUNC project goals, such as providing cultural activities or promoting parent participation.

Figure 9: AHSUNC within an “early childhood development activity hub”

Another model variation sees the AHSUNC project activities and funding being completely integrated within a broader early childhood development program (Figure 10). This situation is typically in partnership with provincial programs (such as Les Centres de la Petite Enfance in Quebec) and AHSUNC funds usually account for a relatively small percentage of the budget. Consequently, AHSUNC activities are less distinct in this model.

Figure 10: AHSUNC contributing financially and integrated within a broader early childhood development program

Data were not available to indicate how many sites operate under each model.

Program efficiency

This section will discuss efficiencies and inefficiencies in the delivery of the AHSUNC program.

Efficiencies of the AHSUNC program

Cost per child: For AHSUNC, the annual cost per child was $7,567^Footnote29 in 2010-11. Although identical comparators do not exist, the cost per child in the AHSUNC program appears to be within range of provincial early childhood education programs. For instance, the cost per child per year is $9,832 in Alberta, $12,827 in Nova Scotia and $10,011 in Ontario.^Footnote30

Program budgets are mostly spent: As previously mentioned, AHSUNC projects are funded through contribution agreements with eligible recipients. As outlined below, there are two types of contribution funding: ongoing project funding and supplemental funding. Within supplemental funding, there is a Regional Enhancement Fund and a National Strategic Fund. Appendix D provides additional detail on each of the funds.

Ongoing project funding

Ongoing AHSUNC project funding is managed by Regional Operations. Over the five year period examined, a total of about $145 million was allocated in contribution funding for AHSUNC projects, and this was entirely spent. An additional two million dollars was received in transfers from other programs within the Public Health Agency; of this, approximately half a million dollars lapsed.

Supplemental project funding

Regional Operations manages the Regional Enhancement Fund and the Centre for Health Promotion manages the National Strategic Fund, both of which are both described below.

Regional Enhancement Fund

The Regional Enhancement Fund provides time-limited funding to existing AHSUNC projects for a variety of purposes (e.g. training and education of frontline project staff, services for children with special needs, increasing program reach, networks and linkages with stakeholders, communication and support to communities, special studies, evaluation and streamlined reporting).

Over the five year period examined, a total of eight million dollars was allocated, and this was entirely spent. An additional one million dollars was received in transfers from other programs within the Public Health Agency; of this additional one million dollars, about one third lapsed.

Finding 6. Over the last five years, implementation of the National Strategic Fund has been challenging

National Strategic Fund

The National Strategic Fund provides time-limited funding to AHSUNC sites for projects that are national in scope. According to the National Strategic Fund Guide for Applicants, this is defined as meeting the needs of more than one region and providing benefits to many or all AHSUNC sites across Canada (e.g. promising practices in program delivery, training, support for special needs children, support for parental involvement, early childhood development services and assessment tools).

Over the five-year period examined, a total of $3.7 million was allocated. Of this, approximately 50 per cent was used to fund AHSUNC strategic projects managed at the national or regional level, 37 per cent was reallocated internally and about 13 per cent lapsed. Interviews with internal and external stakeholders indicated that this was partly due to a lack of capacity of sites to develop projects with a national scope in the short timeline of the solicitation process associated with this fund. It was also related to the challenges of implementing some initiatives at the national level.

Currently, only existing AHSUNC recipients are eligible to apply for this fund. Some interviews and documents suggested the possibility of expanding the eligibility criteria to allow national organizations to apply for the funding. National organizations might have existing capacity to carry out activities that could meet the needs of more than one region and provide benefits to many or all AHSUNC sites across Canada, such as conducting reviews of promising practices or developing broad-based training materials.

As part of this evaluation, a review of official documents describing how the Agency originally intended to use the National Strategic Fund when approved in 2006 indicates that some of the planned programming activities have not been initiated, for instance, research studies. However, the review also found that some national training initiatives and workshops were implemented and resulted in the accreditation of more project staff.

It may be timely to review the National Strategic Fund to ensure that the funds are optimally used to augment the quality of the program.

Finding 7. Resource leveraging and partnerships at the site level are strengths of the program. However, short-term funding agreements and staff turnover, as well as enrolment levels have led to some inefficiencies in program delivery

Project sites leverage funding and in-kind resources:The Public Health Agency encourages program recipients to raise funds from other sources and to seek in-kind contributions. Findings from the National Administrative and Process Evaluation Survey indicate that in 2009-10, sites received a range of in-kind contributions totalling $536,000.^Footnote31 Various in-kind contributions were received including:

• in-kind staff (68 per cent of sites)
• space and use of facility (34 per cent of sites)
• program materials and other donations (27 per cent of sites).

In addition to these in-kind contributions, program sites also reported leveraging over $6.5 million in funding. The leveraged funds enhance AHSUNC projects by, for example, extending the length of the program (extra day per week or full day), hiring an additional staff member and providing access to speech/language and occupational therapists to help support children with special needs. These leveraged funds came from various sources including:

• provincial funding (over $5,400,000)
• business, individuals, not-for-profit (approximately $500,000)
• fundraising (approximately $200,000)
• other federal government departments (approximately $500,000).

Approximately 73 per cent of sites reported leveraging provincial funds, while only 18 per cent of sites leveraged funds from other federal government departments. Approximately 39 per cent of program sites leveraged funding from business, individuals and not-for-profit organizations.

Combined, the additional resources received by the program through leveraging and in-kind donations total over seven million dollars. Therefore, although the Public Health Agency provides $32.1 million in funding through contribution agreements, these additional funds leverage the total to an equivalent of almost $40 million. In relation to comparable Public Health Agency programs (Community Action Program for Children, Canada Prenatal Nutrition Program) however, the AHSUNC program leverages fewer resources. For example, in 2005-06, for every dollar committed by the Public Health Agency for the Canada Prenatal Nutrition Program, the program leveraged approximately 53 cents. For every dollar committed by the Public Health Agency for the Community Action Program for Children, the program leveraged approximately 45 cents. The AHSUNC program leveraged approximately 20 cents for every dollar committed by the Public Health Agency.

Committed champions embedded in program:The program has benefited in large part due to the dedication of many committed site-level personnel across the program. Interview findings indicate that, although staff turnover is a challenge in many sites, there are a number of key dedicated individuals who have remained with the program from the start.

Inefficiencies of the AHSUNC Program

A number of areas for improved efficiency of the AHSUNC program were also identified in this evaluation.

Short-term contribution agreements with late funding announcements:Evaluation findings indicate that over the last five to six years, funding agreements with recipients for the AHSUNC program have typically been limited to one to two years in length. In addition, actual funding announcements for renewal of funding agreements with recipients have been communicated ‘last minute’. These findings were cited by interviewees as factors influencing the efficiency of the AHSUNC program as there is a feeling of uncertainty among recipients and site coordinators about the stability of ongoing funding. As a result of this uncertainty, combined with the fact that there has not been an increase in contribution agreement funding over the last decade, staff turnover at AHSUNC program sites has become a challenge.

Varying enrolment levels:While some sites have waiting lists for enrolment, others are not at full capacity. For example, during the case study visits, none of the 10 visited sites was at full enrolment and some sites reported a drop in enrolment over the last few years. According to self-reported project performance data for 2009-10, AHSUNC functions at 91 per cent capacity, meaning that based on all of the AHSUNC spaces available in Canada there is room to accommodate nine per cent more children. Performance data from this same year also indicate that 23 per cent of the sites function at less than 80 per cent of their enrolment capacity.

High project staff turnover:Staff retention has been a concern for the last five years, especially for trained early childhood educators, because, in addition to the uncertainty mentioned above, AHSUNC centres usually cannot pay wages competitive with other employers. Literature and document review findings indicate that salaries for early childhood educators employed by the AHSUNC program are lower than most provincial early childhood educator salaries. Case study findings revealed that, without stable staffing, training investments (mostly paid for by the Public Health Agency) do not always remain in the program. In addition, program coordinators described how the benefits of stable staffing extend beyond the site because the lessons learned by long-term employees over several years can be shared with other sites. High staff turnover may threaten this knowledge exchange function. In the 2010-11 National Administrative and Process Evaluation Survey results,^{Footnote 32} staff turnover was reported as a concern in 50 per cent of sites (Figure 11). The staff turnover issue was also consistently raised during interviews with program staff and stakeholders.

Figure 11: Percentage of AHSUNC project sites where staff turnover was a concern (n=113)

Source: Public Health Agency of Canada 2010a; Public Health Agency of Canada 2009a

Finding 8. The advisory function that supports program governance could be improved

There is an overall lack of clarity and different perceptions regarding the role of the National Aboriginal Head Start Council.

Originally, the National Aboriginal Head Start Council was intended to provide advice to the Public Health Agency on the AHSUNC program vis-à-vis:

• Aboriginal early child development, education, health, psychology and spirituality
• major issues, concerns and structures of Canadian Aboriginal communities
• management of Aboriginal organizations and community-based programs for Aboriginal peoples
• Aboriginal women’s issues.

In this role, the National Aboriginal Head Start Council was to have provided advice to the Public Health Agency on policies, procedures, services and programs related to AHSUNC (curriculum development, standards for program performance, research priorities and cultural and traditional values).

Evaluation findings indicate that the National Aboriginal Head Start Council is not currently serving the role that was originally intended. Through questionnaire feedback and a group interview, National Aboriginal Head Start Council members expressed concerns about the lack of formal mechanisms and/or current processes in place to facilitate their contributions to program improvement. National Aboriginal Head Start Council members felt that there has increasingly been a lack of engagement by the Public Health Agency on decisions related to the program (e.g. training). Further, when given opportunities to provide insight on decisions related to the program, National Aboriginal Head Start Council members felt that their input was often not taken into consideration.

Management and staff at the Public Health Agency also expressed concerns about the lack of clarity of the National Aboriginal Head Start Council mandate as well as concerns about the effectiveness of the current advisory function.

Possibly contributing to the lack of clarity around the role of the National Aboriginal Head Start Council is the composition of the Council. As per the Council’s terms of reference, the membership is as follows:

• AHSUNC project staff: one community representative from each province and territory that has AHSUNC projects (project directors, coordinators, or educators are elected by AHSUNC bodies in their province or territory)
• one First Nations, Métis, or Inuit Elder from the community in which a national meeting is held
• three ex-officio positions for Public Health Agency staff: one representative (manager) from the Division of Childhood and Adolescence and two representatives from Regional Operations: one program consultant and one children’s programs manager.

Several findings suggest that this composition may not be ideal. If the Public Health Agency requires advice on, for example, education, women’s issues and management of Aboriginal organizations (as noted in the original Council terms of reference), membership of the Council may need to be broadened. Another concern is that the members of this Council are direct recipients of project funds, which could result in conflicts of interest. Although no actual instances of conflict of interest were observed, there was no evidence of mitigating strategies in place for this potential. Finally, having one representative from each province creates an imbalance with respect to the number of sites each member represents.

Program reach

Program documents indicate that the AHSUNC program is intended to reach First Nations, Inuit and Métis preschool-aged children living off-reserve. Based on Statistics Canada’s 2006 Aboriginal Children’s Survey and program enrolment data for 2010-11, the program is reaching approximately eight per cent of the off-reserve Aboriginal population of three to five-year olds (Figure 12). Within the eight per cent of off-reserve Aboriginal children reached, First Nations, Inuit and Métis preschool-aged children are all represented.

Information from multiple sources suggests that the program has the potential to reach more children. At the time of our research, none of the case study sites (n=10) was at full enrolment. Some of the sites have experienced a drop in enrolment over the last few years. In two provinces, coordinators mentioned that the introduction of provincial and territorial full-day learning initiatives may have attracted children who might otherwise have attended AHSUNC. One of the affected sites negotiated with the province and local school board to gain funding for full day kindergarten at their site. This decision extended the reach of the AHSUNC site because the culture and language teachings that were developed for the AHSUNC participants are being extended into the kindergarten program. The case studies did not include interviews with Aboriginal families that have chosen not to enrol their children, or who have withdrawn, so it is difficult to know why the sites are not all full. Some parents who participated in the case studies indicated that, because they work, it can be challenging to have their child attend a program that either runs a few days a week or a few hours a day.

In contrast to the sites that are not full, in 2009-10 close to 50 per cent of AHSUNC sites have a waiting list, which suggests that there is higher demand in some communities than is currently being met. The following figure 13 illustrates the concentration of sites with waitlists and communities with a critical mass (off-reserve communities with sizeable population of Aboriginal children aged zero to six).^Footnote33

Figure 13: AHSUNC sites by waiting list count 2009-10 in relation to off-reserve municipalities with sizeable populations of Aboriginal children aged zero to six

Source: Public Health Agency of Canada 2011

Enlarge Figure 13

The geographical distribution of AHSUNC site locations illustrates unmet need in several Canadian municipalities with a critical mass of Aboriginal children. According to the documentation, AHSUNC reaches 79 of the 134 off-reserve communities with critical mass, or 59 per cent of key target geographic areas. The top six most populated communities with critical mass are reached with at least one site. The following figure (Figure 14) illustrates the ten most populated municipalities without a site. At least eight of these municipalities are reached by the Community Action Program for Children or the Canada Prenatal Nutrition Program. Culturally appropriate activities are offered to Aboriginal participants.

Figure 14: Off-reserve municipalities with critical mass and no AHSUNC site within pick-up distance (60 km): Ten most populated with Aboriginal children aged zero to six

Source: Public Health Agency of Canada 2011

Enlarge Figure 14

When considering efficiency and cost-effectiveness, it is important to consider the factors that affect the quality of the program and its ability to have an impact on the desired outcomes. Research has shown that program success is affected by a number of factors, including program intensity and program quality in terms of staff and curriculum design. With respect to program intensity, dose refers to how often and how much of an intervention has been provided within a set intervention period. Research has demonstrated that a certain minimum dosage of an early childhood intervention is required before quality can be linked with child outcomes and a certain threshold of early childhood education quality needs to be met before positive outcomes for children are seen.^{Footnote 3434} Programs with a measurable impact are those that are more intensive, have more qualified and higher paid teachers, have higher teacher to child ratios (meaning smaller class sizes and more individual attention) and have more time in the classroom over the entire school year.

In other words, program design could be modified to increase reach, such as rotating children on a seasonal basis or using funds to offer summer camps to a larger number of children. However, literature review findings indicate that outcomes can only be observed following a minimum exposure to the program. Research demonstrates there is a minimum amount of exposure to a program required before the program can have an impact. Therefore the evidence does not support reducing program intensity to increase reach.

Coordination

Early childhood development is a multi-jurisdictional issue, with diverse and numerous stakeholders involved in a variety of different functions. The AHSUNC program was founded on the principle of coordination between the federal and provincial/territorial governments and integration with provincial/territorial early childhood development systems.

The international literature review illustrated how national governments in Australia, New Zealand and the United States have introduced frameworks that provide a strategic approach for improving the quality of early childhood education for Indigenous or at-risk children. In each case, these governments have taken a leadership role in disseminating current research and developing program performance standards along with culturally appropriate resource materials (e.g. a bi-cultural curriculum), with a vision for broad implementation of the approach in early childhood and education systems.

The evaluation found that there are varying levels of coordination within the program.

Within the Public Health Agency

The AHSUNC program appears to be well coordinated with other children’s programs within the Public Health Agency, Centre for Health Promotion.

In terms of target population, the Community Action Program for Children and the Canada Prenatal Nutrition Program are strong complementary programs to AHSUNC in terms of reach to Aboriginal children and their families. While AHSUNC is targeted specifically to Aboriginal children, both the Community Action Program for Children and the Canada Prenatal Nutrition Program serve a broader population that includes Aboriginal participants. All three programs are intended to be targeted to a considerable extent to ‘at risk’ clients.

As indicated in Figure 15, key findings from previous evaluations include:

• Evaluation reports have found that 23 per cent of the Canada Prenatal Nutrition Program participants and 13 per cent of Community Action Program for Children participants are Aboriginal which is considerably higher than the proportion of Aboriginal people in the population (four per cent).
• The numbers of Aboriginal clients reached in the Canada Prenatal Nutrition Program (11,500) and the Community Action Program for Children (5,159) are actually larger than the number reached through AHSUNC (4,650). The proportion of Aboriginal participants in the Community Action Program for Children varied considerably across Canada, from a high of nearly 78 per cent in Nunavut, to nearly 70 per cent in Manitoba, 58 per cent in Saskatchewan and 37 per cent in the Yukon.

In terms of overall reach, the distribution of the three programs across Canada by type of geographic area varies. In many communities (especially in larger centres), there are other similar types of programs for pregnant women and young children that are funded by provincial and territorial, municipal or non-governmental sources. This is recognized in the Public Health Agency’s funding strategy with respect to the Canada Prenatal Nutrition Program, which targets its prenatal activities to address the ‘gaps’ in services available. As a result, the distribution of its projects is not based on population size and includes more rural, remote and northern locations.

As depicted in Figure 16, the Public Health Agency’s three maternal and child health programs reach 84 per cent of the off-reserve communities with a critical mass of Aboriginal children. These communities may have one, two, or all three programs available.

Figure 16: Reach of critical mass communities by the Public Health Agency’s programs

Source: Public Health Agency of Canada 2011

While there are clearly connections among these three Public Health Agency programs in that they all provide services to Aboriginal mothers and children at different phases of a child’s development from birth to age five, they do not necessarily provide a ‘continuum’ of programming for Aboriginal children within any specific location. To the extent that two or three of these programs are available within a community and given that all programs give emphasis to lower-income, young and single parents (‘at risk’ populations), it would be reasonable to expect that some of the same Aboriginal mothers and their children may have participated in two or three of these programs. For example, a young pregnant woman may have received support before and after the child’s birth from the Canada Prenatal Nutrition Program (generally for four months before and four months after birth), then attend activities with a young child or toddler through a Community Action Program for Children and a three-year old child could be in Head Start. However, no data were available in the reports examined on the extent to which Aboriginal children and families may receive services from two or more of these programs.

Within the Health Portfolio

Based on interviews and documents reviewed, there does not appear to be any formal operational connection between the two streams of Aboriginal Head Start programming within the Health Portfolio. Meetings occur infrequently and on an ad hoc basis. No formal coordinating mechanisms were found to be in place at any level. As there is mobility of First Nations families and children between urban areas and their First Nations communities, there is a need for coordinated approaches to early childhood development to achieve successful health and social outcomes. Linkages between these two programs could be important to provide continuity and to ease the transition of Aboriginal Head Start graduates to the mainstream school systems.

The evaluation found no evidence of formal links to knowledge development partners within the Health Portfolio, such as the Canadian Institutes of Health Research, or the National Collaborating Centre for Aboriginal Health which receives transfer payments from the Public Health Agency. While linkages to the National Collaborating Centre for Aboriginal Health occur at the site level through project-level networking and information-sharing activities, intentional and systematic linkages to the National Collaborating Centre for Aboriginal Health within the Public Health Agency could not be found. This results in an inconsistent and ad hoc approach to knowledge development and exchange activities which lacks strategic direction and integration with program directions.

Within the federal government

The evaluation revealed no evidence of systematic coordination between the Public Health Agency and other federal departments delivering similar programs, namely Aboriginal Affairs and Northern Development Canada as well as Human Resources and Skills Development Canada. Within the Public Health Agency’s Regional Operations, provincial or regional advisory committees are established to provide advice and guidance on the implementation of the AHSUNC projects. Federal partners may be included in some of these committees, but this is not consistent across the country, nor systematically coordinated at the national level to ensure federal dialogue and exchange at the program level. Furthermore, there is no evidence of the Public Health Agency taking part in partnership tables chaired by other federal departments.

Within the case study sites, there were very few formal linkages identified between AHSUNC site staff and federal government departments other than the Public Health Agency. Of the 10 sites visited, none reported formal links to Health Canada’s Aboriginal Head Start On-Reserve program. One site had linked with two federally funded programs: a “language nest” program that promotes Aboriginal language revival (Aboriginal Languages Initiative funded by Canadian Heritage) and the Aboriginal and Inuit Child Care Initiative (funded by Human Resources and Skills Development Canada).

Within provincial and territorial departments and others

Within each province and territory, there is a network of early childhood development service providers, including Aboriginal organizations involved in programs for children and families, which can provide complementary services to the AHSUNC program. Specialized services for off–reserve Aboriginal communities exist in many areas as well as programs for the non-Aboriginal population. As a result, AHSUNC is one element in a much larger network of programming to support children and their families.

The number and diversity of players and sectors involved in early childhood development requires a high degree of coordination at the provincial and territorial level. To varying degrees, there are coordinating mechanisms in place to address overlaps and duplication among programs. However, the evaluation found no evidence of systematic coordination between the Public Health Agency’s AHSUNC program and other provincial or territorial government programs. The exception to this would be the inclusion of provincial and/or territorial representatives in some, but not all, provincial and regional AHSUNC advisory committees.

At the site level however, there is evidence of consistent partnering with local service providers, health authorities and other early childhood development stakeholders. All case study sites had formal links with the local public health authority (Figure 17). The benefits of this relationship for the AHSUNC children were clear, including the provision of vision, hearing and dental screening, as well as information and training related to injury prevention and nutrition. For example, during the H1N1 epidemic, a local public health authority opened an immunization clinic at the project site to increase access for Aboriginal people. As a result of the effectiveness of this community response, this location has become a regular site for the flu vaccine campaign.

After provincial and territorial government partnerships, the next most common link for AHSUNC has been with municipal or Aboriginal governments. This level of government has been a source of in-kind or financial contributions to several projects. For example, local governments have donated space or provided wage enhancements for staff.

Figure 17: Number of case study sites that reported partnerships (n=10)

Enlarge Figure 17

Several sectors are not involved in most project sites, including academic research institutions, corporate organizations and philanthropic organizations, although this result may be attributable to the open-ended nature of the interview question posed. In addition, links with early child development initiatives or service providers appeared limited at both provincial and federal levels. While AHSUNC sites described extensive links with each other, only three described linking with coalitions or local networks of early childhood development program providers.

Performance measurement

The creation of performance measurement and evaluation positions specific to AHSUNC and the corresponding AHSUNC evaluator’s network (Aboriginal Head Start Evaluation Network) occurred in 2007.

The number of full-time equivalent employees dedicated to AHSUNC performance measurement (not including the full-time equivalents associated with site monitoring) was approximately seven during fiscal year 2010-11.

The decision to centralize the evaluation function in the Public Health Agency in 2009-10 has affected the roles and responsibilities of these resources. Appendix G provides an overview of the evaluation and performance measurement activities that have been conducted since 2005‐06.

Over the past five years, a wealth of information has been collected about the AHSUNC program and its participants. More than 20 AHSUNC performance measurement or evaluation initiatives have been conducted since 2005-06. Many of these studies were initiated and implemented at the local and regional levels (e.g. by provincial AHSUNC committees) and they were mostly designed to respond to local or regional needs. The diversity and depth of these initiatives reflect the community‐driven aspect of the program and have contributed to local capacity building. Each of these studies was reviewed for this evaluation and was found to be methodologically sound. In addition, although no evidence was provided on this, these initiatives appear to have the potential to serve the program as tools for knowledge exchange and continuous improvement.

At the same time, interviews and the review of these reports revealed that there is a need for a more consistent pan‐regional and national approach to performance measurement. There was no evidence of an overarching framework that guides periodic studies and demonstrates a strategic approach to assessment of intermediate and long-term outcomes.

In reviewing the regional performance measurement documents produced over the last five years, evaluators have concluded that the measurement of children’s short- and medium-term outcomes for school readiness is consistent and of high quality. However:

• long-term measures of outcomes related to children’s well-being are insufficient
• outcomes related to parental impact, knowledge exchange and community capacity have not been systematically measured
• focussed studies that advance knowledge of the conditions that promote desired outcomes (e.g. what contexts and factors promote parental involvement and what impact does this have of child and family outcomes) have not been conducted.

Although extensive performance data are collected about activities, processes and reach, there was limited evidence of dissemination of this information internally and externally.

The performance measurement function is well entrenched in the delivery of the AHSUNC program. Additional emphasis on timely dissemination of results and knowledge exchange will further increase the value of this function to the program as well as the efforts to streamline the function and improve its efficiency by reviewing the effectiveness of, for instance, the data collection methods at the site level.

Plans and reports as well as interviews with performance measurement staff indicated that future performance measurement initiatives will focus on streamlining existing tools and processes and using evidence from performance measurement or intervention research initiatives to support timely and effective knowledge transfer and exchange.