PART II

Centre:   Parents Place (Yarmouth Family Resource Centre). Yarmouth, NS Artist:   Amanda Description:   When I go to Parents Place I play with toys and crafts. My Mom likes Holly. My brother likes playing with the toy and making crafts. At Parents Place you have to use your manners. Be polite. No fighting. I like to do art.

PART II: DESCRIPTION OF THE ATLANTIC

2.0 THE ATLANTIC CAPC REGIONAL EVALUATION

CAPC is built on a developmental and collaborative process undertaken with community organizations. Building each community’s unique capacity to respond to the needs of families and children at risk is a critical component of this process.

In Building Communities from the Inside Out, Kretzmann and McKnight (1993) developed the idea that community development is based on capacity-focused development. They point out that all historic evidence indicates that significant community development takes place only when local community people are committed to investing themselves and their resources in the effort. To achieve long-term success and continuity, projects must be seen to be valuable and important at the local community level.

The Community Action Program for Children was designed to increase the empowerment and knowledge development of families and communities through activities that are educational and that help prevent health problems. CAPC’s aim is also to make possible their active participation in designing programs, putting them into practice, and making decisions about them. Community partnerships strengthen this process by allowing people to work on issues of mutual concern and by integrating efforts that are directed toward common, community-based goals.

Participatory action research (PAR) offered an approach to evaluation that was very much compatible with CAPC’s principles and goals, using collective reflection on an issue as a basis for further action.

Community acceptance, involvement, support, and active participation in this evaluation process all were necessary for the success of the Atlantic CAPC Regional Evaluation. Because participatory evaluation requires that the people being evaluated are also those doing the evaluation, it offered a way to ask questions while allowing the community to take ownership and control of the research process.

In keeping with this philosophy, the Atlantic CAPC Regional Evaluation design was based on a participatory action research model that encouraged a cooperative approach and fostered broad-based training, communication and collaboration, as well as community-level participation, empowerment, and action. The design also allowed for active, multi-level involvement at all stages of the evaluation.

2.1 RATIONALE FOR THE PARTICIPATORY APPROACH

The regional evaluation framework was based on a participatory approach that viewed evaluation as a shared process in which people gathered information about themselves and their projects. This holistic, participatory approach contained an inherent commitment not only to participation, but also to building the capacities of individuals, projects, and, ultimately, communities. Communities are strengthened when collective commitment, resources, and skills are increased.

To reflect the community development approach that is at the heart of CAPC, the regional evaluation design was structured to recognize and support

• the value of the reflection, discussion, and processing of information related to program issues that inevitably arise as a result of ongoing communication;
• the individual feelings of value and worth that emerge from communicating, learning, and having a decision-making role; and
• the inclusion of a diverse group of stakeholders, which provides a voice for program users and increases awareness of issues and concerns at the program management level.

This approach was based on the work of other researchers (e.g., Rossi and Freeman, 1989) and was adopted to increase the likelihood that communities would be able to sustain what was learned beyond the evaluation period.

Communication and training were formally integrated to facilitate active participation in this evaluation. This integration represented a unique and critical aspect of the Atlantic CAPC Regional Evaluation. It was expected that emphasizing these supportive processes would contribute to the development of long-term learning and communication networks. In turn, they would build and enhance the capacities of projects to help and support children and their families and communities and which would be sustained beyond the lifetime of this evaluation.

2.2 OVERVIEW OF THE CAPC EVALUATION

The CAPC evaluation was conducted at three levels: national, regional, and local, with the Atlantic CAPC Regional Evaluation serving as a link between the other levels.

National Evaluation
The process of developing a national evaluation framework began in January 1994. This framework was intended to capture the minimal information required to evaluate the program on a national basis. The National Evaluation used standardized, quantitative tools to examine the development and implementation of all CAPC projects, as well as any benefits that could be attributed to the projects. A series of data collection forms (National Forms A to E) were used to collect comparable data from CAPC projects across Canada. This method made it possible to combine information about the impact of CAPC in a way that enabled local sponsors to have a common basis for communicating with the government.

Regional Evaluation
Discussions around an Atlantic CAPC Regional Evaluation began in January 1994. Shortly thereafter, it was decided that a regional approach was preferable because

• all four provinces were funding similar projects;
• and a single evaluation was more cost-effective.

In January 1995, work began on the development of a framework for the Atlantic CAPC Regional Evaluation, and in November 1995, the design and implementation of the evaluation began. Atlantic CAPC Regional Evaluation data was collected from September to November 1996. The Atlantic CAPC Regional Evaluation used qualitative methods to balance and complement the national approach, which allowed those involved to tell the story of their CAPC experience in their own words. The richness of such stories adds depth and context to the quantitative information, but more importantly, it allows the voices of the people involved to form the basis of program understanding, decision making and action.

The Atlantic CAPC Regional Evaluation was participatory, reflecting a shared process among all involved in the evaluation. This participatory approach provided a link between the regional and project-level evaluations. Methods developed for use at the regional level were based on consultations with project staff and participants and then shared so that individual projects could consider what was happening at the regional level and adjust activities at the project level accordingly. This approach allowed projects participating in the Atlantic CAPC Regional Evaluation to share in evaluation activities and to apply new skills and/or ideas to the development of local project and program evaluation activities.

The approach also allowed this evaluation to examine broad patterns and trends within Atlantic Canada. The data presented is not identified with any particular project, and no comparative analysis was done. As agreed by the JMC/PACs, data was not analyzed to make comparisons between provinces; however, the data from both the national and regional evaluations was collected and coded to allow each province to undertake any further analysis it wanted to.

Project Evaluation
Health Canada requires that projects conduct their own evaluations at the local level. To enhance the education and skill building necessary to the participatory process, two training workshops in evaluation methodology and related topics were conducted -- one at the provincial level and one at the regional level. These workshops were designed to build on existing skills within projects, enhancing their ability to carry out their own evaluations.

Specific training needs were explored with project stakeholders through consultations with all 40 CAPC projects. The workshops were then developed to meet these expressed needs and interests. In addition to the training, the Atlantic CAPC Regional Evaluation team provided ongoing technical support to projects throughout the evaluation period.

2.2.1 Putting the Evaluation Pieces Together

As CAPC was evaluated at three levels, national, regional, and local, the evaluation, with its overlapping components was complex. The overall context of this evaluation is shown in Figure 1.

FIGURE 1: The Atlantic CAPC Regional Evaluation

As Figure 1 illustrates, the impact of these overlapping processes for any given individual could be multi-dimensional. For example, a CAPC parent could have been actively involved in one or all three of the following aspects of the Atlantic CAPC Regional Evaluation mentioned below.

2.3 EVALUATING THE REGIONAL EVALUATION PROCESS - EVALUATION WITHIN THE EVALUATION

The Atlantic CAPC Regional Evaluation took place within the context of the National Evaluation, with which, in part, it coincided. Within this context, the Atlantic CAPC Regional Evaluation process had three components:

• the evaluation itself; evaluation activities specific to designing, collecting, analyzing, and interpreting information about the projects;
• the training of project staff and participants to assist in their local evaluations; to facilitate their understanding of, and participation in, this evaluation; and to address their needs related to project development; and
• the communication network, including the development of tools to support both this evaluation and the projects.

Because the interaction of these components was expected to impact at all levels -- on parents, projects, communities, and governments (both provincial and federal) -- an evaluation of this impact was embedded in the Atlantic CAPC Regional Evaluation plan. This "evaluation within the evaluation" examined the impact that participating in all three components of this evaluation had on those involved. Each project was provided with feedback forms containing questions on the three components. Follow-up focus groups were held in September, 1997. Each component is described below.

2.3.1 Evaluation Itself

The evaluation itself, as outlined above, is described in detail throughout this section.

2.3.2 Training

The training component of the evaluation plan had three focuses:

• to build on existing strengths, skills, and abilities of people in CAPC projects, enhance their knowledge of evaluation and increase their comfort level in doing the evaluation;
• to increase the capacity of projects to conduct their own evaluations, increasing self-sufficiency and appreciation of their own skills and experiences; and
• to increase the capacity of projects regarding local developmental and organizational needs.

Emphasis was placed on demystifying the concept of evaluation and enhancing knowledge and skills to enable CAPC projects to conduct their own local evaluations. People involved with a project could then learn to judge their own success and apply that knowledge. The training component was developed through consultations with projects to identify specific topics that would be addressed in workshops. These workshops used an interactive approach that emphasized learning by doing and incorporated a variety of learning approaches, including small group discussions, use of participants’ expertise, application of learned concepts, and consideration of how to use these skills in CAPC work.

Provincial workshops covered the basics of participatory evaluation, data collection, and data analysis and were attended by each CAPC project’s coordinator, a staff person, a parent, and another representative. Participants were actively involved in exploring the benefits of evaluation and how to do one-on-one interviews, conduct focus groups, and organize the information after it is gathered.

The regional workshop, held in May 1996 at Gros Morne National Park, Newfoundland, included a minimum of two representatives from projects in all four provinces. This workshop built on the learning outcomes from the provincial workshops, with more training offered on qualitative and quantitative evaluation methods. In addition, the regional workshop participants could address their developmental and organizational interests and needs through focused, skill building workshops in a variety of areas.

The effectiveness of learning at both the provincial and regional workshops was assessed through participant feedback forms on each session and follow-up focus groups. Observations by the trainers during and after specific workshops were used to make changes to subsequent workshops.

2.3.3 Communication

Participatory evaluation is a two-way process in which information is exchanged between the people doing the work and the people who collectively are to benefit.

Several means of promoting and establishing communication were established to facilitate the open and meaningful communication necessary at all levels of this evaluative process. The regional evaluation team maintained ongoing communication with all 40 CAPC projects through regular contact with the provincial evaluation coordinators. A toll-free line was also established to address any issues or concerns raised by projects about the Atlantic CAPC Regional Evaluation process or project-level evaluation. Fact sheets were developed to update projects on this evaluation process or to elicit their participation or feedback at a specific point in the process. The Regional Evaluation Team also maintained regular contact with the ACES Management Team and ACES through scheduled meetings and conference calls.

Participants in the regional evaluation training workshop had an opportunity for focused training in communications. In addition to the workshops, participants were offered training on basic e-mail usage, and members of the Regional Communications Committee were offered training on electronic newsletter production.

A communications network was set up and managed by the Regional Communications Committee and included e-mail and a regional newsletter. A proposal was developed for a CAPC home page with links to related sites on the Internet.

These communications tools were developed to enhance communication between Atlantic CAPC projects, share evaluation knowledge, access information, and offer mutual support and provided a mechanism through which the Regional Evaluation Team could promote this evaluation and elicit feedback from participants in the evaluation process. Communications were also used to enhance the visibility of CAPC and to support all those involved in their efforts to promote the program.

Evaluation of the communications component was obtained through feedback forms, monitoring of usage of the toll free line, and focus groups held in September 1997.

2.4 PARTICIPANTS IN THE ATLANTIC CAPC REGIONAL EVALUATION

In keeping with the community-development approach, the Atlantic CAPC Regional Evaluation sought to involve representatives of all those involved in the CAPC projects in Atlantic Canada --parents, volunteers, staff, board members, and community partners, as well as the provincial Joint Management/Program Advisory Committees, the Atlantic CAPC Evaluation Sub-committee (ACES), and the ACES Management Team. This involvement of all parties in the multi-level evaluation process is summarized below and detailed in Appendix I. As well, the names of those involved are listed in Appendix V.

Regional Evaluation Team
The Regional Evaluation Team included about 20 professionals, some of whom were contracted at different times depending on the expertise required to carry out the evaluation. Their role was to

• develop and implement the Regional Evaluation framework;
• implement the National Evaluation framework for Atlantic Canada; and
• support CAPC-funded groups in their project-level evaluation efforts

Projects
All 40 CAPC projects were involved in an initial consultation process that determined the key indicators and the focus of the Atlantic CAPC Regional Evaluation. In the implementation phase, project staff, using established selection criteria, identified parents, board members, volunteers, and community groups as potential participants in the interviews and focus groups. In addition, when the results became available, projects participated in confirming the findings and in providing input into initial conclusions and recommendations.

Provincial Project Evaluation Teams
The Provincial Project Evaluation Teams (PPET), were established to provide additional involvement in this evaluation at the project level. PPETs in each province were made up of one project coordinator and three project participants. These teams participated in the first round of data analysis at the provincial level, helping to validate initial themes and selecting key quotes from participant interviews. They also participated in focus groups verifying the findings and reviewing preliminary conclusions and recommendations. An Aboriginal PPET was organized to explore any cultural differences and highlight any specific themes from the Aboriginal parent data.

Atlantic CAPC Evaluation Sub-Committee (ACES)
The Atlantic CAPC Regional Evaluation was one of the activities of the Atlantic CAPC Joint Management Committee (JMC), which is made up of two representatives from the JMC/PAC in each province, as well as federal representatives. It is responsible for the overall management of the program in Atlantic Canada. A subcommittee, the Atlantic CAPC Evaluation Sub-Committee (ACES), made up of federal, provincial, and project representatives from each province, was established to coordinate and monitor this evaluation process in Atlantic Canada.

ACES provided direction, guidance, and advice throughout all stages of this evaluation process and was responsible for the approval of all regional evaluation plans and outputs.

ACES Management Team
Six members of this larger ACES committee were selected as a management team to oversee the evaluation contract on a day-to-day basis by working closely with the Regional Evaluation Team to resolve issues and challenges that arose, discuss progress, review documents, and advise the team throughout the evaluation.

The project was managed through regular meetings between the ACES Management Team and the Regional Evaluation Team that were used to approve approaches taken and to review reports, facilitate discussion and make decisions.

2.5 DEVELOPING THE KEY INDICATORS TO BE ADDRESSED BY THE EVALUATION

The Atlantic CAPC Regional Evaluation drew on multiple sources and on the contributions of many people to develop the key indicators to be addressed by this evaluation. The Regional Evaluation Team gathered information by reviewing documents, conducting consultations, and examining evaluation strategies.

2.5.1 Document Review

To begin, the Regional Evaluation Team undertook a detailed review of project documentation, including

• Program objectives and outcomes from the Request for Proposals: Atlantic Region Evaluation,
• Protocols developed by each of the provinces,
• Review of project proposals,
• The National CAPC Evaluation Framework and related tools, and
• The Regional Evaluation Framework for CAPC-Atlantic Region, a document produced through a participatory process to provide a basis for soliciting evaluation contract proposals and which also contributed to the evaluation plan.

2.5.2 Consultation Process

Between January and May 1995, a Regional Evaluation framework was developed based on a participatory process involving project representatives (staff, parents, and board members). These representatives, through interactive workshops, identified key issues to be addressed within the evaluation framework referred to in Section 2.5.1 (directly above).
Four major evaluation topics emerged from this consultation:

• the extent to which projects enhanced participants’ skills and knowledge;
• the extent to which projects encouraged a sense of community among parents and increased their confidence in their ability to parent;
• the extent to which projects created empowering environments and a sense of project ownership; and
• the position of the project in the larger community.

In November 1995, a consultation was conducted with ACES in which committee members were asked to identify and rank research areas for the evaluation. They identified these areas:

• changes in behaviour;
• level of community support;
• partnerships;
• a profile of the participants;
• the development of knowledge and skill;
• ownership; and
• program functioning.

During January and February 1996, consultations with all 40 projects were undertaken. In group sessions, parents, staff, and board members were asked how they would know if their project was working.

In response, participants said that they would know the project is working if

• they are changing, going through a process - if there is movement
• they can identify what skills they are gaining
• they accept ownership and leadership
• they develop partnerships with the community
• there is a feeling of "identity" in the centre.

All comments made by focus group participants were considered, and five categories of program success were identified:

• personal development of parents, children, and families;
• involvement of participants in the project/program;
• participant ownership of the project;
• community partnerships and linkages; and
• project/program development and management.

Appendix III contains tables incorporating these five categories along with the related,project-generated indicators of success.

2.5.3 The Need to Evaluate both Process and Impacts

It became evident that to more fully understand how effective CAPC projects were and how they contributed to change, it was important to understand both the process through which the projects worked and the impact they had. In other words, this evaluation needed to address not only what the projects did, but how they did it to fully appreciate the project’s impact on participants and others involved.

2.5.4 Evaluation Questions

The results of these consultative processes, along with the national and regional objectives of CAPC and the national evaluation framework, provided the basis for developing the evaluation questions. The primary evaluation question was

Is there improvement in the health and well-being of at-risk children, age 0-6, and their families, in the Atlantic Region?

Whether CAPC ultimately had a positive effect on the health and well-being of children was at the centre of this question. Therefore, many CAPC projects tried to influence the child’s well-being both directly, through children’s programming, and indirectly, by targeting their efforts on improving parental skills through the provision of education, prevention, and intervention programs.

Because much of the focus of programming was on changing the factors and attributes that surround the child, the evaluation focused on gathering data from parents and others involved, rather than from the children themselves. Parents and others were asked to describe how they perceived any changes in their children.

Another reason for taking this approach was that specialized training was required to evaluate the kind of information that could be gathered from young children (e.g., through drawings and stories).While the contributions of children have been represented in this final report by including some of the pictures they drew about CAPC and excerpts from their own descriptions of the drawings, no effort has been made to interpret the drawings. These pictures and accompanying quotes have been incorporated throughout the report to direct the reader to the experiences of children with CAPC.

In addition, the long-term effects of these projects may take years to become evident. Thus, the emphasis of the evaluation was on understanding how the program met its objectives and on gathering information about the process parents and children underwent as participants in the program and its impact on families and communities.

2.6 EVALUATION FRAMEWORK

The original Atlantic CAPC Regional Evaluation framework had been structured around several key areas in terms of both process and impact. Based on this, it was initially planned that evaluation findings would be reported under the categories of individual change, shared change, program/project development, and community partnerships.

However, an analytical scheme that better reflected the data as it emerged during this evaluation process resulted in the categories being changed to implementation and process, individual impacts and effects, impacts and effects of adopting a community development approach to programs and services, and community partnerships.

The tables containing the specific questions addressed by this evaluation in each of the categories are also included in Appendix III.

2.6.1 Implementation and Process

This category examined the process through which the project had been planned and implemented and how it evolved. It considered project delivery structure in terms of

• the programs that had been developed;
• who was being served;
• how effectively projects were meeting the needs of parents and children;
• whether or not the project had succeeded in what it intended to do;
• what was working well; and
• what was not working well.

These criteria were examined, in part, through program records and also in descriptive terms through the experiences of the people involved in the projects (parents, staff, volunteers, board members). Their different perspectives provided insight into project approach, development, successes, and challenges.

2.6.2 Individual Impacts and Effects

CAPC’s impacts and effects were reflected in the changes experienced by the people who participated. In particular, this evaluation looked for indicators of positive change and levels of satisfaction with the projects.

Using a qualitative approach that allowed project participants to tell their stories encouraged reflection on what had happened, how circumstances had changed, and what had been learned. This provided valuable insights into how CAPC projects affected the lives of parents, staff, and others involved.

2.6.3 Impacts and Effects of Adopting a Community Development Approach to Programs and Services

CAPC projects were designed to create a positive environment in which to strengthen the capacities of project participants as a group. A commitment to a participant-driven community development approach to services and programs was central to this objective. Through this approach, participants (parents, board members, volunteers, staff) were able to become more knowledgeable about the project and each other, to establish trust, and to work collaboratively toward mutual goals.

2.6.4 Community Partnerships

Community partnerships were an integral component of CAPC goals and protocols. These linkages established a foundation for future community development and capacity-building. Collaboration and the integration of services provided broad community support for families and children. In addition, empowerment was facilitated because groups worked together toward common, community-based initiatives with decision making and planning carried out at the community level. Linkages among community groups and organizations directly influenced the functioning of projects and the experience of the individuals participating in them. To foster successful cooperation and sustainability, we had to determine what and how community partnerships developed, what collaborative strategies were effective, and what challenges existed.

 

[Previous] [Table of Contents] [Next]