5.0 Limitations of the Methodology

Limitation 1: National Sample Evaluation Bias
The Atlantic CAPC Regional Evaluation was dependent on the National Evaluation for its sample frame, participant profile, and comparative information. This was to provide a cost-effective and objective means of gathering data on a sub-section of the sample, who were randomly selected for the National Evaluation. Seventy-five percent (75%) of the Regional Evaluation respondents in the participant interviews also participated in the National Evaluation. This approach resulted in several limitations for the Atlantic CAPC Regional Evaluation.

The National Evaluation required "new" participants for its sample frame. However, in a number of communities, there was already a high level of participation, so it was difficult to recruit families. Also, new centres were just opening, making it difficult to find eligible participants, i.e., parents who had used the program for any length of time, associated with those projects. It is not known whether these new recruits had characteristics similar to the parents in the longer term participant group. There is also a general sense from the staff and others involved in the evaluation that parents with lower levels of education were more likely to refuse to participate in the National Evaluation.

There is some concern that the demographic data generated by the National Evaluation is inconsistent with the characteristics of the CAPC population in Atlantic Canada, particularly as this data pertains to marital status, age, income, and education. However, at this point, we cannot do a detailed analysis to determine why, or to what extent, this discrepancy exists. Therefore, the demographic information provided should be treated with caution.

Limitation 2:Timing
Some projects, particularly those in New Brunswick, were still in the developmental stages when the Atlantic CAPC Regional Evaluation began. Projects in the developmental stages are less likely to be able to report positive changes in behaviours and attitudes or contribute to the data on the evolution of a participant-driven delivery model.

Limitation 3: Burden of Participation
The Regional Evaluation was conducted at the same time as the National Evaluation. This raised concern that the burden of participating in several levels of evaluation could have impacted on who was able to participate. In addition, the "paper burden" for staff caused by the multiple evaluation levels resulted in some inconsistency in data collection. In particular, the concept of evaluation logs was dropped as impractical, and the planned review of project materials, such as local project evaluation reports and minutes of meetings, was not undertaken due to lack of time and the pressure of other aspects of the evaluation process.

Limitation 4: Insufficient Time
The overall time frame for this evaluation was very tight, given the degree of participation required by a participatory action research model and the multiple levels at which input was desirable .Therefore, some participants were unavailable for interviews when the interviewer was in the area.

It was also difficult to monitor the process and make adjustments in data collection. For example, there was insufficient time to use information from initial interviews to probe for information in later ones. The time frame did not allow the opportunity to have the emerging themes verified and elaborated on by the participant focus group.

At the time of this report, limited information is available from the National CAPC Evaluation. Because of this, we were unable to access supporting/comparative data on parents and projects within the Regional Evaluation timeframe.

 

[Previous] [Table of Contents] [Next]