COMMUNITY ACTION PROGRAM FOR CHILDREN (CAPC)

MADE TO MEASURE: QUALITATIVE DATA AND EVALUATION EMPOWERMENT RESEARCH

5th International Qualitative Health Research Conference
University of Newcastle, Australia
Session 10.4 - Health Services and Policy Evaluation
Friday, April 9, 1999 (11:00 - 11:20)

PDF Format (20 KB)

Please note that this document was published by Health Canada prior to the announcement of the establishment of the Public Health Agency of Canada on September 24, 2004. Any reference to Health Canada should be assumed to be to the Public Health Agency of Canada.

Authors:		Nicole Bernard, Pauline Raven, Michelle Rivard, Yolande Samson, Madine VanderPlaat, Lynn Vivian Book
Presenter:		Yolande Samson Evaluation Consultant, Health Canada 1557 Hollis Street, Suite 709 Halifax, Nova Scotia, Canada B3J 3V4 Yolande_Samson@hc-sc.gc.ca

Over the past decade the rhetoric of “empowerment” has permeated the health promotion, education and social welfare literature. This paper, which represents the reflections of government managers, community workers and the principal researcher, examines the issues that arose at various stages in the evaluation of a community-based health promotion program. It addresses how these were overcome through the reconstruction of our understanding of empowerment. In particular, this paper highlights some of what the research team learned as we struggled to reconcile the principles and politics of a participatory and qualitative approach to research with the conventional demands of program evaluation. The team’s efforts to translate what they learned into practice are also discussed.

I would first like to briefly describe the program being evaluated. The Community Action Program for Children, or CAPC as we often refer to it, is a national program that seeks to improve the health and well-being of Canadian children (from birth to age 6) and their families, particularly those families who require additional support because of difficult life circumstances such as low income or social isolation, or who traditionally have been referred to as “at-risk” families. CAPC uses an empowerment-oriented health promotion strategy and is participant-driven -- participants are the ones who decide how individual projects will be run, what programs they will access and how resources will be allocated. CAPC projects can take numerous forms, but in Atlantic Canada, the region where this evaluation took place, a typical CAPC project usually involves a family resource centre within a central community with outreach sites in rural or remote areas. CAPC projects offer a variety of parent-, child- and family- (parent and child) focused programs and activities such as drop-in play, parenting programs, toy lending libraries, community kitchens and so on.

CAPC is funded by Health Canada, a federal government department, and therefore is required to undergo periodic formal evaluations. The national evaluation used standardized quantitative tools to collect comparable data which measured the progress and impact of all CAPC projects in Canada. The Atlantic CAPC Regional Evaluation was a largely qualitative study, adding richness and depth to the quantitative information being collected at the national level. This regional, two-year, million-dollar study involved five governments working at different levels and over 40 community-based CAPC projects. It is one of the largest empowerment-oriented evaluations ever conducted in Canada.

In conducting the evaluation, it was critical that the evaluation reflect the principles underlying CAPC, that is, the evaluation also had to be empowering in its approach. As a result, the evaluation strategy was based on a participatory action research model that required the community, government managers and the university-based evaluators to work together as a team.

The evaluation of CAPC in Atlantic Canada began as a relatively conventional empowerment-based research project. Our notions of what was empowering and where everyone fit into the process was very much dictated by the traditional participatory action research literature and by our experience with planning, implementing and evaluating community-based government programs. We had a management team consisting of representatives of all parties; we trained members of the community to conduct the research; and we developed teams consisting of researchers, community workers and parents to code and analyze and interpret the data. We had the rather naive belief that as long as the CAPC community had input into all parts of the project, it would be empowering and consistent with an emancipatory ethic. Those of us who worked in government or in academia spent little time examining our own roles beyond agreeing that we would not impose. In other words, the concept of “participatory” was more or less restricted to the CAPC community in its application.

This positioning quickly unraveled. We were forced to seriously reassess our understanding of empowerment and how this understanding should play out in practice. By the time the preliminary round of analysis had been drafted, we realized that we had not given enough attention to the various political agendas. In our concern with allowing the “voices” of the CAPC community to be heard, we had failed to give those voices a meaningful framework within which to speak. What we had were narratives -- wonderfully rich narratives -- describing the CAPC experience and what it meant to participants. What we did not have was an analysis that met the agendas of the evaluators and government managers. The government members of the team wanted to demonstrate the effectiveness of an empowerment-based approach to health promotion and to ensure the continuation of funding for the initiative. The university-based evaluators were primarily concerned with methodological rigour and being able to provide valid evidence in support of the findings. The community workers involved with CAPC agreed with this but also did not want the CAPC experience distorted through the lenses of bureaucracy, academia and efficiency. Many program parents remained unconvinced that the evaluation would be anything more than a typical bureaucratic make-work project and worried about the damage an ineffective report could have on their funding.

About halfway through the evaluation, we began a process through which the various parties negotiated -- and at times battled -- their way toward a new understanding of what an empowerment-oriented philosophy was all about, how everyone fit into the process, and how it would play out in practice.

One of the first requirements was a thorough examination of the various political agendas to see if there was enough commonality and commitment to proceed. Fortunately, despite our differences, every one of us was committed to an emancipatory ethic and we were all willing to explore what that ethic might look like in practice. One of the more critical insights gained at this time was the understanding that meaningful social change requires interaction between human agency and social structures. By focusing solely on the empowerment of the CAPC communities, we had reduced the notion of social change to individual, or at best, group change. We had ignored the fact that as academics and government bureaucrats, we were part of the systemic structure and as such, were in a position to use the CAPC evaluation to make changes within our own disciplines and organizations. By placing ourselves on the sidelines of the empowerment process, we had in effect rendered it powerless.

Ironically, it turned out that the people who had the most work to do in terms of learning to be empowered and empowering were the evaluators and the professionals within government. With this awareness came the recognition that the evaluation should not just focus on project participants but on everyone involved -- that is, it should not measure effectiveness by looking only at changes in the behaviours of participants. If our intent was to empower parents/caregivers we should also expect to see changes in the broader community.

Equally important were the systemic policies and practices related to CAPC in particular, and to social interventions in general. Our understanding of change moved from individualistic behavioural change to social change as a collective experience occurring at all levels. Success or effectiveness needed to be judged by what happened in the larger environment as well as to all individuals and organizations in the collective. Interestingly, once we put ourselves firmly in the middle of the empowering process and learned to think about program participants as agents of change rather than its subjects, the analytical framework that had been missing at the beginning of the process readily emerged.

From a methodological perspective, this increased our awareness of the need to create and test better indicators and outcomes if we truly were to measure the impact of community-based initiatives. We needed to be able to answer the question “What is it about CAPC that creates the positive environment for change -- at a personal, professional and social level.” To do this, we needed to select better indicators of effectiveness, not only at the intervention level but also outside the immediate project site, e.g., “What is it about the impact of CAPC projects on the community and government systems that leads to change in the broader social systems?”

In this way, empowerment is a two-way street (or bi-directional) and the evaluation framework was restructured to reflect this. The significance of what was learned through the Atlantic evaluation of CAPC is that empowerment evaluation using an emancipatory approach is broader than the “project” and includes the measurement of the interaction among complex social, multi-dimensional and political structures and systems. It moves empowerment evaluation into the realm of the political.

We encountered other challenges in putting empowerment theory into practice. For example, how do you handle the volume of rich qualitative data to ensure the “voices” are understood and correctly interpreted? We discovered that there were limitations in using an ethnographic computer program to analyze our data (over 8,000 pages of qualitative information were collected). We used the program to look for key words for counting purposes. While this was somewhat successful, we found that interviews had to be read in their entirety to draw out what was important. When we tried to shape the analysis in a more quantitative way, it became distorted. In analyzing staff interviews, if we simply counted the number of times something was said, what surfaced most often was the amount of time that was spent doing administrative work in the program. However, when you looked at the whole interview, the importance of what participants said was relatively unrelated to administrative work and much more related to the challenges of program implementation. As well, original coding was based on what we thought might be important and therefore we missed emerging themes. This limited our ability to interpret the data.

We also discovered the importance of a multi-disciplinary team in analyzing the results. One can have the most solid data possible but how these data are interpreted and represented is critical. It is important to have a team that consists of participants in the program as well as experts in evaluation, early childhood development and community development, because the data need to be interpreted within the broader context. As we discovered, the downside of this approach, within a participatory/ empowerment model, is that the interpretation of information is also political and this can slow down or paralyze the process, that is, multiple perspectives sometime lead to disagreements about what is the “real truth.”

As an example, in our first draft of the report, there was considerable debate over the language used to describe and represent parents. In these discussions, we became increasingly aware of the underlying values our disciplinary languages reflected and how offensive and disempowering this often was to parents. In particular, we had to make a conscious effort not to objectify parents or talk about them as people who had things done to them. For example, rather than saying “CAPC gives parents,” we used language that reflected the parent’s proactive role, “parents use CAPC to ....” Other examples: instead of “parents get help with ...” we would say “parents access resources related to ...”; instead of “parents learn skills ...” we would say “parents build on existing skills ...” recognizing that they come to the project with experience and skills. Parents and their motivations became the “real truth,” which was a more accurate reflection of reality rather than depicting them as needy, passive, incapable of action or deficit-driven.

We also discovered that the final report is only the beginning of documenting what has occurred in the evaluation process. A great deal of energy goes into writing the report in the participatory manner which is necessary for critical reflection from the different political perspectives. Initially, all the implications of the work may not be apparent. However, as more time is spent thinking about it, the links and connections to the broader body of knowledge become clearer. Understanding the information, transforming it into knowledge within projects, communities, governments and academia, and sharing it with others needs to be a planned and ongoing process. You need to build in time to develop mechanisms or tools to promote and use the results and to act on the recommendations.

Despite all these challenges, we feel there were very positive and significant things that happened as a result of using an empowerment model in evaluating CAPC.

First, we discovered the most important things about the program because we involved the program participants in the analysis and interpretation team. The things the participants considered to be important were, in some cases, different from our original ideas about what was important or what the literature was telling us should be important. The significance of this was that we were able to discover the emerging program themes that evolved over time within complex social, multi-dimensional and political structures. These emerging themes are often critical to project understanding, improvement and success. Had we used another process, we would not have found them.

Second, we discovered that participants define the same activities in different ways and this has an impact on the interpretation of information. One example is participation. A standard approach is to define participation as the number of parents who participated in the program. Those involved with CAPC at the community level saw participation as more than numbers because they understood that many parents, in this often marginalized population, move in and out of programs as circumstances and needs arise. The original notion of participation (for example, counting the number of participants that attend a particular program) clearly takes on a less stringent yet broader definition within the context of the program and people’s lives. This is an example of a word that has different meanings for different people and therefore affects how the collected information is used, interpreted and understood. It also emphasizes the importance of having participants, not just evaluators, actually involved in the interpretation of the data.

Finally, the use of an empowerment evaluation model recognizes that the needs of participants emerge and evolve over time. We discovered through this process that it is critical that the needs of a population not be defined up-front, and that the structure of both the program and evaluation framework not be based on these predetermined needs. Sufficient flexibility needs to be built into the evaluation framework to allow needs to emerge. This impacts on the indicators and outcomes you define beforehand and challenges the notion of the more traditional approach of pre- and post-test. Needs emerge over time as participants gain trust and support within the environment. When this occurs, participants are able to not only articulate their needs but also feel they have a right to have their needs met.

Altogether, this large scale evaluation has taught us a great deal about the principles and politics of an empowerment evaluation, about the challenges and opportunities in putting these into practice and about building an empowerment model that is emancipatory for everyone involved.