Guide to Project Evaluation: A Participatory Approach

Chapter 3: A Framework for Project Evaluation

Project evaluation is challenging work because of the great diversity in the types of projects funded. To be effective, an evaluation Framework must respect and respond to this diversity. It must also provide a consistent and common process that applies across projects, ensures accountability and produces evidence-based results that promote learning about what contributes to better health practices for Canadians.

The evaluation framework presented in this guide meets this challenge.

It is composed of two parts:

• five key evaluation questions
• five evaluation process steps

The five evaluation questions form the core of the framework and can be applied to all types of project activities. The five process evaluation steps outline a systematic approach to the tasks that projects need to complete to answer the evaluation questions. Groups work through the steps to plan and implement the evaluation.

The following two sections discuss the evaluation questions and the process steps. An overview of the evaluation framework is on page 14.

3.1 The five key evaluation questions

The process of developing the answers to the evaluation questions will vary, as each project varies, but the five fundamental questions remain the same.

What? ¹

• 1. Did we do what we said we would do?

Why?

• 2. What did we learn about what worked and what didn't work?

So what?

• 3. What difference did it make that we did this work?

Now what?

• 4. What could we do differently?

Then what?

• 5. How do we plan to use evaluation findings for continuous learning?

1. This approach is based on work done by Ron Labonte and Joan Feather of the Prairie Region Health Promotion Research Centre.

1.Did we do what we said we would do? "What?" (Description of activities)

The responses to this question describe the work done in the project and the relevance of this work in meeting the project goals and objectives. The project success indicators provide the criteria against which success is measured. They assist the project sponsor to collect the information needed to answer this and subsequent evaluation questions. (Chapter 5 discusses how to develop project success indicators.)

Some of the more specific questions that may need to be answered to describe the project work include the following

• What activities were undertaken and how did they link to meeting the project goals and objectives? Examples:
  • Describe the resources that were developed to increase awareness.
  • Describe the training workshops that were conducted for skill development.
  • Describe the new partnerships that were formed to work on accessibility issues.
• What were the major achievements of the project and what resources did they require?
• If the objectives changed during the course of the project, how and why did they change?

2. What did we learn about what worked and what didn't work? "Why?" (Reasons for success)

Participatory evaluation focuses on success, learning and action. Finding out what worked well in a project and what didn't work well practices this principle. Here are some of the questions that could be included in this discussion:

• What strategies worked well for involving the target population in the project. Why?
• What strategies didn't work well for involving the target population in the project. Why?
• What strategies worked best for broadening the base of community support for the project. Why?
• What strategies didn't work well for broadening the base of community support for the project. Why?
• Which activities and strategies did we change. Why?
• What was learned about the relative cost-effectiveness and efficiency of various project strategies and activities?
• How realistic and relevant were the project goals and objectives?
• In what ways did the project planning process work most effectively?
• What did we learn about working together as a group?

3.What difference did it make that we did this work? "So what?" (Impact)

The answers to this question measure a project's success in changing knowledge, attitudes, skills and behaviour. The project success indicators represent the group's assumptions about what changes should be expected from the project work and provide the criteria against which to measure change both during and at the end of the project. (Chapter 5 discusses how to develop success indicators.)

There are two main ways project sponsors can assess impact: by using summarized data related to the success indicators and by asking specific impact questions of people who were involved in the project and who were the target of the project's work.

The following types of questions may be helpful in discussions about this part of the project evaluation:

• What changed as a result of the project?
  • knowledge
  • attitudes
  • skills
  • behaviour
• What changed as a result of the project for
  • members of the target population?
  • community groups?
  • service providers?
  • caregivers?
  • project sponsors and staff?
• Were there any unexpected changes resulting from the project work? Describe them.
• In what ways did this project contribute to increased public participation?
• In what ways did this project help to strengthen community groups?
• To what extent did the project reduce barriers to health?
• What evidence is there to attribute any of the above changes to the project? What other factors outside the project might have contributed to the changes?
• Were other initiatives started, alternative services proposed or new funding resources acquired as a result of this project?
• In what ways did this project contribute to better health practices?
• What new partnerships developed from this project? What was the nature of the partnerships and what was their contribution?
• Is the model or approach continuing beyond the initial funding?
• To what extent is this model or approach transferable to other communities?

4.What could we do differently? "Now what?" (Future of this and other projects)

Evaluation is for learning and often the best learning comes from examining the challenges that projects present. Here are some of the questions that could be included in this discussion:

• What more effective methods for achieving the objectives emerged from the work?
• What additional knowledge development is required to do the work more effectively?
• What additional support from the funders and community sponsoring agencies would have been useful to the project in meeting its goals and objectives?
• Are there more cost-effective ways to achieve the project's objectives?
• Who else could have been involved in the work?
• What could we do to expand the network of people involved in working on this issue?
• Were all the project's needs met?
• Is there a better way of developing realistic project goals and objectives in the initial planning stage?
• How did management and administrative systems change through the project to become more effective?

5.How do we plan to use evaluation findings for continuous learning? "Then what" (Use of evaluation results)

Participatory evaluation includes ways to use the evaluation results throughout the project as well as at the end. Some questions to consider in developing the evaluation are as follows:

• How were evaluation findings used on an ongoing basis to contribute to the planning and implementation of the project strategies and activities?
• How will project findings be used for future knowledge development?
• How will the final evaluation learnings be documented and distributed?
• Are there alternative ways to present the evaluation results so that more people can make use of the learnings?
• How will the evaluation results be used for new project planning?
• How will the evaluation results be used to influence policy and research priorities?

Seeking answers to the five key evaluation questions will guide the evaluation process throughout a project. The learnings from answering the questions can then be used to shape current and future work.

3.2 The five evaluation process steps

The steps to developing answers for the five key evaluation questions are briefly outlined below, and then are further developed in the next five chapters of the guide.

1.Define the project work.

To evaluate a project there must be clear, measurable project goals and objectives that outline what the project plans to accomplish. While this may seem self-evident, many evaluations have gone off the track because this initial work has not been done.

Chapter 4, Defining Project Work, provides ideas on how to strengthen the development of clear project goals and objectives.

2. Develop success indicators and their measures.

The process of defining what constitutes success for a project is another important step in developing evaluations. Project sponsors need to define the success indicators for their projects. The success indicators allow project sponsors to evaluate whether they accomplished what they set out to do and what the impact of their project has been.

Chapter 5, Developing Success Indicators, discusses this process in more detail, gives some examples of specific indicators and describes an activity that could be used to help identify success indicators for projects.

3. Collect the evaluation data

After the first two steps have been completed, it is necessary to decide

• what information the project needs to collect
• who has the information
• how the information will be collected.

Chapter 6, Collecting Evaluation Data, gives a brief overview of types of evaluation instruments and ideas on how to develop evaluation tools that are appropriate for projects. It also outlines some of the tips and cautions for using these tools.

4. Analyse and interpret the data.

As the evaluation data is collected, it should be summarized and analysed and key learnings should be identified. This ongoing process will help projects prepare their final evaluation reports.

Chapter 7, Analysing and Interpreting Data, provides some ideas to help with this process.

5. Use the evaluation results.

Evaluation findings can be used throughout the project to improve the planning and implementing of project activities. By sharing project results with others, each project adds to the body of knowledge about health promotion.

Chapter 8, Using Evaluation Results, provides ideas on how to use evaluation findings during and after the project.

Working through these five steps will provide project sponsors with the information and tools they need to answer the five key evaluation questions. For small projects with limited resources, the process will be simple and straightforward. For large projects with greater resources, the work involved in each step will vary to reflect the complexity of project goals and objectives.

For all projects, project sponsors should:

• set realistic limits on the number of project-specific evaluation questions and on the amount of evaluation information to be collected, as determined by the evaluation resources available to the group
• remember that the quality of information collected, not the quantity, is the most important factor in evaluation.

Remember, the most successful evaluations are clear and easy to understand.

3.3 Tools for using the evaluation framework

To help in applying the evaluation framework, several different tools have been developed for this guide. Examples provided reflect the most common Health Canada project activity types, which are

• needs assessments
• education and awareness
• resource development
• skills development
• developing innovative models
• reducing barriers to health.
• One-page overview of the Framework for Project Evaluation (see Section 3.4)
• Framework Worksheet for the Five Key Evaluation Questions and Examples of Developing the Questions by Project Activity Type (see Appendix 3)
  • The blank worksheet can be used by projects to develop the five evaluation questions. The examples show how the questions can be further developed to reflect the specific evaluation needs of projects

• Framework Worksheet for Success Indicators and Examples of Developing Indicators by Project Activity Type (see Appendix 4)
  • The blank worksheet can be used by projects to develop their own project specific success indicators and their measures. The examples provide ideas for developing success indicators and measures that projects may find useful.

3.4 A framework for project evaluation

An overview of the framework for project evaluation is presented on the next page. This overview is a useful tool that can be used for

• introducing the framework
• reviewing the evaluation work
• preparing evaluation reports

A Framework for Project Evaluation

5 key evaluation questions

What?

• 1. Did we do what we said we would do?

Why?

• 2. What did we learn about what worked and what didn't work?

So what?

• 3. What difference did it make that we did this work?

Now what?

• 4. What could we do differently?

Then what?

• 5. How do we plan to use evaluation findings for continuous learning?

Steps in the project evaluation process

1.	Define the project work	•	clear, measurable project goals and objectives
			Project activity types: needs assessments education and awareness resource development skills development developing innovative models reducing barriers to health
2.	Develop success indicators and their measures	•	process for identifying indicators
		•	ideas for success indicators linked to process and impact
3.	Collect the evaluation data	•	written questionnaire
		•	telephone survey
		•	reaction sheet
		•	interview - face to-face or phone
		•	focus group
		•	participant - observation
		•	project diary
		•	program records
		•	before and after questionnaires
		•	non-traditional methods of documentation
4.	Analyse and interpret the data	•	data analysis
		•	identification of learnings, recommendations, actions
5.	Use the results	•	sharing of the results on an ongoing basis
		•	use of learnings to inform future planning

Table of Contents