Guide to Project Evaluation:  A Participatory Approach

Chapter 6: Collecting Evaluation Data

Participatory evaluation relies on a systematic and rigorous collection of information from project staff and stakeholders. It draws on both quantitative and qualitative data to measure success and to clarify and make decisions about project characteristics, activities and effects.

6.1 Determining information collection needs

Three questions to ask in
determining evaluation information needs:

1. What information is needed?
2. Who has the information?
3. How will the information be collected?

1. What information is needed?

Projects need to collect evaluation information that will provide answers to the five key evaluation questions. The specific type of information to be collected is determined by the work done at the beginning of each project to define the project goals, objectives and success indicators.

2. Who has the information?

Depending on the nature of the project, the people with information useful to the project evaluation will vary widely. People from whom it may be important to collect information include

• project sponsors, staff and volunteers
• program consultants
• target population
• consumers of the service
• general public
• advisory committee members
• other service providers
• partners associated with the project.

3. How will the information be collected?

Project sponsors decide how best to collect evaluation information based on their project's needs and resources. Designing the information collection tools should be done in collaboration with the people who will be using them. Most community projects don't have the time or the resources to put into extensive recording of data and statistics. The goal is to find ways of collecting information that do not put too much of a burden on the people doing the project work but that still provide the information required to answer the evaluation questions.

Characteristics of a good information collection process:

• useful
• practical
• collaborative
• systematic
• ongoing
• accurate
• ethica

6.2 Information collection tools

There is a wide variety of information collection tools that can be used depending on the project's evaluation needs. Examples of tools that have been used in other projects are listed below.

Written survey questionnaire

• structured questionnaires used to reach large numbers of people
• provides quantitative data (numbers) that can be statistically analysed and qualitative information that can be summarized
• used to survey target population in terms of knowledge, attitudes, beliefs and behaviour.

Tips and cautions:

• When developing the questions for the questionnaire, ensure that they are not worded in ways that lead to biased or misleading responses.

• While mass mailing of survey questionnaires has the advantage of reaching large numbers of people, there is no guarantee that people will fill out and return it, so the actual response rate may be low.

• Limiting the number of questions may increase the response rate.

• Using smaller but targeted mailings, followed up by a phone call, may increase the response rate.

• Paying attention to respondents' literacy level, language and visual capacity may increase the response rate.

• All survey questionnaires need to be pilot tested to ensure that the questions succeed in getting the information that is required.

Telephone survey

• can ask for the same types of information as the written survey questionnaire

• has the advantage of increasing accessibility and allowing immediate clarification of questions if the respondent is experiencing any difficulties.

Tips and cautions:

• Telephone interviewers may face resistance from people who are tired of answering this type of call or who are suspicious because of their experiences with telephone soliciting.
• Ensuring that the respondent is provided with clear information on the credibility of the group doing the survey, the purpose of the survey and how the collected information will be used may increase the response rate.
• Finding a convenient time for the respondent to answer the survey questions may increase the response rate.

Reaction sheet

• simple kind of questionnaire that asks questions about people's satisfaction with a particular activity
• easy and fast to administer and summarize
• useful tool for getting an immediate response to new resource materials, workshop models and public education events.

Tips and cautions:

• Avoid using leading questions that prompt positive responses. Instead of asking, "Did you enjoy the workshop?" ask, "Did the workshop provide you with enough information to answer your questions about health promotion?"
• Limit the number of questions to increase the response rate.
• Include open-ended questions to obtain qualitative data. Shape these questions carefully to control the amount of material received. Examples: "Give three words to describe your reaction to this workshop." "What were the two key learnings for you from this workshop?"

Face-to-Face interview

• individual interviews structured around a set of open-ended questions that are developed to guide the interview and to provide consistency in the information collected
• useful method for getting in-depth information on project activities
• provides an opportunity to clarify responses and probe for further information

Tips and cautions:

• This tool can be used with a specific group of people (e.g., project staff to gather their opinions about the strengths and weaknesses of the project) or with key informants who are knowledgeable about the project (e.g., frontline service providers about how best to conduct outreach within their community).
• It is a good method to use with respondents who have low literacy levels and might be uncomfortable with written data collection tools.
• The interviewer needs to be trained not to bias the responses through the use of leading questions.

 Telephone interview

• similar process and function as face-to-face interview but conducted by phone
• less expensive to administer than face-to-face interviews.

Tips and cautions:

• Sending the respondent a copy of the interview guide in advance may promote a more thoughtful discussion.
• Interviews, both in person and by phone, are an alternative to focus groups when you want to avoid group influences on the responses people give.

Focus group

• group discussion in which 10 to 12 people are brought together in a single session of approximately an hour to generate ideas and suggest strategies
• facilitated using a specific agenda of structured questions, similar to the interview guide, that focuses the discussion in the meeting
• used to obtain in-depth understanding of attitudes, behaviour, impressions and insights (qualitative data) on a variety of issues from a group of people, e.g., project staff or a project advisory committee.

Tips and cautions:

• The facilitator must remain neutral and non-judgmental and have the skills to keep the discussion moving and on track.
• This is a particularly useful method for reflecting on evaluation findings and identifying key learnings. It may also be useful for developing preliminary ideas for new programs or for testing messages that will be used in educational and media packages.
• It is not a useful method for developing consensus or making final decisions.

Participant - observation

• involves actual observation rather than asking questions
• used to better understand behaviours, the social context in which they arise and the meanings that individuals attach to them
• observers compile field notes describing what they observe; the analysis focuses on what happened and why.

Tips and cautions:

• This may be the most feasible way to collect data from some hard-to-reach populations (e.g., individuals who frequent public sex environments or drug shooting galleries).
• As with all qualitative techniques, the results may not be fully generalizable to the entire study population.

 Project diary

• project managers, staff or participants are asked to keep a record of their experiences while working on the project
• provides qualitative evaluation data.

Tips and cautions:

• It is important to provide the participants with clear guidelines on keeping a log book: the type of information you are looking for, how it will be used, confidentiality, etc.
• This is a useful method for identifying unintended consequences of a project.
• Some people are very uncomfortable with this method because of the unstructured nature of the writing required.

Program documentation

• analysis of written records (minutes of meetings, telephone logs, intake forms, policy directives, financial records, attendance records)
• can provide information on people's interests, preferences and patterns of usage of services and service locations
• can often, through systematic review, provide important evaluation information, both quantitative and qualitative
• inexpensive source of information.

Tips and cautions:

• This tool is limited in that records document only existing alternatives, they don't show other needs, wants or preferences.
• It is important to identify evaluation information needs at the beginning of a project to ensure that the necessary records are kept throughout the project.

Non-traditional methods of documentation

• non-verbal or non-written evaluation tools used to respect diversity and accessibility issues
• examples include cartooning, drawing, poster making, photography, videotaping, audio taping, scrapbooks.

Tips and cautions:

• Qualitative data collected may be difficult to analyse and generalize.
• This is a useful method for getting responses from respondents who are uncomfortable with written tools.

No single evaluation tool can provide all the evaluation information required. A combination of different tools that suit the project needs and available resources has to be developed. Regardless of which tools are selected, they should reflect the following tips to be effective.

Tips for designing effective evaluation tools

• Keep them short and simple.

• Use plain language with no jargon.

• For tools requiring written responses

  • use large print
  • avoid clutter

• Leave lots of white space

  • provide ample room for responses.

• Ask for key works and key learnings.

• Develop evaluation tools in collaboration with the people who will use them.

• Ask only for information that will be used.

6.3 Sample evaluation tools

Sample evaluation tools are provided in the appendices at the end of this guide. They were developed for use in community-based projects and are included here to give project sponsors some ideas, which they can adapt and build on to develop their own project-specific tools. The questions in each of the sample forms have been shaped specifically to provide data to answer the five key evaluation questions.

Sample evaluation forms provided:

Appendix 6.	Reaction sheet for evaluation workshop
Appendix 7	Guided telephone interview - Community Resource Handbook for Women with Breast Cancer
Appendix 8.	Focus group interview guide - Child Safety Awareness Program
Appendix 9.	Guidelines for keeping a project diary - Child Safety Awareness Program
Appendix 10.	Mail-out questionnaire - Advisory Committee, Health and Disabled Women's Project
Appendix 11.	Mail-out questionnaire - Health Care Providers, Health and Disabled Women's Project

Table of Contents