Guide to Project Evaluation:  A Participatory Approach

Chapter 7: Analysing and Interpreting Data

Most evaluation projects have no problem with collecting large amounts of evaluation information. What they sometimes do have difficulty with is effectively analysing, summarizing and using the results.

The emphasis throughout this guide is on evaluation for learning and action. This section focuses on practical ways that people at the national, regional and community levels can turn evaluation information into usable, accessible summaries and reports that add to the body of knowledge about project success and promote change in attitudes, skills and behaviour. Committing adequate resources at all levels to do the evaluation work is essential if everyone is to benefit from the valuable learnings that can be gained from evaluating health promotion projects.

7.1 Analysing project evaluation information

Analysing evaluation information begins with a review of all the collected data to find the emerging themes or patterns. The five key evaluation questions provide useful categories around which to group information and develop the themes. Look for and record the information that is in the data about how well the project is doing, what is working, what should be done differently and what difference it is making.

Project sponsors may want to record notes on the data on file cards or sheets of paper - one for each question, issue or topic. This makes it possible to see the emerging patterns more easily. Include exact quotations from the interviews and questionnaires. It is essential to stay with what people have said and let the data guide the analysis. Too much detail is better at this stage than not enough. It is always easier to cut down than to add information later.

Once the material has been grouped into themes, it can be analysed to see how the results compare to the changes that were expected as identified by the success indicators. Take the time to reflect on what the analysis reveals. What was learned to answer the "what", "why", "so what", "now what" and "then what" evaluation questions? People who have been involved in the project should be involved in the interpretation of the findings.

Project sponsors or the project evaluator should prepare short summaries of the key learnings from the analysis on a regular basis - for example, every three months or after each project activity. The importance of preparing these brief summaries, which highlight two or three key learnings, cannot be overemphasized. The summaries provide an excellent means of letting the key players in the project know about and begin to use the evaluation findings throughout the project - one of the basic principles of participatory evaluation. By completing summaries of key learnings at regular intervals, the work at the end of the project will be greatly reduced.

Summary - Analysing evaluation information

• Review the collected evaluation material for emerging themes and patterns.
• Use the key evaluation questions to group the material into themes.
• Analyse the material by themes, comparing the results to the changes that were expected as identified by the success indicators.
• Reflect on what the analysis means. Ask other key project players for their interpretations.
• Prepare short summaries of key learnings under each theme.
• Prepare summaries of key learnings on an ongoing basis.
• Submit the summaries to the participants for their feedback and verification of the findings.
• Develop the final analysis.

Analysis of quantitative data

Quantitative data looks at the incidence and quantity of events. Data gathered through quantitative methods (surveys, questionnaires, administrative records) is numerical and may be analysed by calculating averages, ranges, percentages and proportions. Descriptive statistics simply account for what is happening in numerical terms. For example, when evaluating the use of a needle exchange system, an estimate may be made of the average number of people using the facility each week or the percentage of users returning needles. Bar charts, pie charts, graphs and tables can be effective ways to present the statistical analysis in a clear and concise manner.

Analysis of qualitative data

Qualitative data is information that is primarily expressed in terms of themes, ideas, events, personalities, histories, etc. Data is gathered through methods of observation, interviewing and document analysis. These results cannot be measured exactly, but must be interpreted and organized into themes or categories. The primary purpose of qualitative data is to provide information to the people involved in the project. This standard of usefulness is an important one to keep in mind when analysing qualitative data.

Note: Neither the quantitative nor the qualitative approach to the collection and analysis of data is inherently superior. Each has advantages and disadvantages. For both, it is important to know the context within which they have been used in order to understand the analysis. Whenever possible, project evaluations should include several types of information collection tools. The analysis and summaries of key learnings should draw on information collected from all of them.

7.2Preparing useful evaluation reports

Once the evaluation information has been analysed, the next challenge is to present the learnings in ways that are both informative and interesting.

The brief summaries of key learnings, described in the preceding section, are often all that is needed to provide information on an interim basis. However, the final project report requires more data. The next section provides some ideas that might be useful for clarifying the expectations about the final report with project sponsors.

Evaluation report outline

Having an outline at the beginning of a project about how the final report will be developed is extremely useful. It helps shape the thinking about what information is needed and how it will be collected, analysed and used.

There are two questions to consider when planning evaluation reports.

1.Who is writing the report?

Small projects with very limited resources should have different expectations placed on them than larger projects or projects with funding for an external evaluator.

2.Who is the report for?

While every evaluation report should be written in an interesting and clear style, the structure and emphasis of the report may vary depending on who it is for. For example, is it intended primarily for the funder or for the project participants? The former might focus more heavily on learnings about cost-effectiveness strategies; the latter might be more interested in learnings about how to implement a specific health promotion activity.

The following sections form the basic structure - the bare bones - of an evaluation report. Personal stories and quotations from the project participants put a human face on the evaluation results and can make the report much more interesting and user-friendly. Groups can adapt and build on the following guidelines to develop evaluation reports that reflect the unique nature of specific projects.

Example of an outline for a project evaluation report Section 1:

Section 1:	Executive Summary This section is for people who are too busy to read the whole report. One page is best - never more than three. It comes first but is the last piece written. It usually looks at what was evaluated and why and lists the major conclusions and recommendations.
Section 2:	Background Information - Getting started This section provides background leading up to the evaluation: how the project was conceived why it was needed the project goals and objectives who was involved in the work the project organizational structures.
Section 3:	Description of the Evaluation - How we learned This section describes the evaluation approach and how it was chosen evaluation goals and objectives how the evaluator was selected and managed how the data collection tools were designed and used how well the data collection tools worked any limitations of the methodology how people were selected to be interviewed, or to receive questionnaires, etc. who did the interviewing, the number of people interviewed and their situation how questionnaires were distributed and returned.
Section 4:	Evaluation Results - What we learned One way to organize this section is around the first four evaluation questions: Did we do what we said we would do? Outline goals and objectives of the project. Record what happened as a result of the project - e.g., resources developed, training sessions completed, etc. Describe the changes that occurred in relation to the success indicators. What did we learn about what worked and what didn't work? Outline key learnings from the project about making things work. Examples: producing effective resource materials, structuring productive advisory committees, conducting needs assessments in rural and isolated communities, building community ownership of health promotion projects, etc. Identify learnings about what strategies didn't work and why. What difference did it make that we did this work? (outcomes) Outline results from the evaluation that show how the project made a difference to consumers, project sponsors and the wider community. Identify any changes - of attitudes, knowledge, skills or behaviour that occurred from the project work, e.g., how health practices have improved. If appropriate, show how the project contributed to increased public participation and strengthened community groups. Include personal statements and anecdotal material from project evaluations which illustrate the impact an activity has had on project participants. Example: "One thing I plan to use right away in my work which I got from the training is..." What could we do differently? List learnings from the projects about different ways to do the work. Examples: improving the cost-effectiveness of projects, adapting the project model to make it more responsive to volunteers, changing the reporting role for outside evaluators to improve accountability, etc. Reflect on cautions and challenges about doing the project work.
Section 5:	Conclusions and Recommendations - Final thoughts on what we would like others to know Conclude with a summary of the work done and how well the goals and objectives were reached. Include recommendations for further work. Include recommendations on how the evaluation results can be used.
Section 6:	Appendices These may include copies of questionnaires or interview schedules, statistical information, program documents or other reference material important to the evaluation but not important enough to go into the text. It is useful to include a bibliography - list of the sources used to compile the evaluation results, other research studies and articles. A list of who was interviewed or organizations contacted may also be include

7.3 Activity: Analysing and Interpreting Data

Analysing and interpreting data from project evaluations.

Purpose:	To give project sponsors practice in completing the analysis and interpretation of project evaluation results for inclusion in the project evaluation report.
Time:	1-2 hours
Materials:	flipchart raw project evaluation data Guide to Project Evaluation, Chapter
Activity:	Have participants review the raw project data Working in small groups, divide the raw data among groups and have participants analyse the data to find themes (refer to Chapter 7 of the Guide) that relate to the third evaluation question: "What difference did it make that we did this work?" Have participants prepare short summaries of learnings for each theme Have participants develop one example of quantitative analysis and one example of qualitative analysis Bring all participants together to share their results and to discuss their ideas on which information is most useful Have the total group develop a final analysis plan based on the information presented by the small groups Have the total group brainstorm on informative and interesting ways to present the evaluation results, including how to organize the evaluation report for maximum effect

Table of Contents