Guide to Project Evaluation: A Participatory Approach

Chapter 5: Developing Success Indicators

Identifying what success will look like during the developmental phase of a project may seem a little like putting the cart before the horse. Many project sponsors spend a lot of time developing goals and objectives, planning activities and thinking about budgets. The real challenge is to think to the end of the project and name the identifiable changes that they expect to occur as a result of doing the work. These identifiable changes, the success indicators, should be developed as soon as clear project goals and objectives have been established. Therefore, identifying success indicators is the second step in the process of planning high quality project evaluation plans. Project sponsors should identify the success indicators that are most appropriate and best reflect the reality of their own projects.

5.1 Purpose of success indicators and their measures

Success indicators are a group's assumptions about what changes should be expected from doing the project work. These indicators are quantified by specific measures for example, a number, a percentage or a level of satisfaction.

Success indicators and their measures need to link directly to project goals and objectives since they provide the objective and measurable criteria by which groups judge the degree of success they have had in reaching their goals and objectives.

Through their project activities, project sponsors attempt to change the knowledge, attitudes, behaviour or skills of a selected group of people - sometimes referred to as the target group. To measure or evaluate the amount of change, it is useful to know the status of the target group's knowledge, attitudes, behaviour and skills at the beginning of the project. Determining this initial status or starting point is called setting a benchmark. This initial benchmark helps the project determine the amount of change it is trying for in the project. The example below may help to illustrate this process. Other examples of success indicators can be found in Appendix 4 (Examples of Developing Indicators by Project Activity Type) and in Appendix 5 (Success Indicators of Increased Public Participation and Strengthened Community Groups).

Example: Breast Cancer Network Project

• One of the project goals is to promote the development of survivor/directed, self-help groups for women with breast cancer
• A project objective linked to this goal is to raise the awareness of the need for and success of such groups among health care professionals and cancer societies.
• The target group for the project is 200 health care professionals in the project's urban area.
• The project benchmark was established using the questionnaire completed for the initial needs assessment for the project. The results of the questionnaire indicated that 15% of the 200 health care professionals in the community knew about the advantages of self-help groups and referred breast cancer patients to them.
• The project success indicator is to have 60% of the 200 target group health care professionals know about and refer to self-help groups for breast cancer patients.

Project sponsors can use the success indicators to identify some of the specific questions they will ask throughout the project. The information that is collected and summarized in relation to these success indicators can be used by the groups to help answer the first three questions in the evaluation framework: Did we do what we said we would do? What did we learn about what worked and didn't work? What difference did it make that we did this work?

5.2 The process of developing success indicators and their measures

Choosing which indicators are the "best" is not an exact science. The process that project sponsors go through to identify their success indicators is as important as the final list of indicators created. Done well, this process can contribute to the building of commitment and excitement for doing an evaluation. It also helps groups develop reasonable expectations of what can be achieved.

Some guidelines for developing success indicators

Success indicators should:

1. Be results-focused i.e. refer to results or outcomes of the funded activity and not the activity itself.
2. Be challenging but feasible.
3. Involve a meaningful comparison - a comparison over time, a comparison with other similar activities or (preferably) a comparison against a reasonable standard.
4. Be measurable, using quantitative or qualitative measures. In developing indicators, consideration should be given to data availability and data collection, given the resources available.
5. Refer to a result or outcome that can be reasonably attributed to the project activity.
6. Be as valid (directly related to the work done and not attributable to other factors) and reliable (able to be replicated) as possible.
7. Meet the criteria of
   • selectivity i.e. the number of indicators are limited to and focused on the key areas of concern.
   • balance i.e. the indicators refer to a range of project activities and results that together will provide a balanced assessment of project success.
   • usefulness i.e. the potential use of the evaluation information should be taken into account when developing indicators to ensure that they capture the relevant information.

Benefits of developing good success indicators:

• clarification of project goals and objectives to make them measurable
• identification of innovative success indicators that reflect unique community characteristics and needs
• strengthened strategies and workplans to address some of the identified barriers to success
• increased commitment to assess impact question

The activity on the next page has been used with a number of community groups to help them identify success indicators for their projects. Some of the most useful indicators of success have been developed when members of the target population and project sponsors have undertaken this activity together.

5.3 Activity: Defining success indicators

Topic: Defining project success indicators

Purpose:	To give project sponsors a chance to define the indicators of success for their project.
Time:	2 - 3 hours
Materials:	flipchart handout: an outline of the project objectives and a list of the activities or strategies that are to be undertaken to achieve the objective handout: Success Indicators of Increased Public Participation and Strengthened Community Groups (Appendix 5) Note: Appendix 5 is for use as an example only. The project sponsors need to develop their own success indicators that are relevant to their project.
Activity:	Divide the participants into pairs or groups and assign one of the project objectives and the activities associated with that objective to each group. Ask the groups to list five things for each activity that would indicate that the activities were successful. Examples of possible activities: media campaign, organization of a mother's group, development of brochures, needle exchange program, creation of a 1-800 line. Encourage participants to use all their senses in developing indicators: e.g., What changes do they expect to see? hear? feel? Put quantities on criteria when possible. The participants must be realistic about what they hope can be accomplished. Remind them that useful indicators are measurable, specific, easy to collect information on and ultimately can provide useful information to the group. Discuss the success indicators with the total group and add new ones. You may want to review the list of indicators for increased public participation and strengthened community groups to see if any of them are appropriate for the project. With the total group, review the indicators and order them by priority so that only the ones that are most useful and important are selected. The task is to keep the number of success indicators manageable for the project resources. Once there is agreement on the indicators, the group could begin to look at the kind of information that needs to be collected to document the degree of success.

5.4 Success indicators for project activity types

Although there are many different types of projects funded under Health Canada programs, certain project activity types appear more frequently than others. They have been identified as

• needs assessments
• educational and awareness
• resource development
• skills development
• developing innovative models
• reducing barriers to health

Appendix 4 provides examples of individual projects by activity types and their possible indicators and measures of success. Remember, these are only guidelines.

Appendix 5 provides examples of indicators of success for two health promotion program/project impacts - increased public participation and strengthened community groups.

These examples have been included to stimulate thinking and to start the process of developing project-specific success indicators.

Table of Contents