'Evaluation Zone' : Developing an evaluation plan

Indicators of Impact/Outcome

Indicators of Impact/Outcome provide a sign of how well you have achieved the changes you were were hoping for as a result of your project. They are about measuring change. In other words they are a measure of the extent to which you have achieved your objectives and your longer term goal. Indicators of impact relate to your objectives, and indicators of outcome relate to your goal.

Usually indicators of impact are much easier to come up with because they are about more immediate changes you are seeking. Whereas measuring outcomes refer to large and long term changes. Goals are often written as broad statements of desired change, such as to decrease social isolation in a region, or reduce the incidence of domestic violence. They can also relate to a change in health status, such as reduced mortality, morbidity, or improved well being.

Health outcome evalution rarely occurs in small primary health care projects, due to the short term funding, complexity and difficulties of attributing project activities with long term outcomes. See below for more on Impact/Outcome evaluation.

Instead most projects focus on evaluting the impact of their project acccording to their objectives.

Examples of Impact/outcome indicators :

• changes in awareness, knowledge, skills
• increases in the number of people reached
• policy changes
• changes in behaviour
• changes in community capacity
• changes in organisational capacity (skills, structures, resources)
• increases in service usage
• improved continuity of care

To view the impact/outcome indicators on our example of a completed Program Evaluation Plan click on the 'Example Project and Evaluation Plan' in the menu to your left.

More on Impact/Outcome Indicators

For more information about impact/outcome indicators please use the index below which provides and example of a framework for health promotion impact/outcome indicators and some examples.

A FRAMEWORK
   A framework for Health Promotion Outcomes

EXAMPLES OF HEALTH IMPACT/OUTCOME INDICATORS
   Health Promotion Outcomes
   Improved health literacy (awareness, knowledge, skills)
   Improved social influence & action
   Healthier environment
   Healthier personal behaviours
   Healthier environments
   More effective health services

MORE EXAMPLES OF INDICATORS FOR COMMON HEALTH PROMOTION OBJECTIVES
   Improved social support
   Improved quality of life
   Strengthening community participation
   Strengthening partnerships with other organisations
   Improved organisational capacity

REFERENCES

A FRAMEWORK

A framework for health promotion outcomes
The connection between the impact/outcomes of your project and broader health and social outcomes at a population level is a complex one. However understanding where your work fits within this broader context can help shape what is an appropriate evaluation for your project. Don Nutbeam (1998) has come up with an outcome model for health promotion which shows where health outcomes for a health promotion project may fit within broader and more long term outcomes.

The red level represent the level at which project impact/outcome evaluation is appropriate. The health promotion outcomes are more immediate and they represent changes to personal, social, and environmental factors. These then influence change in the next level - intermediate health outcomes which represent the determinants of broader health & social outcomes

This can provide a helpful framework for the development of indicators. Nutbeam's article goes on to provide some examples of indicators within this framework. Those impact/outcome indicators related to health promotion projects are as follows:

EXAMPLES OF HEALTH IMPACT/OUTCOME INDICATORS

Health Promotion Outcomes

These reflect changes to those personal, social and environmental factors which are a means to improving people's control, and thereby changing determinants of health (intermediate outcomes). They also represent the more immediate results of HP activities.

Health literacy :

• improved health knowledge & motivation re lifestyles
• improved knowledge, where to go, what to do to get access to services (health & other)
• improved skills & confidence

Social influence & action :

• mobilisation of human and material resources to overcome structural barriers
• enhanced social support
• reinforce social norms conducive to health, eg. improved social connectedness, improved community competency & community empowerment, greater social capital, greater community ownership of program

Healthy public policy and organisational practice :

• Healthier public policy & organisation practices to improve access, social benefits, appropriate housing
• Development of policy statements, legislation, regulations
• Development of organisational procedures
• Management practice
• Funding & resource allocation
• Institutionalisation of HP programs

Intermediate Health Outcomes

Personal behaviours :

• Improved disease and injury prevention
• Reducing risk of ill health
• Healthier lifestyles

Environments :

Physical environment

• Improved safety
• Improved access to services
• Less hazardous to physical health

Socio-economic environment

• Improved social and economic conditions

Effective heath services :

• Improved access and appropriate use

Nutbeam suggests that evaluation of health promotion action should consider the following

1. Improved personal health literacy

2. Changes to public policy and organisational practices

3. Changes to social norms and community action that increase people's control (individually or collectively) over the determinants of health.

(Nutbeam 1998)

MORE EXAMPLES OF INDICATORS FOR COMMON HEALTH PROMOTION OBJECTIVES

Improving Social Support :

Extent to which social support network provides better quality, or more · information

• practical assistance
• affirmation of personal worth
• someone to lean on in times of crisis
• reciprocity
• desired level of support

(Hawe et al 1990)

Improved Quality of Life :

Participants report as a result of the project/activities, their lives are more · rewarding, enjoyable, productive

(Hawe et al 1990)

Strengthening Community participation :

• Evidence of formation of lobby groups, winning resources
• changes in perceived power and community confidence
• made a difference to problem solving capacity

(Hawe et al 1990)

• commitment (value importance of community)
• articulateness (how well express views)
• participation (active contribution towards definition and achievement of goals
• conflict containment and accommodation (recognition and management of differences)

(Goeppinger et al 1985)

Strengthening partnership with other organisations :

• Increase in number of joint projects or activities
• Evidence of policy change or development of organisational processes to support working together
• Members report an increased capacity to achieve outcomes together that they could not achieve alone

Improved organisational capacity :

Hawe et al (1997) offer the following levels and dimensions of improved capacity building.

1. Health infrastructure or service development - structures, organisation, skills, resources.

2. Program Maintenance and sustainability - capacity to continue to deliver a particular program through a network of agencies, in addition to, or instead of initiating agency.

Indicators

• Evidence of program taken up in the main business of organisation(s) ·
• Maintenance of sun system or personnel practices
• The supportive sub system, eg. housing and facilities offered ·
• Degree of formalisation of accountability structures

3. Problem solving capability of organisations and communities. - a more generic capacity to identify health issues, and develop appropriate mechanisms to address them.

REFERENCES

Eng & Parker (1994) "Measuring Community Competence in the Mississippi Delta; the interface between program evaluation and empowerment Health Education Quarterly 21)

[Goeppinger J., Baglioni AJ. Community Competence : a positive approach to needs assessment American Journal of American Psychology 1985 13(5) ] ·

Hawe P., Noort M., King L., Jordens C. (1997)"Multiplying Health Gains : the critical role of capacity building within health promotion programs." Health Policy 39

Hawe P., Degeling D., Hall J. (1990) Evaluating Health Promotion: A Healthworker's Guide, MacLennan & Petty, Sydney

Don Nutbeam (1998) "Evaluating Health Promotion-progress, problems and solutions", Health Promotion International Volume 13, No. 1