Getting at Impact

A Beginners Guide

By Paul Clements

Back to Table of Contents

Most community organizations face similar questions. In recent years, some funders have started asking for reports on impacts or outcomes – actual benefits for participants during or after their involvement with a program. Funders want to know what changes their money is making, how people’s lives are different. Most community organizations want to know this too: Is what we’re doing worthwhile? Answering such questions is tricky, and most organizations are not used to doing it. But creating an impact measurement process for your organization is both possible and useful.

Moving to impact assessment is a big shift. Community development organizations are more accustomed to tracking inputs and outputs: Did you spend the money in the amounts and categories you said you would? (Answer by line item.) How many housing units did you build? How many people went through the training program? This kind of reporting encourages a counting mentality, as though getting bigger numbers were important by itself.

Changing what we measure and report on can have a big influence on how we carry out our work. This might sound like the tail wagging the dog, but the words and concepts we use can frame the way we think, and as time goes on they can even influence what we see. Focusing only on inputs and outputs can lead organizations to continue building housing when it no longer supports their actual mission, for example, or to miss some of the indirect effects that their development programs are having.

On the other hand, specifying the changes the program is trying to bring about in terms that make sense for participants, agency staff, and other stakeholders can be an occasion to build a stronger consensus on the program strategy and make sure it fits the organization’s mission. This can also shift the balance of power, as funders are given an agenda developed by staff and participants. More important, it focuses everyone’s attention on the shared transformational task. Then the reports you write support thinking and action that build the program.

But measuring impacts is not simple. “A sense of community,” “neighborhood revitalization,” “empowerment,” and “self-esteem” are all tricky concepts to put our fingers on. We often rely on informal assessments, feelings, or anecdotes to assure ourselves and funders that we are on the right track. There is value in those informal assessments, and they should never be skipped or ignored. But sometimes when we are asked to identify program impacts we jump too quickly to things that are easy to quantify, or we ask questions so broad and vague (“On a scale of 1 to 5, how positively do you feel about yourself?”) that we’re not sure what the answers really mean.

It is possible to create a good, formal impact measurement system, however, and there are good reasons to do so. This information anchors ongoing strategic planning and helps the organization demonstrate results to funders and other stakeholders. The information-gathering process heightens awareness of the topics under investigation, and outcome information helps different stakeholders hold each other accountable.

Doing this well will take about five percent of an organization’s human and financial resources. It will probably take more than this in the year you establish your system if you are starting from scratch, but you should expect it to run on about five percent a year – and this should be included in every grant request. If you can’t release these resources or if you run on a small and tight budget, it is probably better to keep your impact assessment informal, based on observations and discussions.

There are several steps to developing an impact evaluation process for your organization. First, you need to figure out what impacts you hope to achieve. Impacts are the changes in people’s conditions due to program activities compared to what we would expect without the program. The number of children immunized is not an impact; the reduction in disease rates is. The number of units you build is not an impact; higher levels of housing stability, improved neighborhood safety, and improved conditions for the families moving into those units are.

Your “benefits picture,” or list of desired impacts, should be created with input from program staff, participants, funders, and other key stakeholders. A benefits picture for a housing program might include reduced monthly housing payments, a greater sense of security and self-esteem for residents, a heightened sense of community, stronger property values, and less crime for the whole neighborhood. It is important for the list of desired impacts in your benefits picture to be complete, since it provides the framework all the way along: for selecting indicators, for seeing how they work together, and for interpreting the information when the evaluation system is in use.

Some benefits may be relatively easy to measure directly, such as rising property values, voting rates, or community use of public spaces. But many will not. Most likely, some of the more important outcomes in your benefits picture will be qualitative and subjective. Sense of community or quality of leadership cannot be directly quantified. People have different ideas about the quality of a house, what self-esteem involves, or what makes a neighborhood attractive.

To measure such intangible benefits, we need indicators. Indicators provide consistent steps for collecting data that tell us something about the phenomenon we are actually interested in. The percentage of people below a certain income is a pretty good indicator of poverty. The number of people who know more than three of their neighbors can be an indicator of community cohesion. We know the information from indicators is not going to be perfect. But it can nonetheless give an organization a good idea of how it is doing. An impact evaluation system should include a range of indicators that fits the benefits picture.

Stop Abusive Family Environments (SAFE) in Welch, West Virginia, started out as a provider of shelter and other temporary services to battered women. In 1999 it decided that the next thing to do was to help its clients get into affordable, stable, permanent housing, and it began a housing development program. It also decided to evaluate the program in terms of the improvement in housing conditions for the women it served.

The quality of a house involves many different factors: physical structure, safety, furnishings, hygiene, space, and social features. SAFE devised a point system: Two or more major problems with plumbing and electrical features or having no insulation would count for three points. Not being allowed to keep a pet if you wanted one would count for one point, as would excessive noise. Not everyone would agree on which of these things were more important, but clearly if you moved out of a house that got 20 points and into one that got only two or three it would be a big improvement. Staff brainstormed about features that contribute to the quality of a home, and then interviewed residents to improve and prioritize the list. When you have worked out an indicator like this, you need to review it with representatives of all your stakeholders to ensure that it reflects their priorities and perspectives and to secure their buy-in.

To help practitioners take the initiative in determining indicators that speak to their needs, the Development Leadership Network (DLN), a professional association of community development practitioners, organized The Success Measures Project (SMP) in 1997. The goal was to establish a set of practitioner-driven “success measures” that would be relevant, comparable, and easy to use – and really get at the question of impact.

SMP focus groups first created generic benefits pictures, focusing mainly on housing, economic development, and community building programs. Then they developed dozens of practical, user-friendly outcome indicators. Twenty-eight organizations have been field-testing these indicators, and many more are using the first edition of the Success Measures Guidebook in which they are described.

Several community development corporations in Detroit, one of the field-test sites, are using SMP indicators to measure the sense of community, the visual attractiveness of their neighborhoods, neighborhood security, and the extent to which basic needs are met by local businesses. For each of these categories, the Guidebook provides ideas about what to measure, ways to get information, and how often to measure. For example, for “sense of community” it suggests using a survey to track residents’ participation in community life, the share of residents who know names of neighbors in adjacent buildings, and residents’ satisfaction. It also recommends keeping track of the number of organized community activities in one year. For neighborhood attractiveness, the Guidebook suggests gathering a focus group of residents to determine a substantial set of small indicators (number of abandoned cars, amount of graffiti, condition of sidewalks), and then doing a neighborhood tour to assess these indicators block by block. It recommends that these tours should happen at least once a year, and warns groups that they will need three to four years of data before trends can start to be identified. The Guidebook provides similar information, with varying amounts of detail, for all its other categories of impacts.

Working from these starting points, the Detroit groups have designed specific survey instruments. They are available from DLN, with sample reports, as models for other organizations.

Once you choose a set of indicators, check them back against your benefits picture to see if you have left any desired impacts out. If you have not found an indicator for one or two of them, that may be all right. Some things you care about may simply be too hard to measure, or the effect of your program may be overwhelmed by other factors. Still, it should be possible to get a good representation of most of the more important program benefits in your benefits picture. Make especially sure that you are not focusing on factors that are easier to measure but less important. Also, keep in mind the results you are not measuring, so you can continue to make informal judgments there. (Click here for some tips on collecting data.)

The information from a survey or questionnaire can show program outcomes but not impacts. Impacts are the actual changes compared to what you would expect without the program. To get at impacts you have to interpret the data using judgment based on your overall knowledge of the program and its environment. In some cases it may be possible to make a comparison to a similar location with no program like your own. As years go by you will have a record of program results that can inform planning, funding requests, and communications about the program in general.

In the long run, programs that can demonstrate their effectiveness are better positioned to grow and to be replicated. As impact assessment becomes more and more common it will make the community development field more professional while also strengthening links with communities. Government offices and foundations will be better able to justify investments in community development, and the public at large will see more clearly the value of community-based approaches.

Dr. Paul Clements was research director for the Success Measures Project. He teaches international development administration at Western Michigan University.