Questions about evaluation and outcome measurement? Come for answers on Friday, December 11, 8-10 a.m. to the Greater Kalamazoo United Way Board Room (709 S. Westnedge Avenue). Any community agency is welcome to attend this free session. Come share concerns and ideas related to evaluation! We can learn from each other. Participation will be limited to the first 40 persons who call to reserve a seat. Call Patricia Hamilton at (616)343-2524. Bring your questions!

Human service organizations face a number of dilemmas as they add outcome measures to the things they already track and report. A University of Iowa newsletter identifies four. How do agencies in the Greater Kalamazoo area deal with these dilemmas?

DILEMMA #1 - Responsibility for defining outcomes.
In the Greater Kalamazoo area program/agency staff have primary responsibility for defining outcomes. They are the experts and should specify the important outcomes of their work. United Way of America's Measuring Program Outcomes cautions agency staff, however, not to work alone. Clients, funders, employers of clients, and other community-based programs also have a stake in the direction of programs and thus need to be involved in establishing significant and realistic client outcomes. Important stakeholders who have not been consulted may justifiably raise questions about staff's choice of outcomes.

DILEMMA #2 - Overpromising outcomes, given the realities of funding and length of time for which services are provided.
Community agencies in Greater Kalamazoo determine their outcomes according to the type of services they provide. Some only focus on short-term assistance for individuals as outcomes. Others can set benchmarks for the long term, because they work with the same clients from year to year and want to track their progress. We are all learning what constitutes a realistic outcome. Some agencies have sought feedback from stakeholders on the viability of defined outcomes for their programs. That feedback has caused rethinking, not only of outcomes, but even of program claims and objectives.

DILEMMA #3 - Convincing funders that prevention outcomes are as worthwhile as direct service outcomes.
Research findings may justify prevention programs, but local community members want to know such programs work here. The fact that something undesirable did not happen to program participants is, by itself, not very convincing. Positive human development has multiple causes; one prevention program can rarely take full credit. Instead, prevention workers report what they do and ask participants to report how they used the information they learned. At-risk participants may be asked what they would have done, if not for the prevention program.

Each prevention program can show how it contributes to desired outcomes. Each can report how participants behave or use what they are given. Each prevention program can report on its role in the community - what it does, what it tries to prevent, and how participants use what they learn. Prevention staff may even be able to show that non-participants have a higher incidence of undesirable development than do participants from a similar demographic group. Use of logic in reporting outcomes can help community members better understand the need to invest early in human development rather than to treat negative offenses or undesirable events down the road.

DILEMMA #4 - Using outcome data against a program when results fall short of expectations, rather than as a tool for program improvement.
With long-term thinking and continuous measurement, Kalamazoo organizations can address this dilemma. As long as programs regularly measure how well they are doing and use their findings to make adjustments, they can demonstrate their efforts to achieve desirable outcomes and eventual successes. We aim to strengthen programs, not penalize programs in the event of negative results in the short term.

Congratulations, Kalamazoo! We're evaluating well!

• When we began the [evaluation] process, we gathered information on 23 specific outcomes per child for the Mentoring/Companion (One-to-One) Program. We quickly realized our focus was too broad and we needed to be more focused and could only be working on a couple of specific outcomes per child at a time. While we simplified by decreasing the number of specific outcomes we were looking at per child for the reporting period, we still find that we need to work on this more.
• With the Bridge Program we realized we had to use a different process for gathering and reporting outcomes than the one we used for the Mentoring/Companion (One-to-One) Program. We have aggregate information on the group and we will be looking at ways to gather and report on changes in individual children.
  --Greta Williams, Big Brothers Big Sisters, A Community of Caring

• Everyone’s  responsibility
• Continually asking good questions, getting answers, and taking action based on those answers
• Integrated into the day-to-day operations of the organization
• A developmental process
• Collaborative and dependent on information sharing
• Time well spent
• Going to ensure the organization’s health and viability in the long run in a changing environment