Evaluation for Learning

News for an Evaluating Community Fall 2000/Winter 2001

OBSERVATION AS AN EVALUATION METHOD

Looking for tips on using observation of participant behavior as an evaluation method? Perhaps your program participants are too young to fill out a survey, or you prefer a trained observer's comments to self-reports. Drs. Jianping Shen and Gunilla Holm of the Western Michigan University College of Education will conduct a workshop on observation as an evaluation method on Friday, March 23, 2001, noon-1 p.m. in the Greater Kalamazoo United Way Board Room, 709 S. Westnedge. Bring your lunch; beverages provided. To reserve a space, call Elaine Griffin at 616/343-2524.

EXPERIENCES WITH OUTCOME MEASUREMENT

Feedback from 381 human service programs around the country indicates that they have found outcome measurement both a reward and a challenge. In 1999 United Way of America contracted with James Bell Associates (JBA) to survey agencies about their experiences with outcome-based evaluation. Questions focused on the benefits of evaluation, barriers and supports encountered, and agencies' use of findings.

The report includes data from programs funded by United Ways in North Carolina, Ohio, Wisconsin, Minnesota, Louisiana and New York. Nearly two-thirds of the programs surveyed have been measuring outcomes for four years or more. Many began evaluation due to prompting from several parties. Four of five programs reported prompting by their United Way, and three of five indicated that internal sources--professional leadership or staff--provided the impetus.

Benefits of Evaluation
Survey respondents agreed that outcome-based evaluation benefited their programs especially in helping to:

  • communicate program results (88%).
  • focus staff effort on common goals and purposes (88%).
  • clarify the purpose of the program (86%).
  • identify effective practices (84%).
  • successfully compete for resources/funding (83%).
  • enhance record-keeping systems (80%).
  • improve service delivery (76%).
The only deleterious effect reported by a majority of programs was that outcome measurement had overloaded their record-keeping capacity (55%).

Barriers and Supports

Barriers identified in the survey include insufficient resources and conceptual challenges. Many respondents agreed that they could have used:
  • more staff time (60%).
  • better computer hardware and software for storing and manipulating outcome data (57%).
  • more time to develop and test the outcome measurement system (50%).
Two-thirds of the programs reported difficulty with identification of:
  • manageable data collection methods (68%).
  • relevant outcome indicators (66%).
  • appropriate outcomes (65%).
Respondents also indicated the types of supports that helped them continue outcome measurement despite the barriers encountered. An overwhelming majority of programs agreed there had been top executive involvement in evaluation (88%) and support by the board of directors (87%). Other factors include ongoing staff commitment to using the results of evaluation (71%) and confidence in the validity of outcome data (70%).

Agencies' Use of Outcome Results

Tracking outcomes gives agencies feedback and data that can help them increase the effectiveness of their services and communicate the value of their activities. Nearly three-fourths of respondents agreed that outcome measurement "has had a positive impact on this program's ability to serve clients effectively," and that "outcome information should be used in making decisions about program funding." About half of the respondents identified barriers to use of outcome findings: lack of flexibility in use of outcome data, and uncertainty about how to identify program strengths/weaknesses (both 48%). Despite the barriers, 9 of 10 program directors surveyed said they would recommend implementation of outcome-based evaluation to the director of a similar program.


TIPS & TOOLS


For a copy of the 12-page report on Agency Experiences with Outcome Measurement, see www.unitedway.org/outcomes/ or request item 0196 when you call Sales Service/America at 703/212-6300 ($1 + shipping.) The United Way of America website has a list of other resources created by UWA, as well as answers to Frequently Asked Questions and an Online Resource Library with links to a variety of related sites on program evaluation, nonprofits, evaluating community initiatives, performance and accreditation, and youth issues.


A survey was included with printed copies of this issue. It was to be completed and returned by February 28, 2001.

Asking Good
Questions





Collecting and
Sharing Useful
Information




Using the
Information for:

  • Improvement
  • Accountability





Continuing
to Repeat
the Cycle







Evaluation for
Learning is:
  • Everyone's
    Responsibility
  • Continually asking good
    questions, getting
    answers, and taking
    action based on those
    answers
  • Integrated into the
    day-to-day operations
    of the organization
  • A developmental
    process
  • Collaborative and
    dependent on
    information sharing
  • Time well spent
  • Going to ensure the
    organization's health
    and viability in the long
    run in a changing
    environment








Evaluation for Learning
 newsletter is a product of the
Greater Kalamazoo Evaluation
Project with support from the
Greater Kalamazoo United Way,
the Irving S. Gilmore Foundation,
and the Kalamazoo Foundation

THE GREATER KALAMAZOO EVALUATION PROJECT PUBLISHES
THIS NEWSLETTER TO FOSTER EVALUATION
THROUGHOUT THE KALAMAZOO AREA.

Send your ideas
and examples to: 		Denise Hartsough
Greater Kalamazoo United Way
709 S. Westnedge Avenue
Kalamazoo, MI 49007-5099 		PH: (616) 343-2524
FAX: (616) 344-7250
GREATER KALAMAZOO EVALUATION PROJECT
c/o Greater Kalamazoo United Way
709 S. Westnedge Avenue
Kalamazoo, MI 49007-5099

[Evaluation for Learning Home]     [Top of Page]