Evaluation 101! Measurement problems solved! Mark you calendar for the next open discussion: September 18,1998, 8 -10 am. in the Board Room at Family & Children Services (1608 Lake Street). Participation will be limited to the first 40 persons who call to reserve a seat. Call Patricia Hamilton at (616) 343-2524.

Questions and Answers from a June 12 meeting included the following:

Q: How can we report evaluation data concisely?
A: First, look at your outcomes. If your outcome is increased employment of clients, report evidence that you are accomplishing that outcome. Examples of what to report: testimonies, survey/telephone interviews. Use a summary of the evidence collected.

Q: What is the difference between outcome objective, performance measure, and desired effect?
A: Desired Effect: what we hope to accomplish with the client (example: increased responsibility in paying bills). Stating the Desired Effect is a tool to help you write the Outcome Objective. If writing the Desired Effect does not seem useful, skip that step and write the Outcome Objective.
Outcome Objective: what happens for the client/benefits to the client (example: client able to handle current utility bills)
Performance Measure: how can we tell (example: pays utility bills)
Method of Measurement: tool for measuring outcome (example: follow-up survey to determine whether clients are more responsibly paying bills after receiving service).

Q: We do a lot of referrals. Are agencies restricted from giving me follow-up information?
A: Depending on the agency and how much information is needed, there are certain restrictions.

Q: We spend a great amount of energy and time following up referrals. Is that a legitimate use of our limited resources?
A: The answer to the question Is the referral process working? is something we need to know. Many of our current practices are based on the assumption that referral works, but do we really know? Yes, follow-up is a legitimate use of limited resources. Otherwise, we may continue to expend resources on an unsuccessful strategy without knowing the strategy doesn't work. To reduce the cost of followups, check a sample. Document cases in which no agency can help and bring this to the attention of funders.

Q: How do you quantify/measure ethereal goals such as group leadership, sense of self, etc.?
A: Brainstorm about somebody who has the quality you want to develop in the client (e.g. sense of self). Identify how that person is different from someone who does not have sense of self. Maybe one of the aspects identified is that someone with sense of self is task-oriented. You can use a counselor to observe clients and decide which ones have the quality you have defined. Staff observations, video clips, and case histories are legitimate ways of measuring your outcomes. We shouldn't downplay the importance of qualitative information. Telling factual stories is a wonderful way to measure outcomes.

Many of us left the meeting on June 12 feeling once again that, by working together, we will improve our evaluation practices.

Tips & Tools
I have worked backwards to see if I can realistically do my outcomes with the resources I have. It turns out I can't. That helps me to re-evaluate my goals (even the agency's goals) or adjust the activities.
--Christina Holmes, Hispanic American Council

The Greater Kalamazoo United Way Resource Library has added workbooks on questionnaire surveys and quantitative (statistical) data analysis. These are easy to-read guides and available for use during business hours.

In Everyday Evaluation on the Run (Allen & Unwin, 9 Atchison St., St. Leonards NSW 2065, Australia, frontdesk@allen-unwin.com.au), author Yoland Wadsworth suggests that "to successfully blend evaluation (thinking) into our everyday lives requires us to deliberately set aside time - a minute at the start of that activity or end of that discussion, a few moments at the end of each meeting, an hour a week, a day during a month's campaign, or a week each year to reflect on the value of what we are doing or planning to do. This time needs to be collectively agreed upon as valuable, labelled legitimate, and treated as precious. Only in this way will we gradually get the evidence we need that will guide and strengthen our practice and reduce any fears and anxieties about [producing outcomes and making progress]."

• Everyone’s  responsibility
• Continually asking good questions, getting answers, and taking action based on those answers
• Integrated into the day-to-day operations of the organization
• A developmental process
• Collaborative and dependent on information sharing
• Time well spent
• Going to ensure the organization’s health and viability in the long run in a changing environment