Evaluation Café
Lively discussions, engaging debates, and dynamic presentations about evaluation over the lunch hour. Bring your lunch – freshly brewed coffee and gourmet cookies provided.
All Evaluation Café events take place from 12:00 to 1:00 p.m. at The Evaluation Center, 4405 Ellsworth Hall.
Past Events: 2002-03 2003-04 2004-05 2005-06 2006-07 2007-08 2008-09 2009-10
Handouts and/or slides available for most presentations.
September 19, 2006
Presenter: Michael Scriven – Professor of Philosophy, Associate Director of The Evaluation Center, and Interdiscipinary Ph.D. in Evaluation Program Director, WMU
field?
September 26, 2006
View the handout for this presentation (PDF)
Presenter: James Sanders – Professor Emeritus of Educational Studies and former Associate Director of The Evaluation Center, WMU
profession, a rolling needs assessment would be useful. By identifying issues and problems in evaluation for which we have no good answers, we can concentrate our research on evaluation and pool wisdom gained from evaluative experiences. If for no other reason, this continuing look at the field of evaluation will provide topics for dissertations, research proposals, and communications among evaluation professionals. Be sure to bring your favorite candidate for this top 10 list.
October 3, 2006
Presenters: Amy Gullickson & Wes Martz – Interdisciplinary Evaluation Doctoral Students; Michael Scriven – Professor of Philosophy, Associate Director of The Evaluation Center, and Interdiscipinary Ph.D. in Evaluation Program Director, WMU,
Abstracts:This session includes abbreviated presentations of papers prepared for the upcoming American Evaluation Association conference.
Strategic Evaluation of Business and Industry: Evaluative Approaches for Improving Organizational Culture (Amy Gullickson)
Employee engagement and satisfaction are integral components for adding value to an organization and its products. Forward thinking corporations are moving away from command-and-control models toward learning organization and complex adaptive systems.
This paper explores how evaluative tools such as needs assessment, after-action review, and minimum specification documents can be used to help increase employee satisfaction and performance and thus, results for shareholders.
Building Shareholder Value Using Formative Evaluation (Wes Martz)
The use of formal evaluation as a tool to drive value from improved operational efficiency presents an opportunity to strengthen an organization’s performance and shareholder value. This presentation explores the application of evaluation outside the scope of human resource development initiatives and considers evaluation as a tool to build shareholder value. Specifically, a case study of a formative evaluation conducted at an operating division of a U.S.-based global manufacturer of industrial products is presented.
The Evaluator's Responsibility for the Consequences of an Evaluation (Michael Scriven)
Evaluations often have consequences, some intended, some unintended. The hard questions concern the extent to which the evaluator is responsible for these consequences. If the evaluation concludes with recommendations, then it's reasonable to suppose, and it's legally likely, that the evaluator will be held (at least partly) responsible for those consequences. But the much more fundamental question is how a typical evaluative conclusion can imply consequences at all. I will examine the traps that evaluatorshave fallen into when too-quickly moving from evaluative conclusions to recommendations, and indicate how and when to avoid the traps – or to avoid making recommendations.
October 10, 2006
View the slides for this presentation (PDF)
Presenter: Wes Martz – Vice President, Corporate Marketing, Kadant Johnson (Three Rivers, MI); Founder, Martz Marketing Group (Kalamazoo, MI); and Interdisciplinary Evaluation Doctoral Student, WMU
October 17, 2006
View the slides for this presentation (PDF)
Presenter: Steven Ziebarth – Professor of Mathematics, WMU
October 24, 2006
View the handouts for this presentation (PDF)
Presenter: Ryoh Sasaki – Interdisciplinary Evaluation Doctoral Student, WMU
community should consider for better aid evaluation are discussed.
November 7, 2006
View the handout for this presentation (PDF)
Presenter: Thomaz Chianca – Interdisciplinary Evaluation Doctoral Student, WMU
Heifer International as an independent party to evaluate their projects in five countries: the U.S., Peru, Thailand, Nepal, and Albania. A specific evaluation approach has been devised, the Heifer Hoofprint Model, that not only responds to Heifer’s main interest in assessing its the impact of its programming on the lives of project recipients, but also provides a comprehensive assessment of merit, worth, and significance of Heifer’s work. The project has involved about 20 evaluators (2/3 from the U.S. and 1/3 from other countries) and has had investments of approximately $300,000. This presentation will focus primarily on the discussion of the Hoofprint Model, the major challenges we faced in designing and implementing the evaluations, and the initial accounts on uses and consequences of those evaluations to different stakeholders, including Heifer headquarters, country offices and local projects, and the WMU Evaluation Center. Participants will be invited to contribute ideas to improve the next round of Heifer “impact” evaluations.
November 14, 2006
View the slides for this presentation (PDF)
Presenter: Daniel Stufflebeam – Distinguished University Professor and Former Director of The Evaluation Center, WMU
Government Auditing Standards. These standards cover program evaluations as well as financial audits. Using the WMU program review context, this presentation will summarize and consider the potential utility for Western of GAO’s most recent version of the Government Auditing Standards.
November 21, 2006
View the slides for this presentation (PDF)
Presenter: Jeffrey Berk – Vice President of Products and Strategy & Susan Johnston – Account Manager, KnowledgeAdvisors, Chicago
KnowledgeAdvisors, a business intelligence company that helps organizations measure and manage their learning investments. Berk is the author behind the organization’s proprietary learning measurement methodologies. He is also the functional architect of the technology product Metrics that Matter, which helps organizations measure the effectiveness of learning investments through automation and technology.
November 28, 2006
Presenters: Denise Hartsough – Director of Community Investment; Suprotik Stotz-Ghosh – Associate Director of Community Investment; Ronda Webber – Community Investment Associate, Greater Kalamazoo United Way
December 5, 2006
View the slides for this presentation (PDF)
Cost Analysis: A Case Study Illustrating How Different Assumptions Influence Evaluative Conclusions
Abstract: This presentation will illustrate with a case study, the differences between rudimentary economic analyses and sophisticated economic analyses. Additionally, the presentation will show how different assumptions affect evaluative conclusions and in some instances, completely reverse an evaluative conclusion. The importance of sensitivity analyses in cost studies will be highlighted, using salary and discount rate to illustrate this concept. Finally, the paper will briefly discuss real-world constraints on data availability and how this problem was resolved in the ABC study, and explain why sophisticated economic analyses may be difficult for stakeholders to understand.
January 16, 2007
Need Funding: Or, What Can Evaluation Offer in Terms of Allocating, Apportioning, and Distributing Federal Research Funds
Abstract: This Evaluation Café is intended to serve as an open forum, or dialogue, to explore the potentials for what evaluation has to offer in the way of allocating, apportioning, or distributing national research funding and setting national research initiatives and agendas. During the Cold War era, the notion of research as an autonomous pursuit, free of interference by sponsors, was asserted by the United States’ Presidential Science Advisor, Vannevar Bush in the 1940s. Bush managed to instill the idea of a generously funded yet self-governing scientific establishment by stressing the importance and inevitable benefits of research. In the last 50 years, the rationale for government support of research has been the contribution of science and technology to military security and national prestige, coupled with a sense—taken mainly on faith—that a strong research community will more than pay for itself in economic and social benefits. Following the end of the Cold War, old questions about the control of the United States’ research agenda and procedures and methods for determining the allocation of resources among fields and disciplines, research institutions, and regions were once again surfaced.
January 23, 2007
Presenter: Michael Scriven – Professor of Philosophy, Associate Director of The Evaluation Center, and Interdiscipinary Ph.D. in Evaluation Program Director, WMU
experiment at a preschool in New Haven. The children were becoming competent touch typists and learning spelling and composition on a couple of mainframe-driven electric IBM typewriters with a tape-drive attachment – and enjoying the experience. In 2001, Larry Cuban, a professor at Stanford, published a book (Oversold and Underused: Computers in the Classroom) in which he concluded that computers, although now ubiquitous, had made no discernible contribution to literacy or other academic subjects. There is no reference in his index to Moore or his work; it has vanished from the radar. There are other cases like this. What is the real truth? And what can we learn from this extraordinary story of paradox and prejudice? It turns out to be a story with shocking implications for business as well as education.
January 30, 2007
View the slides for this presentation (PDF)
Presenter: Ryoh Sasaki – Interdisciplinary Evaluation Doctoral Student, WMU
validity and usefulness of the current aid evaluation practices. However, the ideal mechanism for monitoring and evaluation (M & E) suitable for SWAps is still under discussion. The presenter, who has four years of experience in supporting application of this new approach in Tanzania’s agricultural sector, will present his view on SWAps and discuss issues related to and possible recommendations for the appropriate M & E mechanism.
February 6, 2007
View the slides for this presentation (PDF)
Presenter: Ann Maxwell – Regional Inspector General, Office of Inspector General, U.S. Department of Health and Human Services (Chicago)
February 13, 2007
View the slides for this presentation (PDF)
Presenter: Robin Miller – Associate Professor of Psychology, Michigan State University & Editor, American Journal of Evaluation
outcomes. One popular tool, system dynamics, has principally been used to guide evaluation planning. In this presentation, I will demonstrate how system dynamics may be used to synthesize evaluation data to provide insight about program processes and outcomes. I will present data from an ongoing investigation in which my colleagues and I are applying system dynamics to the evaluation of HIV prevention evidence-based programs.
February 20, 2007
View the slides for this presentation (PDF)
process. He will be present and discuss two processes: Targeted Selection and 360 Feedback. Both methods have been used to successfully evaluate both prospective and incumbent personnel for many of Quantum’s customers
February 27, 2007
View the slides for this presentation (PDF)
Presenter: Mike Trevisan—Professor and Director, Assessment and Evaluation Center, Washington State University & Visiting Scholar, The Evaluation Center, WMU
appraisal of the state of practice of evaluability assessment (EA) as represented in the archived literature will be presented. Twenty studies were located, representing a variety of programs, disciplines, and settings. Most studies employed common EA methodologies, such as document reviews, site visits, and interviews. Other methodologies such as expert testimony and examination of program statistics were also found in the literature. The main purpose for conducting EA mentioned in these studies was to determine whether or not a program was ready for full evaluation. Outcomes included the construction of a program logic model, program development, and or modification. The findings suggest that EA may be employed more widely than previously thought. Recommendations to enhance EA practice are offered.
March 13, 2007
Program Evaluation in the American States
Abstract: This session examines reports written by state government legislative program evaluation organizations in the United States. These reports (variously called performance evaluations, program evaluations, effectiveness and efficiency audits, managerial audits, or, more commonly, performance audits) are an important part of legislative oversight and policymaking. A random sample of reports released by state legislative agencies from 2001 through 2005 were examined to learn their common and differing methods, goals, orientations, timelines, and scope. An approach was developed to metaevaluate these reports using criteria from the program evaluation standards, government performance auditing standards, and the Key Evaluation Checklist. The session also draws on the literature exploring metaevaluation across a large number of evaluation reports. The two goals of this session are 1) to provide a basis of information about state legislative program evaluation unit reports and 2) to test an approach for feasibly and competently metaevaluating state legislative program evaluation reports.
March 20, 2007
Read the paper on which this presentation was based.
Presenter: Lois-ellin Datta – President, Datta Analysis, Captain Cook, HI
randomized control design be used in 2007 for an evaluation of Sesame Street, as the design was used in 1968? At that time, the evaluation was a test of whether television could be an effective way of helping preschoolers get ready for kindergarten, with “ready” to include pre-academic, cognitive, social, and other skills.
Probably not, because Sesame Street is so infused in every-day experiences for children, that a meaningful comparison or control group could not be found in 2007. At least some of the popular 2007 television programs for preschoolers have so many elements akin to Sesame Street that a “compared to what” could not test the original question, long since answered, I believe, “Yes.” Further, the likelihood might be close to zero that a group of randomly selected group of control parents and children would agree – for the sake of an evaluation – not to watch Sesame Street or any other television program. The control group would, in all likelihood, be active…the children, if not the parents!
I’m not sure how many proponents of the widespread applicability of RCT when attribution is a purpose would agree with this argument. One might think, “Quite a few.” In actuality, there doesn’t seem to be adequate recognition of the challenge of the active control group in areas such as education, health, and social services. For example, a congressionally mandated randomized control experiment to prove, once and for all, whether Head Start is worthwhile will soon issue its final report. In technical terms, “Aargh!” For example, Cordray and Lipsey to their immense credit reported in thorough detail what happened in a large, very costly national experiment to test the effectiveness of several approaches to treatment for men with multiple diagnoses when the control group men decided they wanted to choose their own treatments. For example – if you are currently doing an evaluation in a school, find out what else is happening that is relevant to your outcomes in addition to the program you are judging. Dollars to doughnuts, your “experimental” or focal group is someone else’s comparison and vice versa.
The active control group can matter a great deal when it comes to analysis and conclusions. The effects on variance can lead to macro-negative effects, something known since about 1972 when Stallings et al. published their SRCD monograph and other reports on the Follow Through experiment. Arguably, the more active the control group in finding experiences similar to that of the treatment, the more threatened the logic of the RCT or comparison designs. Sometimes discussed as “contamination,” this seems to me a factor both different and more difficult to deal with.
What to do? The Café will include a discussion of some approaches that seem useful and that in at least some instances, have proven to be able to sort out effects that truly exist, thus avoiding unnecessary death by evaluation for worthy ideas.
March 27, 2007
Presenter: Liesel Ritchie – Senior Research Associate, The Evaluation Center, WMU
April 3, 2007
Presenter: Mozdeh Bruss – Associate Professor of Family and Consumer Science, WMU
April 16, 2007
View the slides for this presentation (PDF)
Presenter: Gene Glass – Regents’ Professor of Educational Leadership & Policy Studies, Arizona State University
April 24, 2007
View the slides for this presentation (PDF)
Presenter: Patricia Rogers – Associate Professor in Public Sector Evaluation & Director, CIRCLE (Collaborative Institute for Research, Consulting and Learning in Evaluation), Royal Melbourne Institute of Technology, Australia
implementation environments. This presentation will discuss evaluation of interventions involving complex causal relationships – such as where an intervention is necessary but not sufficient (with other contributing factors needed for success), or sufficient but not necessary (with alternative causal paths available), or where the causal relationships are of interdependence not simple linear causality - and present examples of ways to provide credible evidence of impact without RCTs.