Upcoming Events

All Evaluation Café events take place from 12:00 to 1:00 p.m. at The Evaluation Center, 4405 Ellsworth Hall.

Past Events:   2002-03  2003-04  2004-05  2005-06  2006-07  2007-08  2008-09  2009-10

Handouts and/or slides available for most presentations.

2009-10 Events      Fall

September 15, 2009

Title:  Contemporary Trends & Movements in Evaluation:  Evidence-Based, Participatory & Empowerment, & Theory-Driven Evaluation

Presenter:  Chris Coryn – Director, Interdisciplinary Ph.D. in Evaluation Program, WMU

Abstract:  In recent years, three major movements have dominated evaluation practice. These include evidence-based, participatory and empowerment, and theory-driven evaluation (TDE), in particular. Of these, the former has received the greatest attention, particularly in education and health and medicine, mostly centered on randomized controlled trials (RCTs). Whereas RCTs are premised on the assumption that they are the most appropriate method for generating strong knowledge claims as to "does it work?" (i.e., causal description), TDE is premised on the assumption that the approach generates knowledge claims as to "how?" "for whom?" and "under what conditions?" (i.e., causal explanation) a treatment or intervention works by formulating and testing complex causal hypotheses. Participatory and empowerment approaches, however, are essentially premised on the assumption that by engaging stakeholders in the evaluation process, greater buy-in is generated and, therefore, such evaluations are more likely to be used and have an impact than more scientifically rigorous methods that often exclude stakeholders in the evaluation process. In this Café, the key assumptions, strengths, and weaknesses of these three approaches will be critically examined, including their commonalities, with an emphasis on the specific contexts in which each is appropriate and their implications for contemporary evaluation theory and practice.

September 24, 2009

View the slides for this presentation  (2623 KB PPTX)

Title:  RealWorld Evaluation:  Maximizing Utility in Spite of Inadequate Budget, Time, & Data As Well As Conflicting Political Pressures

Presenter:  Jim Rugh – Independent Consultants

Abstract:  Jim Rugh was a coauthor (with Michael Bamberger and Linda Mabry) of the RealWorld Evaluation book published by Sage in 2006.  That book and the many workshops on the subject at AEA and in many countries around the world have continued to be very popular – probably because many new and even experienced evaluators are looking for practical ideas, backed by experience, on how to conduct adequately credible program evaluations in spite of constraints such as inadequate budget and time, lack of data (such as from baseline and comparison groups), political pressures, and so on.  Jim will summarize the RealWorld Evaluation approach and address lessons learned from many workshops and current discussions among networks of evaluators on conducting impact evaluations, especially of international development programs.

September 29, 2009

View the slides for this presentation  (3679 KB PPT)
View the handout for this presentation  (35 KB DOC)

Title:  Evaluating School Mathematics, Science Textbooks, & Classroom Practice Using an Assessment for Learning Framework

Presenters:  Amy Bentz & Jonathan Engelman – Assessment for Learning Scholars, WMU; Steven Ziebarth – Associate Professor, Mathematics, WMU

Abstract:  This session examines research into the extent to which new mathematics and science textbooks have incorporated AfL practices in their materials and, using a similar analysis framework, how we have developed an observation protocol to help gather data about AfL teaching practices within secondary classrooms.  We will examine one section of the framework.  An opportunity will be provided for attendees to use a portion of the protocol.

October 6, 2009

Title:  (Meta)Evaluation at the International Labour Organization:  Current Practice, Potential for Improvement, & Prospects for In-Depth Study

Presenters:  Daniela Schroeter – Director of Research, The Evaluation Center; Anne Cullen – Senior Research Associate, The Evaluation Center; Kelly Robertson – Project Manager, The Evaluation Center

Abstract:  In this session, we will first discuss the independent appraisal of a sample of the International Labour Organization’s (ILO) 2008 evaluation reports developed for its Evaluation Unit (EVAL), using adapted metaevaluation tools and methodologies. Second, we will emphasize differences between evaluation methodologies used in the past two years and invite attendees to join the discussion of means for improving appraisals for ILO EVAL. Third, we will discuss the challenges of comparing metaevaluation findings across years and raters. Finally, we will introduce our ideas for studying ILO evaluation in more detail.

October 15, 2009

Photo
Not
Available

Title:  Exploring a Utilization-Focused Evaluation Option:  Developmental Evaluation Using Systems Thinking & Complex Nonlinear Dynamics

Presenter:  Michael Quinn Patton – Independent organizational development and evaluation consultant

Abstract:  Systems thinking and attention to complex nonlinear dynamics have created new opportunities—and new challenges—in evaluation. Developmental evaluation offers an alternative to traditional linear logic models and an additional option beyond formative/summative designs especially targeted to innovative initiatives in turbulent and uncertain conditions that display the characteristics of emergent complexities. Patton will discuss developmental evaluation as an option in support of more sophisticated matching of evaluation to the nature of interventions within a utilization-focused evaluation framework.

Students will also have the opportunity to review chapters of and provide feedback on Patton’s new book on Developmental Evaluation just before he submits the final manuscript for publication.

October 20, 2009

Title:  Evaluating Evidence-Based Medicine

Presenter:  Cristian Gugiu – Doctoral student, Interdisciplinary Ph.D. in Evaluation Program, WMU

Abstract:  While there is much to admire about existing efforts at grading the strength of evidence, considerable shortcomings and unsubstantiated logic limit their usefulness and validity.

An abundance of grading guidelines exist for evaluating the quality of evidence produced by a study and for generating informed recommendations on the basis of this evidence, the balance between desirable and undesirable consequences, and consideration of costs and relevant values. On this basis, clinicians are able to develop treatment plans tailored to meet evidence-based standards as well as patient needs and desires. Over time, grading systems have become more sophisticated and complex in the methodology employed to evaluate the strength of evidence. Although in the past five years a consensus appears to be emerging with regard to how evidence should be evaluated, it is important to question research assumptions periodically because methodological hubris can be just as dangerous as medical hubris. This presentation will review several popular grading guidelines used by medical researchers. Furthermore, two new grading guidelines, one for grading research designs and the other for grading the statistical components of a study, will be introduced.

October 27, 2009

Title:  Evaluation - A Quasi-Profession?

Presenter:   – Wolfgang Beywl – Berne University/Univation Cologne

Abstract:  A recent monograph on Evaluation in Germany (Brandt, 2009) presumes that evaluation is far from becoming a profession. A comparison of three German-speaking countries for ten broad evaluation fields (Widmer/Beywl/Fabian, 2009) comes to similar conclusions. A main argument is that there is no unique theoretical core of evaluation in relation to contiguous academic disciplines or management approaches. Wolfgang Beywl will present theses on attributes of evaluation which, in specific combinations, constitute such a theoretical/methodological core. Uniqueness could be established at least for program evaluation in human services sectors, a main branch of the evaluation business.

November 3, 2009

Title:  Documenting Dependency Relationships Among the Standards to Facilitate Metaevaluation

Presenter:  Carl Westine – Doctoral Associate, Interdisciplinary Ph.D. in Evaluation Program, WMU

Abstract:  The 30 standards set forth by The Joint Committee on Standards for Educational Evaluation in The Program Evaluation Standards (PES) (Joint Committee, 1994) form the basis for a checklist to be used for metaevaluation (Stufflebeam, 1999). However, identifying overlap among the standards should simplify the metaevaluation process. Through a systematic content analysis, we learn what the PES reveals about the overlapping nature of the standards. In most standards, specific references of up to ten standards are stated in the textual overview, guidelines, and common errors sections of the PES. Additional references among standards not explicitly stated also are documented. Incongruence in references among standards implies a dependency relationship exists. Moreover, the PES functional table of contents outlines further dependency relationships among the standards. Documenting well-defined dependency relationships has implications for how the standards could be differentially weighted in a condensed and efficient instrument to facilitate metaevaluation.