Dissertation Defenses

Doctoral Dissertation Announcement

---

Candidate:Ryoh Sasaki

Degree of: Doctor of Philosophy

Department: Interdisciplinary Ph.D. in Evaluation

Title: Metaevaluation by Formal Evaluation Theory of Aid Evaluation Work

Committee:
Dr. Paul Clements, Chair
Dr. Michael Scriven
Dr. Liliana Rodriguez-Campos

Date: Wednesday, September 10, 2008 2:00 p.m. - 4:00 p.m.
4410 Wood Hall

Abstract:
This is the metaevaluation of aid evaluation reports generated in a single fiscal year (FY 2004). Sample reports (n=102) basically were chosen by stratified random sampling from the pool of evaluation reports (N=1034). The pool consists of ex post evaluation reports (and some extensive midterm and termination evaluation reports) obtained from Web sites.
The Key Aid Evaluation Checklist (KAEC) was developed based on the well-known Key Evaluation Checklist (KEC) (Scriven, 2005, 2006). As a supplemental checklist, the Subdimensional Checklist (SDC) was also developed and applied to the metaevaluation.
A key conclusion of this metaevaluation is this: Quality of the current aid
evaluation reports in general is regarded as “good” but far from “excellent.” Although some are marked as high quality—i.e., they successfully examine a set of subevaluations and offer overall evaluative conclusions (such as a single grade)—others are very poor quality (they do not determine overall or at least somedimensional evaluative conclusions, but just present analytic conclusions or fact findings).

Some serious flaws exist in the current aid evaluation practice as follows:
? Primary operation of evaluation (subevaluation of each dimension) is relatively well
conducted. On the other hand, secondary operation (integration into a single judgment) is very weak or is simply not conducted.
? Value identification/verification, which usually is conducted by stakeholder participation or political inclusion, is very weak.
? Cost-efficiency subevaluation (either cost-benefit subevaluation or cost-effectiveness subevaluation) is generally weak or simply not conducted, especially if outcome-based evaluation is conducted. On the other hand, the evaluation report is generally regarded as good quality if it conducts serious and extensive cost-efficiency subevaluation.
? Metaevaluation (evaluation of the evaluation report by the evaluator him/herself) has not been accepted widely as an essential part of aid evaluation.Based on the identified flaws, several recommendations are proposed. Finally, the limitations of the main frameworks (KAEC and SDC) are pointed out.

This metaevaluation does not examine the relationship between the collected reports and the overall development portfolio of individual donors. These areas should be considered seriously for further research.