return to
Checklists

PROGRAM EVALUATIONS METAEVALUATION CHECKLIST
(Based on The Program Evaluation Standards)
pdf

Daniel L. Stufflebeam

1999

This checklist is for performing final, summative metaevaluations.  It is organized according to the Joint Committee Program Evaluation Standards.  For each of the 30 standards the checklist includes 6 checkpoints drawn from the substance of the standard.  It is suggested that each standard be scored on each checkpoint.  Then judgments about the adequacy of the subject evaluation in meeting the standard can be made as follows:  0-1 Poor, 2-3 Fair, 4 Good, 5 Very Good, 6 Excellent.  It is recommended that an evaluation be failed if it scores Poor on standards P1 Service Orientation, A5 Valid Information, A10 Justified Conclusions, or A11 Impartial Reporting.   Users of this checklist are advised to consult the full text of The Joint Committee (1994) Program Evaluation Standards, Thousand Oaks, CA:  Sage Publications.

TO MEET THE REQUIREMENTS FOR UTILITY, PROGRAM EVALUATIONS SHOULD:
U1  Stakeholder Identification
  • Clearly identify the evaluation client
  • Engage leadership figures to identify other stakeholders
  • Consult stakeholders to identify their information needs
  • Ask stakeholders to identify other stakeholders
  • Arrange to involve stakeholders throughout the evaluation, consistent with the formal evaluation agreement
  • Keep the evaluation open to serve newly identified stakeholders
6 = Excellent
5 = Very Good 4 = Good  2-3 = Fair 0-1 = Poor
U2 Evaluator Credibility
  • Engage competent evaluators
  • Engage evaluators whom the stakeholders trust
  • Engage evaluators who can address stakeholders’ concerns
  • Engage evaluators who are appropriately responsive to issues of gender, socioeconomic status, race, and language and cultural differences
  • Help stakeholders understand and assess the evaluation plan and process
  • Attend appropriately to stakeholders’ criticisms and suggestions
6 = Excellent
5 = Very Good 4 = Good  2-3 = Fair 0-1 = Poor
U3  Information Scope and Selection
  • Assign priority to the most important questions
  • Allow flexibility for adding questions during the evaluation
  • Obtain sufficient information to address the stakeholders’ most important evaluation questions
  • Obtain sufficient information to assess the program's merit
  • Obtain sufficient information to assess the program's worth
  • Allocate the evaluation effort in accordance with the priorities assigned to the needed information
6 = Excellent
5 = Very Good 4 = Good  2-3 = Fair 0-1 = Poor
U4 Values Identification 
  • Consider all relevant sources of values for interpreting evaluation findings, including societal needs, customer needs, pertinent laws, institutional mission, and program goals 
  • Determine the appropriate party(s) to make the valuational interpretations
  • Provide a clear, defensible basis for value judgments
  • Distinguish appropriately among dimensions, weights, and cut scores on the involved values
  • Take into account the stakeholders’ values
  • As appropriate, present alternative interpretations based on conflicting but credible value bases
6 = Excellent
5 = Very Good 4 = Good  2-3 = Fair 0-1 = Poor
U5 Report Clarity
  • Issue one or more reports as appropriate, such as an executive summary, main report, technical report, and oral presentation
  • As appropriate, address the special needs of the audiences, such as persons with limited
  • English proficiency
  • Focus reports on contracted questions and convey the essential information in each report
  • Write and/or present the findings simply and directly
  • Employ effective media for informing the different audiences
  • Use examples to help audiences relate the findings to practical situations
6 = Excellent
5 = Very Good 4 = Good  2-3 = Fair 0-1 = Poor
U6  Report Timeliness and Dissemination
  • In cooperation with the client, make special efforts to identify, reach, and inform all intended users
  • Make timely interim reports to intended users
  • Have timely exchanges with the pertinent audiences, e.g., the program's policy board, the program's staff, and the program's customers
  • Deliver the final report when it is needed
  • As appropriate, issue press releases to the public media
  • If allowed by the evaluation contract and as appropriate, make findings publicly available via such media as the Internet 
6 = Excellent
5 = Very Good 4 = Good  2-3 = Fair 0-1 = Poor
U7  Evaluation Impact 
  • As appropriate and feasible, keep audiences informed throughout the evaluation
  • Forecast and serve potential uses of findings
  • Provide interim reports
  • Supplement written reports with ongoing oral communication
  • To the extent appropriate, conduct feedback sessions to go over and apply findings
  • Make arrangements to provide follow-up assistance in interpreting and applying the findings
6 = Excellent
5 = Very Good 4 = Good  2-3 = Fair 0-1 = Poor

SCORING THE EVALUATION FOR UTILITY:
Add the following:
Number of Excellent ratings (0-7)
___
x 4 =
 ___
Number of Very Good ratings (0-7)
___
x 3 =
 ___
Number of Good ratings (0-7)
___
x 2 =
 ___
Number of Fair ratings (0-7)
___
x 1 =
 ___
     Total Score
=
 ___
Strength of the Evaluation's provisions for Utility:
26 (93%) to 28  Excellent
19 (68%) to 25  Very Good
14 (50%) to 18  Good
7 (25%) to 13  Fair
 0 (0%) to 5  Poor
TO MEET THE REQUIREMENTS OF FEASIBILITY, PROGRAM EVALUATIONS SHOULD: 
F1  Practical Procedures
  • Minimize disruption and data burden
  • Appoint competent staff and train them as needed
  • Choose procedures in light of known resource and staff qualifications constraints
  • Make a realistic schedule
  • As feasible and appropriate, engage locals to help conduct the evaluation
  • As appropriate, make evaluation procedures a part of routine events
6 = Excellent
5 = Very Good 4 = Good  2-3 = Fair 0-1 = Poor
F2  Political Viability
  • Anticipate different positions of different interest groups
  • Be vigilant and appropriately counteractive concerning pressures and actions designed to impede or destroy the evaluation 
  • Foster cooperation
  • Report divergent views
  • As possible, make constructive use of diverse political forces to achieve the evaluation's purposes
  • Terminate any corrupted evaluation
6 = Excellent
5 = Very Good 4 = Good  2-3 = Fair 0-1 = Poor
F3  Cost Effectiveness 
  • Be efficient
  • Make use of in-kind services
  • Inform decisions
  • Foster program improvement
  • Provide accountability information
  • Generate new insights
6 = Excellent
5 = Very Good 4 = Good  2-3 = Fair 0-1 = Poor

SCORING THE EVALUATION FOR FEASIBILITY:
Add the following:
Number of Excellent ratings (0-3)
___
x 4 =
 ___
Number of Very Good ratings (0-3)
___
x 3 =
 ___
Number of Good ratings (0-3)
___
x 2 =
 ___
Number of Fair ratings (0-3)
___
x 1 =
 ___
     Total Score
=
 ___
Strength of the Evaluation's provisions for Feasibility:
11 (92%) to 12  Excellent
  8 (69%) to 10  Very Good
  6 (50%) to  7  Good
  3 (25%) to 5  Fair
  0 (0%) to 2  Poor
TO MEET THE REQUIREMENTS FOR PROPRIETY, PROGRAM EVALUATIONS SHOULD:
P1  Service Orientation
  • Assess program outcomes against targeted and nontargeted customers’ assessed needs
  • Help assure that the full range of rightful program beneficiaries are served
  • Promote excellent service
  • Identify program strengths to build on 
  • Identify program weaknesses to correct
  • Expose persistently harmful practices
6 = Excellent
5 = Very Good 4 = Good  2-3 = Fair 0-1 = Poor
P2  Formal Agreements
Reach advance written agreements on:
  • Evaluation purpose and questions
  • Audiences
  • Editing
  • Release of reports
  • Evaluation procedures and schedule
  • Evaluation resources
6 = Excellent
5 = Very Good 4 = Good  2-3 = Fair 0-1 = Poor
P3  Rights of Human Subjects
  • Follow due process and uphold civil rights
  • Understand participants’ values
  • Respect diversity
  • Follow protocol
  • Honor confidentiality/anonymity agreements
  • Minimize harmful consequences of the evaluation
6 = Excellent
5 = Very Good 4 = Good  2-3 = Fair 0-1 = Poor
P4  Human Interactions
  • Consistently relate to all stakeholders in a professional manner
  • Honor participants’ privacy rights
  • Honor time commitments
  • Be sensitive to participants’ diversity of values and cultural differences
  • Be evenly respectful in addressing different stakeholders
  • Do not ignore or help cover up any participant's incompetence, unethical behavior, fraud, waste, or abuse
6 = Excellent
5 = Very Good 4 = Good  2-3 = Fair 0-1 = Poor
P5  Complete and Fair Assessment 
  • Assess and report the program's strengths and weaknesses
  • Report on intended and unintended outcomes
  • As appropriate, show how the program's strengths could be used to overcome its weaknesses
  • Appropriately address criticisms of the draft report
  • Acknowledge the final report's limitations
  • Estimate and report the effects of the evaluation's limitations on the overall judgment of the program
6 = Excellent
5 = Very Good 4 = Good  2-3 = Fair 0-1 = Poor
P6  Disclosure of Findings 
  • Clearly define the right-to-know audiences
  • Report relevant points of view of both supporters and critics of the program
  • Report balanced, informed conclusions and recommendations
  • Report all findings in writing, except where circumstances clearly dictate otherwise 
  • In reporting, adhere strictly to a code of directness, openness, and completeness
  • Assure the reports reach their audiences
6 = Excellent
5 = Very Good 4 = Good  2-3 = Fair 0-1 = Poor
P7  Conflict of Interest 
  • Identify potential conflicts of interest early in the evaluation
  • As appropriate and feasible, engage multiple evaluators?  
  • Maintain evaluation records for independent review
  • If feasible, contract with the funding authority rather than the funded program
  • If feasible, have the lead internal evaluator report directly  to the  chief executive officer
  • Engage uniquely qualified persons to participate in the evaluation, even if they have a potential conflict of interest; but take steps to counteract the conflict
6 = Excellent
5 = Very Good 4 = Good  2-3 = Fair 0-1 = Poor
P8  Fiscal Responsibility
  • Specify and budget for expense items in advance
  • Keep the budget sufficiently flexible to permit appropriate reallocations to strengthen the evaluation
  • Maintain accurate records of sources of funding and expenditure and resulting evaluation services and products
  • Maintain adequate personnel records concerning job allocations and time spent on the evaluation project
  • Be frugal in expending evaluation resources
  • As appropriate, include an expenditure summary as part of the public evaluation report
6 = Excellent
5 = Very Good 4 = Good  2-3 = Fair 0-1 = Poor

SCORING THE EVALUATION FOR PROPRIETY:
Add the following:
Number of Excellent ratings (0-8)
___
x 4 =
 ___
Number of Very Good ratings (0-8)
___
x 3 =
 ___
Number of Good ratings (0-8)
___
x 2 =
 ___
Number of Fair ratings (0-8)
___
x 1 =
 ___
     Total Score
=
 ___
Strength of the Evaluation's provisions for Propriety:
30 (94%) to 32  Excellent
22 (69%) to 29  Very Good
16 (50%) to 21  Good
8 (25%) to 15   Fair
0 (0%) to 7  Poor
TO MEET THE REQUIREMENTS FOR ACCURACY, PROGRAM EVALUATIONS SHOULD:
A1  Program Documentation
  • Collect descriptions of the intended program from various written sources and from the client and other key stakeholders
  • Maintain records from various sources of how the program operated
  • Analyze discrepancies between the various descriptions of how the program was intended to function
  • Analyze discrepancies between how the program was intended to operate and how it actually operated
  • Record the extent to which the program's goals changed over time
  • Produce a technical report that documents the program's operations and results
6 = Excellent
5 = Very Good 4 = Good  2-3 = Fair 0-1 = Poor
A2  Context Analysis
  • Describe the context's technical, social, political, organizational, and economic features
  • Maintain a log of unusual circumstances
  • Report those contextual influences that appeared to significantly influence the program and that might be of interest to potential adopters
  • Estimate the effects of context on program outcomes
  • Identify and describe any critical competitors to this program that functioned at the same time and in the program's environment
  • Describe how people in the program's general area perceived the program's existence,  importance, and quality
6 = Excellent
5 = Very Good 4 = Good  2-3 = Fair 0-1 = Poor
A3  Described Purposes and Procedures
  • Monitor and describe how the evaluation's purposes stay the same or change over time
  • As appropriate, update evaluation procedures to accommodate changes in the evaluation's purposes
  • Record the actual evaluation procedures, as implemented
  • When interpreting findings, take into account the extent to which the intended procedures were effectively executed
  • Describe the evaluation's purposes and procedures in the summary and full-length evaluation reports 
  • As feasible, engage independent evaluators to monitor and evaluate the evaluation's purposes and procedures
6 = Excellent
5 = Very Good 4 = Good  2-3 = Fair 0-1 = Poor
A4  Defensible Information Sources
  • Once validated, use pertinent, previously collected information
  • As appropriate, employ a variety of data collection sources and methods
  • Document and report information sources
  • Document, justify, and report the means used to obtain information from each source
  • Include data collection instruments in a technical appendix to the evaluation report
  • Document and report any biasing features in the obtained information
6 = Excellent
5 = Very Good 4 = Good  2-3 = Fair 0-1 = Poor
A5  Valid Information
  • Focus the evaluation on key questions
  • Assess and report what type of information each employed procedure acquires
  • Document how information from each procedure was scored, analyzed, and interpreted
  • Report and justify inferences singly and in combination
  • Assess and report the comprehensiveness of the information provided by the procedures as a set in relation to the information needed to answer the set of evaluation questions
  • Establish meaningful categories of information by identifying regular and recurrent themes in information collected using qualitative assessment procedures
6 = Excellent
5 = Very Good 4 = Good  2-3 = Fair 0-1 = Poor
A6  Reliable Information
  • Identify and justify the type(s) and extent of reliability claimed
  • As feasible, choose measuring devices that in the past have shown acceptable levels of reliability for their intended uses
  • In reporting reliability of an instrument, assess and report the factors that influenced the reliability, including the characteristics of the examinees, the data collection conditions, and the evaluator's biases
  • Check and report the consistency of scoring, categorization, and coding
  • Train and calibrate scorers and analysts to produce consistent results
  • Pilot test new instruments in order to identify and control sources of error 
6 = Excellent
5 = Very Good 4 = Good  2-3 = Fair 0-1 = Poor
A7  Systematic Information
  • Establish protocols and mechanisms for quality control of the evaluation information
  • Verify data entry
  • Proofread and verify data tables generated from computer output or other means
  • Systematize and control storage of the evaluation information
  • Strictly control access to the evaluation information according to established protocols
  • Have data providers verify the data they submitted
6 = Excellent
5 = Very Good 4 = Good  2-3 = Fair 0-1 = Poor
A8  Analysis of Quantitative Information
  • Whenever possible, begin by conducting preliminary exploratory analyses to assure the data's correctness and to gain a greater understanding of the data
  • Report limitations of each analytic procedure, including failure to meet assumptions
  • Employ multiple analytic procedures to check on consistency and replicability of findings
  • Examine variability as well as central tendenciesIdentify and examine outliers, and verify their correctness
  • Identify and analyze statistical interactions
6 = Excellent
5 = Very Good 4 = Good  2-3 = Fair 0-1 = Poor
A9  Analysis of Qualitative Information
  • Define the boundaries of information to be used
  • Derive a set of categories that is sufficient to document, illuminate, and respond to the evaluation questions
  • Classify the obtained information into the validated analysis categories
  • Verify the accuracy of findings by obtaining confirmatory evidence from multiple sources, including stakeholders
  • Derive conclusions and recommendations, and demonstrate their meaningfulness
  • Report limitations of the referenced information, analyses, and inferences
6 = Excellent
5 = Very Good 4 = Good  2-3 = Fair 0-1 = Poor
A10  Justified Conclusions
  • Limit conclusions to the applicable time periods, contexts, purposes, questions, and activities
  • Report alternative plausible conclusions and explain why other rival conclusions were rejected
  • Cite the information that supports each conclusion
  • Identify and report the program's side effects
  • Warn against making common misinterpretations
  • Whenever feasible and appropriate, obtain and address the results of a prerelease review of the draft evaluation report
6 = Excellent
5 = Very Good 4 = Good  2-3 = Fair 0-1 = Poor
A11  Impartial Reporting
  • Engage the client to determine steps to ensure fair, impartial reports
  • Safeguard reports from deliberate or inadvertent distortions
  • As appropriate and feasible, report perspectives of all stakeholder groups and, especially, opposing views on the meaning of the findings
  • As appropriate and feasible, add a new, impartial evaluator late in the evaluation to help offset any bias the original evaluators may have developed due to their prior judgments and recommendations 
  • Describe steps taken to control bias
  • Participate in public presentations of the findings to help guard against and correct distortions by other interested parties
6 = Excellent
5 = Very Good 4 = Good  2-3 = Fair 0-1 = Poor
A12  Metaevaluation
  • Budget appropriately and sufficiently for conducting an internal metaevaluation and, as feasible, an external metaevaluation
  • Designate or define the standards the evaluators used to guide and assess their evaluation
  • Record the full range of information needed to judge the evaluation against the employed standards
  • As feasible and appropriate, contract for an independent metaevaluation
  • Evaluate all important aspects of the evaluation, including the instrumentation, data collection, data handling, coding, analysis, synthesis, and reporting
  • Obtain and report both formative and summative metaevaluations to the right-to-know audiences
6 = Excellent
5 = Very Good 4 = Good  2-3 = Fair 0-1 = Poor
SCORING THE EVALUATION FOR ACCURACY:
Add the following:
Number of Excellent ratings (0-12)
___
x 4 =
 ___
Number of Very Good ratings (0-12)
___
x 3 =
 ___
Number of Good ratings (0-12)
___
x 2 =
 ___
Number of Fair ratings (0-12)
___
x 1 =
 ___
     Total Score
=
 ___
Strength of the Evaluation's provisions for Accuracy:
45 (94%) to 48  Excellent
33 (69%) to 44  Very Good
24 (50%) to 32  Good
12 (25%) to 23  Fair
  0 (0%) to 11  Poor

 
 
This checklist is being provided as a free service to the user. The provider of the checklist has not modified or adapted the checklist to fit the specific needs of the user and the user is executing his or her own discretion and judgment in using the checklist. The provider of the checklist makes no representations or warranties that this checklist is fit for the particular purpose contemplated by user and specifically disclaims any such warranties or representations.