return to
Checklists
PRODUCT EVALUATION CHECKLIST
pdf

Michael Scriven
April 2000
This product evaluation checklist was designed, used, and multiply revised in a very specific context.  It was used to evaluate educational products--developed with federal funding--that were submitted with appropriate documentation about effectiveness to a review panel that made recommendations for the distribution of federal dissemination funds.

Instructions to raters:  Circle one number under Rate.  Check (or double check) one (or more) line(s) under Consider.  Use an "X" to indicate deficiencies instead of strengths.

1.  NEED
Consider:
 ___ Number affected
 ___ Social/educational significance
___ Absence of substitute
___ Multiplicative effects
___ Other		 Rate:
4  Maximum priority, a desperate need
3  Great importance
2  Probability of significant need
1  Possibly significant need
0 No good evidence of significant need		 Comments:
2.  MARKET
Consider:
___ Dissemination plan: 
clarity, feasibility, ingenuity,    economy
___ Size
___ Importance
___ Other		 Rate:
4  Very large and/or very important market will be reached
3  Large and/or important market will be reached
2  Significant market will probably be reached
1  Possible, but not probable, that a significant market will be reached
0  Inadequate evidence to suggest that a significant market will be reached 		 Comments:
3.  TRUE FIELD TRIALS?
Consider:
 ___ Final version?
 ___ Typical user?
___ Typical level of support?
___ Typical setting?
___ Typical time frame?
___ Other		 Rate:
4  Perfectly typical
3  Minor differences
2  Reasonable bet for  generalization
1  Serious weakness
0  Relevance unclear		 Comments:
4. WERE THE EFFECTS MEASURED ON TRUE CONSUMERS?
Consider:
___ Congress?
___ Federal agency?
___ State department?
___ District?
___ Principal?
___ Teacher?
___ Student?
___ Taxpayer?
___ Other		 Rate:
4  Full data on all relevant consumers
3  Fair data on all relevant consumers
2  Good data on the most important consumers
1  Weak data on the most important consumers
0   Only speculation about the most important consumers		 Comments:
5.  WAS SERIOUS COST ANALYSIS DONE?
Consider:
___ Indirect costs included
___ Opportunity costs included
___ Nonmoney costs included
___ Start-up costs included		 Rate:
4   Very thorough analysis
3   Good approximation
2   Rough estimate
1   Poor estimate
0   Potentially fatal omissions		 Comments:
6.  WERE LONG-TERM EFFECTS CHECKED?
Consider:
___ Week to month later
___ Month to year later
___ Year to few years later
___ Many years later
___ On-the-job or life-space sample		 Rate:
4   Good direct evidence about the effects at times needed
3   Some direct evidence about the effects at times needed
2   Follow-up gives reasonable support to a conclusion about the effects when needed
1   Follow-up or other data suggest a conclusion about the effects when needed
0   Useless or no follow-up; no other grounds for inferring long-term		 Comments:
7.  WERE SIDE EFFECTS DILIGENTLY SOUGHT?
Consider:
___ Comprehensive search?
___ Skilled?
___ Independent?
___ Goal-free?
___ During/end/later? 		 Rate:
4   Meets all requirements well
3   Generally good
2   Barely acceptable
1   Some study made, but incomplete
0   No worthwhile study		 Comments:
8.  WAS THE PROCESS SYSTEMATICALLY EVALUATED?
Consider:
___ Implementation index?
___ Ethics?
___ Professional standards?
___ Judge-observer reliability?
___ Affective dimension?
___ Content validity? 		 Rate:
4   Passes with flying colors
3   Appears satisfactory
2   Reasonable risk
1   Serious omission(s)
0   Highly inadequate		 Comments:
9.  WAS CAUSATION ESTABLISHED?
Consider:
___ Randomized- experimental design?
___ Quasi-experimental design?
___ Ex post facto?
___ MO method?
___ A priori interpretation of correlational data?		 Rate:
4   Impeccable
3   Good bet
2   Plausible bet
1   Weak bet
0   Hopeless bet		 Comments:
10.  WERE CRITICAL COMPARISONS MADE?
Consider:
___ No treatment group
___ Existing competitors
___ Projected competitors
___ Created competitors
___ Hypothesized competitors		 Rate:
4   Good data on all important competitors 
3   Good data on the most important competitors
2   Fair data on the most important competitors
1   Lacking data on some of the more important competitors
0   Little or no useful comparative data		 Comments:
11.  OVERALL--STATISTICAL SIGNIFICANCE*
Consider:
___ Appropriate analysis?
___ Appropriate significance level? 		 Rate:
4   Flawless analysis, astronomical significance
3   High significance, well-tested
2   Reasonably significant
1   Marginal significance
0   Not shown to be significant		 Comments:
12.  OVERALL--EDUCATIONAL SIGNIFICANCE*
Consider:
___  Independent judgment?
___ Expert judgment?
___ Judgment based on item analysis?
___ Judgment based on raw scores?
___ Cultural concerns addressed?
___ Teaching to the tests?
___ Testing to the teaching?
___ Congruence with needs?
___ Side effects taken into account?
___ Ethical concerns addressed? 		 Rate:
4   Very high importance demonstrated
3   High importance demonstrated
2   Moderate importance demonstrated
1   Slight or rather uncertain importance
0   Negligible or unknown importance		 Comments:
* It will be especially helpful here to use either checks or 0-4 scores or A-F grades alongside these factors to indicate the basis for the overall rating.
This checklist is being provided as a free service to the user. The provider of the checklist has not modified or adapted the checklist to fit the specific needs of the user and the user is executing his or her own discretion and judgment in using the checklist. The provider of the checklist makes no representations or warranties that this checklist is fit for the particular purpose contemplated by user and specifically disclaims any such warranties or representations.