• Clearly identify the evaluation client
• Engage leadership figures to identify other stakeholders
• Consult potential stakeholders to identify their information needs
• Use stakeholders to identify other stakeholders
• With the client, rank stakeholders for relative importance
• Arrange to involve stakeholders throughout the evaluation
• Keep the evaluation open to serve newly identified stakeholders
• Address stakeholders' evaluation needs
• Serve an appropriate range of individual stakeholders
• Serve an appropriate range of stakeholder organizations

• Engage competent evaluators
• Engage evaluators whom the stakeholders trust
• Engage evaluators who can address stakeholders' concerns
• Engage evaluators who are appropriately responsive to issues of gender, socioeconomic status, race, and language and cultural differences
• Assure that the evaluation plan responds to key stakeholders' concerns
• Help stakeholders understand the evaluation plan
• Give stakeholders information on the evaluation plan's technical quality and practicality
• Attend appropriately to stakeholders' criticisms and suggestions
• Stay abreast of social and political forces
• Keep interested parties informed about the evaluation's progress

• Understand the client's most important evaluation requirements
• Interview stakeholders to determine their different perspectives
• Assure that evaluator and client negotiate pertinent audiences, questions, and required information
• Assign priority to the most important stakeholders
• Assign priority to the most important questions
• Allow flexibility for adding questions during the evaluation
• Obtain sufficient information to address the stakeholders' most important evaluation questions
• Obtain sufficient information to assess the program's merit
• Obtain sufficient information to assess the program's worth
• Allocate the evaluation effort in accordance with the priorities assigned to the needed information

• Consider alternative sources of values for interpreting evaluation findings
• Provide a clear, defensible basis for value judgments
• Determine the appropriate party(s) to make the valuational interpretations
• Identify pertinent societal needs
• Identify pertinent customer needs
• Reference pertinent laws
• Reference, as appropriate, the relevant institutional mission
• Reference the program's goals
• Take into account the stakeholders' values
• As appropriate, present alternative interpretations based on conflicting but credible value bases

• Clearly report the essential information
• Issue brief, simple, and direct reports
• Focus reports on contracted questions
• Describe the program and its context
• Describe the evaluation's purposes, procedures, and findings
• Support conclusions and recommendations
• Avoid reporting technical jargon
• Report in the language(s) of stakeholders
• Provide an executive summary
• Provide a technical report

• Make timely interim reports to intended users
• Deliver the final report when it is needed
• Have timely exchanges with the program's policy board
• Have timely exchanges with the program's staff
• Have timely exchanges with the program's customers
• Have timely exchanges with the public media
• Have timely exchanges with the full range of right-to-know audiences
• Employ effective media for reaching and informing the different audiences
• Keep the presentations appropriately brief
• Use examples to help audiences relate the findings to practical situations

• Maintain contact with audience
• Involve stakeholders throughout the evaluation
• Encourage and support stakeholders' use of the findings
• Show stakeholders how they might use the findings in their work
• Forecast and address potential uses of findings
• Provide interim reports
• Make sure that reports are open, frank, and concrete
• Supplement written reports with ongoing oral communication
• Conduct feedback workshops to go over and apply findings
• Make arrangements to provide follow-up assistance in interpreting and applying the findings

7 (25%) - 13
Fair

• Tailor methods and instruments to information requirements
• Minimize disruption
• Minimize the data burden
• Appoint competent staff
• Train staff
• Choose procedures that the staff are qualified to carry out
• Choose procedures in light of known constraints
• Make a realistic schedule
• Engage locals to help conduct the evaluation
• As appropriate, make evaluation procedures a part of routine events

• Anticipate different positions of different interest groups
• Avert or counteract attempts to bias or misapply the findings
• Foster cooperation
• Involve stakeholders throughout the evaluation
• Agree on editorial and dissemination authority
• Issue interim reports
• Report divergent views
• Report to right-to-know audiences
• Employ a firm public contract
• Terminate any corrupted evaluation

• Be efficient
• Make use of in-kind services
• Produce information worth the investment
• Inform decisions
• Foster program improvement
• Provide accountability information
• Generate new insights
• Help spread effective practices
• Minimize disruptions
• Minimize time demands on program personnel

• Assess needs of the program's customers
• Assess program outcomes against targeted customers' assessed needs
• Help assure that the full range of rightful program beneficiaries are served
• Promote excellent service
• Make the evaluation's service orientation clear to stakeholders
• Identify program strengths to build on
• Identify program weaknesses to correct
• Give interim feedback for program improvement
• Expose harmful practices
• Inform all right-to-know audiences of the program's positive and negative outcomes

• Evaluation purpose and questions
• Audiences
• Evaluation reports
• Editing
• Release of reports
• Evaluation procedures and schedule
• Confidentiality/anonymity of data
• Evaluation staff
• Metaevaluation
• Evaluation resources

• Make clear to stakeholders that the evaluation will respect and protect the rights of human subjects
• Clarify intended uses of the evaluation
• Keep stakeholders informed
• Follow due process
• Uphold civil rights
• Understand participant values
• Respect diversity
• Follow protocol
• Honor confidentiality/anonymity agreements
• Do no harm

• Consistently relate to all stakeholders in a professional manner
• Maintain effective communication with stakeholders
• Follow the institution's protocol
• Minimize disruption
• Honor participants' privacy rights
• Honor time commitments
• Be alert to and address participants' concerns about the evaluation
• Be sensitive to participants' diversity of values and cultural differences
• Be even-handed in addressing different stakeholders
• Do not ignore or help cover up any participants incompetence, unethical behavior, fraud, waste, or abuse

• Assess and report the program's strengths
• Assess and report the program's weaknesses
• Report on intended outcomes
• Report on unintended outcomes
• Give a thorough account of the evaluation's process
• As appropriate, show how the program's strengths could be used to overcome its weaknesses
• Have the draft report reviewed
• Appropriately address criticisms of the draft report
• Acknowledge the final report's limitations
• Estimate and report the effects of the evaluation's limitations on the overall judgment of the program

• Define the right-to-know audiences
• Establish a contractual basis for complying with right-to-know requirements
• Inform the audiences of the evaluation's purposes and projected reports
• Report all findings in writing
• Report relevant points of view of both supporters and critics of the program
• Report balanced, informed conclusions and recommendations
• Show the basis for the conclusions and recommendations
• Disclose the evaluation's limitations
• In reporting, adhere strictly to a code of directness, openness, and completeness
• Assure that reports reach their audiences

• Identify potential conflicts of interest early in the evaluation
• Provide written, contractual safeguards against identified conflicts of interest
• Engage multiple evaluators
• Maintain evaluation records for independent review
• As appropriate, engage independent parties to assess the evaluation for its susceptibility or corruption by conflicts of interest
• When appropriate, release evaluation procedures, data, and reports for public review
• Contract with the funding authority rather than the funded program
• Have internal evaluators report directly to the chief executive officer
• Report equitably to all right-to-know audiences
• Engage uniquely qualified persons to participate in the evaluation, even if they have a potential conflict of interest; but take steps to counteract the conflict

• Specify and budget for expense items in advance
• Keep the budget sufficiently flexible to permit appropriate reallocations to strengthen the evaluation
• Obtain appropriate approval for needed budgetary modifications
• Assign responsibility for managing the evaluation finances
• Maintain accurate records of sources of funding and expenditures
• Maintain adequate personnel records concerning job allocations and time spent on the job
• Employ comparison shopping for evaluation materials
• Employ comparison contract bidding
• Be frugal in expending evaluation resources
• As appropriate, include an expenditure summary as part of the public evaluation report

8 (25%) - 15
Fair

• Collect descriptions of the intended program from various written sources
• Collect descriptions of the intended program from the client and various stakeholders
• Describe how the program was intended to function
• Maintain records from various sources of how the program operated
• As feasible, engage independent observers to describe the program's actual operations
• Describe how the program actually functioned
• Analyze discrepancies between the various descriptions of how the program was intended to function
• Analyze discrepancies between how the program was intended to operate and how it actually operated
• Ask the client and various stakeholders to assess the accuracy of recorded descriptions of both the intended and the actual program
• Produce a technical report that documents the program's operations

• Use multiple sources of information to describe the program's context
• Describe the context's technical, social, political, organizational, and economic features
• Maintain a log of unusual circumstances
• Record instances in which individuals or groups intentionally or otherwise interfered with the program
• Record instances in which individuals or groups intentionally or otherwise gave special assistance to the program
• Analyze how the program's context is similar to or different from contexts where the program might be adopted
• Report those contextual influences that appeared to significantly influence the program and that might be of interest to potential adopters
• Estimate effects of context on program outcomes
• Identify and describe any critical competitors to this program that functioned at the same time and in the program's environment
• Describe how people in the program's general area perceived the program's existence, importance, and quality

• At the evaluation's outset, record the client's purposes for the evaluation
• Monitor and describe stakeholders' intended uses of evaluation findings
• Monitor and describe how the evaluation's purposes stay the same or change over time
• Identify and assess points of agreement and disagreement among stakeholders regarding the evaluation's purposes
• As appropriate, update evaluation procedures to accommodate changes in the evaluation's purposes
• Record the actual evaluation procedures, as implemented
• When interpreting findings, take into account the different stakeholders' intended uses of the evaluation
• When interpreting findings, take into account the extent to which the intended procedures were effectively executed
• Describe the evaluation's purposes and procedures in the summary and full-length evaluation reports
• As feasible, engage independent evaluators to monitor and evaluate the evaluation's purposes and procedures

• Obtain information from a variety of sources
• Use pertinent, previously collected information once validated
• As appropriate, employ a variety of data collection methods
• Document and report information sources
• Document, justify, and report the criteria and methods used to select information sources
• For each source, define the population
• For each population, as appropriate, define any employed sample
• Document, justify, and report the means used to obtain information from each source
• Include data collection instruments in a technical appendix to the evaluation report
• Document and report any biasing features in the obtained information

• Focus the evaluation on key questions
• As appropriate, employ multiple measures to address each question
• Provide a detailed description of the constructs and behaviors about which information will be acquired
• Assess and report what type of information each employed procedure acquires
• Train and calibrate the data collectors
• Document and report the data collection conditions and process
• Document how information from each procedure was scored, analyzed, and interpreted
• Report and justify inferences singly and in combination
• Assess and report the comprehensiveness of the information provided by the procedures as a set in relation to the information needed to answer the set of evaluation questions
• Establish meaningful categories of information by identifying regular and recurrent themes in information collected using qualitative assessment procedures

• Identify and justify the type(s) and extent of reliability claimed
• For each employed data collection device, specify the unit of analysis
• As feasible, choose measuring devices that in the past have shown acceptable levels of reliability for their intended uses
• In reporting reliability of an instrument, assess and report the factors that influenced the reliability, including the characteristics of the examinees, the data collection conditions, and the evaluator's biases
• Check and report the consistency of scoring, categorization, and coding
• Train and calibrate scorers and analysts to produce consistent results
• Pilot test new instruments in order to identify and control sources of error
• As appropriate, engage and check the consistency between multiple observers
• Acknowledge reliability problems in the final report
• Estimate and report the effects of unreliability in the data on the overall judgment of the program

• Establish protocols for quality control of the evaluation information
• Train the evaluation staff to adhere to the data protocols
• Systematically check the accuracy of scoring and coding
• When feasible, use multiple evaluators and check the consistency of their work
• Verify data entry
• Proofread and verify data tables generated from computer output or other means
• Systematize and control storage of the evaluation information
• Define who will have access to the evaluation information
• Strictly control access to the evaluation information according to established protocols
• Have data providers verify the data they submitted

• Begin by conducting preliminary exploratory analyses to assure the data's correctness and to gain a greater understanding of the data
• Choose procedures appropriate for the evaluation questions and nature of the data
• For each procedure specify how its key assumptions are being met
• Report limitations of each analytic procedure, including failure to meet assumptions
• Employ multiple analytic procedures to check on consistency and replicability of findings
• Examine variability as well as central tendencies
• Identify and examine outliers and verify their correctness
• Identify and analyze statistical interactions
• Assess statistical significance and practical significance
• Use visual displays to clarify the presentation and interpretation of statistical results

• Focus on key questions
• Define the boundaries of information to be used
• Obtain information keyed to the important evaluation questions
• Verify the accuracy of findings by obtaining confirmatory evidence from multiple sources, including stakeholders
• Choose analytic procedures and methods of summarization that are appropriate to the evaluation questions and employed qualitative information
• Derive a set of categories that is sufficient to document, illuminate, and respond to the evaluation questions
• Test the derived categories for reliability and validity
• Classify the obtained information into the validated analysis categories
• Derive conclusions and recommendations and demonstrate their meaningfulness
• Report limitations of the referenced information, analyses, and inferences

• Focus conclusions directly on the evaluation questions
• Accurately reflect the evaluation procedures and findings
• Limit conclusions to the applicable time periods, contexts, purposes, and activities
• Cite the information that supports each conclusion
• Identify and report the program's side effects
• Report plausible alternative explanations of the findings
• Explain why rival explanations were rejected
• Warn against making common misinterpretations
• Obtain and address the results of a prerelease review of the draft evaluation report
• Report the evaluation's limitations

• Engage the client to determine steps to ensure fair, impartial reports
• Establish appropriate editorial authority
• Determine right-to-know audiences
• Establish and follow appropriate plans for releasing findings to all right-to-know audiences
• Safeguard reports from deliberate or inadvertent distortions
• Report perspectives of all stakeholder groups
• Report alternative plausible conclusions
• Obtain outside audits of reports
• Describe steps taken to control bias
• Participate in public presentations of the findings to help guard against and correct distortions by other interested parties

• Designate or define the standards to be used in judging the evaluation
• Assign someone responsibility for documenting and assessing the evaluation process and products
• Employ both formative and summative metaevaluation
• Budget appropriately and sufficiently for conducting the metaevaluation
• Record the full range of information needed to judge the evaluation against the stipulated standards
• As feasible, contract for an independent metaevaluation
• Determine and record which audiences will receive the metaevaluation report
• Evaluate the instrumentation, data collection, data handling, coding, and analysis against the relevant standards
• Evaluate the evaluation's involvement of and communication of findings to stakeholders against the relevant standards
• Maintain a record of all metaevaluation steps, information, and analyses

12 (25%) - 13
Fair