return to
Checklists

EVALUATION PLANS AND OPERATIONS CHECKLIST
pdf

Daniel L. Stufflebeam

1999
This checklist is for conducting preliminary, formative metaevaluations.  It is organized according to seven main aspects of an evaluation. By examining an evaluation plan or process against the specific checkpoints in each category, an evaluator can derive direction for strengthening the evaluation plan or operations.
1.  Conceptualization of Evaluation. Evaluators and clients/stakeholders should establish a shared, sound understanding of the guiding concept of evaluation.
Definition How evaluation defined?
Purpose  What purposes(s) will be served?
Values  What values will undergird this evaluation?
Questions  What questions will be addressed?
Information  What information is required?
Audiences  What persons and groups will be served?
Agents  Who will do the evaluation?
Process  How will the evaluation be conducted?
Standards  By what standards will the evaluation be judged, e.g., utility, propriety, feasibility, and accuracy?
2. Sociopolitical Factors. Evaluators and clients should identify and effectively address affected/concerned groups.
Involvement Whose sanction and support is required, and how will it be secured?
Audience communication styles Considering the communication styles of the client and other members of the audience, how can the evaluator best convey the evaluation findings? 
Internal communication How will key audience needs for information on the evaluation’s progress be determined and met, and how will communication be maintained between the evaluators, the sponsors, and the system’s personnel?
Internal credibility Will the evaluation be fair to all system participants and clients and not biased in favor of or against any stakeholder perspective(s)?
External credibility Will the evaluation be free of bias?
Realistic expectations How will the evaluator make clear to stakeholders that realistically only a subset of their information needs will be addressed?
Security What provisions will assure security of the data?
Protocol What communication channels will be honored and employed?
Public relations How will stakeholders be consulted and kept informed about the intents and results of the evaluation?
Political viability How will evaluators stay abreast of social and political forces associated with  the evaluation and use this knowledge when planning and carrying out evaluation procedures?
Evaluator qualifications Does the composition of the evaluation team assure knowledge of context and competence in content and methodological areas?
Stakeholder confidence What checks will be made to ensure that the evaluation plan and the composition  of the evaluation team are responsive and acceptable to the key stakeholders?
3.  Contractual/Legal Arrangements.  Evaluators and clients should establish clear working agreements to ensure efficient collaboration and protect involved parties’ rights.
Client, evaluator, & other roles  Who is the sponsor, who is the evaluator, who are the other audiences, and how are they related to the evaluand?
Evaluation products  What evaluation outcomes are to be delivered and in what form?
Equitable evaluation service  What safeguards assure that the evaluation will serve all levels of stakeholders in  addition to persons in leadership or decision-making roles?
Realistic commitments What clarifications assure that the evaluation can proceed while making reasonable efforts to serve a broad audience but not becoming bogged down in overidentifying and consulting with stakeholders?
Delivery schedule  What is the schedule of evaluation services and products?
Editing reports  Who has authority for editing evaluation reports?
Access to data What existing data may the evaluators use, and what new data may they obtain?
Access to stakeholders Are there sufficient safeguards to assure that evaluators may contact involved stakeholders?
Prerelease reviews  Will the client and representatives of the intended audience(s) be provided appropriate opportunities to review draft reports for clarity and fairness prior to their finalization and release?
Release of reports Who will release the reports, and what audiences may receive them?
Responsibility & authority  Have the system personnel and evaluators agreed on what persons and groups  have both the responsibility and authority to perform the various evaluation tasks?
Finances What is the schedule of payments for the evaluation, and who will provide the  funds?
External audit  Is there provision, as needed, to have the evaluation plan reviewed and the evaluation work audited by another evaluator whose credentials are acceptable to  the client and trusted by the other key stakeholders?
Contract review & revision  Is there appropriate provision for reviewing and amending the contract in  response to emergent developments in the evaluation?
4.  Technical Design. Evaluators should convert a general evaluation plan to a detailed, yet flexible technical plan.
Objectives  What is the evaluand intended to achieve/produce, and in what terms should it be evaluated? 
Variables  What classes of information will be collected, e.g., context, inputs, processes, outcomes?
Program description  Will the object of the evaluation (e.g., the program) be described sufficiently, so that stakeholders will understand its nature?
Investigatory framework Under what conditions will the data be gathered, e.g., experimental design, case  study, survey, site review, examination, etc.?
Instrumentation  What data-gathering instruments and techniques will be employed, and how will the evaluator assure that they address the key evaluation questions?
Sampling  What samples will be drawn, how will they be drawn, and will they meet both utility and technical requirements?
Data gathering How will the data-gathering plan be implemented, and who will gather the data?
Data storage and retrieval  What format, procedures, and facilities will be used to store and retrieve the data?
Data analysis  How will the data be analyzed?
Sources of interpretation Who is charged to interpret findings, e.g., the evaluators, various stakeholders, a regulatory body, etc.?
Bases for interpretation  What bases will be used to interpret findings, e.g., objectives, assessed needs, contractual specifications, laws and regulations, democratic ideals, social norms, performance by a comparison group, technical standards, polls, judgments by reference groups, etc.?
Methods of interpretation What methods will be used to assign value meaning to findings, e.g., focus groups, a Delphi study, advocacy and adversary reports, etc.? 
Reports What reports will be used to disseminate the evaluation findings?
Reporting media  Considering the preferences of the audiences, what are the most appropriate means of reporting findings, e.g., detailed technical reports, summaries, press conferences, study sessions, memos and letters, video presentations, etc.?
Reporting language  Will reports need to be presented in different languages—technical and nontechnical, English and other language(s)—to meet the needs of different audiences?
Reporting format  Will reports be carefully formatted to enhance their readability? 
Responsive design  What ongoing evaluation planning process and resource plan will assure flexibility for adding to or otherwise revising the evaluation questions and obtaining unanticipated, pertinent information?
Delimited design  Is there a clear delimitation of the design, including the purpose of the evaluation and the questions that will be answered?
Attention to trade-offs How will the evaluation address trade-offs between comprehensiveness and selectivity at each stage of the evaluation: planning; budgeting; and collecting, organizing, analyzing, interpreting, and reporting information?
Technical adequacy  What are assurances that the findings will be reliable, valid, and objective?
5.  Management Plan. Evaluators should control and direct the evaluation efficiently and enhance the host agency’s capacity to evaluate.
Organizational mechanism What organizational unit will be employed, e.g., an in-house office of evaluation, a self-evaluation system, a contract with an external organization, or a consortium-  supported evaluation center?
Organizational location  Through what channels can the evaluation influence policy formulation and administrative decision making?
Policies and procedures  What established and/or ad hoc policies and procedures will govern this evaluation?
Staff selection  Who will conduct the evaluation?
Staff composition Will the composition of the staff be responsive to the concerns of key stakeholders?
Credibility of staff Does the plan demonstrate that the staff will be competent, experienced, and credible in the pertinent content, environment, and methodological areas?
Commitment of staff  Does the plan commit staff to the required time and effort and not just their reputations to the evaluation?
Work management What oversight and control will be administered to assure that evaluators devote time and effort, as well as their reputations, to the evaluation?
Facilities  What space, equipment, and materials will be available to support the evaluation?
Data-gathering schedule  What instruments will be administered,  to what groups, according to what schedule?
Maintaining focus  Are there sufficient safeguards to prevent gathering extraneous information?
Reporting schedule What reports will be provided, to what audiences, according to what schedule?
Training  Who will provide what evaluation training to what groups?
Installation of evaluation Will this evaluation be used to aid the host institution to improve and extend its internal evaluation capability?
Budget What is the structure of the budget, is it sufficient but reasonable, and how will it be monitored?
Allocation of resources  Have the resources for the evaluation been appropriately distributed across data collection, analysis, and reporting, placing the most effort on the most important information requirements?
6.  Moral/Ethical Imperatives.  Evaluators and clients/stakeholders should clarify and confirm the evaluation’s role in ethically serving some socially valuable purpose.
Philosophical stance  Will the evaluation be value based, value plural, or value free?
Evaluator’s values  Will the evaluator’s technical standards and values conflict with the client system’s and/or sponsor’s values; will the evaluator face any conflict of interest problems; what will be done about possible conflicts?
Judgments  Will the evaluator judge the program; leave that to the client; or obtain, analyze, and report the judgments of various reference groups?
Objectivity  How will the evaluator avoid being coopted and maintain his or her objectivity?
Equity  How will the evaluator make sure to address and honor the needs and rights of all stakeholders equitably, taking appropriate account of their gender, ethnicity, and language backgrounds?
Cost effectiveness Compared to its potential payoff, will the evaluation be carried out at a reasonable cost?
7. Utility Provisions. Evaluators should plan and execute steps that promote constructive uses of the evaluation findings.
General prospects for utility  Will the evaluation meet utility criteria of relevance, scope, importance, credibility, timeliness, clarity, and pervasiveness?
Mutual understanding  Is it quite certain that the evaluator understands the client’s requirements and that the client understands the extent and limitations of the evaluator’s commitment?
Acceptability of the approach Is there confirmation that the evaluator’s approach is acceptable to the client and key stakeholders?
Responsiveness  Throughout the evaluation, will there be sufficient flexibility and resources to identify and address new audiences and new questions?
Collaborative design  Will the evaluator directly involve clients and other stakeholders in designing and conducting the evaluation?
Boundaries of use  Are there clear stipulations concerning what stakeholder needs will be served and which ones would be outside the evaluation’s boundaries?
Realistic expectations  Will appropriate steps be taken to help stakeholders develop realistic expectations considering available financial, time, and personnel resources?
Service to all stakeholders  Are there adequate provisions to assure that the evaluator will determine the evaluation needs of the various stakeholders and, within feasibility limits, serve all levels of stakeholders?
Tailoring  Are there appropriate provisions for tailoring reports to the needs of the different audiences?
Stakeholder perspectives  What value perspectives do the stakeholders value most, e.g., educational, social, scientific, technical, economic?
Trade-offs Does the evaluation plan adequately consider trade-offs between comprehensiveness and selectivity at every step in the evaluation:  planning, budgeting, and obtaining and reporting information?
Acceptance of the plan Are there provisions for clearly describing the evaluation plan to the full range of stakeholders and demonstrating that the plan is realistic and methodologically sound?
Progress reports  Are there provisions for keeping interested audiences informed about the evaluation’s progress?
   
 This checklist is being provided as a free service to the user. The provider of the checklist has not modified or adapted the checklist to fit the specific needs of the user and the user is executing his or her own discretion and judgment in using the checklist. The provider of the checklist makes no representations or warranties that this checklist is fit for the particular purpose contemplated by user and specifically disclaims any such warranties or representations.