return to
Checklists

UTILIZATION-FOCUSED EVALUATION (U-FE) CHECKLIST
pdf

Michael Quinn Patton

January 2002

Introduction
1. Program/Organizational Readiness Assessment
2. Evaluator Readiness and Capability Assessment
3. Identification of Primary Intended Users
4. Situational Analysis
5. Identification of Primary Intended Uses
6. Focusing the Evaluation
7. Evaluation Design
8. Simulation of Use
9. Data Collection
10. Data Analysis
11. Facilitation of Use
12. Metaevaluation
Introduction

Utilization-Focused Evaluation (U-FE) begins with the premise that evaluations should be judged by their utility and actual use; therefore, evaluators should facilitate the evaluation process and design any evaluation with careful consideration of how everything that is done, from beginning to end, will affect use. Use concerns how real people in the real world apply evaluation findings and experience the evaluation process.  Therefore, the focus in utilization-focused evaluation is on intended use by intended users.  Since no evaluation can be value-free, utilization-focused evaluation answers the question of whose values will frame the evaluation by working with clearly identified, primary intended users who have responsibility to apply evaluation findings and implement recommendations. 

Utilization-focused evaluation is highly personal and situational.  The evaluation facilitator develops a working relationship with intended users to help them determine what kind of evaluation they need.  This requires negotiation in which the evaluator offers a menu of possibilities within the framework of established evaluation standards and principles.

Utilization-focused evaluation does not advocate any particular evaluation content, model, method, theory, or even use.  Rather, it is a process for helping primary intended users select the most appropriate content, model, methods, theory, and uses for their particular situation.  Situational responsiveness guides the interactive process between evaluator and primary intended users.  A utilization-focused evaluation can include any evaluative purpose (formative, summative, developmental), any kind of data (quantitative, qualitative, mixed), any kind of design (e.g., naturalistic, experimental), and any kind of focus (processes, outcomes, impacts, costs, and cost-benefit, among many possibilities).  Utilization-focused evaluation is a process for making decisions about these issues in collaboration with an identified group of primary users focusing on their intended uses of evaluation. 

A psychology of use undergirds and informs utilization-focused evaluation:  intended users are more likely to use evaluations if they understand and feel ownership of the evaluation process and findings; they are more likely to understand and feel ownership if they've been actively involved; by actively involving primary intended users, the evaluator is training users in use, preparing the groundwork for use, and reinforcing the intended utility of the evaluation every step along the way. 

The 12 parts of the checklist are divided into 2 columns. Primary U-FE tasks are identified in the columns on the left.  Because of the emphasis on facilitation in U-FE, particular facilitation challenges are identified in the columns on the right. Underlying premises are made explicit for each step in the U-F process.
1. PROGRAM/ORGANIZATIONAL READINESS ASSESSMENT
Premise: Key people who want the evaluation conducted need to understand and be interested in a utilization-focused evaluation (U-FE). Premise: U-FE requires active and skilled guidance from and facilitation by an evaluation facilitator.

 Primary Tasks:

 Evaluation Facilitation Challenges:
  • Assess primary evaluation clients' commitment to doing useful evaluation based on an explanation of U-FE.
  • Explaining U-FE and enhancing readiness for evaluation generally and U-FE specifically. 
  • Assess if the program is ready to spend time and resources on evaluation.
  • Communicating the value and requirements of U-FE, assessing commitment, and building commitment as needed. 
  • Determine if primary evaluation clients are ready to assess various stakeholder constituencies to select primary intended users of the evaluation.
  • Explaining and facilitating stakeholder assessment; distinguishing between stakeholders in general and primary intended users in particular.
  • Assess what needs to be done and can be done to enhance readiness.
  • Planning, negotiating, and facilitating increased readiness with evaluation clients as needed.
2. EVALUATOR READINESS AND CAPABILITY ASSESSMENT

Premise: Facilitating and conducting a utilization-focused evaluation requires a particular philosophy and special skills.

Premise: Evaluation facilitators need to know their strengths and limitations and develop the skills needed to facilitate utilization-focused evaluations.
Primary Tasks: Evaluation Facilitation Challenges:
  • Assess the match between the evaluator's knowledge and what will be needed in the evaluation.
  • Getting a good match between the evaluator's knowledge and what will be needed in the evaluation.
  • Assess the match between the evaluator's commitment and the likely challenges of the situation.
  • Maintaining focus on and commitment to intended use by intended users as the primary outcome of the evaluation.
  •   Assess the match between the evaluator's skills and what will be needed in the evaluation.
  • Developing facilitation skills to fit the challenges of the specific people and situation.
  • Make sure the evaluators are prepared to have their effectiveness judged by the use of the evaluation by primary intended users.
  • Honest self-reflection by the evaluators.

3. IDENTIFICATION OF PRIMARY INTENDED USERS

Premise: Primary intended users are people who have a direct, identifiable stake in the evaluation and meet the criteria below to some extent. (Caveat: These judgments are necessarily subjective and negotiable.)

Premise: The U-FE facilitator needs to both assess the characteristics of primary intended users and reinforce characteristics that will contribute to evaluation use.

Primary Task:
Find and recruit people who are...

 Evaluation Facilitation Challenges:
  • Interested
  • Determining real interest; building interest as needed; sustaining interest throughout the U-FE process.
  • Knowledgeable
  • Determining knowledge; increasing knowledge as needed.
  • Open
  • Facilitating an evaluation climate of openness.
  • Connected to an important stakeholder constituency
  • Working with primary intended users to examine stakeholder connections and their implications for use.
  • Credible
  • Building and sustaining credibility.
  • Teachable
  • Teaching evaluation and U-FE.
  •   Available for interaction throughout the evaluation process
  • Outlining and facilitating a process that intended users want to be part of.

4. SITUATIONAL ANALYSIS

Premises: Evaluation use is people- and context-dependent.  Use is likely to be enhanced when the evaluation takes into account and is adapted to crucial situational factors such as those below.

Premise: The evaluator has responsibility to identify, assess, understand, and act on situational factors that may affect use.

Primary Tasks:

Evaluation Facilitation Challenges:
  • Examine program's prior experiences with evaluation.
  • Learning the extent to which past evaluations were useful.
  • Look for possible barriers or resistance to use.
  • Looking at typical barriers—people, resources, culture, turbulence—while also looking out for unusual or unexpected barriers.
  • Identify factors that may support and facilitate use.
  • Looking at typical supports—e.g., accountability demands—while also looking out for unusual or unexpected ones.
  • Get clear about resources available for evaluation.
  • Including in the budget resources beyond analysis and reporting to facilitate use.
  • Identify any upcoming decisions, deadlines, or time lines that the evaluation should meet to be useful.
  • Being realistic about time lines.
  • Assess the evaluation knowledge level and experiences of primary intended users.
  • Building into the evaluation process opportunities to increase the knowledge of primary intended users.
  • Understand the political context for the evaluation, and calculate how political factors may affect use.
  • Including attention to both potential uses and potential misuses.
  • Make sure that important constituencies and diverse stakeholder groups for the evaluation are represented among the primary intended users and assess the consequences of any omissions for use.
  • Staying focused on intended use by intended users while assuring that intended users represent important and legitimate interests of diverse stakeholders—done on an ongoing basis as new information surfaces throughout the evaluation.

5. IDENTIFICATION OF PRIMARY INTENDED USES

Premise: Intended use by primary intended users is the U-FE goal of the evaluation.

Premise: The rich menu of evaluation options are reviewed, screened, and prioritized to focus the evaluation.
Primary Tasks: Evaluation Facilitation Challenges:
  • Consider how evaluation could contribute to program improvement.
  • Guiding primary intended users in reviewing formative evaluation options.
  • Consider how evaluation could contribute to making major decisions about the program.
  • Guiding primary intended users in reviewing summative and major decision-oriented evaluation options.
  • Consider how evaluation could contribute by generating knowledge.
  • Guiding primary intended users in considering the possibility of using evaluation to generate lessons learned and evidence-based practices that might apply beyond the program being evaluated.
  • Consider process uses of evaluation.
  • Enhancing communications; building capacity; learning evaluative thinking; nurturing an evaluation culture within the organization; and/or reinforcing the program intervention.

6. FOCUSING THE EVALUATION

Premise: The focus derives from primary intended uses of the evaluation by primary intended users.

Premise:  Primary intended users will often need considerable assistance identifying and agreeing on priority evaluation uses and the major focus for the evaluation.
Primary Tasks: Evaluation Facilitation Challenges:
  • Make sure that all high priority questions are addressed in the evaluation design—or be clear about why they aren't included.
  • Actively involving primary intended users in determining priorities; narrowing the options and determining what specific evaluation questions and issues will be addressed by the evaluation based on priority intended uses.
  • Make sure that the intended uses of answers to the specific evaluation questions are reasonably clear.    
  • Actively involving primary intended users in determining the specific relevance of intended uses of findings.

7. EVALUATION DESIGN

Premises: The evaluation should be designed to lead to useful findings. Methods should be selected and the evaluation designed to support and achieve intended use by primary intended users.

Premise: Evaluators and users have varying responsibilities in the design decision-making process.
Primary Tasks: Evaluation Facilitation Challenges:
  • Select methods appropriate to the questions being asked.
  • Making sure that methods are selected jointly by primary intended users and the evaluator(s).
  • Assure that results obtained by the methods selected will be believable, credible, and valid to primary intended users.    
  • Making sure that primary intended users play an active role in reviewing methods to examine their believability, credibility, and validity.
  • Assure that the proposed methods and measurements are
    • Practical
    • Cost-effective
    • Ethical
  • Making sure that methods and measures are reviewed jointly by primary intended users and the evaluator(s).
  •   Assure that the results obtained from these methods will be able to be used as intended.
  • Facilitating serious review of intended use by primary intended users.
  • Review the evaluation as designed in relation to professional standards and principles.
  • Taking professional standards and principles seriously—-not just treating them as boilerplate or window dressing.
  • Consider seriously whether involving primary intended users or other stakeholders in actual data collection enhance process use.
  • Seeking creative possibilities for enhancing process uses; examining potential trade-offs between utility (process uses specifically) and credibility.

8. SIMULATION OF USE

Premise: Before data are collected, a simulation of potential use can be done with fabricated findings in a real-enough way to provide a meaningful learning experience for primary intended users.

Premise: It's important to move discussions of use from the abstract to the concrete, and a simulation of use based on fabricated data helps do that.
Primary Tasks: Evaluation Facilitation Challenges:
  • Fabricate findings based on the proposed design and measures.
  • Fabricating realistic findings that show varying results and offer good grist for simulated interaction among primary intended users.
  • Guide primary intended users in interpreting the potential (fabricated) findings?
  • Helping primary intended users take the simulation seriously so that they can use the experience to improve design and be better prepared for real use of findings.
  • Interpret the simulation experience to determine if any design changes or additions to the data collection would likely increase utility. 
  • Taking time to do this final, critical check before data collection.
  • As a final step before data collection, have primary intended users make an explicit decision to proceed with the evaluation given likely costs and expected uses.
  • Helping primary intended users seriously ask: Given likely costs and expected uses, is the evaluation worth doing? Assuring that primary intended users feel ownership of the design and measures.

9. DATA COLLECTION

Premise: Data collected should be managed with use in mind.

Premise: It's important to keep primary intended users informed and involved throughout all stages of the process.
Primary Tasks: Evaluator Facilitation Challenges:
  • Keep primary intended users informed of progress.
  • Noting any problems or delays as soon as they are known.
  • Inform primary intended users of important interim findings to maintain interest in the evaluation.
  • Getting intended users to understand that preliminary findings are subject to revision.
  • If involving primary intended users or other stakeholders in actual data collection, manage this process carefully.
  • Offering opportunities to reflect on the process and learn from it; debriefing process learnings as they occur.

10. DATA ANALYSIS

Premise: Analysis should be organized to facilitate use by primary intended users.

Premise:  Facilitating data interpretation among primary intended users increases their understanding of the findings, their sense of ownership of the evaluation, and their commitment to use the results
Primary Tasks: Evaluation Facilitation Challenges:
  • Organize data to be understandable and relevant to primary intended users.
  • Basing organization of data on primary intended uses of the evaluation.
  • Actively involve users in interpreting findings and generating recommendations.
  • Helping users distinguish between findings, interpretations, judgments, and recommendations.
  • Examine the findings and their implications from various perspectives with focus on primary intended uses by primary intended users.
  • Offering opportunities to reflect on the analytical process and learn from it; helping users distinguish varying degrees of certainty in the findings; being open and explicit about data strengths, weaknesses, and limitations.

11. FACILITATION OF USE

Premise: Use doesn't just happen naturally; it needs to be facilitated.

Premise: Facilitating use is a central part of the evaluator's job.
Primary Tasks: Evaluation Facilitation Challenges: 
  • Work with primary intended users to use the findings and learnings from the process in intended ways.
  • Actively facilitating the users' sense of ownership of the findings and their commitment to act on those findings.
  • Examine potential uses and users beyond those intended and originally targeted (dissemination).
  • Reviewing the larger, and possibly changed, stakeholder environment. (There may be a separate action group that the evaluation findings and recommendations would be passed on to for implementation.)
  • Decide on dissemination mechanisms and avenues consistent with intended uses and additional desired uses.
  • Reviewing the larger, and possibly changed, stakeholder environment and resources available to support dissemination; clearly differentiating use from dissemination.
  • Identify possible misuses, and plan action to assure appropriate uses.
  • Being clear about the ethical obligations of being an evaluator.
  • Stay involved beyond formal reporting, and engage in follow-up facilitation as needed to enhance use.
  • Building in from the beginning time and resources to facilitate use beyond just writing a report—additional resources may be needed if new uses or users are added.

12. METAEVALUATION

Premise: Utilization-focused evaluations should be evaluated by whether primary intended users used the evaluation in intended ways.

Premise: A U-FE facilitator can learn something from each evaluation.
Primary Tasks: Evaluation Facilitation Challenges:
  • After the evaluation is completed, followup to determine the extent to which intended use by intended users was achieved.
  • Taking the time for ongoing learning to achieve long term, utilization-focused evaluation excellence.
  • Followup to determine the extent to which additional uses or users were served beyond those initially targeted.
  • Finding time and resources to do the necessary fieldwork.
  • Followup to determine and learn from any misuses or unintended consequences of the evaluation.
  • Helping primary intended users be open and reflective about their U-FE experience,
 
This checklist is being provided as a free service to the user. The provider of the checklist has not modified or adapted the checklist to fit the specific needs of the user and the user is executing his or her own discretion and judgment in using the checklist. The provider of the checklist makes no representations or warranties that this checklist is fit for the particular purpose contemplated by user and specifically disclaims any such warranties or representations.