QUALITATIVE
EVALUATION CHECKLIST
pdf
Michael Quinn Patton
September 2003
5. Where fieldwork is part of the evaluation, determine how to approach fieldwork.
7. Design the evaluation with careful attention to ethical issues.
8. Anticipate analysis--design the evaluation data collection to facilitate analysis.
Qualitative evaluations use qualitative and naturalistic methods, sometimes alone, but often in combination with quantitative data. Qualitative methods include three kinds of data collection: (1) in-depth, open-ended interviews; (2) direct observation; and (3) written documents.
Interviews: Open-ended questions and probes yield in-depth responses about people's experiences, perceptions, opinions, feelings, and knowledge. Data consist of verbatim quotations with sufficient context to be interpretable.
Observations: Fieldwork descriptions of activities, behaviors, actions, conversations, interpersonal interactions, organizational or community processes, or any other aspect of observable human experience. Data consist of field notes: rich, detailed descriptions, including the context within which the observations were made.
Documents: Written materials and other documents from organizational, clinical, or program records; memoranda and correspondence; official publications and reports; personal diaries, letters, artistic works, photographs, and memorabilia; and written responses to open-ended surveys. Data consist of excerpts from documents captured in a way that records and preserves context.
The data for qualitative evaluation typically come from fieldwork. The evaluator spends time in the setting under study—a program, organization, or community where change efforts can be observed, people interviewed, and documents analyzed. The evaluator makes firsthand observations of activities and interactions, sometimes engaging personally in those activities as a "participant observer." For example, an evaluator might participate in all or part of the program under study, participating as a regular program member, client, or student. The qualitative evaluator talks with people about their experiences and perceptions. More formal individual or group interviews may be conducted. Relevant records and documents are examined. Extensive field notes are collected through these observations, interviews, and document reviews. The voluminous raw data in these field notes are organized into readable narrative descriptions with major themes, categories, and illustrative case examples extracted through content analysis. The themes, patterns, understandings, and insights that emerge from evaluation fieldwork and subsequent analysis are the fruit of qualitative inquiry.
Qualitative findings may be presented alone or in combination with quantitative data. At the simplest level, a questionnaire or interview that asks both fixed-choice (closed) questions and open-ended questions is an example of how quantitative measurement and qualitative inquiry are often combined.
The quality of qualitative data depends to a great extent on the methodological skill, sensitivity, and integrity of the evaluator. Systematic and rigorous observation involves far more than just being present and looking around. Skillful interviewing involves much more than just asking questions. Content analysis requires considerably more than just reading to see what's there. Generating useful and credible qualitative findings through observation, interviewing, and content analysis requires discipline, knowledge, training, practice, creativity, and hard work.
Qualitative methods are often used in evaluations because they tell
the program's story by capturing and communicating the participants'
stories. Evaluation case studies have all the elements of a good
story. They tell what happened when, to whom, and with what consequences.
The purpose of such studies is to gather information and generate findings
that are useful. Understanding the program's and participant's stories
is useful to the extent that those stories illuminate the processes
and outcomes of the program for those who must make decisions about
the program. The methodological implication of this criterion is that
the intended users must value the findings and find them credible. They
must be interested in the stories, experiences, and perceptions of program
participants beyond simply knowing how many came into the program, how
many completed it, and how many did what afterwards. Qualitative findings
in evaluation can illuminate the people behind the numbers and put faces
on the statistics to deepen understanding.
Greene, J. C. (2000). Understanding social programs through evaluation. In N. K. Denzin & Y. S. Lincoln (Eds.). Handbook of qualitative research (2nd ed.) (pp. 981- 999). Thousand Oaks, CA: Sage.
Patton, M. Q. (2002). Qualitative research and evaluation methods (3rd ed.). Thousand Oaks, CA: Sage.
Rossman, G. B., & Rallis, S. F. (1998). Learning in the field: An introduction to qualitative research. Thousand Oaks, CA: Sage.
Schwandt, T. A. (2001). Dictionary of qualitative inquiry (2nd revised ed.) Thousand Oaks, CA: Sage.
Stake, R. (1995). The art of case study research. Thousand Oaks, CA: Sage.
Williams, D. D. (Ed.) (1986). Naturalistic evaluation. New Directions for Program Evaluation, 30.
This checklist is being provided as a free service to the user. The provider of the checklist has not modified or adapted the checklist to fit the specific needs of the user, and the user is executing his or her own discretion and judgment in using the checklist. The provider of the checklist makes no representations or warranties that this checklist is fit for the particular purpose contemplated by users and specifically disclaims any such warranties or representations.