9. Analyze the data so that the qualitative findings are clear, credible, and address the relevant and priority evaluation questions and issues.
  • Purpose guides analysis. Keep the analysis focused on primary evaluation questions.
  • Be sensitive to stages and sequence of analysis.
    • Generative and emergent stage: Ideas for making sense of the data that emerge while still in the field constitute the beginning of analysis; they are part of the record of field notes. In the course of fieldwork, ideas about directions for analysis will occur. Patterns take shape. Possible themes spring to mind. Hypotheses emerge that inform subsequent fieldwork. Record these.
    • Confirmatory stage: Later stages of fieldwork bring closure by moving toward confirmatory data collection—deepening insights into and confirming (or disconfirming) patterns that seem to have appeared.
    • Systematic analysis following fieldwork: Write case studies and conduct cross-case analyses based on rigorous review of field notes, interview transcripts, and document analysis.
  • Purpose guides reporting.
    • Summative evaluations will be judged by the extent to which they contribute to making decisions about a program or intervention, usually decisions about overall effectiveness, continuation, expansion, and/or replication at other sites. A full report presenting data, interpretations and recommendations typically is required.
    • Formative evaluations conducted for program improvement may or may not require a detailed, written report for dissemination. Findings may be reported primarily orally. Summary observations may be listed in outline form or an executive summary may be written, but the time lines for formative feedback and the high costs of formal report writing may make a full, written report unnecessary. Staff and funders often want the insights of an outsider who can interview program participants effectively, observe what goes on in the program, and provide helpful feedback. The methods are qualitative, the purpose is practical, and the analysis is done throughout fieldwork; no written report is expected beyond a final outline of observations and implications.
  • Organize the voluminous data from fieldwork. Inventory what you have.
    • Are the field notes complete?
    • Are there any parts that you put off to write later and never got to that need to be finished, even at this late date, before beginning analysis?
    • Are there any glaring holes in the data that can still be filled by collecting additional data before the analysis begins?
    • Are all the data properly labeled with a notation system that will make retrieval manageable? (Dates, places, interviewee identifying information, etc.)
    • Are interview transcriptions complete?
    • Check out the quality of the information collected.
    • Back up the data.
  • Determine which, if any, computer-assisted qualitative data management and analysis will be used. (Qualitative software programs facilitate data storage, coding, retrieval, comparing, and linking—but human beings do the analysis.) Checklist of considerations in selecting software:
    • How you enter your data (typing directly, imported from word processing, scanning; flexible or fixed formatting)
    • Storage differences (internal versus external databases)
    • Coding variations (on-screen coding versus assigning the codes first)
    • Differences in ease of organizing, reorganizing and relabeling codes
    • Variations in whether memos and annotations can be attached to codes (especially important for team analysis)
    • Data-linking mechanisms and ease vary (connecting different data sources or segments during analysis)
    • Ease of navigating and browsing
    • Ease, speed, and process of search and retrieval
    • Important display variations (e.g., with and without context)
    • Tracking details (recording what you’ve done for review)
  • Distinguish description, interpretation, and judgment.
    • Aim for “thick” description—sufficient detail to take the reader into the setting being described. Description forms the bedrock of all qualitative reporting.
    • Use direct quotations so that respondents are presented in their own terms and ways of expressing themselves.
    • Keep quotations and field incident descriptions in context.
    • Assure that interpretations follow from the qualitative data. Qualitative interpretation begins with elucidating meanings. The analyst examines a story, a case study, a set of interviews, or a collection of field notes and asks: What does this mean? What insights and answers are provided about central evaluation questions?
    • Make the basis for judgments explicit.
  • Distinguish and separate case studies from cross-case analysis.
    • Make cases complete. A case study consists of all the information one has about each case: interview data, observations, the documentary data (e.g., program records or files, newspaper clippings), impressions and statements of others about the case, and contextual information—in effect, all the information one has accumulated about each particular case goes into that case study.
    • Make case studies holistic and context sensitive. Case analysis involves organizing the data by specific cases (individuals, groups, sites, communities, etc.) for in depth study and comparison. The qualitative analyst’s first and foremost responsibility consists of doing justice to each individual case. (Each case study in a report stands alone, allowing the reader to understand the case as a unique, holistic entity. At a later point in analysis it is possible to compare and contrast cases, but initially each case must be represented and understood as an idiosyncratic manifestation of the evaluation phenomenon of interest.)
    • Ground cross-case analysis in the individual case studies.
    • Identify cross-case patterns and themes with citations and illustrations from the case studies.
  • Distinguish inductive from deductive qualitative analysis.
    • Inductive analysis involves discovering patterns, themes, and categories in one’s data. Findings emerge out of the data through the analyst’s interactions with the data.
    • Deductive analysis involves analyzing data according to an existing framework, e.g., the program’s logic model.
    • Build on the strengths of both kinds of analysis. For example, once patterns, themes, and/or categories have been established through inductive analysis, the final, confirmatory stage of qualitative analysis may be deductive in testing and affirming the authenticity and appropriateness of the inductive content analysis, including carefully examining deviant cases or data that don’t fit the categories developed.
  • Distinguish convergence and divergence in coding and classifying.
    • In developing codes and categories, begin with convergence—figuring out what things fit together. Begin by looking for recurring regularities in the data. These regularities reveal patterns that can be sorted into categories.
      • Judge categories by two criteria: internal homogeneity and external heterogeneity. The first criterion concerns the extent to which the data that belong in a certain category cohere in a meaningful way. The second criterion concerns the extent to which differences among categories are clear.
      • Prioritize categories by utility, salience, credibility, uniqueness, heuristic value, and feasibility.
      • Test the category system or set of categories for completeness and coherence. The individual categories should be consistent; a set of categories should comprise a whole picture. The set should be reasonably inclusive of the qualitative data and information collected and analyzed.
      • Test the credibility and understandability of the categories with someone not involved in the analysis. Do the categories make sense?
    • After analyzing for convergence, the mirror analytical strategy involves examining divergence. This is done by processes of extension (building on items of information already known), bridging (making connections among different items), and surfacing (proposing new information that ought to fit and then verifying its existence). The analyst brings closure to the process when sources of information have been exhausted, when sets of categories have been saturated so that new sources lead to redundancy, and when clear regularities have emerged that feel integrated.
      • Carefully and thoughtfully consider data that do not seem to fit including deviant cases that don’t fit the dominant identified patterns.
    • Integrate the analysis. This sequence, convergence then divergence, should not be followed mechanically, linearly, or rigidly. The processes of qualitative analysis involve both technical and creative dimensions.
  • Construct a process-outcomes matrix for the program.
    • Distinguish process descriptions from outcome documentation.
    • Show linkages between processes and outcomes.
  • Integrate and reconcile qualitative and quantitative findings as appropriate.
    • Note where qualitative and quantitative findings reinforce one another.
    • Note and explain differences
  • Use strategies to enhance the rigor and credibility of analysis.
    • Consider and discuss alternative interpretations of the findings.Carefully consider and discuss cases and data that don’t fit overall patterns and themes.
    • Triangulate the analysis. Options include:
      • Check out the consistency of findings generated by different data collection methods, i.e., methods triangulation.
      • Check out the consistency of different data sources within the same method, i.e., triangulation of sources.
      • Use multiple analysts to review findings, i.e., analyst triangulation.
      • Use multiple perspectives or theories to interpret the data, i.e., theory/perspective triangulation.
  • Determine substantive significance. In lieu of statistical significance, qualitative findings are judged by their substantive significance. The analyst makes an argument for substantive significance in presenting findings and conclusions, but readers and users of the analysis will make their own value judgments about significance. In determining substantive significance, the analyst addresses these kinds of questions:
    • How solid, coherent, and consistent is the evidence in support of the findings? (Triangulation, for example, can be used in determining the strength of evidence in support of a finding.)
    • To what extent and in what ways do the findings increase and deepen understanding of the program being evaluated?
    • To what extent are the findings consistent with other knowledge? (A finding supported by and supportive of other work has confirmatory significance. A finding that breaks new ground has discovery or innovative significance.)
    • To what extent are the findings useful for the evaluation’s purpose?
  • Determine if an expert audit or metaevaluation is appropriate because the stakes for the evaluation are high (e.g., major summative evaluation) and the credibility of the qualitative findings will be enhanced by external review.
    • Quality audit: An external audit by a disinterested expert can render judgment about the quality of data collection and analysis.
    • An audit of the qualitative data collection process results in a dependability judgment.
    • An audit of the analysis provides a confirmability judgment.
    • Any audit would need to be conducted according to appropriate criteria given the evaluation’s purpose and intended uses.
QEC Home
9