4. Make major design decisions so that the design
answers important evaluation questions for intended users. Consider
design options and choose those most appropriate for the evaluation’s
purposes.
|
- Pure or mixed methods design: Determine whether
the evaluation will be purely qualitative or a mixed method design
with both qualitative and quantitative data.
|
- Units
of analysis: No matter what you are studying, always collect
data on the lowest level unit of analysis possible; you can aggregate
cases later for larger units of analysis. Below are some examples
of units of analysis for case studies and comparisons.
- People-focused: individuals; small, informal groups (e.g., friends,
gangs); families
- Structure-focused: projects, programs, organizations, units
in organizations
- Perspective/worldview-based: People who share a culture; people
who share a common experience or perspective (e.g., dropouts,
graduates, leaders, parents, Internet listserv participants, survivors,
etc.)
- Geography-focused: neighborhoods, villages, cities, farms, states,
regions, countries, markets
- Activity-focused: critical incidents, time periods, celebrations,
crises, quality assurance violations, events
Time-based: Particular days, weeks, or months; vacations; Christmas
season; rainy season; Ramadan; dry season; full moons; school
term; political term of office; election period
(Note: These are not mutually exclusive categories)
|
- Purposeful
sampling strategies: Select information-rich cases for in-depth
study. Strategically and purposefully select specific types and numbers
of cases appropriate to the evaluation’s purposes and resources.
Options include:
- Extreme or deviant case (outlier) sampling: Learn from unusual
or outlier program participants of interest, e.g., outstanding
successes/notable failures; top of the class/dropouts; exotic
events; crises.
- Intensity sampling: Information-rich cases manifest the phenomenon
intensely, but not extremely, e.g., good students/poor students;
above average/below average.
- Maximum variation sampling: Purposefully pick a wide range of
cases to get variation on dimensions of interest. Document uniquenesses
or variations that have emerged in adapting to different conditions;
identify important common patterns that cut across variations
(cut through the noise of variation).
- Homogeneous sampling: Focus; reduce variation; simplify analysis;
facilitate group interviewing.
- Typical case sampling: Illustrate or highlight what is typical,
normal, average.
- Critical case sampling: Permits logical generalization and maximum
application of information to other cases because if it's true
of this one case, it's likely to be true of all other cases.
- Snowball or chain: Identify cases of interest from sampling
people who know people who know people who know what cases are
information-rich, i.e., good examples for study, good interview
subjects.
- Criterion sampling: Pick all cases that meet some criterion,
e.g., all children abused in a treatment facility; quality assurance.
- Theory-based or operational construct sampling: Find manifestations
of a theoretical construct of interest so as to elaborate and
examine the construct and its variations, used in relation to
program theory or logic model.
- Stratified purposeful sampling: Illustrate characteristics of
particular subgroups of interest; facilitate comparisons.
- Opportunistic or emergent sampling: Follow new leads during
fieldwork; taking advantage of the unexpected; flexibility.
- Random purposeful sampling (still small sample size): Add credibility
when potential purposeful sample is larger than one can handle;
reduces bias within a purposeful category (not for generalizations
or representativeness).
• Sampling politically important cases: Attract attention
to the evaluation (or avoid attracting undesired attention by
purposefully eliminating politically sensitive cases from the
sample).
- Combination or mixed purposeful sampling: Triangulation; flexibility;
meet multiple interests and needs.
|
- Determine
sample size: No formula exists to determine sample size.
There are trade-offs between depth and breadth, between doing fewer
cases in greater depth, or more cases in less depth, given limitations
of time and money. Whatever the strategy, a rationale will be needed.
Options include:
- Sample to the point of redundancy (not learning anything new).
- Emergent sampling design; start out and add to the sample as
fieldwork progresses.
- Determine the sample size and scope in advance.
|
- Data collection methods: Determine the mix of
observational fieldwork, interviewing, and document analysis to be
done in the evaluation. This is not done rigidly, but rather as a
way to estimate allocation of time and effort and to anticipate what
data will be available to answer key questions.
|
- Resources available: Determine the resources available
to support the inquiry, including:
- financial resources
- time
- people resources
- access, connections
|