Global
Review: Publications
What’s Happening in AJE (2003-2004)
Lori A. Wingate
|
AJE Web site: http://www.sciencedirect.com/science/journal/10982140 |
The American Journal of Evaluation (AJE) is the flagship publication of the
American Evaluation Association, the world’s largest organization for
professional evaluators. As such, AJE
plays an important role in defining the relatively young discipline of
evaluation and influencing the work and thought of many practicing evaluators,
many of whom have never had any formal training in evaluation.
In the Evaluation Thesaurus Scriven
(1991) provides an analogy for understanding how various disciplines, and the
levels of activities within those disciplines, relate to one another. In this
analogy, he suggests we think of disciplines as estates in the “country of the
mind.” He explains, “The houses on an estate have a ground floor representing
applied work; a floor above that which is devoted to developing instruments,
methods, and techniques, and a top floor where the theoretical work is done. Up in the attic, out of sight for most of the time, is the den of metatheory” (pp. 13-14).
I used this framework to analyze the contents of AJE articles (from Spring 2003 through the present issue, which is Autumn 2004). I categorized the articles (65 in all) according to whether they focused on practice, methods, theory, or metatheory, and one additional category—history. The breakdown is shown Figure 1. Below I describe these categories and summarize the articles associated with those categories, highlighting what I believe to be the most important articles.
|
|
|
Figure 1. Focus of 2003-2004 AJE articles |
Practice
“Practice” articles
deal with ways of working with stakeholders and clients, ethical challenges,
evaluation contexts, managerial aspects of evaluation, and evaluation use. Almost half (46.2 percent) of the articles published in AJE since 2003 focus primarily on such
practical aspects of the evaluation profession.
Eight of the 30
articles in the Practice category are part of AJE’s “Ethical Challenges”
series, in which the section editor, Michael Morris, presents a brief scenario
in which an evaluator faces an ethical challenge. In response, two
commentators, in two separate articles, analyze the nature of the ethical
problem and describe what they believe to be the appropriate response by the
evaluator in the scenario, especially in light of the American Evaluation
Association’s Guiding Principles for Evaluators and The Program Evaluation Standards by the Joint Committee on Standards for Educational Evaluation
(1994).
Seven articles in
the Practice category focus on evaluation use, with five of these appearing as
a series in a single issue. These use-oriented articles explore the many facets
of evaluation utilization. They provide exemplars of useful evaluation,
identify factors that promote and impede evaluation use, and weigh the sometimes
conflicting values of evaluation utility and scientific rigor. Evaluation is an
inherently applied discipline—intended to be used—but it is something that many
people shy away from, or downright fear. Given these conflicting conditions, it
is no surprise that many evaluators are interested in improving evaluation
utilization. I categorized two other use-oriented articles (by Henry [2003] and
Henry and Mark [2003]) in the “Metatheory” category,
because they go beyond the practical issues related to use and venture into a
theory about evaluation influence, which I discuss in greater detail in that
section.
The remaining
articles that I included in the Practice category address a variety of issues
that have emerged out of the experience of real people engaged in the practice
of evaluation—for example, how certain evaluation contexts present particular
challenges or opportunities, the managerial aspects of evaluation (e.g.,
contracts, resource constraints), and how to communicate effectively with stakeholders.
One article that stands as particularly useful is by Bamberger, Rugh, Church, and Fort (2004). They offer several practical
solutions for common problems that evaluators face when working under severe
constraints. Their recommendations are most relevant for impact evaluations in
which the use of control groups, baseline data, and random sampling would be
ideal but not feasible due to timing, resources, and/or availability of data.
Articles
focusing on practice offer readers insights into the real world of evaluation,
where textbook methods and theory meet politics, red tape, ethical dilemmas,
and stakeholders and clients who may or may not be interested in participating
in evaluation or using its results. These types of articles provide
readers with opportunities to learn from others’ mistakes and successes in the
uncertain world of evaluation practice. They offer students and established
evaluators insights into how evaluation happens in the real world—lessons often
not provided in textbook expositions on theory and methods.
Methods
“Methods” articles
focus on a particular approach to data gathering and/or analysis. Seventeen of
the 65 AJE articles (26.2 percent) deal primarily with methods. Such articles
typically describe an innovative method or a modification of an existing
method. These articles were equally divided between qualitative (8) and
quantitative methods (8), with one article featuring a blend of both.
The qualitative
methods covered by the articles include concept mapping, site visits,
qualitative phone interviewing, the “most significant change” technique,
methods for reconstructing and analyzing program theories, the Delphi
technique, methods of values inquiry, and methods for formatively evaluating
educational technology.
Four of the seven
articles on quantitative methods discussed methods used to overcome problems
associated with randomized controlled trials, including the use of longitudinal
data on program outcomes to estimate program effects, two different methods for
analyzing impacts on beneficiary subgroups, and an approach for blending
experimental and quasi-experimental methods. Other articles focused on the
development of intervention-specific measures, techniques for assessing the
quality of program implementation, and the use of post-plus retrospective
pretests for measuring change.
The one article
that focused on a method that incorporates the use of both qualitative and
quantitative data described the development and use of a rubric for evaluating
collaboration.
Methods articles
highlight innovative and cutting edge approaches to evaluation data gathering
and analysis. Journal articles and professional conferences are probably the
most important ways practicing evaluators learn about new and useful methods.
The methods are typically described in the context of a particular evaluation,
which may help readers to discern the method’s applicability to the areas in
which they work.
Theory
“Theory” articles
center on the use of a particular evaluation approach or model. Evaluation
theory was the focus of just two articles (3.1 percent) published in AJE since 2003. One provides an in-depth
look at an evaluation that blended two approaches to evaluation—theory- driven
and utilization-focused. The other theory-focused article offers an adaptation
of Michael Fetterman’s empowerment evaluation model
(by Carolyn Sullins, a Senior Research Associate at
The Evaluation Center). Both deal with practical applications of theory, but
the emphasis in on the applied theory, rather than the specific methods or
findings. (There are other AJE
articles that feature the use of a particular theory, but the thrust of these
articles is on practice, not theory.)
No articles in the
timeframe examined (2003-2004) focused exclusively on an evaluation theory/model/approach
in its pure form. As Christie and Alkin
(2003) remark in their article about using a theory-driven approach in a
user-oriented evaluation, “theories are rarely, if ever, flawlessly translated
into practice” (p. 381). Given this, “in order to
develop a deeper understanding of how evaluation theories are best applied in
practice, it is important to describe cases where evaluation theories have been
used in practice” (p. 381). That, indeed, is the nature of these two
Theory articles.
It was somewhat
surprising to me that only 2 articles out of 65 focused purely on evaluation
theory. It is an important area of inquiry would seem to warrant more space in AJE.
Metaetheory
Scriven
(1991) defines metatheory as a “‘theory’ about the
nature of a field of inquiry, engineering, or craft. It deals with matters such
as the definition of the field’s boundaries, its differences from neighboring
fields or disciplines, the reason why certain methods work well for it and
others are inappropriate…..it is the self-concept of the discipline” (p. 232).
Seven (10.8 percent) articles in AJE
directly discuss or contribute to the evaluation discipline’s self-concept, or metatheory.
Two of the Metaetheory articles focus on use. Both articles address
the issue of evaluation use not simply as a practical
matter, but as a sort of lens through we can view the role of evaluation
discipline. Henry and Mark (2003) address the shortcomings in the existing
literature on evaluation use, particularly the “inattention to the intrapersonal,
interpersonal, and society change processes through which evaluation findings
and process may translate into steps toward social betterment” (p. 294). They
urge evaluators to look beyond immediate use of findings as the primary
utilitarian purpose of evaluation, and instead focus on social betterment as
the ultimate desired outcome. They outline a general theory of evaluation
influence. Similarly, Henry (2003) offers several examples of evaluations that
have been influential and offers a “clearer picture of what evaluation should
look like in the future” (p. 515).
Two articles that I placed in the Metatheory category have to do with evaluation education. These articles do not directly contribute to the metatheory of evaluation in terms of content, but the way in which and what students and others learn about evaluation—its practice, methods, and theory, and history—is probably the primary vehicle by which evaluation metatheory develops. One article provides an overview of a one-year evaluation course that employs a mentoring approach. The other, by Christie and Rose (2003), provides an account of an informal discussion group. This group, facilitated by Marvin Alkin at UCLA, includes both students and faculty members who meet every other week to discuss an article in a recent issue of the American Journal of Evaluation. In addition to providing a venue in which members can share and test ideas, relate theory to practice, refine thinking, and hypothesize (among other things), the group also promotes socialization into the field. Such groups, write Christie and Rose, “are an alternative mechanism for encouraging the kinds of dynamic dialogue that facilitates the advancement of both theoretical and practical notions of a field, such as evaluation, that is so dependent up on the interchange of ideas” (p. 238).
In his article on
the Joint Committee evaluation standards, Stufflebeam
(2004) addresses the applicability of the Program, Personnel, and Student
Evaluation Standards to other cultural contexts. These are essentially
standards for evaluation practice, but they have played an important role in
shaping the field’s self-concept. At issue is whether the Standards can or
should be transferred to other cultural contexts, and Stufflebeam
argues they should not. The widespread interest in doing so is a testament to
the Standard’s relevance to the discipline’s self-concept.
Stake (2004)
addresses the role of advocacy in evaluation. He outlines six types of
advocacies found to some extent in most evaluations. Roughly, they are advocacy
for (1) a program’s success, (2) the evaluation discipline, (3) rationality,
(4) evaluation use, (5) the alleviation of underprivilege,
and (6) democracy. He argues that these advocacies shape evaluators’
interpretations of findings, which are “are enriched by personal experience”
(p. 107). He concludes the article by stating, “Comprehensive, idiosyncratic
interpretations are small steps toward saving the world” (p. 107).
The final article dealing with metatheory
views evaluation itself as an important object of inquiry and provides a
framework for researching the processes, contexts, obstacles, and knowledge
claims in public sector evaluations. In this article, Segerholm
(2003) reviews existing research on evaluation and concludes that it is “fairly
scarce” and tends to focus on particular aspects of the evaluation cycle (i.e.,
initiation, implementation, results, and utilization) (p. 356). Likewise, she
notes, metaevaluations (evaluations of evaluations)
usually focus on a single evaluation. Segerholm
argues that we need more research on evaluation to “gain knowledge and a more
thorough understanding of the phenomenon and practice of evaluation in
general” (p. 357).
History
In addition to Scriven’s disciplinary categories of practice, methods,
theory, and metatheory, I added History as a fifth
category. I found this to be necessary because articles that focus on the
development of the evaluation field cut across all the other categories,
dealing with evaluation practice, methods, and theory, as well as influential
personalities in the field; groundbreaking evaluations; important books; and
key agencies, organizations, and educational institutions. These articles also contribute to the
development and refinement of evaluation’s metatheory,
since they help interpret and shape the field’s self-concept. Nine (13.8
percent) of the AJE articles since
2003 delve into the history of evaluation.
Most of the
articles included in this category (6 out of 9) are oral history accounts of
evaluation leaders collected for The Oral History Project—an effort by Robin
Miller, Jean King, Melvin Mark, and Stacey Stockdill
to document the “genealogy” of program evaluation. These oral history articles
have featured interviews with Lois-ellin Datta and William Shadish, as
well as brief articles by Laura Leviton, Roger Straw,
Charles Reichardt, and Melvin Mark, who reflect on
their experience in the Methodology and Program Evaluation program in the
Psychology Department at Northwestern. Additional evaluation leaders will be
featured in future issues, leading to the compilation of a rich and detailed
history of the development of the evaluation field.
Margaret Mead’s
evaluation of the 1947 Salzburg Seminar on American Civilization is the focus
of the three other History articles.
References
Bamberger, M., Rugh, J., Church, M., & Fort, L. (2004). Shoestring evaluation: Designing impact
evaluations under budget, time and data constraints. American Journal of Evaluation, 25(1),
5-37.
Christie,
C. A., & Alkin, M. C. (2003). The user-oriented evaluator’s role in formulating a program
theory: Using a theory-driven approach. American Journal of Evaluation, 24(3), 373-385.
Christie,
C. A., & Rose, M. (2003). Learning about evaluation through
dialogue: Lessons from an informal discussion group. American Journal of Evaluation, 24(2), 235-243.
Henry,
G. T. (2003) Influential evaluations. American Journal of Evaluation, 24(4), 515-524.
Henry,
G. T., & Mark. M. M. (2003) Beyond use:
Understanding evaluation’s influence on attitudes and actions. American Journal of Evaluation, 24(3),
293-314.
Joint
Committee on Standards for Educational Evaluation. (1994). The program evaluation standards (2nd ed).
Scriven, M. (1991). Evaluation thesaurus.
Segerholm,
C. (2003). Researching
evaluation in national (state) politics and administration: A critical
approach. American Journal of Evaluation, 24(3),
353-372.
Stake,
B. (2004). How far dare an evaluator go toward saving the world? American Journal of
Evaluation, 25(1), 203-107.
Stufflebeam, D. L. (2004). A Note on the Purposes, Development, and Applicability of the Joint
Committee Evaluation Standards. American
Journal of Evaluation, 25(1), 99-102.
Evaluation: The International
Journal of Theory, Research and Practice (2003-2004)
Daniela C. Schröter
Evaluation
is a quarterly, European-based journal that in addition to interdisciplinary
and multidisciplinary peer-reviewed articles occasionally provides Special Issues, Visits to the World of
Practice, News from the Community, Book
Reviews, Speeches and Addresses, and Debates,
Notes, and Queries. The publisher highlights the journal’s interdisciplinarity (across fields of research),
interconnectivity (across fields of practice), and internationality (across
countries and continents). The journal appears quarterly and can be purchased
for £51 per individual (approximately $92) and £372 for institutions
(approximately $671) per annual subscription.
In the following,
the most recent four issues of Evaluation–Volumes
9(4), 2003 to 10(3), 2004–are analyzed in terms of theory-specific,
research-specific, and practice-related articles published and themes from the News from the Community and Speeches and Addresses.
Articles reflecting
theory-orientation illuminate general or abstract principles of the nature or
practice of evaluation. There will not be a distinction here between theory and
meta-theory.
Contributions
categorized as research are those that discus mainly methodological issues or
evaluation tools and how they are employed in specific contexts. It is believed
that this methodological approach best summarizes the core of research, that
is, “disciplined investigation” (Scriven 1991, p.
312).
Practice-reports
are those that describe a specific evaluation project. While some of those
articles may attempt to build evaluation theory or contribute to methodology,
specific evaluation projects are the core of discussion.
Overall, there were
22 articles from contributors of 11 countries published within the last year.
Of these contributions, 6 are theory-oriented, 8 are
research/methodology-oriented, and 8 are accounts of practice (see Figure 1).
Figure 1: Distribution
of articles
Theory
All six theoretical
articles were published in a Special Issue of Evaluation (Vol. 10(1)), which contains a selection of papers
presented at the European Evaluation Society’s 5th biennial
conference entitled Learning, Theory, Evidence: Three Movements in Contemporary Evaluation.
The first of these articles is Stern’s (2004) personal reflection on “What
Shapes European Evaluation?” Stern generalizes features of the European culture
and infers tasks for policy and practice of the European community of
evaluators. The second contribution, “Theory-Based
Evaluation and Learning: Possibilities and Challenges” (Van der Knaap, 2004) attempts to
overcome the split between positivist and constructivist paradigms through
theory-driven evaluation in public policy which also considers the complexity
of society thus providing learning opportunities to governments. Mark and Henry
(2004) present a framework of evaluation influence and the underlying
mechanisms to be considered. Stame (2004) revises
theory-driven evaluation and provides ideas for coping with complexities in
hierarchical governmental programs. Virtanen and Uusikylä (2004) try to re-conceptualize the traditional
logic of cause and effect under consideration of “ontological premises, conceptions
of causality, and relationships to rational theories of action of different
program evaluation paradigms (p.77)." The last article in this category is
Rieper’s (2004) discussion of Popper’s significance
to and influence in evaluation by illustrating his impact on two opposite
figures in evaluation, Campbell and Lindblom.
There are eight
articles within the last year across the four issues that contain research and
methodology-oriented contributions to Evaluation.
Seven articles provide frameworks, approaches, or tools for evaluation in
specific contexts such as public policy, social programs, training, and aid
(Ling, 2003; Thurston & Potvin, 2003; Brousselle, 2004; Crawford et al, 2004; Bulgarelli
& Gori, 2004; Johnson et al, 2004; Saunders &
Heflinger, 2004). Davies (2004) provides a
methodological approach to representing theories of organizational change to
evaluate impacts of international aid agencies. This article is especially
interesting because it provides other approaches to depicting program theory in
addition to the traditional logic model. Davies (2004) argues that some
theories may have to be represented in their complexity through networks rather
than directional simplifications. While his contribution focuses on international
aid agencies, it may be useful to other contexts; programs of all types are
initiated through organizations, none of which are simple, unidirectional
entities. Overall, the research/methodological contributions embrace the
interdisciplinary nature of the journal.
The
practice-oriented contributions to the last four issues of Evaluation contain specific examples of evaluation in policy,
health, environment, welfare, and crime prevention evaluation among others.
These are of specific use for those readers who have an interest or are active
in the specific areas, but also provide ideas for evaluation work in other
contexts. As such these contributions reflect the interconnectivity feature of
the journal.
Each issue within
the last year included News from the
Community containing special evaluation-specific developments in individual
countries (e.g., developments in France in autumn 2003, in Vol. 9,4),
announcements of conferences and awards to be distributed, calls for papers
(e.g., to the 6th biennial EES conference, Vol. 10,1), references to
specific research (e.g., the IOCE initiative to conduct a survey, Vol. 10, 2),
recommendations for and announcements of training, seminars, or workshops; and
special honors in evaluation. Overall,
News from the Community is not limited to the European community but
embraces the evaluation community internationally.
Speeches and
Addresses
This component of
the journal includes addresses and speeches that were presented at evaluation
events and are to be disseminated to a wider audience—the readers of Evaluation
(see Sage). Within the last year there
have been three speeches and addresses. Issue 9(4) includes two contributions: Pawson (2003) and Stek (2003). Pawson’s address is based on a presentation given at the 5th
biennial EES conference and discusses tasks for developing theory in
evaluation, which include: (1) developing a theory-driven approach to
systematic review of existing evaluations, (2) improving the transferability of
findings in program evaluation to other programs, and (3) elucidating program
complexity. Stek’s (2003) contribution “Evaluation at
the World Bank and Implications for Bilateral Donors” was presented at the “25th
anniversary of the IOB, the evaluation body of the Netherlands Ministry of
Foreign Affairs and the Ministry for Development Co-operation” (p.491).
The next
contribution followed in issue 10(3), a “keynote address to the 9th
UK Evaluation Society Annual Conference” (p. 366), in which Sanderson (2004)
presented “perspectives on rationality” while arguing for “a more ‘practical’
conception of rationality as a basis for an appropriate institutional framework
for ‘getting evidence into practice,’ serving the emphasis the UK government
puts on “evidence-based policy making (p.366)."
References
Stern,
Elliot (Ed.), (2003). Evaluation: The International Journal of Evaluation
Theory, Research and Practice, 9(4)-10(3).
Stern,
Elliot (2004). What shapes European evaluation: A personal reflection.
In: Evaluation: The International Journal of Evaluation Theory, Research and
Practice, 10(1): 7-15.
Appendix I Summary of classifications
Theory
Vol. 10(1)
Stern, E.: What
shapes European Evaluation? A Personal Reflection
Van der Knaap, P.: Theory-Based Evaluation and Learning:
Possibilities and Challenges
Mark, M.M. and Henry, G.T.: The Mechanisms and Outcomes of
Evaluation Influence
Stame, N.: Theory-Based
Evaluation and Types of Complexity
Virtanen, P. and Uusikylä, P.: Exploring the Missing Links between Cause and
Effect: A Conceptual Framework for Understanding Micro–Macro Conversions in Programme Evaluation
Rieper, O.: Exploring
Popper’s Relevance for the Evaluation Community
Research
and Methodology
Vol. 9(4)
Ling, T.: Ex Ante Evaluation and the Changing Public
Audit Function: The Scenario Planning Approach
Thurston, W.E. and Potvin,
L.: Evaluability
Assessment: A Tool for Incorporating Evaluation in Social Change Programmes
Vol. 10(1)
Davies, R.: Scale, Complexity and the Representation of
Theories of Change
Vol. 10(2)
Brousselle, A.: What
Counts is not Falling... but Landing: Strategic Analysis: An Adapted Model for
Implementation Evaluation
Crawford, P.; Perryman, J. and Petocz, P.: Synthetic Indices: A Method for Evaluating
Aid Project Effectiveness
Bulgarelli, A. and Gori, E.: Information Systems for the Evaluation of the
Effectiveness and Efficiency of Vocational Training Programmes
Vol. 10(3)
Johnsen, A.; Robertsen, K.; and Äsland, D.Y.: Contracting
Out Municipal Auditing: Conceptual Framework and Assessment
Saunders, R.C. and
Practice
Vol. 9(4)
Bustelo, M: Evaluation
of Gender Mainstreaming: Ideas from a Meta-Evaluation Study
Alexander, H.: Health-Service Evaluations:
Should We Expect the Results to Change Practice?
Mickwitz, P.: A
Framework for Evaluating Environmental Policy Instruments: Context and Key
Concepts
Vol. 10(2)
Skinner, D.: Primary and Secondary Barriers to the
Evaluation of Change: Evidence from Two Public Sector Organizations
Ashworth, K.; Cebulla,
A.; Greenberg, D. and Walker, R.: Meta-Evaluation: Discovering What Works Best
in Welfare Provision
Vol. 10(3)
Marra, M.: The
Contribution of Evaluation to Socialization and Externalization of Tacit
Knowledge: The Case of the World Bank
Halliday, J.; Asthana, S.M.N. and Richardson, S: Evaluating Partnership: The
Role of Formal Assessment Tools
The Japanese Journal of
Evaluation Studies
John S.
Risley
General Summary of Activities
The Japanese
Evaluation Society (JES) —available at www.idcj.or.jp/JES/—began
publishing The Japanese Journal of
Evaluation Studies (JJES) in 2001.
The journal is published twice each year. Complete copies of each issue of JJES can be found at the JES web site.
The complete text of most articles is in Japanese. Some articles (about 15% of
all articles since 2001) are printed in English. All other articles are
accompanied by an abstract in English. Each issue of JJES also contains an English-language index.
JJES is dominated by articles written by
authors from Japanese organizations. Nearly 90% of the 64 articles published
since the journal’s founding are by Japanese authors. The large majority of
these articles concern examples of evaluation practice, methodology, and theory
as they relate to
I reviewed all
issues of JJES and categorized each
article according to Lori Wingate’s (in this issue) adaptation of Michael Scriven’s analogy for understanding disciplines. Wingate
identified four categories of focus for journal articles—practice, methods,
theory, metatheory—that I used below and one
category—history—that I eliminated because no articles fit the description.
Practice issues
dominated the 64 articles from the journal’s first four years (50%). The
practice articles dealt with a range of evaluation issues concerning Japanese
government activities. Local and national government attempts to institute
systems for evaluating their policies were a consistent theme. The evaluation
of development assistance programs run through the Japanese International
Cooperation Agency (JICA) was also a popular topic.
One out of four
articles concerned theory. Typically, an author would highlight a model of
evaluation (utilization-focused, empowerment, etc.) and discuss how the model
was being used in
I categorized
fifteen articles (23.5%) as methods articles. Many of these articles addressed
methodological issues around developing indicators for performance measurement.
Other articles focused on methods that were helpful in evaluating gender
equality in programs and policies.
The metatheory
category included only one article. This article[1],
one of the few English-language articles, concerned the preparation of professional
evaluators. The author, James W. Altschud of
Journal of Evaluation and Program Planning
Journal
Description
Below is the explanation of the purpose and intent of the Journal of Evaluation and Program Planning
(JEPP) and a description of the types
of articles published in this journal as presented by the publisher.
Purpose and Intent of the Journal
JEPP is based on the principle that the
techniques and methods of evaluation and planning transcend the boundaries of
specific fields and that relevant contributions to these areas come from people
representing many different positions, intellectual traditions, and interests.
In order to further the development of evaluation and planning, JEPP publishes articles from the private
and public sectors in a wide range of areas: organizational development and
behavior, training, planning, human resource development, health and mental,
social services, mental retardation, corrections, substance abuse, and
education. The primary goals of the journal are to assist evaluators and
planners to improve the practice of their professions, to develop their skills
and to improve their knowledge base.
Types
of Articles Published
JEPP publishes articles, "special issues" (usually a section of an issue), and book reviews. Articles are of two types: 1) reports on specific evaluation or planning efforts, and 2) discussions of issues relevant to the conduct of evaluation and planning.
Reports on individual evaluations include presentations of the evaluation setting, design, analysis and results. Because of JEPP's focus and philosophy, however, they also devote a special section to "lessons learned". This section contains advice to other evaluators about how you would have acted differently if you could do it all over again. The advice may involve methodology, how the evaluation was implemented or conducted, evaluation utilization tactics, or any other wisdom that you think could benefit your colleagues. More general articles provide information relevant to the evaluator/planner's work. This includes theories in evaluation, literature reviews, critiques of instruments, or discussions of fiscal, legislative, legal or ethical affecting evaluation or planning.
Special issues are groups of articles which cover a particular topic in depth. They are organized by "special issue editors" who are willing to conceptualize the topic, find contributors, set up a quality control process, and deliver the material. Often several editors share responsibility for these tasks. Suggestions for special issues are encouraged.
Book reviews cover
any area of social science or public policy which may interest evaluators and
planners. As the special issues, suggestions for books and book reviewers are
encouraged.
This journal publishes four times a
year.
‘Somewhere Out There:’ A Survey of the Oft Lone Journey of Evaluators Working in
Business and Industry Settings
Trude J. Fawson, Vanessa Moss-Summers and Sharon Marie May
Evaluation and Program Planning
Volume
27, Issue 3 , August 2004, Pages 335-340
Summary: The purpose of this study was
to (1) locate evaluators working in the for-profit sector and; (2) to obtain
detailed information about what they do, how they do their jobs, and the
barriers and supports that they encounter in their work environments.
This article was
very helpful for a new professional in evaluation because it gave the “status
quo” of the life of a professional evaluator in a for-profit organization. As
the article states, there is not a lot of work done on evaluation in the
for-profit sector so this article provided useful insight. Particularly since
most organizations are for-profit.
This article was
very comprehensive. Information was gathered from members of the AEA, the ROI
Network of the American Society for Training and Development, the International
Society of Performance Improvement and a variety of industries such as
airlines, banks, technology companies, insurance, health care/pharmaceutical,
manufacturing, retail, and transportation.
First, the article
reported on the nature of the job of evaluation. The researchers found that
even the largest global corporations “were unlikely to have large, full time
staffs of evaluators.” Only one third of the respondents consider evaluation as
their primary focus. In addition, the evaluators working in the corporate
setting felt “isolated with few peers to rely on for mentoring and support.”
These evaluators also felt that they had a ways to go to receive the
recognition and respect that their jobs should bring.
Secondly, the
article collected information on the types of tasks the evaluators in for-profit
organizations perform. They include analyzing data, collecting data, collecting
information about participant reactions to new programs/products/interventions,
creating measurement instruments, and recommending solutions/interventions.
“The evaluator within the for-profit sector was most likely to design
instruments, collect data, analyze data, and recommend solutions.”
Next, the
challenges and frustrations that were reported for the professional evaluator
were similar to many professionals. With corporate downsizing, there is a
concern for lack of time and resources to complete a worthwhile evaluation.
There was also the concern that the demands for immediate results from an
evaluation were unrealistic. Often times, programs were cancelled before
important changes could occur. Management fear of negative reports and pressure
from top managers to make the results “look good” were also hurdles for
evaluators.
For a successful
evaluation, the article found that the client organization must have leadership
that is open and willing to participate in honest evaluation. The evaluators
must be knowledgeable, ethical, willing to work in a collaborative role, have a
high level of skill, be objective, and be able to communicate well with the
organization and other team members.
Finally, the most
important statement from the article was, “Although inroads are being made, our
profession has a tough sales job ahead of it.” In addition, the authors' state:
“Great
opportunities exist for evaluators in corporate
Not Drinking the Poison You Name:
Reflections on Teaching Ethics to Evaluators in For-Profit Settings
Michael Morris
Evaluation and Program Planning
Volume
27, Issue 3 , August 2004, Pages 365-369
The first sentence in the abstract of this
article states: “Recent corporate scandals represent a type of ethical
challenge frequently encountered by evaluators within and outside of for-profit
settings: the pressure to misrepresent findings.” The article refers to the
“Enron Era” and the scandalous affairs that seem to happen more frequently in
the for-profit world. Yet, it discusses how we must train future evaluators,
with the leadership of the AEA’s Guiding Principles
for Evaluators, to be professionals of integrity.
The article cited a
Morris and Cohn 1993 study that asked a national sample of evaluators the
ethical problem that they encountered most frequently. Their response was
“being pressured by a key stakeholder (usually the primary client) to alter
one’s presentation of the findings was, by far, the challenge most often reported.”
While we can certainly hope that this type of ethical problem is a rarity, we,
as evaluators must be aware of the type of pressure we may receive from the
stakeholders of our evaluation projects.
This article makes
suggestions for successful preparation of professional evaluators. Evaluators
must received more training in areas of finance and accounting if they intend
to practice their skill in the business world. Evaluators must be familiar with
multiple approaches for analyzing data and be able to communicate well their
findings. Evaluators must have respect for all stakeholders in an evaluator project,
not just the powerful. Evaluators must remember that “core societal values are
embedded in the professional principles and standards that guide them as
evaluators.” What evaluators do goes beyond the organization they are studying.
Evaluators should keep an “ethics journal” to review and share with colleagues.
This article concludes with suggesting that an ethics course should be included in a professional evaluators training. Ultimately, large or small, all evaluators will be faced with an ethics issue at some point. We should be able to assume that for the sake of the professional and the profession, the evaluator will make the correct choice.
New Directions for Evaluation (Vol. 102)
Mary Keating
Views
on Cultural Competence
A summary of the Summer
2004 edition of New Directions for Evaluation entitled “In Search of Cultural
Competence in Evaluation—Toward
Principles and Practices”
“Culture is present
in evaluation not only in the contexts in which programs are implemented but
also in the designs of these programs and the approach, stance, or methods
evaluators choose to use in their work. A common thread between culture and
evaluation is the concept of values. Culture shapes values, beliefs, and
worldviews. Evaluation is fundamentally an endeavor of determining values,
merit, and worth.” (p. 6)
The most recent
edition of New Directions for Evaluation
reviews the impact that cultural influences may have in evaluations. The
journal begins with an overview of the historical and current ideas surrounding
cultural competence and then looks at the topic from the perspective of
different cultures: African American, American Indian and Alaskan tribal communities,
and Latinos. The journal then examines issues with program evaluation in
multicultural settings and potential next steps for the future.
Many evaluators may
attempt to consider cultural influences on a project and think only of the
physical characteristics of a community such as food, music, celebrations, and
clothing. However, there is a much deeper layer “…that influences customs,
norms, practices, and social institutions, including psychological processes,
language, caretaking practices, media, educational systems, and organizations (Fiske, Kitayama, Markus, and
Nesbitt 1998, p. 6).” Many of these factors are often either topics
themselves of evaluation or relevant values that should be considered in doing
an evaluation’s KEC. Evaluators are asked to consider the stakeholders’ and
clients’ values when processing an evaluation—but how, if they have no heritage
in that culture?
“The APA’s multicultural guidelines (2003) posit that culture is
the embodiment of a world view through learned and transmitted beliefs, values,
and practices, including religious and spiritual traditions. It also
encompasses a way of living informed by the historical, economic, ecological
and political forces acting on a group.” (p. 6)
When there are
value differences they often lead to value conflicts that can impact an
evaluation from problem definition all the way to recommendations. The journal
gives several examples of Eastern versus Western cultures or even the impact of
Caucasian researchers interpreting the experiences of low income minorities. The
editors also debate the approach used for many policy evaluations involving
funding that are done on needs based rather than a strengths based approach. Rather
than trying to understand and determine the needs, it may be more beneficial to
focus on the strengths and how those positive attributes can be leveraged in a
program. The evaluators must possess an “active awareness, understanding, and
appreciation for the context at hand, and uses responsive and inclusive means
to conduct evaluation" (p.11).
In the second
chapter of the journal, “A Journey to Understand the Role of Culture in Program
Evaluation: Snapshots and Personal Reflectors of one African American
Evaluator” by Stafford Hood, the author states: “It is my continuing belief
that few evaluative approaches of the past (or for that matter the present)
have seriously considered race, culture, poverty, or contextual context as
anything more than ‘error variance’” (p. 22). Hood reviews various historical
views from leading experts such as Cronbach, Scriven, Stufflebeam, and Tyler. He
refers to evaluation standards that recommend evaluators understand the
audience and their needs as part of the evaluation process. Hood takes it one
step further in identifying stakeholders:
1. Include less powerful groups or individuals as stakeholders, such as racial, cultural, or language minority groups,
2. Determine how they would like to use the results and for whom the information will be particularly useful,
3. Include
clients and stakeholders in designing and conducting the evaluation as well as
interpreting the results (p. 30)
Evaluators should
also have to meet PES criteria in order to perform culturally diverse
evaluations. There are guidelines that state that the evaluator must be “knowledgeable
of the social and political forces affecting the less powerful stakeholders and
use this information in designing and conducting the evaluation.” (p. 30) However,
Hood questions whether these criteria are being enforced in the evaluator
interview process.
Lastly, the AfrEA has started to amend many
The third module of
the latest New Directions for Evaluation
journal is entitled “Culturally Competent in Indian Country” by Joan LaFrance. Similar to the cultural differences described
thus far, this module stresses the uniqueness of tribal sovereignty and the
need for cultural integration into the evaluation practice. Throughout the
article, La
The fourth article, “Developing &
Implementing Culturally Competent Evaluation: A Discussion of Multicultural
Validity in 2 HIV Prevention Programs for Latinos,” Ross F Conner examines the
importance in distinguishing internal, external and construct validity.
“Cultural issues and differences can be
important factors in understanding which variables did or did not cause
differences in programs (internal validity), which effects generalize over
other settings and times (external validity), and what effects mean for higher
order constructs and implications (construct validity). Multicultural validity
therefore extends the issues evaluators need to be attentive to if they are to
draw valid conclusions, set out well-grounded implications, and make accurate
recommendations.” (p. 52)
The writer goes on
to explain key factors in the two programs that were run and once again proves
that participant inclusion throughout the evaluation to enable cultural context
is critical for success. One additional point discussed in this module is the
inclusion of both the literal and figurative language of the participants in
the evaluation process. This inclusion of speech allows the cultural influences
to permeate and not be hidden during the examination.
The remaining
articles continue to support the need for cultural inclusion and emersion in
evaluation programs. These articles reinforce the three future needs that are
identified by the editors: policies regarding cultural competence and
developing practice guidelines, a pool of qualified multicultural evaluators,
and more reports and best practice sharing of multicultural evaluation programs
(p. 15). Evaluators should be reminded that cultural context has an impact on
both international programs and domestic programs. Domestic programs that are
in a community unfamiliar to the evaluator may have just as many cultural
competency complexities as foreign lands. Evaluators must realize the need to
understand cultural values in all scenarios. The definition of cultural
competence as articulated below, embodies the current
issues surrounding evaluation and the severe impact that cultural context has
played in the past and could in the future if not addressed:
“A systematic, responsive inquiry that is actively cognizant, understanding, and appreciative of the cultural context in which the evaluation takes place; that frames and articulates the epistemology of the evaluative endeavor; that employs culturally and contextually appropriate methodology; and that uses stakeholder generated, interpretive means to arrive at the results and further use of the findings.” (p. 13)
References
Thompson-Robinson,
M., Hopson, R., & SenGupta, S. (Eds.). (2004). In
search of cultural competence in evaluation: Toward principles and practices. New Directions for
Evaluation, 102.
Education and Personnel Evaluation: A Review of
Leading Journals for the Period 2003-2004
Nadini Persaud
This literature review highlights interesting research
during the period 2003-2004, based on the Journal of Personnel Evaluation in
Education (last issue produced June 2003), Journal of Human Resources, Journal
of Teacher Education, NASSP Bulletin and
Education Administration Quarterly.
Psychological and
Contextual Influences on the Teacher Selection Interview: A Model for Future
Research by D. A. Delli and E. M. Vera
The article “Psychological and Contextual Influences on the
Teacher Selection Interview: A Model for Future Research” suggested that psychological influences can bias the interview
process. According to the authors, research suggests that variables such as attractiveness, likeability, expertness, and
similarity can influence interviewers’ perceptions of prospective employees
and applicants’ perceptions of the interviewer and the job itself. The article
notes that interviews, not paper qualifications, have been rated as the most
important criteria in making hiring decisions; therefore education
administrators need to be aware of these issues to ensure that the selection
process is not biased by psychological influences. The authors suggest that in
certain professions such as teaching and nursing where the ratio of candidates
to jobs is very low, selection processes may deviate from the norm and be
subject to contextual influences (interview structure, interview format and the
influence of local and federal policies on interview outcomes) as well as
psychological influence. The authors conclude by noting that awareness of both
the psychological and contextual issues that can influence the hiring of
teachers can help ensure that the best qualified teachers are recruited.
Summative
Evaluation of Student Teachers: An Enduring Problem by J. Raths
and F. Lyman
The article “Summative Evaluation of Student Teachers: An
Enduring Problem” noted that
education faculty are not taking sufficient care to prevent weak and
incompetent student teachers from attaining state licenses. Factors
contributing to the problem include:
§
Conflation of summative and formative
evaluation: As
defined by Scriven (1967), summative evaluation
involves decisions such as final grades, promotion to the next class, acceptance to a university, granting a license, giving merit
pay increases and other similar performance incentives. Formative evaluation is
akin to coaching and is designed to improve student performance or improve the
products on which students are working. According to the authors, in the mode
of formative evaluation, teachers rarely issue overall judgments about the
quality of a performance or a product, but instead point out particular
strengths and/or weaknesses, suggest how weakness might be addressed, and
encourage reflection. The authors also note that there is an inevitable
conflict of interest when coaches make summative evaluations of the
performances of their students, because initially they start out acting as a coach
but later become the evaluator.
§
Minor roles within the larger teacher
education profession:
Teachers may not be full time, may not be faculty and may not have received
adequate coaching. They have their own duties and see the supervision process
of student teachers as secondary.
§
Uncertainty: There is no clear
distinction between what constitutes competent and incompetent student
teachers. This makes it difficult to make a high-stakes judgment about an
individual student teacher armed only with vague decision criteria, and
administrators want to avoid the threat of lawsuits and grievance petitions.
§
Rhetoric of excellence: Language on evaluation
forms give emphasis to “excellence” and “high quality” in teaching, but
provides minimal guidance to enable supervisors to determine what constitutes
incompetence and what is really excellence and high quality.
Responding
to incompetence clarification, the authors make two distinct contributions: (a)
they place incompetence in a continuum of teaching behaviors from criminality
and malpractice through best practice and (b) the teaching acts in specific
settings are pronounced as indicators of teacher incompetence. The article
concludes with suggested caveats that need to be taken into account as the
definitions of incompetence are shared or adopted.
International
Comparisons of the Relationships among Educational Effectiveness, Evaluation
and Improvement Variables: An Overview by C. Teddlie,
This study focused on teacher effectiveness research,
school effectiveness research, teacher evaluation, staff development, teacher
improvement and school improvement in five countries:
The Effects of
Chronological Age and Information Media on Teacher Screening Decisions for
Elementary School Principals by
The article “The Effects of Chronological Age and
Information Media on Teacher Screening Decisions for Elementary School
Principals” was an experimental
study that examined traditional versus modern methods of screening decisions
made by elementary school principals. The study failed to detect any
statistically significant multivariate effects for chronological age of teacher
candidates; however, it did show that principals may well overlook many
qualified teacher candidates who choose to use electronic mediums to
communicate teacher credentials to organizational representatives, at the
screening stage of the selection process.
Continuing
Assessment of Teacher Reactions to a Standards-Based Teacher Evaluation System
by H. G. Henemann III and A. T. Milanowski
“Continuing Assessment of Teacher Reactions to a
Standards-Based Teacher Evaluation System” was a study that provided a
description and evaluation of the first two years of district-wide
implementation of a standards-based teacher evaluation system in the
The Impact of
Teacher Training on Student Achievement: Quasi-Experimental Evidence from School
Reform Efforts in
This study used a regression discontinuity strategy to
estimate the effect of teacher training on math and reading performance of
elementary students in 20% of
The Efficacy of
Portfolios for Teacher Evaluation and Professional Development: Do They Make a
Difference? by P.D. Tucker, J. H. Stronge, C. R. Gareis and C. S. Beers
This study, conducted in a small school district,
introduced the use of portfolios as part of a new evaluation system to
determine the efficacy of portfolios in evaluating teacher performance both for
accountability and professional development purposes. The conclusion was that
the portfolio system was found to be particularly useful in documenting teacher
performance in the areas of assessment and professionalism. However, survey and
focus group data indicated that while teachers and administrators viewed
portfolios as fair and accurate measures of performance, both had concerns
about their feasibility given the time demands of portfolio development.
Evaluating the Evaluators: Teachers'
Perceptions of the Principal's Role in Professional Evaluation by S. Zimmerman and M. Deckert-Pelton
Eighty-six educators from five northwest
References
Delli, D.A. & Vera, E. M. (2003). Psychological and contextual influences on the teacher
selection interview: A model for future research. Journal of Personnel Evaluation in Education 17(2): 137-155.
Jacob, B. & Lefgren,
L. (2004). The impact of teacher
training on student achievement: Quasi-experimental evidence from school reform
efforts in
Henemann III, H. G. & Milanowski,
A. T. (2003). Continuing
assessment of teacher reactions to a standards-based teacher evaluation system.
Journal of Personnel Evaluation in
Education 17(2): 173-195.
Raths, J. & Lyman, F (2003). Summative evaluation of student teachers: An enduring
problem. Journal of Teacher Education
54(3): 206-216.
Teddlie, C,
Tucker, P.D, Stronge
J.H., Gareis, C.R. and Beers, C.S. (2003). The efficacy of portfolios for teacher evaluation and
professional development: Do they make a difference? Educational Administration Quarterly 39(5): 572-602.
Young,
Zimmerman,
S., Deckert-Pelton, M. (2003) Evaluating
the evaluators: teachers' perceptions of the principal's role in professional
evaluation. NASSP Bulletin 87(636): 28- 36.