Global Review: Regions
Evaluation Activities in Africa
Paul A.
Lamphear
The major purpose
of the association is to build individual and institutional capacity in policy,
program and project evaluation in
The first
conference of the African Evaluation Association, held in
It appears that
these lectures, the training seminars, and 80 papers that were submitted on
evaluation, helped to ‘jumpstart’ the development of the national associations.
Although there does not appear to be any evaluation journals specific to or
published in Africa, AfrEA has worked on several
projects to increase evaluation capacity and foster a consistent professional
approach for evaluators. In 2002, the association completed the "African
Evaluation Guidelines", a cultural adaptation of the
At the 2nd AfrEA conference in 2002, a variety of international
evaluators were invited, including keynote speakers Prof. Anna Madison of
Cornell University, USA, Ada Ocampo,
Leader of the Latin American Evaluation Network, and Penny Hawkins, President
of the Australasian Evaluation Association. After 5 days of trainings and paper
submissions, the association recommended that the African Evaluation Guidelines
(AEG) be adopted by all the National Networks, by Government and Public bodies,
and by UN Agencies and other Multinational Organizations performing evaluation
in
The Niger Network of Monitoring and
Evaluation (ReNSE) has over 140 documents (in
English or French) and in 2004 has published their first newsletter (in
French), available from their website, to promote expertise in Nigerian
evaluators.
AfrEA
supports dissemination of monitoring and evaluation resource materials focused
on
AfrEA
has significant connections with international organizations and has activities
currently sponsored by:
African Development Bank (AfDB)
Agence
Intergouvernementale de la Francophonie
Catholic Relief Services
Canadian Institutes for Health
Canadian International Development Agency
(CIDA)
CARE International
Danish Agency for Development Assistance
(DANIDA)
Family Health International (FHI)
International Development Research Centre
(IDRC)
World Conservation
Norwegian Ministry of Foreign Affairs
UNAIDS
UNCHS
UNDP
UNICEF
UNIFEM
World Bank
The third national Conference
of the African Evaluation Association will be held in
Evaluation Activities in Australasia
John
S. Risley
General
Summary of Activities
The Australasian
Evaluation Society (AES) produces, and posts on their website (www.aes.asn.au), an e-newsletter approximately
twice per year. The AES also holds an annual conference, usually in September
or October. The 2004 conference is in October near
From reading the
editorials and other non-refereed articles in the Evaluation Journal of Australasia (EJA) it appears that the evaluation profession in
Evaluation Journal of
A recent editorial
in Evaluation Journal of Australasia (EJA) noted the history of AES
publications. The society launched the EJA
in 1989. Then from 1993 through 2000 AES published both EJA and Evaluation News &
Comment. In 2001 these publications merged to form the new series of EJA. The journal is published by the AES
bi-annually (though recently there have been delays in publishing new
editions). AES posts the two copies preceding their most recent issue on their
web site. The journal includes refereed and non-refereed articles, editorials, interviews with evaluators from both within and without the
region, book reviews, research reports, and information about the annual AES
conference.
Issues addressed in
EJA included much information
concerning cultural appropriateness, indigenous peoples, and diversity in
evaluation. This may be a reflection of the recent AES conference themes. There
is some material drawing distinctions about evaluation aspects specific to
Subjects of
refereed articles in recent issues of EJA
include: evaluation of options for changing port ownership in
Some of the
refereed articles had very little to say about evaluation. For example, one of
these articles (Burton & Rajan, 2002) concerned a
case study evaluation of 15 people seriously injured in workplace accidents.
The authors described the project’s goal as exploring the social and economic
consequences to society from these workplace injuries. The article discussed
the methodology of the study, the experiences of the researchers, and the
lessons learned from their research experiences. The methodology was basically
a semi-structured interview of injured workers, their family members,
employers, etc. The lessons learned by the researchers were: 1) interviewing
can be exhausting, 2) diversity of the project team was essential, and 3) it
was difficult to remain objective after seeing the suffering of the injured
workers.
One interesting
article (Sigsgaard, 2002) addressed an unusual
methodology (in evaluation research), the Most Significant Change (MSC)
methodology. The author, Peter Sigsgaard, works at a
Danish NGO called “MS” on measurement and evaluation issues. He gave examples
of his experience using MSC in evaluating partnership-based economic
development programs in
Sigsgaard
(2002) contrasts this approach with one previously used by MS in evaluating
these programs, in which they would conceive of indicators to measure and then
cast about looking for these indicators. This led to lots of time spent looking
for, and not finding, specific data.
It makes intuitive
sense to ask program consumers what changes are occurring due to the program.
It does highlight the need to be careful how one measures program changes.
References
Sigsgaard,
P. (2002). Monitoring without indicators: an ongoing testing
of the MSC approach. Evaluation Journal of
The State of Evaluation Canada
Chris
L. S. Coryn[4]
Background and General Context of Organized
Evaluation in
The Canadian
Evaluation Society (CES)—
|
ü
Ontario |
|
|
ü
Manitoba |
|
|
ü
Alberta |
|
|
ü
Quebec |
|
Since 1991 the CES membership has grown to over 1,750 individual Canadian and student members, as well as over 100 international members (CES, 2004).
The CES offers a
wide range of resources and services for practicing evaluators and students of
the discipline including: a comprehensive Web site (available in English and
French); an evaluation report bank (academic, government, and private sector
reports); a fully-searchable database—the Grey
Literature Bank (unpublished documents of interest to evaluators); a professional
development series of workshops; an annual
conference (including the upcoming 2005 joint conference with the American
Evaluation Association); and the Canadian
Journal of Program Evaluation. The CES efforts are strongly supported by
the Government of Canada, which has its own specialized evaluation unit; Evaluation and Data
Development (EDD). EDD is one of the largest evaluation shops in the
Federal Government of Canada, and focuses primarily on governmental initiatives
including analysis of government policy and evaluation of government programs,
foe example, Human Resources Development Canada (HRDC)
programs. Other government contingencies which influence the Canadian
evaluation field include the National
Science and Engineering Research Council, the Social
Science Research Council, Transport Canada,
Industry Canada, Health Canada, the Treasury Board Secretariat, and the Canadian
International Development Agency; each of which are also sponsors of the CES.
Informed decision making is further facilitated by Statistics Canada a
provider—federally legislated—of statistical data for the whole of Canada and
each of its provinces that is intended to inform Canadian citizens and other
key stakeholders regarding Canada's population, resources, economy, culture,
and society.
In the summer of
2001 the CES announced their new vision, mission, and goals for the future
(Canadian Evaluation Society Newsletter, Summer 2001):
Vision:
The Canadian Evaluation Society will be the leader for evaluation in
Goals:
1.
Leadership—To provide leadership to individuals and organizations in
support of evaluation theory and practice in
2.
Knowledge—To improve the state of evaluation theory and practice.
3.
Advocacy—To promote the importance of an evaluation culture.
4.
Professional
Development—To promote and facilitate the
enhancement of evaluation capacity for members and non-members.
The CES also supports various student initiatives including
the CES
Student Case Competition and student
paper contest (for undergraduate and graduate students in the field of
evaluation). The CES Student Case Competition (initiated in 1996), is an annual
event in which teams of three to five students from
Canadian colleges and academic institutions compete in the analysis of an
evaluation case file. In a preliminary competition, all teams receive on the
same day the key to an evaluation case file that has been hidden on the Web.
They have five hours to prepare an analysis and then submit it by e-mail for
judging by an expert panel. The three best teams are invited to participate in
a final round, held at CES's annual conference, in
which they must analyze a new case and present findings and recommendations
before a live audience. The team that makes the best presentation takes
possession of the Case Competition Trophy for a year, receives prizes, and is
given visibility in various publications.
Evaluation Education Programs in
As of 2000 (CES), over 25 Canadian
institutions/colleges/universities offered more than 100 evaluation-related
courses across a wide array of academic disciplines (e.g., psychology,
political science, public administration, economics)—a complete institution,
department, and course list is available at http://www.evaluationcanada.ca/txt/outline200106.pdf.
Professional Development of Canadian
Evaluators
The CES plans to
focus on two key areas in the upcoming years: (1) professional development of
its members, and (2) advocacy on behalf of the evaluation function. The
articulation of a Core Body of
Knowledge (CBK) will guide the Society's professional development and
advocacy activities (Canadian Evaluation Society, 2004). The CBK comprises
theories, skills, and best practices that people must possess to plan, carry,
out, and report on valid and reliable evaluations of programs or policies in
governments, not-for-profit organizations, and businesses.
Essential Skills. Much of the emphasis on professional development is funneled through the CES Essential
Skills Series. Regional chapters offer this series as well as any other
form of training they consider adequate for their members. These essential
skills include:
1.
Understanding
Program Evaluation
§
Key terms and
concepts
§
Benefits of program
evaluation
§
Basic steps in the
evaluation process
§
Major approaches to
program evaluation
§
Formatting
evaluation questions
§
Designing an
evaluation
§
Evaluating with
limited resources
§
Analyzing and
reporting evaluation results
§
Reducing resistance
to evaluation
§
Involving staff and
clients in the evaluation process
§
Increasing
evaluation utilization
§
Making evaluations
ethical and fair
2.
Building an
Evaluation Framework
§
Identifying who the
client is and what the client needs
§
Basic concepts of
needs assessment
§
Major approaches to
assessing client needs
§
Evaluation methods
for "getting close to the client"
§
Building an
evaluation framework through logic models
§
Involving managers
and staff in building an evaluation framework
§
Relating program
design to client needs
§
Defining program
components
§
Formulating
indicators for program success
§
Using the evaluation
framework for linking program performance to client needs
3.
Improving Program
Performance
§
Using evaluation as
a management tool for improving program performance and enhancing internal
accountability
§
Basic concepts of
monitoring and process evaluation
§
Monitoring program
performance with existing administrative data and information systems
§
Developing ongoing
data collection instruments and procedures
§
Linking process
evaluation to program decision-making
§
Assessing client
satisfaction
§
Understanding continuous
quality improvement
§
Using program
evaluation for building a "learning organization"
4.
Evaluating for
Results
§
Defining program
results
§
Major approaches to
evaluating results
§
Developing results
measures
§
Designing outcome
evaluations
§
Validity and reliability
§
Appropriate use of
quantitative and qualitative techniques
§
Relating program
results to program costs
§
Understanding
program benefits
§
Measuring program
equity and responsiveness to community needs
§
Communicating
evaluation findings
§
Using evaluations to
improve program effectiveness and accountability
(Canadian Evaluation Society, 2004)
Certification
of Evaluators in
This latter issue — developing a form of
certification for members — would be a major step for the CES. Therefore,
it was the subject of an in-depth study
of the experience of several other organizations with certification (Long
& Kishchuk, 1997). A second study, carried out in
1999, reports on a pilot
survey of clients and employers (Stierhoff, 1999) on their views
regarding certification of evaluators.
Canadian
Journal of Program Evaluation
The Canadian Journal of Program Evaluation (CJPE) was launched in 1986 and is published
twice a year (available at www.cjpe.ca).
CJPE is sponsored by the CES and the
§
Articles
on all aspects of the theory and practice of evaluation, including methodology,
evaluation standards, implementation of evaluations, reporting and use of
studies, and the audit or meta-evaluation of evaluation.
§
Research
and Practice Notes that provide practical examples of the applications
of particular methodologies or procedures within the context of a particular
study or group of studies.
§
Book
Reviews of relevance to the practice in
(Canadian Journal of Program Evaluation,
2004)
Review of the past
eight issues (from Spring 2001 to Spring 2004) of CJPE revealed a number of insights into
the journal's thematic trends. The journal does, in fact, promote and publish articles
on theory, practice, implementation, and standards, for example. Notable
examples include Christie & Rose (2003)—The language of evaluation theory: Insights gained from an empirical study
of theory and practice, Levin-Rozalis (2003)—Evaluation and research: Differences and
similarities, Morris (2002)—The
inclusion of stakeholders in evaluation: Benefits and drawbacks.
In 2001, the CJPE devoted a special issue to
provincial evaluation policy and practice in
§
Has not acquired an identity of its own
§
Tends to neglect key issues
§
Loses emphasis on rigor
§
Is dominated by program monitoring
§
Is insufficiently connected with management
needs
Perspectives across
The Western Canadian
Perspective. (Malatest, 2004)
Strength: Development of evaluation
methodologies—in recent years the provincial and federal agencies have
recognized the requirement of good evaluation.
Weakness: Inadequate planning of
program evaluations—awareness and use of evaluation tools are often an
afterthought.
Threat: Reduced program evaluation
capacity—the ability to design and manage complex evaluation activities has
been compromised (e.g., lack of resources).
Program Evaluation in
Strength: Growing sophistication—evaluators
are more skilled and better qualified.
Weakness: Dependence on performance
measurement—to the exclusion of more relevant, complex outcomes.
Threat: Devaluation—avoidance of
serious evaluation (e.g., focus on accountability rather than improvement).
Program Evaluation in
Strength: Commitment—
Weakness: The paradigm—the current
approach is to assist the government in determining redirection of funding.
Threat: Capacity—Public and non-profit
organizations need to demonstrate effectiveness, yet they are limited in their capacity
to meet this demand.
Teaching and Learning Evaluation
in
Strength: Self-definitional
capacity—the time for evaluation to define itself and establish itself as a
distinct discipline is "now."
Weakness: Lack of disciplinary
focus—disciplines view evaluation differently rather than having a common
ground.
Threat: Disconnection—evaluation as
part of management is under threat (e.g., lack of common ground).
This paper is an outsider's perspective of evaluation in
References
Bradley,
S. E. (2001). Evaluation in the government of
Cabatoff, K. (2001). The long march from evaluation to
accreditation: Québec's new government management framework. The Canadian Journal of Program Evaluation,
16(special issue), 73-88.
Canadian Evaluation Society (2004). Canadian evaluation society website.
Available at http://www.evaluationcanada.ca/site.cgi?s=1&ss=0&_lang=an
Canadian
Evaluation Society (2001-2004). Canadian evaluation society quarterly newsletter. Available
at http://www.evaluationcanada.ca/site.cgi?s=4&ss=3
&lang=an
Canadian Evaluation Society (2004). CES guidelines for ethical conduct.
Available at http://www.evaluationcanada.ca/site.cgi?section=5&ssection=4&_lang=
an
Canadian International Development Agency (2004). Canadian International
Development Agency website. Available at http://www.acdi-cida.gc.ca/index.htm
Canadian Journal of Program Evaluation (2004). Canadian journal of program
evaluation website. Available at http://www.evaluationcanada.ca/
site.cgi?s=4&ss=2&_lang=an
Cousins, J. B. (2004). Personal communication.
Evaluation Data and Development (2004). Evaluation Data and Development website.
Available at http://www11.hrdc-drhc.gc.ca/pls/edd/hrdc.main
Gauthier,
B.,
Health
Industry
Hicks, K. (2001). Program evaluation in the government
of the Northwest Territory, 1967-2000. The Canadian Journal of Program Evaluation, 16(special issue),
107-114.
Long, B. & Kishchuk,
N., (1997). Professional certification: A report to the national council of the
Canadian evaluation society on the experience of other organizations. Canadian Evaluation Society.
McDavid, J. C. (2001). Program evaluation
in
Mowry, S.,
Clough, K., MacDonald, B., Pranger, T., & Griner, D. (2001). Evaluation policy and practice in
provincial governments province of
Ross,
A. (2001). Evaluation in
Segsworth, B. (2001). Evaluation policy and practice in
Social Sciences Research Council (2004). Social Science Research Council website.
Available at http://www.sshrc.ca/
Stierhoff, K. A. (1999). The certification of program
evaluators: A pilot survey of clients and employers. Canadian
Evaluation Society.
Statistics
Transport
Treasury Board of
Warrack, B. (2001). Program evaluation in the
Evaluation in Europe : An
Overview
Daniela Schröter[5]
The
Landscape of European Evaluation
The umbrella organization of evaluation in
The web site of the EES provides a good overview
about the evaluation community including lists of European and international
evaluation associations and networks, evaluation journals, events, and other
online resources. Currently,
the EES provides links to 13 national or multinational European organizations
as well as 5 regional networks within the
|
|
|
European
Evaluation Society, Danish Evaluation Society, Finnish Evaluation Society, French Evaluation
Society,
German Evaluation Society, International Program
Evaluation Network, Irish Evaluation Network, Italian Evaluation
Association, Polish Evaluation
Society, Spanish Public Policy
Evaluation Society, Swedish Evaluation
Society, Swiss Evaluation Society, UK Evaluation Society (the following are regional UK networks: Cymru Evaluation Network, Scottish
Evaluation Network, London Evaluation
Network, Midlands Evaluation
Network, North West Evaluation Network),
Walloon Evaluation Society |
|
Figure 1.
National and Multinational Evaluation Societies in |
Evaluation in
The EES holds conferences
biennially. From September 30 to
The Development of
Evaluation in
Leeuw (2004) asked if European evaluation
is still an “infant industry” and illuminates the European type of “evaluation
industry”. His book chapter will serve as the foundation for the following
sections. Rist, Furubo, and
Sandahl’s (2002)[7]
assessed countries worldwide on eight dimensions to determine levels of
development in evaluation. The dimensions included:
·
Evaluation
activity
·
Supply
of evaluators
·
Training
capacity
·
National
discourse
·
Organized
evaluation meetings
·
Evaluation
infra-structure within the public sector
·
Evaluation
infra-structure within parliament
·
Evaluations
carried out by Supreme Audit Offices (see Leeuw 2004,
63).
While not all European countries were assessed within this
study results indicated most intense evaluation efforts in North and West
European countries. However, data was either insufficient or indicated only
moderate training capacity for evaluators in Europe, which as Leeuw argues is plausible in view of the fact that
evaluation has not been established well at the university level in form of
evaluation studies. On the other hand, national discourse and organized
meetings were available and as indicated by other contributions in this issue
of JMDE not only stimulate debate and
discussions, but also provide platforms for trainings. Additionally, Rist et al. found that evaluation in the public sector was
more widely available than evaluation within parliament. Last but not least,
evaluations carried out by Supreme Audit Offices were most developed in
Historically,
The European Evaluation Market
Based on a study conducted in 1999, Leeuw
describes the European evaluation market as a growing market. While the
response rate in the study was rather limited, findings indicated that the
evaluation market was growing faster on the European and national levels than
in regions. Most evaluations conducted were related to policy and respondents
indicated that methods utilized usually derived from the evaluators’ specific
subject areas. Moreover, the regional evaluation market was perceived as rather
fragmented and it was thought that international competition on the European
evaluation market would be constrained due to cultural factors. For instance, Leeuw pointed out that one respondent said that it was even
hard to hire a British evaluator for an Irish setting. This is due to language
constraints and an understanding of the different organizational cultures. On
the European level, this leads to evaluations which are conducted by teams of
evaluators from multiple nations. Leeuw refers to
such arrangements as “(quasi)professions” (p.68). Moreover, Leeuw
argues that top-down processes thwart good evaluation practice. While
evaluation in
Evaluation on the
European Union Level[8]
On the European level, initial forms of program evaluations
began in the 80s, were focused on research and technology development programs,
and were based on practices prevalent in first wave evaluations. A shift
occurred in 1995, when a new evaluation scheme was introduced that demanded
evaluation of research and framework programs in form of annual monitoring and
five-year periodic assessments. Leeuw states:
The assessments
can be understood as a combination of an ex post evaluation of the previous
program, an intermediate evaluation of the current program and an ex ante
appraisal of future activities (2004, 69).
However, while
evaluation on the Union level always focused on regulatory policy, formal
evaluation systems or databases for the Directorates General are insufficient
and “the Council and Parliament pass[ed] a small number of ‘sunset’ regulations
which include a formal evaluation clause given a deadline (especially in the
field of Competition Policy)” (Leeuw, 2004, 69). The
results of reporting, however, are neither called nor could be classified as
evaluation. Other foci, especially cost-effectiveness and cost-benefit
evaluations were yielded by management reforms in the 90’s and are “supervised
by the Directorate General for Budgets and Financial Control” (Leeuw, 2004, 71).
In 1996, steps for
more systematic evaluations of policies were undertaken and a “decentralized
model in which the operational Directorates General are responsible for
establishing systematic evaluation procedures for the programs they are
executing” was developed to improve evaluation practice (Leeuw,
2004, 71). As a result, each Directorate had to designate one evaluation
official who is responsible for establishing an annual evaluation plan and for
determining program to be evaluated. The Directorates’ evaluation plans are
assembled into the “Commission’s Annual Evaluation Program”. The Directorate
General for Budget “coordinates evaluation activities and maintains an
overview of the evaluation findings across the Commission services. It also
provides methodological
guidance and support, helps with procurement
of evaluation expertise and maintains evaluation
networks within and outside the Commission (see website).
Unique features of the Evaluation Commission include a broad definition of the
concept of evaluation and its direct link to budget:
Not only does it
[evaluation] encompass ex post and midterm evaluation, but it also cover ex
ante exercises… evaluation projects are to be framed so that they correspond to
identifiable entities in the Community budget and to be timed so that results
are available when they are relevant for budgetary decisions (Leeuw, 2004, 72).
Current Issues in
European Evaluation[9]
Leeuw
refers to different elements of current developments in
Most central topics
for evaluation within
·
The
increasing importance of evaluation for civil society
·
Evaluation
for Parliaments (Do parliament decisions have effects?)
·
Evaluation
for public policy partnerships
·
Decentralization
of evaluation
·
Potentials
for evaluation of social programs from a non-managerial standpoint
·
Evaluation
of information and communication technology products, processes, and outcomes
(web-based communication, training, the internet as knowledgebase)
·
Auditing
versus evaluation
·
Evidenced–based
evaluation
·
Learning
from evaluation
·
Effective
implementation and utilization of performance management systems in public
management.
Overall, evaluation appears to be a vast growing market in
Leeuw, F.L. (2004). Evaluation in
News from the community (2004). In: Evaluation: The International Journal of Evaluation Theory, Research
and Practice, 10(3): 380-381.
Stern,
Elliot (Ed.). Evaluation: The
International Journal of Evaluation Theory, Research and Practice, 9(4)-10(3).
Stern, Elliot (2004). What shapes European evaluation: A personal reflection. In: Evaluation: The International Journal of Evaluation Theory, Research and Practice, 10(1): 7-15.
The European Evaluation Society (2004). The European Evaluation Society website. Available at: http://www.europeanevaluation.org/
Evaluation Activities in the United Kingdom
John
S. Risley
General Summary of Activities
The UK Evaluation
Society (UKES; www.evaluation.org.uk)
was founded in 1994 and is composed of over 150 individual and corporate
members. Most of these are individual members. UKES hosts
an annual conference each year in December and jointly conducts seminars and
conferences with other professional organizations. The society also sponsors an
e-mail discussion list, Eval Chat, publishes a thrice
yearly newsletter, The Evaluator, and
produces Evaluation: The International
Journal of Theory, Research and Practice.
UKES
has five regional networks. Three of these networks, the Scottish
Evaluation Network, the London Evaluation Network, and the North West
Evaluation Network are established. The other two, the Cymru
Evaluation Network (
The UKES website
offers a host of information and links on evaluation topics, including:
·
evaluation guidelines for good practice from
different national evaluation associations,
·
a list of postgraduate courses on evaluation
taught throughout the
·
links to 21 national/regional evaluation society
websites,
·
an evaluation glossary (including an entry on
“chatty bias”)
·
a short but wide-ranging bibliography of
evaluation books
Evaluation:
The International Journal of Theory, Research and Practice
The journal Evaluation is published quarterly by
Sage. Through the end of October it is available free online at
evi.sagepub.com. I reviewed the last two years of Evaluation (the January 2003 issue through the July 2004 issue) and
categorized each article according to Lori Wingate’s adaptation of Michael Scriven’s analogy for understanding disciplines. Wingate
identified four categories of focus for journal articles—practice, methods,
theory, metatheory—that I used below and one
category—history—that I eliminated because no articles fit the description.
Practice issues
dominated the 37 articles from the last two years (48.6 percent). The practice
articles mainly dealt with the related issues of evaluation use and stakeholder
participation. An article by Taut & Brauns (2003)
examines social and psychological explanations for resistance to evaluation and
offers strategies for overcoming evaluation resistance.
Many articles I
categorized in the practice area concerned evaluation in different
fields—healthcare, bidding for public services, welfare policy. These articles
did not discuss different evaluation approaches or models, so I did not
categorize them under theory.
Over one-fifth of
the articles (21.6 percent) concerned theory. Three of these eight articles
concerned theory-based evaluations—with two generally favorable and one
generally unfavorable toward the approach—while other evaluation approaches
addressed included qualitative, desk screening and implementation evaluation.
Hearn, Lawler and Dowswell (2003) addressed the
dominance of the positivist approach to most healthcare evaluation and argued
that an inclusion of “nonpositivist, qualitative, and
process-oriented evaluation” would improve our understanding of health programs
and policies.
I categorized six
articles (16.2 percent) as methods articles. Interestingly, all of these
articles focused on quantitative methods of data collection and analysis. Sverdrup (2003) discussed the use of time-series databases
of complaints data to evaluate laws and regulations.
The metatheory category included five articles (13.5 percent)
across 2003-2004. Virtanen and Uusikylä
(2004) address the “paradigm crisis” in evaluation that stems from evaluators’
different assumptions about causality. These authors describe four alternative
models (which they term ideal models) for evaluation considering: 1) how
explicitly causality has been taken into account, and 2) how well the model
enhances public-sector accountability.
The model
reflecting both a strong link between causality and the evaluation design and
an emphasis on public accountability is termed “transparent democracy”.
“Scientific inquiry” signifies a strong link between the evaluation design and
causality without an emphasis on accountability. The “explorative inquiry”
model is characterized by a high degree of emphasis on accountability and a
difficulty in distinguishing causal effects. Finally, an evaluation using the
“symbolic evaluation” model serves a symbolic purpose rather than a “true
pursuit of learning.” (89)
References
Hearn, J., Lawler,
J., & Dowswell, G. (2003) Qualitative
evaluations, combined methods and key challenges: General lessons from the
qualitative evaluation of community intervention in stroke rehabilitation. Evaluation. 9:
30-54.
Sverdrup,
S. (2003). Towards an evaluation of the effects of laws: Utilizing time-series
data of complaints. Evaluation.
9: 325-339.
Taut,
S., & Brauns, D. (2003). Resistance to
evaluation: A psychological perspective. Evaluation. 9: 247-264.
Virtanen, P., & Uusikylä, P. (2004) Exploring the missing links between
cause and effect: A conceptual framework for understanding micro–macro
conversions in programme evaluation. Evaluation. 10:
77-91.
Evaluation in Eastern Europe
and the Middle East
P. Cristian Gugiu
The state of evaluation in
Compared to
Evaluation
Journals and Newsletters
East European Journals
Several representatives of the European
Evaluation Society (EES) report that no one knows of any journal or newsletter
publications in
According to Barbara Rosenstein, Ph. D., Chairperson of the
Israeli Association for Program Evaluation (IAPE), the IAPE has published, to
date, eight newsletters, in both Hebrew and English on evaluation.
Israeli Journal: Studies in Educational Evaluation
Studies in
Educational Evaluation (SEE) is published in English. The majority of articles were not
published by Israelis. Authors were dispersed throughout the world including
the
A great many of the articles were purely research articles,
a few of them described an evaluation case study, and a fair number of them
discussed a specific methodology that could be used in evaluation.
Evaluation
Societies
European
Evaluation Society (http://www.europeanevaluation.org/)
The primary goal of the European Evaluation
Society (EES) is to promote theory, practice and utilization of high quality
evaluation especially, but not exclusively, within the European countries. This
goal is obtained by bringing together academics and practitioners from all over
EES held its sixth conference on September 30 to
|
|
Over three-quarters of the
presenters came from West European countries including Belgium (2.4 percent),
Denmark (2.7 percent), Finland (4.2 percent), France (4.8 percent), Germany
(9.3 percent), Greece (0.3 percent), Iceland (0.3 percent), Ireland (2.1
percent), Italy (15.9 percent), Netherlands (5.4 percent), Norway (1.5
percent), Portugal (2.7 percent), Spain (5.7 percent), Sweden (5.7 percent),
Switzerland (4.5 percent), and the United Kingdom (9.0 percent). The remaining
presenters included countries from Asia (Japan, 0.6 percent; Korea, 0.9
percent), Australasia (Australia, 2.4 percent; New Zealand, 0.6 percent),
Africa (Angola, 0.3 percent; Guinea Bissau, 0.3 percent; Kenya, 0.3 percent;
Nigeria, 0.9 percent), East Europe (Austria, 2.4 percent; Bosnia and
Herzegovina, 0.3 percent; Czech Republic, 0.3 percent, Poland, 1.2 percent),
the Middle East (Egypt, 0.3 percent; Israel, 0.3 percent, Palestine, 0.3
percent), North America (Canada, 0.9 percent; United States, 5.1 percent), and
Latin America (Colombia, 0.6 percent; Mexico, 1.8 percent).
|
|
There were slightly more male presenters than
female presenters.[10]
However, this statistic was primarily influenced by the large number of West
European presenters. Five of the seven other regions had an equal or greater
number of female presenters than male presenters.
|
|
An examination of the type of jobs presenters
worked in revealed that the majority of them worked for a university or college
in their native country. The two next largest groups included people who worked
in private industry or for the government.[11]
It was interesting to note the differences in distribution of job type among
the eight regions. For seven of the eight regions, presenters typically worked
at a university. However, for
|
|
Polish Evaluation
Society (http://www.pte.org.pl/)
The Polish Evaluation Society (PES) began in 2001
and set out to build an evaluation culture and popularize evaluation as a
social and democratic process. To this end, it sought to (a) organize studies,
courses and trainings; (b) conduct evaluation research; (c) exchange
experiences with other societies, institutions and organizations; (d) organize
meetings, seminars and conferences, (e) publish in the area of evaluation, and
(f) provide consulting and advising services.
The Polish Evaluation Society has very strict
rules as to the educational qualifications of its members. Most members are
still strongly connected with the academic environment, either via didactic
activity or scientific research (
Members of PES are professional evaluators who
also conduct marketing research and other research on social character. They
have wide experience in the field of the evaluation which they gained in the
process of conducting a variety of research for Polish and international
organizations such as Polish Children and Youth Foundation, Public Interest
Institute, government organizations such as European Integration Committee, the
ministry of Education, service sector companies such as Daewoo, and EU
institutions such as European Parliament and European Commission. Members of
PES use different paradigms and research perspectives. A Rich variety of the
activities and approaches is an advantage of this organization.
Romanian
National Assessment and Examination Service (http://www.edu.ro/snee.htm)
The National Assessment and Examination Service
(NAES) was established in 1998 by the Romanian
Government as the first national, independent body providing professional
expertise in educational assessment and examinations in
NAES is actively involved in national and
international projects (e.g. the British Council, QUATRO Fontys—PTH Eindhoven) and maintains professional contacts with
universities, research institutes, governmental and nongovernmental
institutions and organizations in the field (e.g. CITO—The Netherlands, EDC—USA
etc.). Their headquarters in
Israeli Association for
Program Evaluation (http://www.iape.org.il/)
The Israeli Association for Program Evaluation
(IAPE) is a non-profit, professional organization comprised of academics,
practitioners and users of program and project evaluation in a variety of
fields—psychology, education, social services, health, business,
and others. The goals of the organization include (a) increasing the use of
program evaluation and its findings, (b) encouraging the development of the
theory of program evaluation, (c) advancing the essential recognition of
program evaluation as a means of improving the effectiveness of social and
educational interventions, (c) promoting the recognition of program evaluation
as a profession, (d) serving the communities and the populations involved in
program evaluation, (e) contributing to the influence of program evaluation on
decision making, (f) supporting and influencing evaluation practice in Israel, and
(g) creating and developing professional ties among evaluators and users of
evaluation in Israel. To this end, the IAPE has sought to (a) organize
conferences focusing on issues of concern to the evaluation community; (b)
create an electronic and regular mail network that provide information about
issues concerning evaluation in Israel and abroad; (c) establish connections
with evaluation organizations throughout the world; (d) participate in the
worldwide forum of evaluation associations, (e) circulate of a list of members
to evaluation consumers in Israel; and (f) publish a newsletter containing
articles, discussions, and events of interest to the evaluation community in
Israel.
Evaluation in Latin America and the Caribbean : An Overview
of Recent Developments
Thomaz Chianca and Brandon Youker[12]
In the past ten
years, evaluation, as a professional field, has undergone significant
development in several countries in
The first
professional evaluation organization that was formed in the region was the
Central American Evaluation Association (ACE) in 1989 that has had its
headquarters, since then, in
Only eight years
later, in 2002 new evaluation professional organizations were established in
LAC. Given their specific contexts,
PREVAL (Program for Strengthening the
Regional Capacity for Evaluation of Rural Poverty Alleviation Projects in Latin
America and the Caribbean)—a joint effort between IFAD and, from 1995-2000, the
Inter-American Institute for Cooperation on Agriculture (IICA), and, from
2000-2007, with the Centro de Estudios para la Promoción del Desarrollo (Center of Studies for Development Promotion)—has
played an strategic role in the region since 1995, contributing directly to the
creation of the national evaluation networks in Peru and Colombia. In its first
two phases (1995-2000 and 2000-2004), PREVAL focused its work on strengthening
the evaluation capacity of IFAD projects to reduce rural poverty in the region.
In its third phase (2004-2007), PREVAL will broaden its objectives to work more
closely with governments, organizations offering technical assistance in
monitoring and evaluation, as well as national evaluation and monitoring
networks/associations in the region. PREVAL has established an important network
of evaluators working with projects aimed at alleviating rural poverty, and has
produced an important body of knowledge in this area published in Spanish. It
is also important to recognize the key role played by the International
Organization for Cooperation in Evaluation (IOCE)—comprising all national
and regional evaluation organizations around the globe—in fostering the
establishment of evaluation organizations in the region. IOCE held an important
planning meeting in
In September 2003,
representatives from the four existing evaluation organizations in the region
got together in
There are at least four
electronic discussion lists on evaluation in the LAC region: RELAC, PREVAL, the
Brazilian Evaluation Network,
and the ILPES/CEPAL.
It is not
over-optimistic to assume that very soon we will witness a significant increase
in the number of evaluation professional organizations in LAC.
Use
of Professional Evaluation in Key Societal Sectors in LAC
There has been
significant growth in the use of professional evaluations by the government,
the nonprofit sector, and at least in the field of personnel evaluation, in
large private businesses. In the government arena, initiatives related to
national educational evaluation/assessment systems, innovations in government
administration systems, and social development programs supported by international
cooperation agencies are major factors influencing such growth.
In education, the
establishment of evaluation mechanisms has been extensive from basic (K-12) to
higher education in many countries within the region. In
The idea of
reducing the size of the state and making it more effective and efficient
(state reform) has strongly influenced virtually all countries in the region.
Such an idea brings along a strong push for the establishment of control
systems on expenditures as well as for implementation of planned activities
that usually involve monitoring and, to some extent, evaluation. Several
countries have created structures, usually subordinated to the ministry of
planning, that are in charge of dealing with monitoring and internal evaluation
of governmental efforts. Examples of such structures are the System of
Information, Evaluation, and Monitoring of Social Programs (
In the area of
social development, virtually all programs supported by international
cooperation agencies such as the World Bank, Inter-American Development Bank,
World Health Organization, and United States Agency for International
Development (USAID), are required to be evaluated both internally as well as by
using external evaluators. These organizations have played a major role in
introducing innovations in evaluation as well as advocating for the use of
quality professional evaluations within government funded initiatives. Several
examples of such evaluations are already publicly available from the agencies’
websites (e.g., USAID and the World
Bank-Operations Evaluation Department (OED). The Latin American Institute
for Social and Economic Planning (ILPES),
subordinated to the United Nations Economic Commission for Latin America and
the Caribbean (CEPAL), has been an
important reference in providing evaluation support to country-level government
evaluators by offering supporting materials (publications); evaluation
training; and networking opportunities for professionals working in evaluations
of governmental social-development programs in the region.
Initially
influenced by international foundations investing in the region the fast—growing
nonprofit sector in
The W.K. Kellogg Foundation is one of the
international foundations that have significantly invested in the development
of evaluation in LAC. In 1995 and 1997, the foundation sponsored two groups of
LAC evaluators (a total of approximately 40 professionals) in in-depth training
programs in evaluation at The
Evaluation Center—Western Michigan University. Some of the participants
of such training opportunities are assuming leadership roles in evaluation in
the creation of evaluation organizations in their home countries.
Several foundations
and institutes are commissioning and/or developing evaluations throughout the
region. In Brazil, for instance, some of the nonprofit organizations that are
very active in evaluation include: Fundação Carlos Chagas, Fundação Cesgranrio, Instituto Ayrton Senna, Fundação ABRINQ Foundation, Instituto FONTE,
Fundação Roberto Marinho, and Fundação IOCHPE.
Another interesting
movement influencing the growth of evaluations in the third sector is the
increasing number of private businesses investing in social initiatives, based
on the idea of social responsibility. Such organizations have a different
culture (focus on control and efficiency) from the nonprofit organizations
investing in the sector, and have made an important push to support the
establishment of monitoring and evaluation systems in the initiatives sponsored
by them. In
The extent of
evaluation use in the private sector is not very public. It is evident that
several corporations and other private business have made serious efforts to
evaluate their products, projects and personnel. Reports on such efforts,
however are not easily accessible and the evaluators working in this area have
almost no contact with other evaluators working in the public and nonprofit
sectors. No doubt more extensive exchange of experiences between these
professionals has great potential to be beneficial to all, but some important
barriers such as prejudices from both sides (e.g., ‘private sector only look at
profits;’ ’public and nonprofits are always inefficient’) need to be overcome
before such approximation has any chance of succeeding.
Body
of Original Publications in Evaluation
Though there are
virtually no evaluation specific journals in LAC, there are several journals
related to education, health, and social sciences with strong evaluation
content. Some examples include:
· La Revista de Ciencias
Sociales (Journal
of Social Sciences—Costa Rica)
· Revista
Ensaio – Avaliação e Políticas Públicas em Educação (Evaluation and Public Policy in
Education—Brazil)
· Estudos em
Avaliação Educacional (Educational Evaluation Studies—Brazil)
·
Cadernos
de Saúde Pública
(Journal of Public Health—
·
Revista
Avaliação Psicológica
(Journal of Psychological Evaluation—
·
Revista da Rede de Avaliação
Institucional (Journal of the
Institutional Evaluation of Higher Education Network—
·
Cuadernos
de Investigación de la Escuela de Gerencia Social (Journal of Inquiry of
the School of Social Management—Venezuela)
· Revista del Instituto de
Investigaciones en Ciencias de la Educación (Journal of the
Education Science Investigation Institute—Argentina)
· Acción y Reflexión Educativa (Educative Action and Reflection—Panama)
·
Planejamento e Políticas Públicas (Planning
and Public Policy—
· Revista de Administração Pública (Journal of
Public Administration—Brazil)
The footnoted
social science journals[14]
have regularly published the intellectual products of LAC evaluators.
It is critical to
acknowledge the substantial collection of accessible evaluation publications
such as books, manuals, newsletters, technical reports, etc. that are available
in most Latin American countries. There
are several websites such as the Latin American Institute for Social and
Economic Planning (ILPES), and the Programme for
Strengthening the Regional Capacity for Evaluation of Rural Poverty Alleviation
Projects in Latin America and the Caribbean (PREVAL) that provide an extensive
collection of evaluation publications in the field of evaluation throughout the
region.
There are two
excellent annotated bibliographies that provide published reference materials
that address several aspects of evaluation in LAC. The first publication, The Annotated Bibliography of International Programme Evaluation, edited by Russon
& Russon[15]
has a chapter by Antoinette B. Brown and Ada Ocampo, on
There are a few LAC
universities and training institutions that offer masters level programs,
specifically in evaluation. At the masters’ level, there are at least five
universities offering such program:
·
Professional
Masters in Evaluation of Social Programs and Projects. Universidad de
Costa Rica.
·
Masters in
Socio-Economic Evaluation of Investment Projects. Universidad Panamericana.
·
Masters of Science in Project
Management and Evaluation. University of the
·
Masters in Project
Evaluation. Universidad del CEMA,
· Masters in Social Projects
Evaluation. Universidad Autónoma de Guadalajara.
Guadalajara, Jalisco, México.
At the
certification level there are quite a few programs offered in different
countries including:
·
Course on
Evaluation of Social Programs and Projects. Centro de Empreendedorismo Social e Administração
em Terceiro Setor—CEATS (
·
Diploma in Evaluation
of Projects. Universidad de Concepción.
·
Diploma
in Evaluation of Social Projects. Pontificia Universidad Católica
de Chile.
·
Diploma in
Planning and Evaluation of Projects. Universidad de Chile.
·
Diploma in
Planning and Evaluation of Socioeconomic Projects. Centro de Análisis y Evaluación de Política
Pública—CAEP—Monterrey, Mexico. (
·
International
Certificate of Project Planning, Evaluation, and Management—Inter-American Development
Bank. Centro de Investigaciones Territoriales
·
Post Graduate in Formulation
and Evaluation of Projects. Universidad Americana.
There are also several
short-term evaluation training courses facilitated by different organizations
within the region. Some of the best sources to identify such training
opportunities include: (a) Nota Informativa
del ILPES sobre Evaluación
de Proyectos y Programas (ILPES
Informative Note on Program and Project Evaluation); (b) PREVAL; and FONTE Institute. The following is a
sample of the recently offered short-term courses in some LAC countries:
·
X International Course on Planning
and Evaluating Public Investment Projects. Offered by CEPAL/ILPES.
Sept 27 to Oct 22,
·
Internet-based
course on Planning and Evaluation of Agricultural and Agri-Industrial
Projects. Offered by REDCAPA and Austral
University of Chile. Sep 1 to
·
International
Course on Logic Model, monitoring and Evaluation. Offered by
ILPES/CEPAL and the Spanish Cooperation Agency (AECI). Jun 21 to
·
International
Course on Use of Socio-Economic Indicators for the Evaluation of Impact of
Poverty Reduction Programs. Offered by ILPES/CEPAL and the Spanish Cooperation
Agency (AECI). May 3—14.
·
Utilization-Focused
Evaluation by Michael Quinn Patton. Sponsored by the Brazilian
Evaluation Network, UNICEF-Brazil, and FONTE Institute.
·
Collaborative
Evaluation by Rita O’Sullivan. Sponsored by the Brazilian Evaluation
Network, UNICEF-Brazil, and FONTE Institute.
The report makes no
claims to be comprehensive and does lack significant information, mainly about
the state-of-art of the evaluation field in the
It does, however
provide unquestionable evidence of the impressive advances the whole region has
made in the evaluation field in the recent past. Evidently,
even though not to the same degree in each country, it is reasonable to say
that basic conditions have been established to make such advances even more
comprehensive and effective in the future.
The current efforts
to establish national and regional evaluation organizations, the growing number
of quality publications in both Spanish and Portuguese on evaluation, the
increasing use of professional evaluation by different organizations in all
societal sectors and the broad recognition of evaluation as important for
improving society are some of the factors influencing such advances. One major
challenge still to be faced in order to have evaluation in a better position as
a recognized professional field is the creation of more formal graduate-level
training for evaluators in a wider range of countries.
This paper is a
work in progress that will be modified and/or improved as we gain new
information. If you would like to provide additional information or point out
any errors or misunderstanding in the text, please do not hesitate to contact Thomaz Chianca (thomaz.chianca@wmich.edu) or Brandon
Youker (brandon.w.youker@wmich.edu).