Global Review: Regions

 

 

 

 

 

 

 

 

 

 

 

 

 

                              


Evaluation Activities in Africa

Paul A. Lamphear

 

Africa has seen a significant growth in Evaluation networking in the past five years with the founding of the African Evaluation Association (AfrEA) in 1999. As of this writing, AfrEA has 16 national associations under its umbrella, each with evaluator networks supporting their respective countries. The associations are in a wide disparity of development maturity, but are all aimed at supporting socio-economic development programs in their countries, as in this description of the Uganda Evaluation Association as posted in www.kabissa.org.

The major purpose of the association is to build individual and institutional capacity in policy, program and project evaluation in Uganda through local and global networking, training, skills development and other avenues for professional development in evaluation practice[1].

The first conference of the African Evaluation Association, held in Nairobi in connection with the creation of AfrEA, was attended by more than 300 evaluators from 35 countries. Michael Quinn Patton was invited as keynote speaker, promoting a focus on ethics and guiding values for professional evaluation associations. He also emphasized the need for a culturally defined set of standards for evaluation and the necessity for utilization-focused evaluation.[2] Patton referred to the historically poor “expatriate” evaluations with poorly briefed evaluators, performed too quickly, with inadequate reports.

It appears that these lectures, the training seminars, and 80 papers that were submitted on evaluation, helped to ‘jumpstart’ the development of the national associations. Although there does not appear to be any evaluation journals specific to or published in Africa, AfrEA has worked on several projects to increase evaluation capacity and foster a consistent professional approach for evaluators. In 2002, the association completed the "African Evaluation Guidelines", a cultural adaptation of the US “Program Evaluation Standards”, published in both English and French.

At the 2nd AfrEA conference in 2002, a variety of international evaluators were invited, including keynote speakers Prof. Anna Madison of Cornell University, USA, Ada Ocampo, Leader of the Latin American Evaluation Network, and Penny Hawkins, President of the Australasian Evaluation Association. After 5 days of trainings and paper submissions, the association recommended that the African Evaluation Guidelines (AEG) be adopted by all the National Networks, by Government and Public bodies, and by UN Agencies and other Multinational Organizations performing evaluation in Africa. The AEG provides a checklist of 30 items essential for quality control in evaluation and it has now been used by several national governments in major development program evaluations.

The Niger Network of Monitoring and Evaluation (ReNSE) has over 140 documents (in English or French) and in 2004 has published their first newsletter (in French), available from their website, to promote expertise in Nigerian evaluators.

AfrEA supports dissemination of monitoring and evaluation resource materials focused on Africa in the areas of Agriculture, Conservation, Gender, HIV/AIDS, and Poverty. Additionally, AfrEA encourages members to participate in the Xceval listserve, a discussion forum for evaluators from developing countries.[3]

AfrEA has significant connections with international organizations and has activities currently sponsored by:

African Development Bank (AfDB)

Agence Intergouvernementale de la Francophonie

Catholic Relief Services

Canadian Institutes for Health

Canadian International Development Agency (CIDA)

CARE International

Danish Agency for Development Assistance (DANIDA)

Family Health International (FHI)

International Development Research Centre (IDRC)

World Conservation Union (IUCN)

Norwegian Ministry of Foreign Affairs

UNAIDS

UNCHS

UNDP

UNICEF

UNIFEM

World Bank

The third national Conference of the African Evaluation Association will be held in Cape Town, South Africa, from December 1st through December 4th, 2004, and is looking for speakers and attendees.

 


Evaluation Activities in Australasia

John S. Risley

 

General Summary of Activities

The Australasian Evaluation Society (AES) produces, and posts on their website (www.aes.asn.au), an e-newsletter approximately twice per year. The AES also holds an annual conference, usually in September or October. The 2004 conference is in October near Adelaide, Australia and will focus on “Diverse Voices in Evaluation.” Last year’s conference emphasized evaluation and indigenous peoples. Many pre-conference workshops are offered. AES has regional representatives from throughout Australia and New Zealand. There is a New Zealand Listserv—Evaluation Aotearoa—that discusses “evaluation research.” It only has a few posts per month, mostly dealing with Auckland Evaluation Group activities.

From reading the editorials and other non-refereed articles in the Evaluation Journal of Australasia (EJA) it appears that the evaluation profession in Australasia differs from the profession in the United States in two main ways. First, evaluators come from more diverse academic and professional backgrounds in Australasia than in the United States. Second, Australasian evaluators are much less likely to be associated with a university and much more likely to be employed by a government agency than are American evaluators.

 

 

Evaluation Journal of Australasia

A recent editorial in Evaluation Journal of Australasia (EJA) noted the history of AES publications. The society launched the EJA in 1989. Then from 1993 through 2000 AES published both EJA and Evaluation News & Comment. In 2001 these publications merged to form the new series of EJA. The journal is published by the AES bi-annually (though recently there have been delays in publishing new editions). AES posts the two copies preceding their most recent issue on their web site. The journal includes refereed and non-refereed articles, editorials, interviews with evaluators from both within and without the region, book reviews, research reports, and information about the annual AES conference.

Issues addressed in EJA included much information concerning cultural appropriateness, indigenous peoples, and diversity in evaluation. This may be a reflection of the recent AES conference themes. There is some material drawing distinctions about evaluation aspects specific to Australasia, but many articles are written by authors outside the region about subjects not specific to the region.

Subjects of refereed articles in recent issues of EJA include: evaluation of options for changing port ownership in Belfast, an evaluation of a respite care program in Christchurch, evaluating the cultural appropriateness of human service delivery programs in Australasia, and the TRIAGE (Technique for Research of Information by Animation of a Group of Experts) technique. A few refereed articles were short (3 pages and 5 pages) compared to articles in the American Journal of Evaluation, for example.

Some of the refereed articles had very little to say about evaluation. For example, one of these articles (Burton & Rajan, 2002) concerned a case study evaluation of 15 people seriously injured in workplace accidents. The authors described the project’s goal as exploring the social and economic consequences to society from these workplace injuries. The article discussed the methodology of the study, the experiences of the researchers, and the lessons learned from their research experiences. The methodology was basically a semi-structured interview of injured workers, their family members, employers, etc. The lessons learned by the researchers were: 1) interviewing can be exhausting, 2) diversity of the project team was essential, and 3) it was difficult to remain objective after seeing the suffering of the injured workers.

One interesting article (Sigsgaard, 2002) addressed an unusual methodology (in evaluation research), the Most Significant Change (MSC) methodology. The author, Peter Sigsgaard, works at a Danish NGO called “MS” on measurement and evaluation issues. He gave examples of his experience using MSC in evaluating partnership-based economic development programs in Africa, Asia and Central America. Using MSC you ask people to identify positive or negative changes they have observed within a given “domain of interest.” People are then asked which change, positive or negative, they think is most significant and why. More important or very large changes that are reported are verified by further investigation.

Sigsgaard (2002) contrasts this approach with one previously used by MS in evaluating these programs, in which they would conceive of indicators to measure and then cast about looking for these indicators. This led to lots of time spent looking for, and not finding, specific data.

It makes intuitive sense to ask program consumers what changes are occurring due to the program. It does highlight the need to be careful how one measures program changes.

References

Burton, J., & Rajan, R. (2002). Revealing the hidden costs: research experiences from a case study evaluation project. Evaluation Journal of Australasia (new series), 2(2), 69-73.

Sigsgaard, P. (2002). Monitoring without indicators: an ongoing testing of the MSC approach. Evaluation Journal of Australasia (new series), 2(1), 8-15.


The State of Evaluation in Canada

Chris L. S. Coryn[4]

 

Background and General Context of Organized Evaluation in Canada

The Canadian Evaluation Society (CES)—Canada's official professional organization for evaluation—serves as the country's core for evaluation related activities. CES is similar to many other evaluation associations around the globe, but differs from the American Evaluation Association in that the majority of its members are from the government sector and evaluators practicing in NGOs, para-government, and the public and private sectors. Presently, the CES has 12 regional chapters, including:

ü     Newfoundland & Labrador

ü     Ontario

ü     Prince Edward Island

ü     Manitoba

ü     Nova Scotia

ü     Saskatchewan

ü     New Brunswick

ü     Alberta

ü     Quebec

ü     British Columbia

ü     National Chapter

ü     Northwest Territories

Since 1991 the CES membership has grown to over 1,750 individual Canadian and student members, as well as over 100 international members (CES, 2004).

The CES offers a wide range of resources and services for practicing evaluators and students of the discipline including: a comprehensive Web site (available in English and French); an evaluation report bank (academic, government, and private sector reports); a fully-searchable database—the Grey Literature Bank (unpublished documents of interest to evaluators); a professional development series of workshops; an annual conference (including the upcoming 2005 joint conference with the American Evaluation Association); and the Canadian Journal of Program Evaluation. The CES efforts are strongly supported by the Government of Canada, which has its own specialized evaluation unit; Evaluation and Data Development (EDD). EDD is one of the largest evaluation shops in the Federal Government of Canada, and focuses primarily on governmental initiatives including analysis of government policy and evaluation of government programs, foe example, Human Resources Development Canada (HRDC) programs. Other government contingencies which influence the Canadian evaluation field include the National Science and Engineering Research Council, the Social Science Research Council, Transport Canada, Industry Canada, Health Canada, the Treasury Board Secretariat, and the Canadian International Development Agency; each of which are also sponsors of the CES. Informed decision making is further facilitated by Statistics Canada a provider—federally legislated—of statistical data for the whole of Canada and each of its provinces that is intended to inform Canadian citizens and other key stakeholders regarding Canada's population, resources, economy, culture, and society.

In the summer of 2001 the CES announced their new vision, mission, and goals for the future (Canadian Evaluation Society Newsletter, Summer 2001):

Vision: The Canadian Evaluation Society will be the leader for evaluation in Canada and a major contributor in the global evaluation community

Mission: The society is a Canada-wide non-profit bilingual association dedicated to the advancement of evaluation theory and practice.

Goals:

1.     LeadershipTo provide leadership to individuals and organizations in support of evaluation theory and practice in Canada and the global community.

2.     KnowledgeTo improve the state of evaluation theory and practice.

3.     AdvocacyTo promote the importance of an evaluation culture.

4.     Professional DevelopmentTo promote and facilitate the enhancement of evaluation capacity for members and non-members.

The CES also supports various student initiatives including the CES Student Case Competition and student paper contest (for undergraduate and graduate students in the field of evaluation). The CES Student Case Competition (initiated in 1996), is an annual event in which teams of three to five students from Canadian colleges and academic institutions compete in the analysis of an evaluation case file. In a preliminary competition, all teams receive on the same day the key to an evaluation case file that has been hidden on the Web. They have five hours to prepare an analysis and then submit it by e-mail for judging by an expert panel. The three best teams are invited to participate in a final round, held at CES's annual conference, in which they must analyze a new case and present findings and recommendations before a live audience. The team that makes the best presentation takes possession of the Case Competition Trophy for a year, receives prizes, and is given visibility in various publications.

Evaluation Education Programs in Canada

As of 2000 (CES), over 25 Canadian institutions/colleges/universities offered more than 100 evaluation-related courses across a wide array of academic disciplines (e.g., psychology, political science, public administration, economics)—a complete institution, department, and course list is available at http://www.evaluationcanada.ca/txt/outline200106.pdf.

Professional Development of Canadian Evaluators

The CES plans to focus on two key areas in the upcoming years: (1) professional development of its members, and (2) advocacy on behalf of the evaluation function. The articulation of a Core Body of Knowledge (CBK) will guide the Society's professional development and advocacy activities (Canadian Evaluation Society, 2004). The CBK comprises theories, skills, and best practices that people must possess to plan, carry, out, and report on valid and reliable evaluations of programs or policies in governments, not-for-profit organizations, and businesses.

Essential Skills. Much of the emphasis on professional development is funneled through the CES Essential Skills Series. Regional chapters offer this series as well as any other form of training they consider adequate for their members. These essential skills include:

1.     Understanding Program Evaluation

§   Key terms and concepts

§   Benefits of program evaluation

§   Basic steps in the evaluation process

§   Major approaches to program evaluation

§   Formatting evaluation questions

§   Designing an evaluation

§   Evaluating with limited resources

§   Analyzing and reporting evaluation results

§   Reducing resistance to evaluation

§   Involving staff and clients in the evaluation process

§   Increasing evaluation utilization

§   Making evaluations ethical and fair

2.     Building an Evaluation Framework

§   Identifying who the client is and what the client needs

§   Basic concepts of needs assessment

§   Major approaches to assessing client needs

§   Evaluation methods for "getting close to the client"

§   Building an evaluation framework through logic models

§   Involving managers and staff in building an evaluation framework

§   Relating program design to client needs

§   Defining program components

§   Formulating indicators for program success

§   Using the evaluation framework for linking program performance to client needs

 

 

3.     Improving Program Performance

§   Using evaluation as a management tool for improving program performance and enhancing internal accountability

§   Basic concepts of monitoring and process evaluation

§   Monitoring program performance with existing administrative data and information systems

§   Developing ongoing data collection instruments and procedures

§   Linking process evaluation to program decision-making

§   Assessing client satisfaction

§   Understanding continuous quality improvement

§   Using program evaluation for building a "learning organization"

4.     Evaluating for Results

§   Defining program results

§   Major approaches to evaluating results

§   Developing results measures

§   Designing outcome evaluations

§   Validity and reliability

§   Appropriate use of quantitative and qualitative techniques

§   Relating program results to program costs

§   Understanding program benefits

§   Measuring program equity and responsiveness to community needs

§   Communicating evaluation findings

§   Using evaluations to improve program effectiveness and accountability

(Canadian Evaluation Society, 2004)

Certification of Evaluators in Canada. As a body representing program evaluators across Canada and promoting the program evaluation function in Canadian institutions, the CES is concerned with the sustainability, growth and strengthening of the profession. In recent years, this concern has led the Society to consider issues related to increasing professionalization, through means such as professional development programs, development and adoption of practice standards and ethical guidelines, and certification of members. This issue remains unresolved, but is becoming increasingly acute in the wake of recent developments in the federal government sector that have raised the profile of auditing (Cousins, 2004).

This latter issue — developing a form of certification for members — would be a major step for the CES. Therefore, it was the subject of an in-depth study of the experience of several other organizations with certification (Long & Kishchuk, 1997). A second study, carried out in 1999, reports on a pilot survey of clients and employers (Stierhoff, 1999) on their views regarding certification of evaluators.

Canadian Journal of Program Evaluation

The Canadian Journal of Program Evaluation (CJPE) was launched in 1986 and is published twice a year (available at www.cjpe.ca). CJPE is sponsored by the CES and the University of Calgary. Individual issues and articles can be downloaded by non-members for a nominal cost. CJPE seeks to promote the theory and practice of program evaluation in Canada by publishing:

§         Articles on all aspects of the theory and practice of evaluation, including methodology, evaluation standards, implementation of evaluations, reporting and use of studies, and the audit or meta-evaluation of evaluation.

§         Research and Practice Notes that provide practical examples of the applications of particular methodologies or procedures within the context of a particular study or group of studies.

§         Book Reviews of relevance to the practice in Canada.

(Canadian Journal of Program Evaluation, 2004)

Review of the past eight issues (from Spring 2001 to Spring 2004) of CJPE revealed a number of insights into the journal's thematic trends. The journal does, in fact, promote and publish articles on theory, practice, implementation, and standards, for example. Notable examples include Christie & Rose (2003)—The language of evaluation theory: Insights gained from an empirical study of theory and practice, Levin-Rozalis (2003)—Evaluation and research: Differences and similarities, Morris (2002)—The inclusion of stakeholders in evaluation: Benefits and drawbacks.

In 2001, the CJPE devoted a special issue to provincial evaluation policy and practice in Canada. Accounts of provincial evaluation activity were provided for British Columbia (McDavid, 2001), Alberta (Bradley, 2001), Manitoba (Warrack, 2001), Ontario (Segsworth, 2001), Quebec (Cabatoff, 2001), Prince Edward Island (Mowry, Clough, MacDonald, Pranger, & Griner, 2001), Newfoundland (Ross, 2001) and the Northwest Territories (Hicks, 2001). "Being the first ever account of evaluation activity at the provincial level, this collection of articles represented a very important contribution to the knowledge of evaluation practice in Canada" (Gauthier, Barrington, Bozzo, Chaytor, Cullen, Lahey, Malatest, Mason, Mayne, Myers, Porteous, & Roy, 2004). A number of general and specific conclusions were drawn about the state of affairs in Canadian evaluation as a result of this special issue and were summarized in The lay of the land: Evaluation practice in Canada today (Gauthier et. al., 2004). The authors conclude that program evaluation in Canada:

§        Has not acquired an identity of its own

§        Tends to neglect key issues

§        Loses emphasis on rigor

§        Is dominated by program monitoring

§        Is insufficiently connected with management needs

Regional Perspectives

Perspectives across Canada's various regions are briefly summarized below. This summary includes: (1) strengths, (2) weaknesses, (3) threats, and (4) opportunities of and for evaluation in western Canada, Alberta, and Ontario, as well as potentials for evaluation teaching and learning.

The Western Canadian Perspective. (Malatest, 2004)

Strength: Development of evaluation methodologies—in recent years the provincial and federal agencies have recognized the requirement of good evaluation.

Weakness: Inadequate planning of program evaluations—awareness and use of evaluation tools are often an afterthought.

Threat: Reduced program evaluation capacity—the ability to design and manage complex evaluation activities has been compromised (e.g., lack of resources).

Opportunity: Managing for outcomes—activities in British Columbia and Alberta have been strengthened by strong government-wide commitment to measure and report on the key outcomes for almost all ministries and/or departments.

Program Evaluation in Alberta. (Barrington, 2004)

Strength: Growing sophistication—evaluators are more skilled and better qualified.

Weakness: Dependence on performance measurement—to the exclusion of more relevant, complex outcomes.

Threat: Devaluation—avoidance of serious evaluation (e.g., focus on accountability rather than improvement).

Opportunity: Linking accountability and evaluation—evaluators believe that they can make evaluation more rigorous and more useful.

Program Evaluation in Ontario. (Mason, 2004)

Strength: Commitment—Ontario government is being steered toward evaluation by political interest.

Weakness: The paradigm—the current approach is to assist the government in determining redirection of funding.

Threat: Capacity—Public and non-profit organizations need to demonstrate effectiveness, yet they are limited in their capacity to meet this demand.

Opportunity: Collaboration and partnerships—potential to pool evaluation resources across different funders and different funding interest.

Teaching and Learning Evaluation in Canada. (Chaytor, 2004)

Strength: Self-definitional capacity—the time for evaluation to define itself and establish itself as a distinct discipline is "now."

Weakness: Lack of disciplinary focus—disciplines view evaluation differently rather than having a common ground.

Threat: Disconnection—evaluation as part of management is under threat (e.g., lack of common ground).

Opportunity: Demand for skills—recognition of the value of evaluation and the demanding skills required.

This paper is an outsider's perspective of evaluation in Canada. Any errors or omissions are entirely unintentional. Comments, questions, and criticism are certainly welcomed. As new and additional information becomes available it will be made available in upcoming issues of the Journal of MultiDisciplinary Evaluation. If you would like to provide additional information or insight regarding the state of evaluation in Canada please contact the the author via e-mail at: christian.coryn@wmich.edu

 

 

References

Bradley, S. E. (2001). Evaluation in the government of Alberta: Adapting to the "new way." The Canadian Journal of Program Evaluation, 16(special issue), 29-44.

Cabatoff, K. (2001). The long march from evaluation to accreditation: Québec's new government management framework. The Canadian Journal of Program Evaluation, 16(special issue), 73-88.

Canadian Evaluation Society (2004). Canadian evaluation society website. Available at http://www.evaluationcanada.ca/site.cgi?s=1&ss=0&_lang=an

Canadian Evaluation Society (2001-2004). Canadian evaluation society quarterly newsletter. Available at http://www.evaluationcanada.ca/site.cgi?s=4&ss=3 &lang=an

Canadian Evaluation Society (2004). CES guidelines for ethical conduct. Available at http://www.evaluationcanada.ca/site.cgi?section=5&ssection=4&_lang= an

Canadian International Development Agency (2004). Canadian International Development Agency website. Available at http://www.acdi-cida.gc.ca/index.htm

Canadian Journal of Program Evaluation (2004). Canadian journal of program evaluation website. Available at http://www.evaluationcanada.ca/ site.cgi?s=4&ss=2&_lang=an

Cousins, J. B. (2004). Personal communication.

Evaluation Data and Development (2004). Evaluation Data and Development website. Available at http://www11.hrdc-drhc.gc.ca/pls/edd/hrdc.main

Gauthier, B., Barrington, G., Bozzo, S. L., Chaytor, K., Cullen, J., Lahey, R., Malatest, R., Mason, G., Mayne, J., Myers, A., Porteous, N. L., & Roy, S. (2004). The lay of the land: Evaluation practice in Canada today. The Canadian Journal of Program Evaluation, 19(1), 143-178.

Health Canada (2004). Health Canada website. Available at http://www.hc-sc.gc.ca/

Industry Canada (2004). Industry Canada website. Available at http://www.ic.gc.ca/

Hicks, K. (2001). Program evaluation in the government of the Northwest Territory, 1967-2000. The Canadian Journal of Program Evaluation, 16(special issue), 107-114.

Long, B. & Kishchuk, N., (1997). Professional certification: A report to the national council of the Canadian evaluation society on the experience of other organizations. Canadian Evaluation Society.

McDavid, J. C. (2001). Program evaluation in British Columbia in a time of transition. The Canadian Journal of Program Evaluation, 16(special issue), 3-28.

Mowry, S., Clough, K., MacDonald, B., Pranger, T., & Griner, D. (2001). Evaluation policy and practice in provincial governments province of Prince Edwards Island. The Canadian Journal of Program Evaluation, 16(special issue), 89-100.

Ross, A. (2001). Evaluation in Newfoundland: Then is then now is now. The Canadian Journal of Program Evaluation, 16(special issue), 101-106.

Segsworth, B. (2001). Evaluation policy and practice in Ontario. The Canadian Journal of Program Evaluation, 16(special issue), 59-72.

Social Sciences Research Council (2004). Social Science Research Council website. Available at http://www.sshrc.ca/

Stierhoff, K. A. (1999). The certification of program evaluators: A pilot survey of clients and employers. Canadian Evaluation Society.

Statistics Canada (2004). Statistics Canada website. Available at http://www.statcan.ca/start.html

Transport Canada (2004). Transport Canada website. Available at http://www.tc.gc.ca/

Treasury Board of Canada Secretariat (2004). Treasury Board of Canada Secretariat website. Available at http://www.tbs-sct.gc.ca/

Warrack, B. (2001). Program evaluation in the Manitoba government: past, present, and future. The Canadian Journal of Program Evaluation, 16(special issue), 45-58.

 


Evaluation in Europe: An Overview

Daniela Schröter[5]

 

The Landscape of European Evaluation

The umbrella organization of evaluation in Europe is the European Evaluation Society (EES). Founded in 1994 in Hague (Netherlands), the EES elects presidents for two-year mandates and provides a central secretariat for two to three years in different locations. The EES welcomes all individuals interested in evaluation, professionally or academically. Members of the EES receive a Newsletter,[6] a one year subscription to Evaluation: the International Journal of Theory, Research and Practice, and reductions on EES conference fees and other activities.

The web site of the EES provides a good overview about the evaluation community including lists of European and international evaluation associations and networks, evaluation journals, events, and other online resources. Currently, the EES provides links to 13 national or multinational European organizations as well as 5 regional networks within the United Kingdom (see Figure 1). A Portuguese evaluation society will be established (News from the Community, 2004) and listserv discussions of the German Evaluation Association (DeGEval) indicate that an Austrian Evaluation Society has been formed.

European Evaluation Society, Danish Evaluation Society, Finnish Evaluation Society, French Evaluation Society, German Evaluation Society, International Program Evaluation Network, Irish Evaluation Network, Italian Evaluation Association, Polish Evaluation Society, Spanish Public Policy Evaluation Society, Swedish Evaluation Society, Swiss Evaluation Society, UK Evaluation Society (the following are regional UK networks: Cymru Evaluation Network, Scottish Evaluation Network, London Evaluation Network, Midlands Evaluation Network, North West Evaluation Network), Walloon Evaluation Society

Figure 1. National and Multinational Evaluation Societies in Europe

Evaluation in Europe appears to be highly influenced by the political environment. One of the most constraining elements to effective communication across the European evaluation community is the diversity of language. In addition, Elliot Stern (2004, p. 9-10) referred attendees at the 5th conference of the European Evaluation Society 2003, to four contextual dimensions that challenge and shape evaluation in contemporary Europe specifically in public policy and civil society. These are (1) national specificity or convergence (identity), (2) cultural diversity and its limit (solidarity), (3) decentralization or supranational solutions (legitimacy), and (4) the strong state with the weak means (complexity). The task of the EES is to help minimize and overcome any barriers the European evaluation community faces.

The EES holds conferences biennially. From September 30 to October 2, 2004, the sixth conference took place in Berlin, Germany with the title Governance, Democracy and Evaluation. There were about 423 evaluators (as indicated on listserv discussions of the German Evaluation Society); 334 of these stem from 36 countries and 5 organizations and presented on issues related to Governance, Democracy and Evaluation (see List of Presenters). More specifically the call for papers welcomed contributions related to program complexity, accountability, standards and guidelines, policy implementation, knowledge management, and education in evaluation that address needs of national and international level governments. The program of the conference reflects these issues, and the proceedings may shed light on the specifics.

The Development of Evaluation in Europe

Leeuw (2004) asked if European evaluation is still an “infant industry” and illuminates the European type of “evaluation industry”. His book chapter will serve as the foundation for the following sections. Rist, Furubo, and Sandahl’s (2002)[7] assessed countries worldwide on eight dimensions to determine levels of development in evaluation. The dimensions included:

 

·        Evaluation activity

·        Supply of evaluators

·        Training capacity

·        National discourse

·        Organized evaluation meetings

·        Evaluation infra-structure within the public sector

·        Evaluation infra-structure within parliament

·        Evaluations carried out by Supreme Audit Offices (see Leeuw 2004, 63).

While not all European countries were assessed within this study results indicated most intense evaluation efforts in North and West European countries. However, data was either insufficient or indicated only moderate training capacity for evaluators in Europe, which as Leeuw argues is plausible in view of the fact that evaluation has not been established well at the university level in form of evaluation studies. On the other hand, national discourse and organized meetings were available and as indicated by other contributions in this issue of JMDE not only stimulate debate and discussions, but also provide platforms for trainings. Additionally, Rist et al. found that evaluation in the public sector was more widely available than evaluation within parliament. Last but not least, evaluations carried out by Supreme Audit Offices were most developed in Sweden, the Netherlands, and the United Kingdom.

Historically, Sweden, Germany, and the United Kingdom are considered first and second wave evaluation countries where evaluation developed in the 70s and 80s. Since then, many other European countries have been established as frontrunners in evaluation, especially the Scandinavian countries (Denmark, Norway, Finland), other West European countries (Ireland, the Netherlands, and Switzerland) as well as South European countries (France, Spain, and Italy), and the numbers are growing.

The European Evaluation Market

Based on a study conducted in 1999, Leeuw describes the European evaluation market as a growing market. While the response rate in the study was rather limited, findings indicated that the evaluation market was growing faster on the European and national levels than in regions. Most evaluations conducted were related to policy and respondents indicated that methods utilized usually derived from the evaluators’ specific subject areas. Moreover, the regional evaluation market was perceived as rather fragmented and it was thought that international competition on the European evaluation market would be constrained due to cultural factors. For instance, Leeuw pointed out that one respondent said that it was even hard to hire a British evaluator for an Irish setting. This is due to language constraints and an understanding of the different organizational cultures. On the European level, this leads to evaluations which are conducted by teams of evaluators from multiple nations. Leeuw refers to such arrangements as “(quasi)professions” (p.68). Moreover, Leeuw argues that top-down processes thwart good evaluation practice. While evaluation in North America is outcome and impact oriented, European evaluation focuses on resources and administrative processes. Leeuw points out that there may be a slight drift into auditing, focusing on form rather than substance.

 

 

 

Evaluation on the European Union Level[8]

On the European level, initial forms of program evaluations began in the 80s, were focused on research and technology development programs, and were based on practices prevalent in first wave evaluations. A shift occurred in 1995, when a new evaluation scheme was introduced that demanded evaluation of research and framework programs in form of annual monitoring and five-year periodic assessments. Leeuw states:

The assessments can be understood as a combination of an ex post evaluation of the previous program, an intermediate evaluation of the current program and an ex ante appraisal of future activities (2004, 69).

However, while evaluation on the Union level always focused on regulatory policy, formal evaluation systems or databases for the Directorates General are insufficient and “the Council and Parliament pass[ed] a small number of ‘sunset’ regulations which include a formal evaluation clause given a deadline (especially in the field of Competition Policy)” (Leeuw, 2004, 69). The results of reporting, however, are neither called nor could be classified as evaluation. Other foci, especially cost-effectiveness and cost-benefit evaluations were yielded by management reforms in the 90’s and are “supervised by the Directorate General for Budgets and Financial Control” (Leeuw, 2004, 71).

In 1996, steps for more systematic evaluations of policies were undertaken and a “decentralized model in which the operational Directorates General are responsible for establishing systematic evaluation procedures for the programs they are executing” was developed to improve evaluation practice (Leeuw, 2004, 71). As a result, each Directorate had to designate one evaluation official who is responsible for establishing an annual evaluation plan and for determining program to be evaluated. The Directorates’ evaluation plans are assembled into the “Commission’s Annual Evaluation Program”. The Directorate General for Budget “coordinates evaluation activities and maintains an overview of the evaluation findings across the Commission services. It also provides methodological guidance and support, helps with procurement of evaluation expertise and maintains evaluation networks within and outside the Commission (see website). Unique features of the Evaluation Commission include a broad definition of the concept of evaluation and its direct link to budget:

Not only does it [evaluation] encompass ex post and midterm evaluation, but it also cover ex ante exercises… evaluation projects are to be framed so that they correspond to identifiable entities in the Community budget and to be timed so that results are available when they are relevant for budgetary decisions (Leeuw, 2004, 72).

Current Issues in European Evaluation[9]

Leeuw refers to different elements of current developments in Europe, including an increasing importance of civil societies, strengthened public management, and “polity”. The most interesting aspect here is polity, especially because distinct political traditions in European nations need to be considered. Moreover, Leeuw refers to the valuing component of evaluation, which is especially inherent in political processes in which decisions are being made, values are chosen, and priorities set. However, the traditional practice of social research is challenged in their value-free doctrine.

Most central topics for evaluation within Europe as stated by Leeuw are:

·        The increasing importance of evaluation for civil society

·        Evaluation for Parliaments (Do parliament decisions have effects?)

·        Evaluation for public policy partnerships

·        Decentralization of evaluation

·        Potentials for evaluation of social programs from a non-managerial standpoint

·        Evaluation of information and communication technology products, processes, and outcomes (web-based communication, training, the internet as knowledgebase)

·        Auditing versus evaluation

·        Evidenced–based evaluation

·        Learning from evaluation

·        Effective implementation and utilization of performance management systems in public management.

Overall, evaluation appears to be a vast growing market in Europe. However, as a discipline evaluation is still an “infant,” not only on the European level but internationally.

References

Leeuw, F.L. (2004). Evaluation in Europe. In: Stockmann, R. (Ed.). Evaluationsforschung: Grundlagen und ausgewaehlte Forschungsfelder. 2nd edition. Opladen: Leske + Budrich.

News from the community (2004). In: Evaluation: The International Journal of Evaluation Theory, Research and Practice, 10(3): 380-381.

Stern, Elliot (Ed.). Evaluation: The International Journal of Evaluation Theory, Research and Practice, 9(4)-10(3).

Stern, Elliot (2004). What shapes European evaluation: A personal reflection. In: Evaluation: The International Journal of Evaluation Theory, Research and Practice, 10(1): 7-15.

The European Evaluation Society (2004). The European Evaluation Society website. Available at: http://www.europeanevaluation.org/

 


Evaluation Activities in the United Kingdom

John S. Risley

 

General Summary of Activities

The UK Evaluation Society (UKES; www.evaluation.org.uk) was founded in 1994 and is composed of over 150 individual and corporate members. Most of these are individual members. UKES hosts an annual conference each year in December and jointly conducts seminars and conferences with other professional organizations. The society also sponsors an e-mail discussion list, Eval Chat, publishes a thrice yearly newsletter, The Evaluator, and produces Evaluation: The International Journal of Theory, Research and Practice.

UKES has five regional networks. Three of these networks, the Scottish Evaluation Network, the London Evaluation Network, and the North West Evaluation Network are established. The other two, the Cymru Evaluation Network (Wales) and the Midlands Evaluation Network are just forming.

The UKES website offers a host of information and links on evaluation topics, including:

·        evaluation guidelines for good practice from different national evaluation associations,

·        a list of postgraduate courses on evaluation taught throughout the U.K.,

·        links to 21 national/regional evaluation society websites,

·        an evaluation glossary (including an entry on “chatty bias”)

·        a short but wide-ranging bibliography of evaluation books

Evaluation: The International Journal of Theory, Research and Practice

The journal Evaluation is published quarterly by Sage. Through the end of October it is available free online at evi.sagepub.com. I reviewed the last two years of Evaluation (the January 2003 issue through the July 2004 issue) and categorized each article according to Lori Wingate’s adaptation of Michael Scriven’s analogy for understanding disciplines. Wingate identified four categories of focus for journal articles—practice, methods, theory, metatheory—that I used below and one category—history—that I eliminated because no articles fit the description.

Practice issues dominated the 37 articles from the last two years (48.6 percent). The practice articles mainly dealt with the related issues of evaluation use and stakeholder participation. An article by Taut & Brauns (2003) examines social and psychological explanations for resistance to evaluation and offers strategies for overcoming evaluation resistance.

Many articles I categorized in the practice area concerned evaluation in different fields—healthcare, bidding for public services, welfare policy. These articles did not discuss different evaluation approaches or models, so I did not categorize them under theory.

Over one-fifth of the articles (21.6 percent) concerned theory. Three of these eight articles concerned theory-based evaluations—with two generally favorable and one generally unfavorable toward the approach—while other evaluation approaches addressed included qualitative, desk screening and implementation evaluation. Hearn, Lawler and Dowswell (2003) addressed the dominance of the positivist approach to most healthcare evaluation and argued that an inclusion of “nonpositivist, qualitative, and process-oriented evaluation” would improve our understanding of health programs and policies.

I categorized six articles (16.2 percent) as methods articles. Interestingly, all of these articles focused on quantitative methods of data collection and analysis. Sverdrup (2003) discussed the use of time-series databases of complaints data to evaluate laws and regulations.

The metatheory category included five articles (13.5 percent) across 2003-2004. Virtanen and Uusikylä (2004) address the “paradigm crisis” in evaluation that stems from evaluators’ different assumptions about causality. These authors describe four alternative models (which they term ideal models) for evaluation considering: 1) how explicitly causality has been taken into account, and 2) how well the model enhances public-sector accountability.

The model reflecting both a strong link between causality and the evaluation design and an emphasis on public accountability is termed “transparent democracy”. “Scientific inquiry” signifies a strong link between the evaluation design and causality without an emphasis on accountability. The “explorative inquiry” model is characterized by a high degree of emphasis on accountability and a difficulty in distinguishing causal effects. Finally, an evaluation using the “symbolic evaluation” model serves a symbolic purpose rather than a “true pursuit of learning.” (89)

References

Hearn, J., Lawler, J., & Dowswell, G. (2003) Qualitative evaluations, combined methods and key challenges: General lessons from the qualitative evaluation of community intervention in stroke rehabilitation. Evaluation. 9: 30-54.

Sverdrup, S. (2003). Towards an evaluation of the effects of laws: Utilizing time-series data of complaints. Evaluation. 9: 325-339.

Taut, S., & Brauns, D. (2003). Resistance to evaluation: A psychological perspective. Evaluation. 9: 247-264.

Virtanen, P., & Uusikylä, P. (2004) Exploring the missing links between cause and effect: A conceptual framework for understanding micro–macro conversions in programme evaluation. Evaluation. 10: 77-91.

 


Evaluation in Eastern Europe and the Middle East

P. Cristian Gugiu

 

The state of evaluation in Europe is ever changing. In 1994, there was only one national evaluation society in Europe. Since then, the evaluation scene in Europe has blossomed to include 10 or 11 national societies—most of which are located in West Europe. Recent efforts, most notably by the European Evaluation Society, have been directed towards integrating all of these organizations under one umbrella. Through training and capacity building, the European Evaluation Society hopes to increase the number of engaged professionals in Europe, develop an academic support base, and strengthen the links to the policy community.

Compared to Western Europe, the state of the evaluation field in the Middle East appears to be less developed. In some way, the state of evaluation in the Middle East resembles that of Eastern Europe. Few of these countries have created national evaluation societies, taught evaluation in schools of higher learning, or published evaluation journals. The present paper intends to examine the current state of evaluation in Europe and the Middle East.

Evaluation Journals and Newsletters

East European Journals

Several representatives of the European Evaluation Society (EES) report that no one knows of any journal or newsletter publications in Eastern Europe.

 

Middle East Journals

According to Barbara Rosenstein, Ph. D., Chairperson of the Israeli Association for Program Evaluation (IAPE), the IAPE has published, to date, eight newsletters, in both Hebrew and English on evaluation.

Israeli Journal: Studies in Educational Evaluation

Studies in Educational Evaluation (SEE) is published in English. The majority of articles were not published by Israelis. Authors were dispersed throughout the world including the United States, United Kingdom, Australia, Spain, the Netherlands, and Germany.

A great many of the articles were purely research articles, a few of them described an evaluation case study, and a fair number of them discussed a specific methodology that could be used in evaluation.

Evaluation Societies

European Evaluation Society (http://www.europeanevaluation.org/)

The primary goal of the European Evaluation Society (EES) is to promote theory, practice and utilization of high quality evaluation especially, but not exclusively, within the European countries. This goal is obtained by bringing together academics and practitioners from all over Europe and from any professional sector, thus creating a forum where all participants can benefit from the cooperation and bridge building. The society was founded in Hague in 1994. The first official board was elected in autumn 1995 and started its work in January 1996.

EES held its sixth conference on September 30 to October 2, 2003 in Berlin, Germany. The conference took place at the University of Applied Sciences and featured a total of 334 presenters from 36 countries.

Over three-quarters of the presenters came from West European countries including Belgium (2.4 percent), Denmark (2.7 percent), Finland (4.2 percent), France (4.8 percent), Germany (9.3 percent), Greece (0.3 percent), Iceland (0.3 percent), Ireland (2.1 percent), Italy (15.9 percent), Netherlands (5.4 percent), Norway (1.5 percent), Portugal (2.7 percent), Spain (5.7 percent), Sweden (5.7 percent), Switzerland (4.5 percent), and the United Kingdom (9.0 percent). The remaining presenters included countries from Asia (Japan, 0.6 percent; Korea, 0.9 percent), Australasia (Australia, 2.4 percent; New Zealand, 0.6 percent), Africa (Angola, 0.3 percent; Guinea Bissau, 0.3 percent; Kenya, 0.3 percent; Nigeria, 0.9 percent), East Europe (Austria, 2.4 percent; Bosnia and Herzegovina, 0.3 percent; Czech Republic, 0.3 percent, Poland, 1.2 percent), the Middle East (Egypt, 0.3 percent; Israel, 0.3 percent, Palestine, 0.3 percent), North America (Canada, 0.9 percent; United States, 5.1 percent), and Latin America (Colombia, 0.6 percent; Mexico, 1.8 percent).

There were slightly more male presenters than female presenters.[10] However, this statistic was primarily influenced by the large number of West European presenters. Five of the seven other regions had an equal or greater number of female presenters than male presenters.

An examination of the type of jobs presenters worked in revealed that the majority of them worked for a university or college in their native country. The two next largest groups included people who worked in private industry or for the government.[11] It was interesting to note the differences in distribution of job type among the eight regions. For seven of the eight regions, presenters typically worked at a university. However, for Eastern Europe, the majority of presenters came from private industry. Possible explanations for this difference include the lack of university programs specializing in evaluation, the low number of professional evaluation associations, and historical factors such as socialism and recent wars.

Polish Evaluation Society (http://www.pte.org.pl/)

The Polish Evaluation Society (PES) began in 2001 and set out to build an evaluation culture and popularize evaluation as a social and democratic process. To this end, it sought to (a) organize studies, courses and trainings; (b) conduct evaluation research; (c) exchange experiences with other societies, institutions and organizations; (d) organize meetings, seminars and conferences, (e) publish in the area of evaluation, and (f) provide consulting and advising services.

The Polish Evaluation Society has very strict rules as to the educational qualifications of its members. Most members are still strongly connected with the academic environment, either via didactic activity or scientific research (Warsaw University, Lublin Catholic University, B. Jański School of Administration and Enterprise). Members of PES continuously enrich their knowledge by taking part in trainings, seminars and conferences both in Poland and abroad (UK, The Netherlands, Denmark) and also by co-operation with other similar organizations in the world. (United Kingdom Evaluation Society, European Evaluation Society, IOCEInternational Organization for Cooperation in Evaluation, PLS Ramboll ManagementDenmark, Eureval-C3ECentre for European Evaluation ExpertiseFrance).

Members of PES are professional evaluators who also conduct marketing research and other research on social character. They have wide experience in the field of the evaluation which they gained in the process of conducting a variety of research for Polish and international organizations such as Polish Children and Youth Foundation, Public Interest Institute, government organizations such as European Integration Committee, the ministry of Education, service sector companies such as Daewoo, and EU institutions such as European Parliament and European Commission. Members of PES use different paradigms and research perspectives. A Rich variety of the activities and approaches is an advantage of this organization.

 

Romanian National Assessment and Examination Service (http://www.edu.ro/snee.htm)

The National Assessment and Examination Service (NAES) was established in 1998 by the Romanian Government as the first national, independent body providing professional expertise in educational assessment and examinations in Romania. NAES is responsible for the design and implementation of the new educational evaluation system, namely for: (a) current assessment in pre-university education; (b) school leaving examinations (Capacitate exam and Baccalaureate exam); (c) national assessments at the end of educational cycles (now at the end of 4th grade); and (d) continuous teacher training in the field of assessment and examinations.

NAES is actively involved in national and international projects (e.g. the British Council, QUATRO FontysPTH Eindhoven) and maintains professional contacts with universities, research institutes, governmental and nongovernmental institutions and organizations in the field (e.g. CITOThe Netherlands, EDCUSA etc.). Their headquarters in Bucharest provides assessment technologies and facilities for development projects and studies in assessment and examination and their staff offers competence and expertise for cooperation to all those interested, in Romania and abroad.

Israeli Association for Program Evaluation (http://www.iape.org.il/)

The Israeli Association for Program Evaluation (IAPE) is a non-profit, professional organization comprised of academics, practitioners and users of program and project evaluation in a variety of fieldspsychology, education, social services, health, business, and others. The goals of the organization include (a) increasing the use of program evaluation and its findings, (b) encouraging the development of the theory of program evaluation, (c) advancing the essential recognition of program evaluation as a means of improving the effectiveness of social and educational interventions, (c) promoting the recognition of program evaluation as a profession, (d) serving the communities and the populations involved in program evaluation, (e) contributing to the influence of program evaluation on decision making, (f) supporting and influencing evaluation practice in Israel, and (g) creating and developing professional ties among evaluators and users of evaluation in Israel. To this end, the IAPE has sought to (a) organize conferences focusing on issues of concern to the evaluation community; (b) create an electronic and regular mail network that provide information about issues concerning evaluation in Israel and abroad; (c) establish connections with evaluation organizations throughout the world; (d) participate in the worldwide forum of evaluation associations, (e) circulate of a list of members to evaluation consumers in Israel; and (f) publish a newsletter containing articles, discussions, and events of interest to the evaluation community in Israel.


Evaluation in Latin America and the Caribbean: An Overview of Recent Developments

Thomaz Chianca and Brandon Youker[12]

 

In the past ten years, evaluation, as a professional field, has undergone significant development in several countries in Latin America and the Caribbean (LAC). Four considerations provide clear evidence of such development: (1) establishment of professional evaluation organizations; (2) intensified dissemination of ideas and use of professional evaluation in the three key societal sectors: government, private, and philanthropic; (3) increased number of evaluation-related publications; and (4) growing establishment of short-term and graduate-level training programs in evaluation.

Evaluation Organizations

The first professional evaluation organization that was formed in the region was the Central American Evaluation Association (ACE) in 1989 that has had its headquarters, since then, in Costa Rica. ACE’s main objective is to foster the evaluation of programs and projects to improve efficacy and efficiency of the use of societal resources. The main activities developed by ACE include seminars, workshops, and courses to disseminate evaluation knowledge in Central America.

Only eight years later, in 2002 new evaluation professional organizations were established in LAC. Given their specific contexts, Brazil, Colombia and Peru opted to create networks of evaluators, less formal and more flexible organizations than associations or societies. United Nations agencies such as Unicef (United Nations Children's Fund), Unesco (United Nations Educational, Scientific and Cultural Organization), and IFAD (International Fund for Agriculture Development), along with international and indigenous nonprofit organizations (e.g., foundations, institutes, etc.), played a decisive role in the creation of these three evaluation networks. Most membership is comprised of evaluators working in social and educational programs funded either by the government or by foundations.

PREVAL (Program for Strengthening the Regional Capacity for Evaluation of Rural Poverty Alleviation Projects in Latin America and the Caribbean)—a joint effort between IFAD and, from 1995-2000, the Inter-American Institute for Cooperation on Agriculture (IICA), and, from 2000-2007, with the Centro de Estudios para la Promoción del Desarrollo (Center of Studies for Development Promotion)—has played an strategic role in the region since 1995, contributing directly to the creation of the national evaluation networks in Peru and Colombia. In its first two phases (1995-2000 and 2000-2004), PREVAL focused its work on strengthening the evaluation capacity of IFAD projects to reduce rural poverty in the region. In its third phase (2004-2007), PREVAL will broaden its objectives to work more closely with governments, organizations offering technical assistance in monitoring and evaluation, as well as national evaluation and monitoring networks/associations in the region. PREVAL has established an important network of evaluators working with projects aimed at alleviating rural poverty, and has produced an important body of knowledge in this area published in Spanish. It is also important to recognize the key role played by the International Organization for Cooperation in Evaluation (IOCE)—comprising all national and regional evaluation organizations around the globe—in fostering the establishment of evaluation organizations in the region. IOCE held an important planning meeting in Barbados with all major international leaders and its inaugural assembly in Lima, Peru in 2003[13]. The creation of both the Peruvian and the Colombian evaluation networks were strongly supported by IOCE.

In September 2003, representatives from the four existing evaluation organizations in the region got together in São Paulo, Brazil, to create the Latin American and Caribbean Evaluation Network (RELAC). The city of Lima, Peru, will host RELAC’s first evaluation conference in October 20-23, 2004. The conference’s theme (“Evaluation, Democracy, and Governability: Challenges for Latin America”) and main sessions (e.g., democratic evaluation, methods for evaluating human rights programs, evaluation capacity building in social initiatives, monitoring, evaluation and systematizing as a political and social process to strengthen democracy in LAC, etc.) reflect RELAC’s strong focus on promoting a social agenda for the region and having evaluation as a major tool. The conference will gather more than 100 evaluators from all over the region including representatives from eight countries that are trying to create their own national evaluation organizations: Argentina, Bolivia, Chile, Ecuador, Honduras, Nicaragua, Uruguay, and Venezuela.

There are at least four electronic discussion lists on evaluation in the LAC region: RELAC, PREVAL, the Brazilian Evaluation Network, and the ILPES/CEPAL.

It is not over-optimistic to assume that very soon we will witness a significant increase in the number of evaluation professional organizations in LAC.

Use of Professional Evaluation in Key Societal Sectors in LAC

There has been significant growth in the use of professional evaluations by the government, the nonprofit sector, and at least in the field of personnel evaluation, in large private businesses. In the government arena, initiatives related to national educational evaluation/assessment systems, innovations in government administration systems, and social development programs supported by international cooperation agencies are major factors influencing such growth.

In education, the establishment of evaluation mechanisms has been extensive from basic (K-12) to higher education in many countries within the region. In Brazil, for instance, the ministry of education has in place at least four major evaluation initiatives, applied countrywide, to assess the quality of education. In higher education there are two initiatives: (1) institutional evaluation (assessment of the general conditions of all higher education institutions in the country), (2) evaluation of undergraduate education (includes a national exam for senior students in each professional area and an accreditation strategy to renew or provide new licenses for universities and colleges). The other two are related to the basic education system: (3) national evaluation system for basic education (bi-annual assessment of the quality of K-12 schools, based on a national random sample), and (4) national exam for high school students (senior high school students have the option to take the exam that will serve as one of the criteria for acceptance in a university—similar to the ACT and SAT tests in the U.S.). Several countries also have official connections with regional as well as worldwide education assessment initiatives such as the Latin American Laboratory for Evaluating the Quality of Education, the International Program for Student Evaluation, and the World Education Indicators Program.

The idea of reducing the size of the state and making it more effective and efficient (state reform) has strongly influenced virtually all countries in the region. Such an idea brings along a strong push for the establishment of control systems on expenditures as well as for implementation of planned activities that usually involve monitoring and, to some extent, evaluation. Several countries have created structures, usually subordinated to the ministry of planning, that are in charge of dealing with monitoring and internal evaluation of governmental efforts. Examples of such structures are the System of Information, Evaluation, and Monitoring of Social Programs (Argentina); Secretary of Strategic Investments (Brazil); Interssectorial Committee for Modernizing Public Administration (Chile); National System for Management and Results Evaluation (Colombia); and the National Evaluation System (Costa Rica).

In the area of social development, virtually all programs supported by international cooperation agencies such as the World Bank, Inter-American Development Bank, World Health Organization, and United States Agency for International Development (USAID), are required to be evaluated both internally as well as by using external evaluators. These organizations have played a major role in introducing innovations in evaluation as well as advocating for the use of quality professional evaluations within government funded initiatives. Several examples of such evaluations are already publicly available from the agencies’ websites (e.g., USAID and the World Bank-Operations Evaluation Department (OED). The Latin American Institute for Social and Economic Planning (ILPES), subordinated to the United Nations Economic Commission for Latin America and the Caribbean (CEPAL), has been an important reference in providing evaluation support to country-level government evaluators by offering supporting materials (publications); evaluation training; and networking opportunities for professionals working in evaluations of governmental social-development programs in the region.

Initially influenced by international foundations investing in the region the fast—growing nonprofit sector in Latin America has increasingly used and advocated for the use of evaluation as a way of assuring quality of the programs they fund, and also to inform their decisions regarding strategic funding.

The W.K. Kellogg Foundation is one of the international foundations that have significantly invested in the development of evaluation in LAC. In 1995 and 1997, the foundation sponsored two groups of LAC evaluators (a total of approximately 40 professionals) in in-depth training programs in evaluation at The Evaluation Center—Western Michigan University. Some of the participants of such training opportunities are assuming leadership roles in evaluation in the creation of evaluation organizations in their home countries.

Several foundations and institutes are commissioning and/or developing evaluations throughout the region. In Brazil, for instance, some of the nonprofit organizations that are very active in evaluation include: Fundação Carlos Chagas, Fundação Cesgranrio, Instituto Ayrton Senna, Fundação ABRINQ Foundation, Instituto FONTE, Fundação Roberto Marinho, and Fundação IOCHPE.

Another interesting movement influencing the growth of evaluations in the third sector is the increasing number of private businesses investing in social initiatives, based on the idea of social responsibility. Such organizations have a different culture (focus on control and efficiency) from the nonprofit organizations investing in the sector, and have made an important push to support the establishment of monitoring and evaluation systems in the initiatives sponsored by them. In Brazil, the Grupo de Institutos, Fundações e Empresas—GIFE (Group of Institutes, Foundations, and Enterprises) and the Ethos Institute are examples of two large organizations created to mobilize funds and to support companies and private foundations who invest in social, cultural and environmental projects of public interest, making such organizations partners in building a sustainable and fair society.

The extent of evaluation use in the private sector is not very public. It is evident that several corporations and other private business have made serious efforts to evaluate their products, projects and personnel. Reports on such efforts, however are not easily accessible and the evaluators working in this area have almost no contact with other evaluators working in the public and nonprofit sectors. No doubt more extensive exchange of experiences between these professionals has great potential to be beneficial to all, but some important barriers such as prejudices from both sides (e.g., ‘private sector only look at profits;’ ’public and nonprofits are always inefficient’) need to be overcome before such approximation has any chance of succeeding.

Body of Original Publications in Evaluation

Though there are virtually no evaluation specific journals in LAC, there are several journals related to education, health, and social sciences with strong evaluation content. Some examples include:

·     La Revista de Ciencias Sociales (Journal of Social Sciences—Costa Rica)

·     Revista Ensaio – Avaliação e Políticas Públicas em Educação (Evaluation and Public Policy in Education—Brazil)

·     Estudos em Avaliação Educacional (Educational Evaluation Studies—Brazil)

·     Cadernos de Saúde Pública (Journal of Public Health—Brazil)

·     Revista Avaliação Psicológica (Journal of Psychological Evaluation—Brazil)

·     Revista da Rede de Avaliação Institucional (Journal of the Institutional Evaluation of Higher Education Network—Brazil)

·     Cuadernos de Investigación de la Escuela de Gerencia Social (Journal of Inquiry of the School of Social Management—Venezuela)

·     Revista del Instituto de Investigaciones en Ciencias de la Educación (Journal of the Education Science Investigation InstituteArgentina)

·     Acción y Reflexión Educativa (Educative Action and ReflectionPanama)

·     Planejamento e Políticas Públicas (Planning and Public Policy—Brazil)

·     Revista de Administração Pública (Journal of Public Administration—Brazil)

The footnoted social science journals[14] have regularly published the intellectual products of LAC evaluators.

It is critical to acknowledge the substantial collection of accessible evaluation publications such as books, manuals, newsletters, technical reports, etc. that are available in most Latin American countries. There are several websites such as the Latin American Institute for Social and Economic Planning (ILPES), and the Programme for Strengthening the Regional Capacity for Evaluation of Rural Poverty Alleviation Projects in Latin America and the Caribbean (PREVAL) that provide an extensive collection of evaluation publications in the field of evaluation throughout the region.

There are two excellent annotated bibliographies that provide published reference materials that address several aspects of evaluation in LAC. The first publication, The Annotated Bibliography of International Programme Evaluation, edited by Russon & Russon[15] has a chapter by Antoinette B. Brown and Ada Ocampo, on Latin America and the Caribbean. The reviewed documents (books, manuals, journal articles, etc.) are grouped into three sections: (1) manuals and guides, (2) participation, and (3) case studies and evaluation reports. The second publication, Annotated Bibliography on Project Evaluation by Viñas[16] which includes a broad range of documents in the review, is divided into fourteen categories: (1) definition of basic concepts; (2) design, implementation and evaluation; (3) environment impact; (4) evaluation approaches; (5) evaluation design; (6) gender; (7) indicators; (8) methods and instruments for information collection and analysis; (9) monitor and evaluation systems; (10) organizational development; (11) participation by beneficiaries; (12) producing reports and presenting conclusions; (13) types of evaluation; and (14) use of evaluations.

Capacity Building in Evaluation

There are a few LAC universities and training institutions that offer masters level programs, specifically in evaluation. At the masters’ level, there are at least five universities offering such program:

·     Professional Masters in Evaluation of Social Programs and Projects. Universidad de Costa Rica. San José, Costa Rica.

·     Masters in Socio-Economic Evaluation of Investment Projects. Universidad Panamericana. Mexico City, Mexico.

·     Masters of Science in Project Management and Evaluation. University of the West Indies—Cave Hill Campus. Bridgetown, Barbados.

·     Masters in Project Evaluation. Universidad del CEMA, Buenos Aires, Argentina.

·     Masters in Social Projects Evaluation. Universidad Autónoma de Guadalajara. Guadalajara, Jalisco, México.

At the certification level there are quite a few programs offered in different countries including:

·     Course on Evaluation of Social Programs and Projects. Centro de Empreendedorismo Social e Administração em Terceiro Setor—CEATS (Center of Social Entrepreneurship and Administration for the Third Sector—Brazil) and the FONTE Institute.

·     Diploma in Evaluation of Projects. Universidad de Concepción. Concepción, Chile.

·     Diploma in Evaluation of Social Projects. Pontificia Universidad Católica de Chile. Santiago, Chile.

·     Diploma in Planning and Evaluation of Projects. Universidad de Chile. Santiago, Chile.

·        Diploma in Planning and Evaluation of Socioeconomic Projects. Centro de Análisis y Evaluación de Política Pública—CAEP—Monterrey, Mexico. (Center of Analysis and Evaluation of Public Policy)

·        International Certificate of Project Planning, Evaluation, and Management—Inter-American Development Bank. Centro de Investigaciones Territoriales del Ecuador—CITE. (Territorial Investigation Center of Ecuador).

·     Post Graduate in Formulation and Evaluation of Projects. Universidad Americana. Managua, Nicaragua.

There are also several short-term evaluation training courses facilitated by different organizations within the region. Some of the best sources to identify such training opportunities include: (a) Nota Informativa del ILPES sobre Evaluación de Proyectos y Programas (ILPES Informative Note on Program and Project Evaluation); (b) PREVAL; and FONTE Institute. The following is a sample of the recently offered short-term courses in some LAC countries:

·     X International Course on Planning and Evaluating Public Investment Projects. Offered by CEPAL/ILPES. Sept 27 to Oct 22, Santiago, Chile.

·     Internet-based course on Planning and Evaluation of Agricultural and Agri-Industrial Projects. Offered by REDCAPA and Austral University of Chile. Sep 1 to Nov 30 2004.

·     International Course on Logic Model, monitoring and Evaluation. Offered by ILPES/CEPAL and the Spanish Cooperation Agency (AECI). Jun 21 to Jul 2, 2004. Cartagena de Indias, Colombia.

·     International Course on Use of Socio-Economic Indicators for the Evaluation of Impact of Poverty Reduction Programs. Offered by ILPES/CEPAL and the Spanish Cooperation Agency (AECI). May 3—14. Santa Cruz de la Sierra, Bolivia.

·     Utilization-Focused Evaluation by Michael Quinn Patton. Sponsored by the Brazilian Evaluation Network, UNICEF-Brazil, and FONTE Institute. Salvador, BahiaBrazil. March 2004.

·     Collaborative Evaluation by Rita O’Sullivan. Sponsored by the Brazilian Evaluation Network, UNICEF-Brazil, and FONTE Institute. São Paulo, SP—Brazil. September 2003.

Final Comments

The report makes no claims to be comprehensive and does lack significant information, mainly about the state-of-art of the evaluation field in the Caribbean countries. We hope to fulfill the existing information gaps in the next revisions of and addenda to this paper.

It does, however provide unquestionable evidence of the impressive advances the whole region has made in the evaluation field in the recent past. Evidently, even though not to the same degree in each country, it is reasonable to say that basic conditions have been established to make such advances even more comprehensive and effective in the future.

The current efforts to establish national and regional evaluation organizations, the growing number of quality publications in both Spanish and Portuguese on evaluation, the increasing use of professional evaluation by different organizations in all societal sectors and the broad recognition of evaluation as important for improving society are some of the factors influencing such advances. One major challenge still to be faced in order to have evaluation in a better position as a recognized professional field is the creation of more formal graduate-level training for evaluators in a wider range of countries.

This paper is a work in progress that will be modified and/or improved as we gain new information. If you would like to provide additional information or point out any errors or misunderstanding in the text, please do not hesitate to contact Thomaz Chianca (thomaz.chianca@wmich.edu) or Brandon Youker (brandon.w.youker@wmich.edu).

 



[2] Full transcript of Patton’s lectures are available at http://www.afrea.org/.

[3] Xceval-subscribe@yahoogroups.com

[4] The author would like to thank J. Bradley Cousins (editor of the Canadian Journal of Program Evaluation) for his insight and assistance.

[5] This is a first draft of an illustration of evaluation in Europe. The author anticipates receiving feedback from those evaluators who are experts on issues of European evaluation. Please send any comments, recommendations, feedback, and additional information to the state of art in Europe or individual European countries to: daniela.schroeter@wmich.edu. I appreciate your feedback and will gladly revise this paper accordingly.

[6] At this point in time there are three newsletters available online at http://www.european evaluation.org/news/newsletters.html. All of these newsletters are specifically addressing conference related issues in preparation for the biennial events.

[7] The author did not have the study at hand and utilized Leeuw (2004) summary and comments of that study only.

[8] Based on Leeuw (2004).

[9] Based on Leeuw (2004)

[10] The gender of the presenter was determined by (a) an inspection of their name, (b) an internet search for a website that mentioned the presenter’s gender, or (c) an internet search for the gender most common for the presenter’s first name. Consequently, there may be a small degree of inaccuracy in the categorization of gender.

[11] The type of job the presenter worked in was determined by the e-mail domain they provided to EES or by an internet search on their name or the name of the organization listed as their employer with EES. For four presenters, however, it was not possible to determine their job type.

[12] The authors want to thank Craig Russon, Marco Segone, Rogério Silva, Xinia Picado, Emma Rotondo, and Audley Espeut for their great comments and suggestions to improve an early version of this paper.

[13] Russon, C. & Love, A. (2003) The Inaugural Assembly of the International Organization for Cooperation in Evaluation: The Realization of a Utopian Dream. Occasional Paper #20. The Evaluation Center, Western Michigan University: Kalamazoo, Michigan.

[14] A very useful source to find out about Latin American journals in education and related areas is the Centro de Recursos Documentales InformáticoCREDI (Center for Digital Documental Resources), sponsored by the Organization of Ibero-American States.

[15] Russon C. & K. Russon. Eds. (2000) The Annotated Bibliography of International Programme Evaluation. Norwel: Kluwer, Academic Publishers

[16] Viñas, V.E. (1997). Annotated Bibliography on Project Evaluation. San José, Costa Rica : International Fund for Agricultural Development (IFAD), Inter-American Institute for Cooperation on Agriculture (IICA), Programme for Strengthening the Regional Capacity for Evaluation of Rural Poverty Alleviation Projects in Latin America and the Caribbean (PREVAL).