JMDE

Japan Evaluation Society: Pilot Test of an Accreditation Scheme for Evaluation Training

Masafumi Nagao

The Japan Evaluation Society (JES) has conducted a pilot test of an accreditation scheme for short-term evaluation training programs. The program chosen for this test was a 4-day school evaluation training course organized by a public teacher training center in cooperation with Hiroshima University. The course was designed to impart functional competence to school teachers for co-ordinating self-evaluation exercises in their schools. The course contents were adapted from the ‘Essential Skills Series (ESS)’ program of the Canadian Evaluation Society (CES), which provided technical assistance for the formulation of the course design, preparation of the textual materials and actual conducting of the course, including on-site guidance and evaluation of its results on the basis of a formal agreement with JES. The course was given twice—first in July 2003 and then in August 2004. A special committee established in JES elaborated tentative procedures for accreditation and examined reports submitted by the course organizer and a JES member dispatched to observe the course. The committee has judged that the course cleared the hurdle for accreditation and, based on this pilot test, is making a recommendation to JES Board that an Accreditation Scheme should be formally established. JES believes that its accreditation scheme will, by virtue of the norms it sets for evaluation training, have a direct impact on the quality of the short-term evaluation training programs conducted by public sector bodies and specialized training organizations. It also hopes that the scheme will lead to multiplication of evaluation training programs by providing qualification incentives to individuals interested in evaluation training. In addition to school evaluation, aid evaluation and government performance evaluation are considered target areas for the accreditation scheme.

Masafumi Nagao is a professor at the Center for the Study of International Cooperation in Education (CICE), Hiroshima University and can be contacted at nagaom@hiroshima-u.ac.jp.

Aotearoa/New Zealand Starting National Evaluation Conferences

Pam Oliver, Maggie Jakob-Hoff, and Chris Mullins

Aotearoa/New Zealand had its first national evaluation conference in September 2004. Organised by the Auckland Evaluation Group, the meeting was held at the Tauhara Centre, near Lake Taupo (central North Island). This was chosen partly because it is a central location for North Island evaluators to get to, and partly because of the nature of the conference centre itself—a retreat and spiritual centre located in a quiet setting with native bush and overlooking the lake.

The conference theme in 2004 was “Radical directions, gnarly questions and half-baked ideas” and the programme included a keynote workshop, several other facilitated workshops and discussion groups, and a number of more informal discussion groups using an Open Space Technology process. Twenty-five people attended the conference and spent two and a half days engaged in discussions which participants found inspiring and highly worthwhile.

Entitled “Really useful stuff”: A new role for evaluators in building evaluative capacity in New Zealand, the keynote workshop was presented by Bill Ryan, Associate Professor and Director of Programmes in Victoria University’s School of Government in Wellington. It was followed by an engaging discussion which focused on participants’ endeavours to date at building evaluation culture into their everyday evaluation practice, including examples of both successful attempts and frustrations, and suggestions about what evaluators would like to be able to do, in an ideal evaluation environment.

In addition to the programmed workshops—ranging from “looking at ways of educating our clients around evaluation tendering”, and “aspects of ethical decision-making in evaluation”, to “building reflective practice through evaluation”—the Open Space events also resulted in some focused discussion on topics of key importance to the development of the profession, such as “who evaluates the evaluators?”

A second conference is scheduled for June 29^th-July 1^st, 2005 at the Tauhara Centre and is being organized jointly by members of the Auckland and Wellington evaluation groups. More information will be available in the next few weeks.

Conference Explores the Intersection of Evaluation and Research with Practice and Native Hawaiian Culture

Matthew Corry

One of the great potential sources of innovative thinking in evaluation is events where indigenous peoples come together to explore the ways in which evaluation and research can be designed in a way that is culturally relevant and useful to the communities served by these activities.

In recent years, there have been several hui (gatherings) where Native Hawaiian and New Zealand Māori researchers and evaluators have come together to explore this theme. Currently, a Māori/Native Hawaiian working group is compiling a series of papers on the topic for possible publication in a monograph series.

Last October, the Policy Analysis & System Evaluation (PASE) department of Kamehameha Schools organized a conference at their Hawai‘i (Big Island) campus. The purpose of the gathering was to gain a better understanding of Hawaiian well-being by bringing together multiple viewpoints from diverse disciplines.

The event attracted a broad cross-section of researchers, educators, and cultural practitioners from the fields of education, health, family, economics, leadership, environmental studies, cultural practices, politics, and spirituality. “The variation of research presented was wonderful,” said one participant. “I am amazed at how far we have come [to be able to] present from a Hawaiian perspective.”

An overarching theme throughout the conference was the charge to rely on küpuna (elders) wisdom and to balance that knowledge with scientific learning. Presenters also insisted that native voices are necessary to provide a more complete and accurate portrait of känaka maoli (indigenous activities).

“It was very special to have the conference on our campus and to be talking about the wellbeing of our children while they were among us building their own futures through education. It made tangible the direct connections between our research and the education of Hawaiian children,” says PASE director Shawn Malia Kana‘iaupuni, PhD. “Everything we do—the surveys, the longitudinal studies, statistical analyses, technical reports, evaluations, and the sharing of findings with other researchers—is to help achieve a better understanding of how to make a difference for the keiki (children) and families we serve.”

For more information about PASE, call (808) 541-5372 or visit www.ksbe.edu/pase. To view presentations from this year’s conference, visit http://www.ksbe.edu/pase/researchproj-ksrschcon.php.

Washington, DC: Evaluation of Driver Education

Northport Associates

February 16-17, 2005

Project to Develop Guidelines for Improved Evaluation of Driver Education

This is intended to provide a brief overview of this project and of the consultative workshop which took place last week in Washington, D.C.

Guidelines Project

The AAA Foundation for Traffic Safety and BMW of North America are funding a research project to develop guidelines for improving the evaluation of driver education (DE) programs. Northport Associates is conducting this project in consultation with an advisory group and other experts.

Over the long history of driver education, there have been a moderate number of evaluations, including quasi-experiments, random controlled trials, and ecological time series studies. Reviews of evaluations typically conclude that young people who complete DE programs crash at about the same rate as those who do not receive formal education. Do some types of driver education programs lead to better educational outcomes and safety impacts than others? How can driver education programs be improved in order to yield safer young drivers? Lack of systematic programs of research and methodological weaknesses in previous evaluations have left these questions partially or completely unanswered.

Northport Associates, lead contractor on the project, is reviewing the DE evaluation literature, examining methods and theories of driver education programs, identifying and assessing evaluation methods, measures, and data sources, and preparing a final report and guidelines for future evaluations of driver education programs.

Driver Education—A Challenging Evaluand

Evaluating and improving DE is highly challenging, but the potential benefits are very high. Road trauma is a costly public health problem, particularly among youth. Young, inexperienced drivers are at high risk—16 year olds have 10 times the crash rate per mile of experienced adults. Risk declines rapidly over the first few hundred miles of driving, but the learning curve is long, taking up to 10 years to finally level off.

The limitations in skills and abilities that contribute to elevated risk are known. Novice drivers are less able to control attention, scan the environment effectively, detect potential hazards early, make critical decisions quickly, and maintain consistency in critical thought and action. They often raise their risk through overconfidence and choices such as driving too fast, accepting small gaps in traffic, and leaving inadequate safety margins. Both skill deficiencies and risky choices contribute to their excess risk.

Driver education has long been seen as a societal response to the tragic losses of novice drivers. Traditional driver education takes place before the driver becomes licensed. Indeed, one of its principal purposes is to prepare beginners for license testing. Typical U.S. DE programs consist of 30 classroom hours and 6 hours in car. Content covers legal requirements, vehicle handling, interacting with traffic, and efforts to motivate beginners to fear the consequences of crashes. Classroom methods most often include teacher-centred lectures, with some discussion and support with film and video, and sometimes simulators.

In recent years, there have been major changes in the technological, business, and regulatory environments of driver education and also in driver licensing, with the move to graduated licensing suggesting graduated training. There appears to have been limited development of new DE content, but instructional and delivery methods are rapidly changing. Self-instruction, computer-based instruction, simulation, and web-based instruction are increasingly becoming available. In some jurisdictions, recent regulatory provisions recognize a formal course delivered by parents.

While more education is always a popular prescription for improving safety, demonstrated effectiveness in reducing the risk of drivers of all ages through education alone is rare. It is widely believed, if not yet proven, that carefully designed multifaceted, multi-level behavior-change programs are required. Simply increasing knowledge and skill does not make safer drivers—“better” drivers do not necessarily crash less.

Because the needed comprehensive programs require coordination across bureaucratic boundaries, organizational interests and behavior become issues as important as individual behavior change. Organizational constraints provide great challenges for DE program development and to their evaluation. Utilization of evaluation results is also problematical in the DE field. Disappointing evaluations led to (or justified) reduced support for driver education in the 1980s, when a more rational response would have been to redouble efforts to make the programs more effective.

Content, structure, standards, governance, and market incentives are critical issues for driver education globally. Significant further development is needed for DE to fully satisfy the expectations placed upon it by society. More comprehensive evaluation and continuous improvement are seen as critical to progress. Developing guidelines for evaluation of complex programs is always challenging, but seems especially so for the highly diverse driver education field, whose two main goals—independent mobility and safety—are antithetical.

Guiding the Guidelines—The Consultation Processes

Consultation for the development of guidelines consists of an internet discussion board (www.northportassociates.com/aaafts) and a two day workshop. The discussion board has been active and received contributions from driver education experts and evaluators. Much of the board discussion has focused on appropriate objectives and success criteria for DE—safety impacts or other outcomes and impacts.

The consultative Workshop took place February 16-17, 2005 in the AAA’s Washington, D.C. offices. Two dozen invited participants included academics, consultants and research staff from:

· National Highway Traffic Safety Administration

· National Institutes of Health

· Insurance Institute for Highway Safety

· American Driver and Traffic Safety Education Association

· State Departments of Motor Vehicles

· Driving School Association of the Americas

· University traffic research institutes: UNC-HSRC, TTI, UMTRI

· Traffic Injury Research Foundation of Canada

· The Evaluation Center—Western Michigan University

· Georgia State University, Institute Of Public Health

· Private sector safety research consultants

The Workshop members were asked to help answer fundamental questions from their own diverse perspectives. Questions and a few representative answers are shown below.

1. What has worked in driver education evaluation?

· RCTs focused on safety impacts

· Quasi-experiments with statistical control

2. What should be done differently?

· More comprehensive, systematic evaluation, e.g., British Columbia GDl/DE ongoing evaluation/program development process

· More formative evaluation, e.g., SPC process in the 1970s

· Use of established evaluation standards—Joint Committee Standards for Educational Evaluation

· Look at different approaches, e.g., Success Case Method

· Methodological improvements, e.g., sample sizes

· Hybrid designs, e.g., RCTs with modeling to compensate for limitations in control group equivalence

3. What do we want driver education evaluation to accomplish?

· To track DE effectiveness, “see if it works”

· To help improve DE, e.g., feedback & continuous improvement

· To compare performance, e.g., across states.

· Defend policy, e.g., choices regarding investment, etc.

· Recognize needs versus objectives, e.g., real driver needs versus arbitrary objectives

4. What are the key evaluation targets, indicators and measures?

· Product & process, e.g., needs assessment, program quality & consistency, quality management

· Learning outcomes—knowledge & skills, e.g., risk perception, insight

· Behavioral outcomes—risk response, e.g., speed & space choices

· Societal impacts—safety & mobility, e.g., licensing & crash rates

5. Who are the key users for DE evaluation guidelines?

· Evaluators

· Policy makers, legislatures

· Parents

· Consumer protection

· Insurers & policy holders

· State level administrators

· DE program managers

· Researchers

6. What is the best format for the guidelines?

· Emulate good models, e.g., CDC, Ottawa Health Unit

· Moderate in size

· User friendly to a wide range of users, e.g., clear definitions

· Examples of good & bad practice

· Cover simple needs, e.g., program quality checklist

· Support higher level evaluations, e.g., data acquisition for intermediate objectives and impacts

The Driver Education Evaluation Guidelines project will proceed through drafting and review of materials, as well as ongoing consultation. It is scheduled to be completed in the Fall of 2005.

Principal investigator:

Larry Lonero, Northport Associates, e-mail: npa@eagle.ca, telephone: 905-377-8883

AAAFTS Project Manager:

Dr. Scott Osberg, e-mail: sosberg@aaafoundation.org, telephone: 202-638-5944

Selected Bibliography

Ammenwerth, E., Iller, C., & Mansmann, U. (2003). Can evaluation studies benefit from triangulation? A case study. International Journal of Medical Informatics, 70(2-3), 237-248.

Anderson, D., Abdalla, A., Goldberg, C. N., Diab, T., & Pomietto, B. (2000). Young drivers: A study of policies and practices. Fairfax, VA: Center for the Advancement of Public Health, George Mason University.

Donelson, A. C., & Mayhew, D. R. (1987). Driver improvement as post-licensing control: The state of knowledge. Toronto, ON: Ontario Ministry of Transportation and Communications.

Fisher, D. L., Laurie, N. E., Glaser, R., Connerney, K., Pollatsek, A., Duffy, S. A., et al. (2002). Use of a fixed-base driving simulator to evaluate the effects of experience and PC-based risk awareness training on drivers' decisions. Human Factors, 44(2), 287-302.

Glad, A. (1988). Phase 2 Driver Education: Effect on the risk of accident. Oslo, Norway: Norwegian Center for Research, Department of Transportation.

Green, L. W., & Kreuter, M. W. (1991). Health promotion planning: An educational and environmental approach. Mountain View, CA: Mayfield Publishing.

Joint Committee on Standards for Educational Evaluation. (1994). The Program Evaluation Standards. How to assess evaluations of educational programs (2nd ed.). Thousand Oaks, CA: Sage Publications.

Keskinen, E., Hatakka, M., & Katila, A. (1998). The Finnish way of driver improvement: Second phase of training. Paper presented at the International Driver Improvement Workshop, Bergisch Gladbach, Germany: Bundesanstalt fur Strassenwesen.

Lonero, L. P., Clinton, K. M., & Black, D. (2000). Training to improve the decision making of young novice drivers. Volume II: Literature review: Consistency not capacity: Cognitive, motivational and developmental aspects of decision making in young novice drivers. Washington, DC: U.S. Department of Transportation, National Highway Traffic Safety Administration.

Lonero, L. P., Clinton, K. M., Peck, R. C., Mayhew, D., Smiley, A., & Black, D. (unpublished). Driver improvement programs: State of knowledge and trends. Ottawa, Ontario: Transport Canada, Government of Canada.

Lonero, L. P., Clinton, K. M., Persaud, B. N., Chipman, M. L., & Smiley, A. M. (unpublished). A longitudinal analysis of Manitoba Public Insurance Driver Education Program. Winnipeg, Manitoba: Manitoba Public Insurance.

Lonero, L. P., Clinton, K. M., Wilde, G. J. S., Roach, K., McKnight, A. J., Maclean, H., et al. (1994). The roles of legislation, education, and reinforcement in changing road user behaviour. Toronto, Ontario: Ontario Ministry of Transportation.

Lonero, L., & Clinton, K. (1998). Changing road user behavior: What works and what doesn’t, from www.drivers.com

Lund, A. K., & O’Neill, B. (1986). Perceived risks and driving behaviour. Accident Analysis and Prevention, 18(5), 367-370.

Mayhew, D. R., & Simpson, H. M. (1997). Effectiveness and role of driver education and training in a graduated licensing system. Ottawa, Ontario: Traffic Injury Research Foundation.

Patton, M. Q. (1997). Utilization-focused evaluation: The new century text (3rd ed.). Thousand Oaks, CA: Sage Publications.

Rogers, P. J., Hacsi, T. A., Petrosino, A., & Huebner, T. A. (2000). Program theory in evaluation: Challenges and opportunities. New Directions for Evaluation (87).

Rossi, P. H., Lipsey, M. W., & Freeman, H. E. (2004). Evaluation. A systematic approach (7th ed.). Thousand Oaks, CA: Sage Publications.

Scriven, M., quoted in Davidson, E. J. (2004). Evaluation methodology basics. Thousand Oaks, CA: Sage Publications.

Shope, J. T., & Molnar, L. J. (2003). Graduated driver licensing in the United States: Evaluation results from the early programs. Journal of Safety Research, 34(1S), 63-69.

Smiley, A., Lonero, L., & Chipman, M. (2004). Final Report. A review of the effectiveness of driver training and licensing programs in reducing road crashes. Paris, France: MAIF Foundation.

Stock, J. R., Weaver, J. K., Ray, H. W., Brink, J. R., & Sadof, M. G. (1983). Evaluation of Safe Performance Secondary School Driver Education Curriculum Demonstration Project (DOT HS-806 568). Washington, DC: U.S. Department of Transportation, National Highway Traffic Safety Administration.

Stufflebeam, D. L. (2001). Evaluation Models. New Directions for Evaluation (89), 1-106.

THCU (The Health Communication Unit). (2002). Evaluating health promotion programs. Toronto, ON: Centre for Health Promotion, University of Toronto.

Trochim, W. (2001). The research methods knowledge base. Cincinnati, OH: Atomic Dog Publishing.

Turner, C., McClure, R., & Pirozzo, S. (2004). Injury and risk-taking behavior - A systematic review. Accident Analysis and Prevention, 36(1), 93-101.

W.K. Kellogg Foundation. (1998). W.K. Kellogg Foundation Evaluation Handbook. Retrieved November, 2004, from http://www.wkkf.org/Pubs/Tools/Evaluation/Pub770.pdf

Waller, P. F. (1975). Education for driving: An exercise in self delusion? Prepared for the Driver Research Colloquium, Highway Safety Research Institute, University of Michigan, Ann Arbor, Michigan, June 4 & 5. Chapel Hill, N.C.: Highway Safety Research Center, University of North Carolina.

Waller, P. F. (1983). Young drivers: Reckless or unprepared? Paper presented at the International Symposium on Young Driver Accidents: In Search of Solutions, November, Banff, Alberta.

African Evaluation Association

AfrEA[1]

The African Evaluation Association (AfrEA) held its Third Conference in Cape Town, South Africa, on December 1-4, 2004. The conference was attended by 479 people from 61 countries, of which 47 were in Africa. Around 250 participants also attended the 10 pre-conference professional development workshops held on 29-30 November.

The Conference was hosted in conjunction with the Public Service Commission of the South African Government, and was supported by another 21 local and international organisations, including SIDA, DFID, GTZ, the World Bank, the Nelson Mandela Foundation and the African Capacity Building Foundation.

Conference proceedings will be available in the first few months of 2005, in both English and French. A narrative conference report is available at Third AfrEA Conference Report . A listserv has been established to continue communication and information sharing among those interested in monitoring and evaluation in Africa. Details can be obtained from the AfrEA Secretariat at info@afrea.org.

AfrEA is planning several follow-up activities in partnership with national associations and other interested organizations before its next conference, which will be held in Niger in 2006. For additional information on evaluation in Africa visit the AfrEA Website or see the first issue of JMDE (Part III: Global Review—Regions) at http://evaluation.wmich.edu/jmde/JMDE_Num001.html.

International Association for Impact Assessment

Brandon W. Youker

International Association for Impact Assessment (IAIA) defines impact assessment as “the process of identifying the future consequences of current or proposed action.” “IAIA is a forum for advancing innovation, development and communication of best practice in impact assessment. Its international membership promotes development of local and global capacity for the application of environmental assessment in which sound science and full public participation provide a foundation for equitable and sustainable development.”

I. Impact Assessment and Its Relationship with Evaluation

IAIA is a “shadow” professional organization of AEA. Like AEA members, IAIA members are concerned with evaluation issues as they pertain to predicting long-term outcomes, particularly as it relates to the human environment. Below is a discussion of the relationship between impact assessment and evaluation from the perspective of a “traditional” evaluator.

Impact assessment, as defined by IAIA, is pertinent to evaluation; yet differences between the definition of “impact” by evaluation experts and impact assessment experts cause confusion. IAIA defines impact assessment as, “the process of identifying the future consequences of current or proposed action.” Conversely, evaluation most frequently refers to retrospective studies. In Evaluation Methodology Basics: The Nuts and Bolts of Sound Evaluation (2005), E. Jane Davidson defines impact as, “change or (sometimes) lack of change caused by the evaluand. This term is similar in meaning to the terms outcome and effect. This term impact is often used to refer to long-term outcomes, (Davidson, 2005, p. 241).” The IAIA definition of impact assessment and the Davidson definition of impact at least seem to agree on long-term outcomes as a focus of assessing impact. However, the IAIA’s definition is inconsistent with that of the Evaluation Thesaurus (1991), which defines an impact evaluation as “an evaluation focused on outcomes or payoff rather than process, delivery or implementation evaluation, (Scriven, 1991, p. 190).” Scriven’s definition mentions nothing of long-term or future outcomes; rather, an impact evaluation is focused on actual outcomes and does not examine other program or policy components. It is clear that impact assessments would surely investigate beyond solely outcomes and also study the process and implementation of the planned intervention. Despite the confusion over the multiple meanings and uses for the term impact, several aspects of impact assessments are evaluative in nature and include evaluation-type tasks.

An impact assessment is a type of evaluation and may have utility for a certain evaluators. As the author previously reported, IAIA uses impact assessment to be the study, prediction, and evaluation of long-term outcomes. Therefore, an impact assessment is a process that determines the value of policies and programs in relation to future consequences. To elaborate further, impact assessments select relevant values and determine merit criteria in evaluating both the planned intervention and several alternative interventions to find the “best” (greatest benefit with least cost) potential intervention. IAIA’s description and definition of impact assessment leads the author to conclude that an impact assessment is in fact an evaluation. It is an evaluation of interventions (program/policy) and alternatives based on long-term or future outcomes.

Furthermore, impact assessments frequently incorporate evaluation methodology, use evaluation reports, and/or conduct (the typically defined) program and policy evaluations. Evaluations may be particularly germane in developing monitoring and management systems for these outcomes.

Evaluation experts may find IAIA and impact assessments especially relevant if they have interest in studying the future outcomes of social; bio-physical; health; or policy, as it pertains to specific human activities. Methodology of impact assessments is beyond the scope of this paper, for additional information on impact assessment methodology, see the IAIA Website.

II. IAIA Members and activities

Introduction

IAIA was founded in 1980 aiming to provide an international forum for researchers, practitioners and others who utilize impact assessments. Its more than 2,500 members from 100 plus-countries include corporate planners and managers, public interest advocates, government planners and officials, private consultants and policy analysts, and university professors and students.

IAIA Partnerships and Interested US Agencies

IAIA has strategic partnerships formed with the Canadian International Development Agency, the Netherlands Association for Environmental Professionals, the World Bank, US Council on Environmental Quality, the Canadian Environmental Assessment Agency, and various UN agencies.

Examples of other US federal agencies that may conduct social impact assessments or utilize its principles: US Bureau of Reclamation, US Forest Service, US Department of Transportation, US Environmental Protection Agency, and US Council on Environmental Quality.

IAIA Topical Sections

There are 11 sections of IAIA that provide more in depth coverage of a topical debate. The Sections of IAIA are biodiversity and ecology; environmental management systems; health impact assessment; integrated assessment of trade-related policies; indigenous peoples; strategic environmental assessment; local and regional government policy and impact assessment; disasters and conflicts; environmental legislation, institutions and policies; public participation; and social impact assessment.

Additional IAIA Activities

o Several topical listservs provide networking opportunities and dialogue regarding impact assessments for IAIA members.

o IAIA presents an annual conference and each year IAIA chooses a different global location. Additionally, it offers regional conferences, trainings, and professional exchange opportunities.

o An IAIA newsletter is published quarterly. It provides information regarding association activities and events, as well as professional news related to impact assessments.

o Quarterly, IAIA produces Impact Assessment and Project Appraisal, a journal containing peer-reviewed research articles, professional practice ideas, article and book reviews, editorials, and a professional practice section. The journal focuses on the environment, social, health, technology assessment, sustainability, project appraisal, case studies, cost-benefit analysis, and other impact assessment-related material. The journal is only available to members of IAIA or vis-à-vis purchase.

o The IAIA Website offers “Key Citations” of background reference material related to the various areas of impact assessments. Environmental Impact Assessment’s index of Websites is a preliminary index of useful Internet sites used as a preliminary guide for environmental impact assessments.

References

Davidson, E.J. (2005). Evaluation methodology basics: The nuts and bolts of sound evaluation. Thousand Oaks, CA: Sage Publications, Inc.

IAIA (2005). International Association for Impact Assessment (IAIA) Website. Available at http://www.iaia.org/

Scriven, M. (1991). Evaluation thesaurus (4^th ed.). Newbury Park, CA: SAGE Publications, Inc.

An Update on Evaluation in Canada

Chris L. S. Coryn

News and Events

The most pertinent news on the Canadian front is of course the 2005 joint Canadian Evaluation Society (CES)/American Evaluation Association (AEA) conference: Crossing Borders, Crossing Boundaries. The conference will be held October 24-30, 2005 in Toronto, Ontario, Canada. Proposals for the conference are due no later than March 11, 2005. For complete information on the conference including strands, topical interest groups (TIGS), submission procedures, and general information please visit the CES/AEA 2005 conference Website.

The most recent CES Newsletter (Vol. 4, September 2004) focuses on unpublished evaluation-related articles, conference presentations, book reviews, interviews, and dissertations. This newsletter concentrates on contributions from the European and Australasian Evaluation Societies (EES and AES, respectively) and is divided into five sections; (1) Benefits, Consequences, and Effects of Evaluation, (2) Evaluation Outputs, (3) Evaluator Knowledge, Skills, and Competencies, (4) Evaluator Professionalism, and (5) Evaluation Process Issues. Over 90 complete papers and presentations (primarily from the conferences of the ESS and the AES) are available through links within the newsletter.

The Grey Literature Database and Evaluation Report Bank

The CES Website offers a wide array of useful resources for evaluators. Of notable interest is the Grey Literature database; a fully searchable bank of almost 500 evaluation-related documents encompassing diverse areas of interest to evaluators; for example, theory, policy analysis, ethics, and communication and reporting. The Grey Literature database is searchable by recent conferences (from 2000-2003; including those of the European Evaluation Society, the Australasian Evaluation Society, and the Canadian Evaluation Society), topic, author, date, or language (English, French, and Spanish). These documents include, for example, full-length evaluation reports, conference presentations, and unpublished papers. Also available on the CES Website is the Evaluation Report Bank, a resource for evaluation reports related to Canadian academia, government, and the private sector. Both the Grey Literature database and Evaluation Report Bank are well organized, easy to navigate, and accessible free of charge to non-CES members.

An Update on Evaluation in Europe

Daniela C. Schroeter

I apologize for exclusion of those national societies whose language I have not mastered and would like to encourage evaluators from the European countries to contribute a report about the state of evaluation in their country to JMDE.

Details of events and news items listed below can be found on the corresponding Web sites.

Evaluation and the European Union

Within the European Union (EU), evaluation is promoted by the Directorate General for Budget, which commissions and conducts ex ante (prospective), interim, and ex post (retrospective) evaluations of programs, projects, and policies of the General Directorates within the EU. The Web site of the evaluation unit provides information about evaluation activities and ongoing studies, policies and procedures, guides on evaluation in different contexts (General Directorates) and on evaluation specific methodology, procurement of evaluation via “calls for expressions of interest,” information on evaluation networks, including links to specific commissions within the EU, the expert network of member states, evaluation societies within Europe, and links to other organizations conducting evaluation (e.g., OECD, UNDP, UNICEF, the World Bank, the GAO, and the International Monetary Fund).

The European Evaluation Society

In October, 2004 the EES held its 6^th Conference in Berlin, Germany where over 420 participants from all over the world attended. Attendees’ feedback and the conference program can be viewed on the EES Web site. Papers and presentations from the conference are available through links within the Canadian Evaluation Society’s Newsletter (Vol. 4, September 2004).

The most recent Newsletter of the EES (December, 2004) reports on the 6^th conference of the EES and introduces the new board for the year 2005. Additionally, the EES Residential Summer School—Evaluating innovative policy instruments: change, complexity, and policy learning, which takes place September 25-30, 2005 in Seville, Spain, is announced. Between 24-30 experienced evaluators, policy designers, and change managers are invited to participate and learn from international evaluation experts. More detailed information on how to register will be posted on the EES Web site.

Moreover, the EES published a summary of a proposed strategy for evaluation, training, and education in Europe (see http://www.european evaluation.org/docs/Strategy_for_Education_and_Training.pdf). The EES board promotes (i) the provision of evaluation training through the society, (ii) support and consulting services for developing evaluation training by national evaluation societies in Europe, (iii) collaboration with institutions of higher education to develop degrees in evaluation, and (iv) functioning as a connector between potential collaborators to develop evaluation training for those in need.

The German Evaluation Society

The German Evaluation Society (DeGEval) announced various training opportunities. Of specific interest are e-learning opportunities provided by the Center for European Evaluation Expertise (Eureval-C3E). Moreover, the DeGEval announced an invitation for a conference on Multifunctionality of Landscape-Analysis, Evaluation, and Decision Support, on May 18-19, 2005 in Giessen, Germany.

Most recently, the following event were announced on the listserv of the DeGEval:

·                          EU-sponsored series of three conferences and four training opportunities on the "Evaluation of Sustainability," EASY-ECO, 2005-2007. For details and further information, please visit www.sustainability.at/easy/.

·                          The first EASY-ECO conference “Impact Assessment for a New Europe and Beyond” takes place June 15-17, 2005 in Manchester. The call for papers is published at: http://www.sed.manchester.ac.uk/idpm/news/#easyeco.

The Spanish Evaluation Society

The Spanish Evaluation Society announced:

· Fifth Annual Campbell Collaboration Colloquium on Supply and Demand for Evidence in Lisbon, Portugal, February 23-25, 2005

· Third Argentine Congress of Public Administration in San Miguel de Tucumán, Argentina with the topic Society, State, and Administration

The Swiss Evaluation Society

The Swiss Evaluation Society (SEVAL) announced:

· SEVAL congress—Evaluation in Education, June 3, 2005 in Berne, Switzerland

· International Conference on "Visualising and Presenting Indicator Systems," March 14 – 16, 2005, Swiss Federal Statistical Office, Neuchâtel, Switzerland

· Outcome Mapping: Practical, Flexible, Participatory Approach to
Monitoring and Evaluation, a 4-day workshop, Febuary 21-25, 2005
Lebensgarten, Germany

Others events listed on SEVAL’s Web site include training opportunities provided in Switzerland.

The UK Evaluation Society

The UK Evaluation Society (UKES) announced various events sponsored by regional networks:

· London Evaluation Network: Participatory Approaches to Evaluation, February 3 or 10, 2005 in the Institute of Commonwealth Studies

· Scottish Evaluation Network: What’s working? Improving the contribution of research and evaluation to organizational learning, February 25, 2005, in Our Dynamic Earth, Holyrood Road, Edinburgh

· North West Evaluation Network: Evaluation today: emerging themes and improving practice, July 1, Manchester, NWEN annual conference.

Additionally and as announced on the EvalTalk listserv, The 5^th International Conference on Evaluation for Practice takes place July 13-15, 2005 at the University of Huddersfield, England. The conference will focus on evaluation practice and its implications in areas such as social services, social work, education, and health services among other. The keynote speaker is Professor Michael Scriven.

An Update on Evaluation in the Latin American and Caribbean Region

Thomaz Chianca

This short paper updates the evaluation scene in the Latin American and Caribbean (LAC) region. Specifically, it covers four aspects of that scene: (1) evidence on the growing number of job opportunities for evaluators in the region, (2) news about national-level evaluation organizations, (3) professional development courses available in 2005, and (4) the internal evaluation system supported by the Brazilian federal government.

1. Work opportunities for evaluators in LAC

There is an increasing number of job postings being advertised in three major LAC evaluation discussions lists—RELAC (Latin American and the Caribbean Evaluation Network), PREVAL (Program for Strengthening the Regional Capacity for Evaluation of Rural Poverty Alleviation Projects in Latin America and the Caribbean), and the Brazilian Evaluation Network. Just in the past three months, at least eleven job postings have been advertised. Ten of them were procurements for external program evaluators and one was a search for a full-time coordinator of monitoring and evaluation. Government agencies and nonprofit organizations (NGOs, foundations, and institutes) were the two main clients offering such opportunities. It is interesting to notice the broad range of areas covered by such announcements, including education, environment, child labor, agriculture, and socioeconomic development.

Such augmentation in the number of job postings announced in these evaluation lists can be considered in two ways: it is not only an increase in the existing market for professional evaluators in the region, but also a general indicator that local evaluators are being sought to fill the available positions. Table 1 presents some additional details on the job positions advertised.

Table 1. Employment Opportunities in LAC

*Sector*	*Client*	*Position*	*Description*
Government	Ministry of Education—Brazil	Evaluation consultant	Assess intermediary results of methodology of Active School Program—improving the quality of public schools
	Ministry of Education—Brazil	Evaluation consultant	Develop evaluation methodology to assess strategic planning of education secretaries within the FUNDESCOLA Project
	Ministry of Education—Brazil	Evaluation consultant	Independent evaluation of GESTAR I Program—improving quality of Math and Language education
	Brazilian Environmental and natural Resources Institute—Brazil	Evaluation consultant	Evaluation of projects related to the Promising Initiatives Component of the Pro-Várzea Program in the Amazon region
	Fideicomisos Instituidos en Relación con la Agricultura (FIRA)—Mexican Government	Evaluation consultant	Evaluation of Mexican government programs in livelihood/agriculture.
Nonprofits	World Vision—Brazil	Evaluation consultant	Develop monitoring system based on Logic Model for a social-economic development program
	Acción Sin Fronteras (ASF)—Peru	Coordinator for Monitoring and Evaluation	Provide leadership and manage all activities related to monitoring and evaluation within ASF
	Instituto Avaliação—Brazil	Evaluation consultant	Evaluation of projects in the area of race-ethnicity and education
Private	Killefit Consult—Colombia	Evaluation consultants	Several evaluations of projects in the area of environment and rural development in LAC countries
Private	Juárez and Associates—USA	Evaluation consultants	Technical assistance in monitoring and evaluation of international education initiative to combat child labor in Latin American and other countries
Development Agency	OIT—International Labor Organization	Evaluation consultant	External final evaluation of program for eradicating child labor in Guatemala

2. LAC Evaluation Organizations

Since November 2004, one new national evaluation organization was formed and three others are taking the initial steps towards their official creation. The Nicaraguan Monitoring and Evaluation Network (RENICSE) became the fifth organization officially created in the region, joining Brazil, Colombia, Costa Rica and Peru. Bolivia, Chile and Venezuela have already started consistent efforts to create their evaluation networks. RELAC, PREVAL, and the UNICEF continue to play important roles in the establishment of national organizations.

While searching for information to write this update, I came across some references to an evaluation association in Argentina, the “Asociación Argentina de Evaluación,” including a training course they are providing (see “Evaluation Training” below for details). At the closing of this issue of the journal I had not received any additional detail regarding the organization from the provided contact person—María Isabel Andrés (mandal@mecon.gov.ar). We hope to have some more to report on the Argentinean Evaluation Association in the next issue of JMDE.

The following are some short news items about the existing national evaluation organizations in the region.

Brazil: The Brazilian Evaluation Network has now about 300 members divided among its five state chapters in Bahia, Minas Gerais, Pernambuco, Rio de Janeiro, and São Paulo. The Network maintains a website (www.avaliabrasil.org.br) and a discussion list (ReBraMA-subscribe@yahoogrupos.com.br). The most recent activity promoted by the Network was an evaluation seminar involving professionals working in the social development area during the World Social Forum in Porto Alegre (Brazil) in January 2005. For more information about this organization contact Rogério Silva (rrsilva@fonte.org.br).

Colombia: The Colombian Planning, Monitoring, Evaluation and Systematization Network (SIPSE) has its headquarters in Cali at the San Buenaventura University and consists of professionals working in academia, government agencies, nonprofit organizations, international cooperation agencies, and grassroots organizations. SIPSE has as its main goal applying planning, monitoring, evaluation and systematization approaches as a way to foster participative democracy. The programs of the two national events organized by SIPSE can be found at consorcio@consorcio.org.co. For more information about SIPSE contact Gloria Vela (gvela@cable.net.co).

Costa Rica: Costa Rica hosts the Central American Evaluation Association (ACE)—the first formal evaluation organization created in LAC. ACE has recently launched its first newsletter (http://www.geolatina.org/ace) and is regaining space as a major reference for the region. More information about ACE can be obtained by contacting Welmer Ramos (ramosacu@racsa.co.cr) or Ana Laura Ibaja Jiménez (sisube@racsa.co.cr).

Nicaragua: The Nicaraguan Monitoring and Evaluation Network (RENICSE) has hosted three meetings and is working towards defining its vision, mission, and strategic objectives. Eduardo Centeno Cruz (ecenteno@ibw.com.ni) can provide further details about RENICSE.

Peru: The Peruvian Monitoring and Evaluation Network has developed its annual plan for 2005. It works via a coordinating committee that rotates every six months charged with fostering participation and articulations for accomplishing the annual plan. The Network has an active electronic discussion list. Additional information on the Network’s activities and accomplishments can be provided by Emma Rotondo (rotondoemma@yahoo.com.ar).

RELAC: The Latin American and the Caribbean Monitoring, Evaluation and Systematization Network (RELAC) had its first international conference in Lima, Peru, in October 2004. The conference had 142 participants from 22 countries. A Coordination Committee for RELAC was formed with the support of representatives from 16 countries that already have or are in their way to create national evaluation organizations. A preliminary plan of action for 2005 was approved that include: (i) defining operational norms, (ii) defining strategies for establishing alliances, (iii) designing communication and support strategies for new national organizations, and (iv) establishing working groups (e.g., evaluation standards, systematizing, public policy, and capacity building). A CD-Rom with all the material presented at the Conference, including videos, as well as the strategic planning for RELAC will soon be available upon request at: preval3@desco.org.pe. For additional information on RELAC contact one of the members of the coordination committee: Consuelo Ballesteros—South Cone (consueloballesteros@vtr.net), Gloria Vela—Colombia (gvela@cable.net.co), Welmer Ramos—Central America (ramosacu@racsa.co.cr), Luis Soberón—Peru (lsober@terra.com.pe), Rogerio Silva—Brazil (rrsilva@fonte.org.br), Marco Segone (msegone@unicef.org), Ada Ocampo (ada.ocampo@undp.org), Emma Rotondo (rotondoemma@yahoo.com.ar), and Oscar Jara (oscar.jara@alforja.or.cr)

3. Evaluation training

The availability of a number of on-line evaluation courses in Spanish is probably the most interesting fact regarding evaluation training in LAC not covered in the last issue of JMDE. At least three institutions are offering such courses in 2005:

(1) In Argentina, the University Nacional del Litoral and the Center for Development and Technological Technical Assistance Centro for Public Organizations (TOP) are offering an on-line certification course in Outcome and Impact Evaluation of Public Organizations and Programs. More information at: www.top.org.ar/curso_virt6.htm

(2) PROGRESO Social Projects, Management, and Resources, a Peruvian nonprofit organization, is offering an on-line course in Qualitative Methods for Evaluation. Access their website for additional information: www.progreso-peru.org

(3) The Inter-American Development Bank (BID) is offering a series of four on-line courses covering: (i) Logic Models for Project Design, (ii) Project Monitoring and Evaluation, (iii) Evaluation of Environment Impact, and (iv) Institutional Analysis. The courses have two quite attractive features. They are free of charge and are available year around. For details access: www.iadb.org/int/rtc/ecourses/esp/index.htm

Some courses not covered in the October 2004 JMDE issue include:

The Brazilian Ministry of Social Development, the National School of Public Administration and the National School of Public Health in association with the Institute of Social Studies, (The Netherlands) are offering a certification course on Evaluation of Social Programs targeting primarily public administrators. March to June 2005 in Brasilia, DF, Brazil.

The Center for Studies in Economic Development (CEDE)—University of Los Andes in Bogotá, Colombia is offering a specialization course in Social Evaluation of Projects.

The Latin American Institute for Social and Economic Planning (ILPES) will be offering at least three short-term evaluation courses in 2005: (i) Use of Socioeconomic Indicators to Evaluate the Impact of Poverty Reduction Program—July 2005, Cartagena de Indias, Colombia; (ii) Logic Model, Monitoring and Evaluation—Aug/Sep 2005, Antigua, Guatemala; (iii) Planning and Evaluation of Public Investments Projects—Oct 2005, Santiago, Chile.

The Argentinean Evaluation Association (AAE) will be offering a year-long (March-December 2005) specialization course on project identification, elaboration, and evaluation as part of the Capacity Building Program of the Secretary of Economic Policy of the Argentinean Government. For details contact María Isabel Andrés (mandal@mecon.gov.ar).

4. Evaluation within the Brazilian federal government

The Brazilian government under the leadership of the Ministry of Planning, Budget and Management has taken important steps towards the development of an evaluation culture within the public administration system. Important efforts have been made in the direction of establishing internal evaluation strategies for all federal programs under the umbrella of the federal Pluri-Annual Plan (PPA) for 2004-2007. An evaluation manual has been produced offering a framework to orient all program managers to assess their efforts as a way to improve their practices. The idea is to establish a flexible monitoring system with the input from the different ministries

External evaluations of four major federal programs are expected to be implemented in 2005. Studies to develop a more participative evaluation system within the federal sphere are underway. The government is also trying to learn from other countries’ experiences—a group of staff members recently visited the US, Canada and the United Kingdom. An evaluation seminar, open to the public and including the participation of evaluation specialists from Canada, was organized to share lessons learned from such visits.

The advances produced in this area in Brazil are unquestionable and show a clear interest in promoting long-lasting changes. Hopefully such efforts will culminate in the establishment of an independent evaluation agency, on the model of the American General Accounting Office (GAO) that will be able to provide candid accounts of the merit, relevance, and significance of the federally funded programs. Such evaluations will help strengthen government’s accountability as well as provide quality information to help decision makers make better use of the scarce resources available.

Additional information regarding the work done by the Brazilian Ministry of Planning can be obtained from Andreia Rodrigues dos Santos (andreia.santos@planejamento.gov.br).

Final Note

I would like to thank the many people that provided me with key information to develop this update: Ana Laura Ibaja Jiménez (Costa Rica), Andréia Rodrigues dos Santos (Brazil), Eduardo Centeno Cruz (Nicaragua), Emma Rotondo (Peru), Gloria Vela (Colombia), Rogério Renato Silva (Brazil), and Welmer Ramos (Costa Rica).

If you have additional information or corrections on any of the topics covered by this article or by the previous one, or if you want to send additional contributions regarding evaluation in Latin America and the Caribbean please do not hesitate to contact me at Thomaz.Chianca@wmich.edu.

[1] This piece is based on the News section of the African Evaluation Association Website. Available at http://www.afrea.org/.

Katrina Bledsoe	Shawn Kana'iaupuni
Nicole Bowman	Ana Carolina Letichevsky
Robert Brinkerhoff	Mel Mark
Tina Christie	Masafumi Nagao
J. Bradley Cousins	Michael Quinn Patton
Lois-Ellen Datta	Patricia Rogers
Stewart Donaldson	Nick Smith
Gene Glass	Robert Stake
Richard Hake	James Stronge
John Hattie	Dan Stufflebeam
Rodney Hopson	Helen Timperley
Iraj Imam	Bob Williams