Global Review: Regions

Evaluation Activities in Africa

Paul A. Lamphear

Africa has seen a significant growth in Evaluation networking in the past five years with the founding of the African Evaluation Association (AfrEA) in 1999. As of this writing, AfrEA has 16 national associations under its umbrella, each with evaluator networks supporting their respective countries. The associations are in a wide disparity of development maturity, but are all aimed at supporting socio-economic development programs in their countries, as in this description of the Uganda Evaluation Association as posted in www.kabissa.org.

The major purpose of the association is to build individual and institutional capacity in policy, program and project evaluation in Uganda through local and global networking, training, skills development and other avenues for professional development in evaluation practice^{^[1]}.

The first conference of the African Evaluation Association, held in Nairobi in connection with the creation of AfrEA, was attended by more than 300 evaluators from 35 countries. Michael Quinn Patton was invited as keynote speaker, promoting a focus on ethics and guiding values for professional evaluation associations. He also emphasized the need for a culturally defined set of standards for evaluation and the necessity for utilization-focused evaluation.^{^[2]} Patton referred to the historically poor “expatriate” evaluations with poorly briefed evaluators, performed too quickly, with inadequate reports.

It appears that these lectures, the training seminars, and 80 papers that were submitted on evaluation, helped to ‘jumpstart’ the development of the national associations. Although there does not appear to be any evaluation journals specific to or published in Africa, AfrEA has worked on several projects to increase evaluation capacity and foster a consistent professional approach for evaluators. In 2002, the association completed the "African Evaluation Guidelines", a cultural adaptation of the US “Program Evaluation Standards”, published in both English and French.

At the 2nd AfrEA conference in 2002, a variety of international evaluators were invited, including keynote speakers Prof. Anna Madison of Cornell University, USA, Ada Ocampo, Leader of the Latin American Evaluation Network, and Penny Hawkins, President of the Australasian Evaluation Association. After 5 days of trainings and paper submissions, the association recommended that the African Evaluation Guidelines (AEG) be adopted by all the National Networks, by Government and Public bodies, and by UN Agencies and other Multinational Organizations performing evaluation in Africa. The AEG provides a checklist of 30 items essential for quality control in evaluation and it has now been used by several national governments in major development program evaluations.

The Niger Network of Monitoring and Evaluation (ReNSE) has over 140 documents (in English or French) and in 2004 has published their first newsletter (in French), available from their website, to promote expertise in Nigerian evaluators.

AfrEA supports dissemination of monitoring and evaluation resource materials focused on Africa in the areas of Agriculture, Conservation, Gender, HIV/AIDS, and Poverty. Additionally, AfrEA encourages members to participate in the Xceval listserve, a discussion forum for evaluators from developing countries.[3]

AfrEA has significant connections with international organizations and has activities currently sponsored by:

African Development Bank (AfDB)

Agence Intergouvernementale de la Francophonie

Catholic Relief Services

Canadian Institutes for Health

Canadian International Development Agency (CIDA)

CARE International

Danish Agency for Development Assistance (DANIDA)

Family Health International (FHI)

International Development Research Centre (IDRC)

World Conservation Union (IUCN)

Norwegian Ministry of Foreign Affairs

UNAIDS

UNCHS

UNDP

UNICEF

UNIFEM

World Bank

The third national Conference of the African Evaluation Association will be held in Cape Town, South Africa, from December 1st through December 4th, 2004, and is looking for speakers and attendees.

Evaluation Activities in Australasia

John S. Risley

General Summary of Activities

The Australasian Evaluation Society (AES) produces, and posts on their website (www.aes.asn.au), an e-newsletter approximately twice per year. The AES also holds an annual conference, usually in September or October. The 2004 conference is in October near Adelaide, Australia and will focus on “Diverse Voices in Evaluation.” Last year’s conference emphasized evaluation and indigenous peoples. Many pre-conference workshops are offered. AES has regional representatives from throughout Australia and New Zealand. There is a New Zealand Listserv—Evaluation Aotearoa—that discusses “evaluation research.” It only has a few posts per month, mostly dealing with Auckland Evaluation Group activities.

From reading the editorials and other non-refereed articles in the Evaluation Journal of Australasia (EJA) it appears that the evaluation profession in Australasia differs from the profession in the United States in two main ways. First, evaluators come from more diverse academic and professional backgrounds in Australasia than in the United States. Second, Australasian evaluators are much less likely to be associated with a university and much more likely to be employed by a government agency than are American evaluators.

Evaluation Journal of Australasia

A recent editorial in Evaluation Journal of Australasia (EJA) noted the history of AES publications. The society launched the EJA in 1989. Then from 1993 through 2000 AES published both EJA and Evaluation News & Comment. In 2001 these publications merged to form the new series of EJA. The journal is published by the AES bi-annually (though recently there have been delays in publishing new editions). AES posts the two copies preceding their most recent issue on their web site. The journal includes refereed and non-refereed articles, editorials, interviews with evaluators from both within and without the region, book reviews, research reports, and information about the annual AES conference.

Issues addressed in EJA included much information concerning cultural appropriateness, indigenous peoples, and diversity in evaluation. This may be a reflection of the recent AES conference themes. There is some material drawing distinctions about evaluation aspects specific to Australasia, but many articles are written by authors outside the region about subjects not specific to the region.

Subjects of refereed articles in recent issues of EJA include: evaluation of options for changing port ownership in Belfast, an evaluation of a respite care program in Christchurch, evaluating the cultural appropriateness of human service delivery programs in Australasia, and the TRIAGE (Technique for Research of Information by Animation of a Group of Experts) technique. A few refereed articles were short (3 pages and 5 pages) compared to articles in the American Journal of Evaluation, for example.

Some of the refereed articles had very little to say about evaluation. For example, one of these articles (Burton & Rajan, 2002) concerned a case study evaluation of 15 people seriously injured in workplace accidents. The authors described the project’s goal as exploring the social and economic consequences to society from these workplace injuries. The article discussed the methodology of the study, the experiences of the researchers, and the lessons learned from their research experiences. The methodology was basically a semi-structured interview of injured workers, their family members, employers, etc. The lessons learned by the researchers were: 1) interviewing can be exhausting, 2) diversity of the project team was essential, and 3) it was difficult to remain objective after seeing the suffering of the injured workers.

One interesting article (Sigsgaard, 2002) addressed an unusual methodology (in evaluation research), the Most Significant Change (MSC) methodology. The author, Peter Sigsgaard, works at a Danish NGO called “MS” on measurement and evaluation issues. He gave examples of his experience using MSC in evaluating partnership-based economic development programs in Africa, Asia and Central America. Using MSC you ask people to identify positive or negative changes they have observed within a given “domain of interest.” People are then asked which change, positive or negative, they think is most significant and why. More important or very large changes that are reported are verified by further investigation.

Sigsgaard (2002) contrasts this approach with one previously used by MS in evaluating these programs, in which they would conceive of indicators to measure and then cast about looking for these indicators. This led to lots of time spent looking for, and not finding, specific data.

It makes intuitive sense to ask program consumers what changes are occurring due to the program. It does highlight the need to be careful how one measures program changes.

References

Burton, J., & Rajan, R. (2002). Revealing the hidden costs: research experiences from a case study evaluation project. Evaluation Journal of Australasia (new series), 2(2), 69-73.

Sigsgaard, P. (2002). Monitoring without indicators: an ongoing testing of the MSC approach. Evaluation Journal of Australasia (new series), 2(1), 8-15.

The State of Evaluation in Canada

Chris L. S. Coryn[4]

Background and General Context of Organized Evaluation in Canada

The Canadian Evaluation Society (CES)—Canada's official professional organization for evaluation—serves as the country's core for evaluation related activities. CES is similar to many other evaluation associations around the globe, but differs from the American Evaluation Association in that the majority of its members are from the government sector and evaluators practicing in NGOs, para-government, and the public and private sectors. Presently, the CES has 12 regional chapters, including:

ü Newfoundland & Labrador	ü Ontario
ü Prince Edward Island	ü Manitoba
ü Nova Scotia	ü Saskatchewan
ü New Brunswick	ü Alberta
ü Quebec	ü British Columbia
ü National Chapter	ü Northwest Territories

Since 1991 the CES membership has grown to over 1,750 individual Canadian and student members, as well as over 100 international members (CES, 2004).

The CES offers a wide range of resources and services for practicing evaluators and students of the discipline including: a comprehensive Web site (available in English and French); an evaluation report bank (academic, government, and private sector reports); a fully-searchable database—the Grey Literature Bank (unpublished documents of interest to evaluators); a professional development series of workshops; an annual conference (including the upcoming 2005 joint conference with the American Evaluation Association); and the Canadian Journal of Program Evaluation. The CES efforts are strongly supported by the Government of Canada, which has its own specialized evaluation unit; Evaluation and Data Development (EDD). EDD is one of the largest evaluation shops in the Federal Government of Canada, and focuses primarily on governmental initiatives including analysis of government policy and evaluation of government programs, foe example, Human Resources Development Canada (HRDC) programs. Other government contingencies which influence the Canadian evaluation field include the National Science and Engineering Research Council, the Social Science Research Council, Transport Canada, Industry Canada, Health Canada, the Treasury Board Secretariat, and the Canadian International Development Agency; each of which are also sponsors of the CES. Informed decision making is further facilitated by Statistics Canada a provider—federally legislated—of statistical data for the whole of Canada and each of its provinces that is intended to inform Canadian citizens and other key stakeholders regarding Canada's population, resources, economy, culture, and society.

In the summer of 2001 the CES announced their new vision, mission, and goals for the future (Canadian Evaluation Society Newsletter, Summer 2001):

Vision: The Canadian Evaluation Society will be the leader for evaluation in Canada and a major contributor in the global evaluation community

Mission: The society is a Canada-wide non-profit bilingual association dedicated to the advancement of evaluation theory and practice.

Goals:

1. Leadership—To provide leadership to individuals and organizations in support of evaluation theory and practice in Canada and the global community.

2. Knowledge—To improve the state of evaluation theory and practice.

3. Advocacy—To promote the importance of an evaluation culture.

4. Professional Development—To promote and facilitate the enhancement of evaluation capacity for members and non-members.

The CES also supports various student initiatives including the CES Student Case Competition and student paper contest (for undergraduate and graduate students in the field of evaluation). The CES Student Case Competition (initiated in 1996), is an annual event in which teams of three to five students from Canadian colleges and academic institutions compete in the analysis of an evaluation case file. In a preliminary competition, all teams receive on the same day the key to an evaluation case file that has been hidden on the Web. They have five hours to prepare an analysis and then submit it by e-mail for judging by an expert panel. The three best teams are invited to participate in a final round, held at CES's annual conference, in which they must analyze a new case and present findings and recommendations before a live audience. The team that makes the best presentation takes possession of the Case Competition Trophy for a year, receives prizes, and is given visibility in various publications.

Evaluation Education Programs in Canada

As of 2000 (CES), over 25 Canadian institutions/colleges/universities offered more than 100 evaluation-related courses across a wide array of academic disciplines (e.g., psychology, political science, public administration, economics)—a complete institution, department, and course list is available at http://www.evaluationcanada.ca/txt/outline200106.pdf.

Professional Development of Canadian Evaluators

The CES plans to focus on two key areas in the upcoming years: (1) professional development of its members, and (2) advocacy on behalf of the evaluation function. The articulation of a Core Body of Knowledge (CBK) will guide the Society's professional development and advocacy activities (Canadian Evaluation Society, 2004). The CBK comprises theories, skills, and best practices that people must possess to plan, carry, out, and report on valid and reliable evaluations of programs or policies in governments, not-for-profit organizations, and businesses.

Essential Skills. Much of the emphasis on professional development is funneled through the CES Essential Skills Series. Regional chapters offer this series as well as any other form of training they consider adequate for their members. These essential skills include:

1. Understanding Program Evaluation

§ Key terms and concepts

§ Benefits of program evaluation

§ Basic steps in the evaluation process

§ Major approaches to program evaluation

§ Formatting evaluation questions

§ Designing an evaluation

§ Evaluating with limited resources

§ Analyzing and reporting evaluation results

§ Reducing resistance to evaluation

§ Involving staff and clients in the evaluation process

§ Increasing evaluation utilization

§ Making evaluations ethical and fair

2. Building an Evaluation Framework

§ Identifying who the client is and what the client needs

§ Basic concepts of needs assessment

§ Major approaches to assessing client needs

§ Evaluation methods for "getting close to the client"

§ Building an evaluation framework through logic models

§ Involving managers and staff in building an evaluation framework

§ Relating program design to client needs

§ Defining program components

§ Formulating indicators for program success

§ Using the evaluation framework for linking program performance to client needs

3. Improving Program Performance

§ Using evaluation as a management tool for improving program performance and enhancing internal accountability

§ Basic concepts of monitoring and process evaluation

§ Monitoring program performance with existing administrative data and information systems

§ Developing ongoing data collection instruments and procedures

§ Linking process evaluation to program decision-making

§ Assessing client satisfaction

§ Understanding continuous quality improvement

§ Using program evaluation for building a "learning organization"

4. Evaluating for Results

§ Defining program results

§ Major approaches to evaluating results

§ Developing results measures

§ Designing outcome evaluations

§ Validity and reliability

§ Appropriate use of quantitative and qualitative techniques

§ Relating program results to program costs

§ Understanding program benefits

§ Measuring program equity and responsiveness to community needs

§ Communicating evaluation findings

§ Using evaluations to improve program effectiveness and accountability

(Canadian Evaluation Society, 2004)

Certification of Evaluators in Canada. As a body representing program evaluators across Canada and promoting the program evaluation function in Canadian institutions, the CES is concerned with the sustainability, growth and strengthening of the profession. In recent years, this concern has led the Society to consider issues related to increasing professionalization, through means such as professional development programs, development and adoption of practice standards and ethical guidelines, and certification of members. This issue remains unresolved, but is becoming increasingly acute in the wake of recent developments in the federal government sector that have raised the profile of auditing (Cousins, 2004).

This latter issue — developing a form of certification for members — would be a major step for the CES. Therefore, it was the subject of an in-depth study of the experience of several other organizations with certification (Long & Kishchuk, 1997). A second study, carried out in 1999, reports on a pilot survey of clients and employers (Stierhoff, 1999) on their views regarding certification of evaluators.

Canadian Journal of Program Evaluation

The Canadian Journal of Program Evaluation (CJPE) was launched in 1986 and is published twice a year (available at www.cjpe.ca). CJPE is sponsored by the CES and the University of Calgary. Individual issues and articles can be downloaded by non-members for a nominal cost. CJPE seeks to promote the theory and practice of program evaluation in Canada by publishing:

§ Articles on all aspects of the theory and practice of evaluation, including methodology, evaluation standards, implementation of evaluations, reporting and use of studies, and the audit or meta-evaluation of evaluation.

§ Research and Practice Notes that provide practical examples of the applications of particular methodologies or procedures within the context of a particular study or group of studies.

§ Book Reviews of relevance to the practice in Canada.

(Canadian Journal of Program Evaluation, 2004)

Review of the past eight issues (from Spring 2001 to Spring 2004) of CJPE revealed a number of insights into the journal's thematic trends. The journal does, in fact, promote and publish articles on theory, practice, implementation, and standards, for example. Notable examples include Christie & Rose (2003)—The language of evaluation theory: Insights gained from an empirical study of theory and practice, Levin-Rozalis (2003)—Evaluation and research: Differences and similarities, Morris (2002)—The inclusion of stakeholders in evaluation: Benefits and drawbacks.

In 2001, the CJPE devoted a special issue to provincial evaluation policy and practice in Canada. Accounts of provincial evaluation activity were provided for British Columbia (McDavid, 2001), Alberta (Bradley, 2001), Manitoba (Warrack, 2001), Ontario (Segsworth, 2001), Quebec (Cabatoff, 2001), Prince Edward Island (Mowry, Clough, MacDonald, Pranger, & Griner, 2001), Newfoundland (Ross, 2001) and the Northwest Territories (Hicks, 2001). "Being the first ever account of evaluation activity at the provincial level, this collection of articles represented a very important contribution to the knowledge of evaluation practice in Canada" (Gauthier, Barrington, Bozzo, Chaytor, Cullen, Lahey, Malatest, Mason, Mayne, Myers, Porteous, & Roy, 2004). A number of general and specific conclusions were drawn about the state of affairs in Canadian evaluation as a result of this special issue and were summarized in The lay of the land: Evaluation practice in Canada today (Gauthier et. al., 2004). The authors conclude that program evaluation in Canada:

§ Has not acquired an identity of its own

§ Tends to neglect key issues

§ Loses emphasis on rigor

§ Is dominated by program monitoring

§ Is insufficiently connected with management needs

Regional Perspectives

Perspectives across Canada's various regions are briefly summarized below. This summary includes: (1) strengths, (2) weaknesses, (3) threats, and (4) opportunities of and for evaluation in western Canada, Alberta, and Ontario, as well as potentials for evaluation teaching and learning.

The Western Canadian Perspective. (Malatest, 2004)

Strength: Development of evaluation methodologies—in recent years the provincial and federal agencies have recognized the requirement of good evaluation.

Weakness: Inadequate planning of program evaluations—awareness and use of evaluation tools are often an afterthought.

Threat: Reduced program evaluation capacity—the ability to design and manage complex evaluation activities has been compromised (e.g., lack of resources).

Opportunity: Managing for outcomes—activities in British Columbia and Alberta have been strengthened by strong government-wide commitment to measure and report on the key outcomes for almost all ministries and/or departments.

Program Evaluation in Alberta. (Barrington, 2004)

Strength: Growing sophistication—evaluators are more skilled and better qualified.

Weakness: Dependence on performance measurement—to the exclusion of more relevant, complex outcomes.

Threat: Devaluation—avoidance of serious evaluation (e.g., focus on accountability rather than improvement).

Opportunity: Linking accountability and evaluation—evaluators believe that they can make evaluation more rigorous and more useful.

Program Evaluation in Ontario. (Mason, 2004)

Strength: Com