CREATING A WORLDWIDE EVALUATION COMMUNITY

By The Presidents Panel

January 1999


Paper #15

Occasional Paper Series



CONTENTS

Introduction and Narrative Description
by Craig Russon and Arnold Love

Position Papers:

American Evaluation Association (AEA)
by Donna Mertens

Associazione Italiana di Valutazione (AIV)
by Nicoletta Stame

Australasian Evaluation Society (AES)
by Sue Funnell

Canadian Evaluation Society (CES)
 by Jean-René Bibeau

European Evaluation Society (EES)
by Hellmut Wollman

Malaysian Evaluation Society (MES)
by Aru Rasappan

United Kingdom Evaluation Society (UKES)
by Nick Tilley

I&CCE TIG
by Mahesh Patel

 

 

Introduction and Narrative Description

Craig Russon and Arnold Love

In the fall of 1997, a discussion was held on the EVALTALK listserv about the international nature of the evaluation profession. One of the principal issues discussed was the creation of a partnership of international evaluation organizations.

Shortly after the discussion, the International and Cross-Cultural Evaluation Topical Interest Group (I&CCE) sent invitations to the presidents of nine national evaluation organizations, inviting them to sit on a panel and debate the pros and cons of a partnership during the 1998 conference of the American Evaluation Association. In addition to the presidents, the I&CCE designated one of its cochairs to sit on the panel to represent the interests of developing countries and regions of the world that do not have a formal evaluation association.

A special listserv was created for the exclusive use of the Presidents Panel participants on the mainframe computer at Western Michigan University. The listserv enabled the subscribers to discuss preliminary issues and to agree on the format for the panel.

On November 6, 1998, during a plenary session of the American Evaluation Association conference, the presidents of several national evaluation organizations sat on a panel and debated the pros and cons of entering into a partnership. The organizations represented on the panel were the American Evaluation Association, Associazione Italiana di Valutazione, Australasian Evaluation Society, Canadian Evaluation Society, African Evaluation Association, and the United Kingdom Evaluation Society. Dr. Arnold Love, former president of the Canadian Evaluation Society and principal organizer of the International Evaluation Congress that was held in Vancouver, Canada, in 1995, moderated the panel.

Position papers had been submitted in advance by the organizations represented on the panel as well as the European Evaluation Society and Malaysian Evaluation Society. This document contains a summary of the Presidents Panel and the position papers.

Narrative Description of the Presidents Panel

At the outset of the plenary session, Craig Russon, cochair of the I&CCE, introduced himself and gave the background of the Presidents Panel. He recognized the contributions of an advisory panel that consisted of the following members: David Williams of BYU; Michael Bamberger of The World Bank; Ellen Konrade, the AEA TIG Coordinator; Mahesh Patel of UNICEF ESARO; Paula Bilinsky of the Peace Corps; and Jim Rugh of CARE. The strong support of Daniel Stufflebeam, Director of the Western Michigan University Evaluation Center, was also recognized. Lastly, Russon introduced the moderator of the Presidents Panel, Arnold Love. Dr. Love is the former president of the Canadian Evaluation Society (CES) and principal organizer of the 1995 International Evaluation Congress in Vancouver, BC.

Arnold Love introduced the panelists: Mahesh Patel, I&CCE representative to the panel; Sue Funnell, president of the Australasian Evaluation Society; Mita Marra, representing the Italian Evaluation Society; Jean-René Bibeau, president of the Canadian Evaluation Society; Nick Tilley, president of the UK Evaluation Society and who also represented the European Evaluation Society; and Donna Mertens, president of the American Evaluation Society. Love stated that from time to time he would represent the position of Aru Rasappan, president of the Malaysian Evaluation Society (MES). The boards of the participating evaluation societies were thanked for making it possible for their presidents to attend the conference.

Lastly, Love described the format of the panel. He said that each president and representative would be asked to make introductory remarks, after which six major questions would be asked of panel members on a rotating basis. Then other panelists would be invited to discuss the issue at hand. Fifteen minutes would be reserved at the end for questions from the audience.

Mahesh Patel made his introductory remarks first. Patel clarified that, not only was he on the panel to represent the I&CCE, but also the African Evaluation Association-Kenyan Branch. The African Evaluation Association is interested in learning from other evaluation organizations but also has much that it can share with them. Examples given include infant mortality reduction, literacy, international cooperation, compliance with treaties and conventions, human rights fulfillment and violations, use of social change, and innovative evaluation techniques.

Sue Funnell began her introductory remarks by saying that members of the Australasian Evaluation Society (AES) think that a world community in evaluation is an idea whose time may have come. Her hope for the panel was to explore the idea thoroughly and move ahead with the initiative or lay it to rest for the time being. The AES Board sees three basic principles as important for any kind of world evaluation community: (1) the community must be an international partnership of evaluation organizations rather than a federation; (2) existing structures and telecommunications should be used instead of high cost new superstructures; and (3) the partners would value and learn from the diversity rather than try to impose or apply pressure to comply with dominant norms, standards, methodology, etc.

In her introductory remarks, Mita Marra stated that she was representing the president of the Associazione Italiana di Valutazione (AIV), Nicoletta Stame, who was unable to attend the conference. She said that the AIV was strongly supportive of an international community of evaluation associations. The AIV viewed the initiative as an opportunity to increase the exchange among people who do evaluations.

Jean-René Bibeau stated, in his introductory remarks, that CES supports an international community of evaluators. Bibeau drafted its position paper in October and submitted it to the CES board for its consideration. The paper was then posted to the CES web site to engender discussion among the membership. However, there had not been enough time to get feedback.

In his introductory remarks, Nick Tilley reiterated that he was representing the United Kingdom Evaluation Society (UKES) and to a limited extent the European Evaluation Society (EES). Tilley had just come from the EES-AIV joint conference in which 29 countries were represented. There is already a good deal of international dialogue taking place in Europe. The UKES board's position on the matter is that there is a good deal of benefit in posturing dialogue across different positions and different methodologies and that a good deal can be obtained from buy-in. The board was skeptical about the value of something as formally constituted as a federation. A loose arrangement that facilitates dialogue is preferable to a formal organization. There was concern that costs of forging some new international organization would outweigh the benefits that might be derived by individual members. The idea of dialogue and of sharing views was thought to be important as was the idea that this needs to be done as equals, not dominated by any single country. Tilley stated that many of the European national societies have been formed recently and there is some concern about trying to create an international organization while they are still engaged in the time-consuming process of forging individual national set-ups.

Donna Mertens was the last panelist to make her introductory remarks. Mertens said that she had put the idea of an international partnership before her board, who had given it a positive response. Such a partnership would offer a breadth of perspectives and understanding, as well as create new opportunities for involvement. Evaluation theory and methods cannot be transported abroad without adapting them to the local culture. It is important to open the avenues for this kind of meaningful participation in the development of an evaluation community.

After the panelists made their introductory remarks, Arnold Love directed his first question to Sue Funnell: What is the AES vision for a greater world evaluation community? Funnell stated that the AES vision is one of a world community of evaluators who learn from diversity of practices across the globe. The world community would really be a product of a partnership of all evaluation organizations.

Jean-René Bibeau spoke in support of that vision and added that it should have diverse backgrounds and experience as the basis for interacting with each other, a common code of ethics, a common understanding of their mission, and acknowledgment of common goals and confidence in exchanging expertise, approaches, and information.

Nick Tilley questioned the desirability of sharing common ethics, common missions, and common goals. Evaluation, he said, takes place in rather different contexts in different countries and for different purposes. Part of the diversity of evaluation might include diversity in terms of goals and perhaps missions. On ethical issues, there is a certain level of generality. It would be easy to agree on general matters. However, he said, there may less agreement on specific matters.

Donna Mertens acknowledged that during the discussions on the evaluation presidents' listserv, there was concern about evaluation becoming monolithic, dominated by one or a few organizations, and homogenized. However, she said she did not think it necessary for all of the organizations to adopt the same approaches or the same standards. Her vision for a world community was one in which national organizations worked together to strengthen the presence, the practice, and theory of evaluation. Such a community could make an strong contribution to the improvement of the effectiveness of programs, policies, and services. The benefit of this improvement would accrue to domestic as well as international evaluators. Increasing worldwide understanding of evaluation would promote the establishment of new evaluation associations in other countries. All of us would be beneficiaries.

Arnold Love shared the MES position on this question. He said the MES would like to see "a global entity that represents the interests of evaluation." MES probably goes the furthest in wanting a formal organization that represents all fields of evaluation. Love then went on to pose the second question, "What specific mission or goals do you anticipate an international entity having?" He invited Donna Mertens to present the AEA position.

When going from vision to mission, Mertens said that she asked herself, "Why is this important for members of the AEA, as well as evaluators around the world?" She found a partial answer in a forthcoming volume of New Directions for Evaluation entitled "Framing Participatory Evaluation" (Cousins & Whitmore, in press). In one of the chapters, Sharon Brisolara (in press) reported that one form, practical participatory evaluation, arose in the U.S., U.K., Canada, Norway, and Sweden. Another form, transformative participatory evaluation, arose in Latin America, India, and Africa. This is an example of how culture can influence evaluation theory and practice.

Mertens went on to say that she thought an important goal was to provide forums for sharing information. This would provide an opportunity to share information about evaluation theory and practice, ethics, and standards, and about cultural dimensions. Comparing concerns across nations would help to deepen everyone's understanding about these issues. A second goal identified by Mertens was to provide support to emerging communities of evaluators in their struggle to establish formal evaluation associations in their home countries. In the absence of formal organizations, we could provide alternate mechanisms for involvement in the evaluation community.

After Merten's presentation, Sue Funnell spoke in favor of a mission to improve the profile and status of evaluators as professionals and of evaluation as a profession. This would improve the capacity of evaluation to bring about improved results to the world community. One indicator of that capacity would be opinion leaders in government, international agencies, NGO, and private sector business leaders placing a high value on the role of evaluation in decision making, knowledge generation, and accountability. Three specific goals might be (1) to increase the rates and scope of improvements to theory, practice, and use of evaluation; (2) an increase in the rate and scope of growth of public knowledge; and (3) to increase size, influence, and services of evaluation organizations--both existing and developing organizations.

Jean-René Bibeau added that another goal should be to build a common understanding of the profession. In the CES position paper, he had argued for attaining a universal ground for the profession. At the same time, he agreed with the words that had been spoken regarding respect for diversity of social and economic context everywhere on the planet.

Mahesh Patel cited the many Year 2000 initiatives of the United Nations that seek to make the world a better place. He stated that he thought the structure of an international evaluation entity would follow the U.N. model. Patel suggested that the topical interest groups of each society could cooperate on policy issues. He also underscored the importance of providing support to nascent evaluation societies. He expressed concern that the evaluation network of which he was a part in Kenya was suddenly forced to constitute itself as a formal society by the advent of the Presidents Panel.

Mita Marra suggested that a statute from AIV might be appropriate as a goal for an international community: "Promoting the culture of evaluation in public administration as well as among professionals. Providing training and education and evaluation theories and methodologies. Improving the quality of evaluation. Linking the people in the evaluation field. And thirdly, the utilization of evaluation."

Nick Tilley stated that the socioecological literature is critical of notions of professionalism and exclusion in claims of competence and expertise. He expressed discomfort about appropriating all the trappings of a profession. He thought that the notion of improving evaluation would be fairly contestable because of differences in opinion as to what constitutes a good evaluation. He would prefer to increase diversity rather than expect convergence.

Arnold Love thanked the panelists and then posed the third question, "What are the major options for structuring a partnership?" He invited Mahesh Patel to present the I&CCE position.

Patel said that he saw two possible scenarios for international cooperation. The first was an international community of evaluators and the second was a federation of evaluation associations. He thought that the concrete options for achieving a world evaluation community depended on which scenario emerged.

Patel saw an international evaluation community as a group with interest in international evaluation that focused on the professional development of its members. In contrast, a federation would focus more on institutional linkages and institutionally oriented support to associations in developing countries. The unit of membership for the international evaluation community would be individuals and, to a certain extent, it would compete with national evaluation organizations. The unit of membership of the federation would be national evaluation organizations. The federation would be complementary to national evaluation organizations--not supplementary. An international evaluation community would have an elected president and board of directors. The leadership of a federation would rotate from member to member. However, a federation would require a permanent secretariat to provide for an element of continuity.

Patel identified a number of problems with an international community and a federation. Some people would join an international community, others would not. It would grow slowly and wouldn't really move us forward. A federation would have a heavy structure, and equality among large and small organizations might be an issue.

Lastly, Patel identified problems with the whole notion of national evaluation organizations. How does one define a national organization? Does it consist of 5 members, or 20, or 50? Must national organizations have a president and a constitution? In countries with military governments, it is dangerous to form a group with a recognizable president. Informal networks would be much more desirable. Almost as an afterthought, Patel said that he saw the creation of an international community or a federation happening by the Year 2000.

After Patel's presentation, Jean-René Bibeau said that CES would be in favor of proceeding slowly with any initiative. He felt that a transition barrier from now till the Year 2000 would be appropriate. This would give the organizations time to understand the issues completely.

Nick Tilley added that the notion of a federation implied a closer link and a more coordinated, more unified approach than UKES would like. His board would favor something like a league. A commonwealth would suggest a sort of coherent and more organized body. Tilley said that UKES saw the benefits of international conferences because of the EES experience. However, it was concerned about the venue of such conferences. Conferences could not only be held in the developed countries. Then there are the problems of language. How would we provide for non-English speakers? Lastly, where would the center of the international community of evaluators be located? It could not be based in the U.S. or Europe. It would have to rotate.

Donna Mertens pointed out a fundamental dilemma: an international community of evaluators would not build on capacity that is already in place through existing organizations; and a federation would leave out evaluators in countries where formal organizations do not yet exist. She suggested that the Joint Committee on Standards for Educational Evaluation might be a suitable model.

Lastly, Sue Funnell said that AES would support an alliance operating as a partnership among current and emerging organizations. However, it would not support a federation as a body with a governance role in relation to member organizations.

Arnold Love thanked the panelists and then posed the fourth question, "What are the important issues that need to be resolved before making a decision to seriously pursue the idea of building a world community of evaluation?" He invited Nick Tilley to present the UKES position.

Tilley reiterated several points that he had already made. UKES was concerned about the exclusion of evaluators who live in countries where there is no national society, resolution of the language issue, and domination by any single country. After Tilley's presentation, Mita Marra voiced AIV's support for broad inclusion.

Sue Funnell added that there were three issues that were of concern to the AES. First, any initiative must add value to existing services and must perform functions that cannot be achieved more cost-effectively by other processes. Second, a model must be adopted that would be politically sustainable. Third, and last, the model must be operationally feasible given the level of funding, involvement of personnel, and logistics of common action.

Jean-René Bibeau said that CES shared the some of the concerns voiced by UKES and AES. He added that the use of technology could be used to mitigate some of these concerns. Donna Mertens stated that cost of any initiative and the sources of funds were also an important issue.

Arnold Love thanked the panelists and then posed the fifth question, "What kind of contribution is your organization willing to make?" He invited Jean-René Bibeau to present the CES position.

Bibeau stated that CES was ready to support a partnership of national evaluation organizations. However, it must first be convinced that there would be sufficient equitable benefits for the costs. After Bibeau's presentation, Donna Mertens said that AEA support could include use of the association's annual meeting scheduled for 2000 in Hawaii. Sue Funnell stated that AES had discussed a minimum pro-rata contribution calculated on the basis of membership size and fees to cover basic running costs. The AIV, said Mita Marra, could not afford a monetary contribution. However, it could make human resource contributions. Nick Tilley said that UKES was also not in a position to contribute to the financing. He was concerned that those who contribute the bulk of funding would exert undue influence. He added that it was not unreasonable that those who do not pay, do not warrant a seat. However, the exclusion of those who cannot pay would lead to international bias.

Arnold Love thanked the panelists and then posed the sixth, and last, question, "If the decision to move ahead is made, what are the essential steps needed to manage the process?" He invited Mita Marra to present AIV's position. Marra said that the AIV supported the organization of a partnership in a conference format.

After Marra's presentation, Donna Mertens said that representatives from the national organizations and from countries without organizations should be identified. This group would agree upon a mechanism for planning. The process might include examining the structure of other international organizations. Then a proposal for continuing the planning process would be made. It was essential, Mertens said, to keep the memberships of their respective organizations informed of events.

The AES, said Sue Funnell, had envisioned a similar process. She added that it might be desirable to do some small scale pilot projects to ensure that the organizations could work together and build momentum for the process. As the concept broadened, reviews of viability and implications could be conducted. Jean-René Bibeau referred the audience to the terms of reference that the CES included in its position paper.

Mahesh Patel suggested that a summit meeting of presidents be convened by the summer of 1999. During the summit, the presidents could develop a framework for the partnership. The Presidents Panel could be reconvened at the end of 1999 to report the results of the deliberations. In 2000 one of the major evaluation organizations might consider devoting a meeting to the theme of international evaluation, and the federation could be launched at that time. Lastly, Patel suggested that the I&CCE serve as interim secretariat while details of the federation are elaborated.

At the conclusion of Patel's remarks, Arnold Love opened the floor to questions from the audience. After 15 minutes, Love thanked the panelists for their participation and the audience for attending the Presidents Panel. He invited everyone to proceed to the reception next door where a silent auction was held to raise money for a travel fund to help evaluators from developing countries attend future AEA conferences. With that, Love adjourned the plenary session.

 

American Evaluation Association (AEA)
Working Document for Discussion
Building an International Evaluation Community

Donna M. Mertens
Gallaudet University
President, American Evaluation Association

Over the past year, the presidents of nine evaluation organizations from around the world have been participating in a discussion via email (and sometimes in person) of issues related to establishing an international community of evaluators--some think of it as a federation of evaluation organizations, others want to avoid what they view as an overly formalized, potentially oppressive administrative structure that a federation might pose. Representatives from evaluation organizations from the United States, Canada, Australia, England, Africa, and Italy participated in a panel during a plenary session at the annual meeting of the American Evaluation Association in November 1998 in Chicago entitled: The Presidents Panel on Creating a World-Wide Evaluation Community.

In preparation for the plenary session, we were asked to respond to six "talking points." I shared the outline I had prepared with the board of the American Evaluation Association prior to the November meeting, and then held a discussion of my ideas during the board meeting. The board was supportive of the ideas in concept and desirous of knowing more specifically what is planned in this effort to build an international community of evaluators. Many AEA members were able to hear the plenary in Chicago, and they provided me with feedback following that session. The purpose of distributing this document now is to obtain reactions from a wider representation of the AEA membership as we begin the planning process of building a worldwide evaluation community. The text of this document represents my current thinking and I welcome your reactions. Please contact me directly via email with your reactions at Donna.Mertens@Gallaudet.edu.

Private enterprise has long recognized the need for a global presence to insure health and prosperity. The private sector's stance supporting the need for strong global relationships mirrors in some respects the potential benefits that the evaluation community and those we serve stand to gain by fostering a global presence for evaluation worldwide. To achieve an international community of evaluators, I will share with you my vision of such a community, missions and goals associated with achieving that vision, major options for structuring the partnership, important issues that need to be resolved, contributions that AEA is willing to make, and the essential steps needed to manage the planning process.

Vision for the Greater World Evaluation Community

In contemplating the title of this plenary, Creating a Worldwide Community of Evaluators, I asked myself why this is important for the members of the American Evaluation Association, as well as for evaluators around the world. I found a partial answer to the query by looking at examples of the kind of learning that can occur when communication on an international level is improved in the evaluation community. One such example can be found in a forthcoming edition of a volume of New Directions in Evaluation entitled "Framing Participatory Evaluation." The editors, Brad Cousins and Bessa Whitmore (in press), describe two contrasting developments in participatory evaluation that they call practical participatory evaluation and transformative participatory evaluation. These two approaches share some common characteristics, such as both seek to foster participation in the evaluation for the purpose of increasing understanding of program functions and processes, and to develop skills in systematic inquiry. The two traditions did grow up in very different regions of the world and have significant differences.

In a chapter on the history of participatory evaluation, Sharon Brisolara notes that practical participatory evaluation arose primarily in the United States, Great Britain, Canada, Norway, and Sweden, and reflects the cultural lens predominant in those countries regarding social science research. In the practical participatory approach, the involvement that is sought is mainly that of the program decision makers, such as program sponsors, managers, developers, and implementors. The transformative participatory evaluation approach arose in Latin America, India, and Africa, and reflects the cultural and social conditions of those countries. Brisolara describes the conditions under which transformative participatory evaluation emerged: "Thus, much of their work was conducted in farmlands, squatter settlements, urban ghettos, and within the field of international development. Many Latin American and African countries were overwhelmed by war and poverty during these decades. Revolutions, such as those in Cuba and Nicaragua, the advance of liberation theology in Latin American countries, and the movements and struggles that preceded independence from colonial powers for many African countries contributed to . . . . ideas and actions" that characterize this approach.

Transformative participatory evaluation is one approach that explicitly recognizes the need for involvement of all legitimate stakeholders, including those with the least power. The participatory approach is not the only way that transformative theory can be integrated into evaluation work. In my presidential address ("Inclusive evaluation: Implications of transformative theory for evaluation," Mertens, 1998), I described the work of feminists, ethnic and racial minorities, and people with disabilities who have developed other approaches to evaluation that are based on transformative principles. However, transformative participatory evaluation does provide one example of an approach that was forged in the face of extremely difficult circumstances with the goal of social transformation for the least advantaged.

Yet, some Western critics have dismissed transformative participatory evaluation because, in their view, it violates the principle of objectivity by attempting to represent divergent perspectives at the community level. As I noted in my presidential address and is noted by Brisolara, transformative participatory evaluators disagree with this criticism. They strive to provide a pluralistic view of the values, beliefs, and knowledge systems through dialogue and negotiation to guard against the undue influence of one dominant voice. In order to advance the theory and practice of evaluation, it is necessary to bring the debates to a more accessible level so that viewpoints as to what is true evaluation can be elevated beyond the experiences of one part of the world. At first glance, the problems encountered in the northern countries might seem very different from those in the southern countries; I would argue that there may be greater generality in conditions when one looks below the surface. The Southern Hemisphere is not the only venue in which poverty and its associated powerlessness occurs. Thus, there is room for learning all around. And my vision is for a community of evaluators that will learn from each other and embrace diversity. The following bulleted points outline my vision further:

Missions and Goals That are Essential for That Vision

Major Options for Structuring the Partnership

Important Issues That Need to be Resolved Before Making a Decision to Pursue Seriously the Idea of Building a World Evaluation Community

Contributions (Financial or Otherwise) AEA is Willing to Make

If a Decision is Made to Move Ahead, What are the Essential Steps Needed to Manage the Process?

At this time, there are more questions than answers. I welcome your thoughts and reactions and suggestions with regard to any of the points made in this working paper. You can send your reactions to me via email: Donna.Mertens@Gallaudet.edu.

 

AEA Presidents Panel
Associazione Italiana di Valutazione (AIV)

Nicoletta Stame, President

Dear Colleagues and Friends,

It was a great community of evaluation associations. This invitation means the pleasure to receive the invitation to participate in this panel about the creation of an international organization so you know what we - just beginners - are trying to do. And it is important for us to have the opportunity to share with you our feelings and our concerns.

In the field of evaluation everything is new and recent in Europe. Associazione Italiana di Valutazione was one of the first evaluation associations to be set up in Europe, in February 1997, after the UK Evaluation Society (UKES) and the European Evaluation Society (EES). We, in our country, have felt the need to upgrade the activities that were taking place in a scattered way by building upon the experiences going on in other evaluation communities. Those who participated in AEA conferences in Boston, Vancouver, and Atlanta had come to know how important it is to discuss professional, theoretical, and political matters in evaluation among people who have different persuasions, institutional roles, and social backgrounds. We took a timely decision to form an association, however not an unexpected one: others had shown us the way.

AIV has undergone a twofold process. On the one hand, AIV is the result of the networking of professional and academic people who got interested in the subject. They began to exchange ideas within seminars and workshops that have taken place since winter 1996. For example, it is worth recalling the Socrates seminars and their related publications.

On the other hand, AIV is the response to the demand for evaluation that has gradually arisen in our country. It is due to the European Union Structural Programs requirements which request ex-ante, midterm and ex-post evaluations. However, this demand has been driven by the public sector reforms, initiated by the Department of Public Administration, since 1992.

I must state in advance that our association did not have the opportunity for a full discussion about the international community of evaluation associations among its members. So, I will give you what I believe is the general opinion of our members.

First of all, I will state the AIV mission and vision, quoting from our statute:

Complementing our statute are the "Guiding Principles for Evaluators," which deal with ethics and standards.

Coming to the main points:

Vision for a Greater World Evaluation Community

I prefer the idea of a "partnership bound by a memorandum of understanding" between existing organizations rather than a new bureaucratic organization and a task force of representatives on a rotational basis rather than a formal body.

Indeed, an international community can grow only if there is a real exchange between different experiences and if it is possible to find out what we can learn from each other. We are interested in this community because we feel part of the actual international process of institutionalization of evaluation, which has had an unequal development.

Mission and Goals

I think the main goal should be "Learning through diversity" by knowing what the others are doing in common fields of interest.

Major Options

I prefer international conferences, debating themes of common interest.

Issues

The most important issue is how to reach organizations or informal networks that are not in the core group of the promoters of this initiative. I see from my own and the other Europeans' experience how difficult it is to think in terms that we believe overwhelm our actual situation, although we would benefit a lot from this connection.

Contributions

As you can imagine, we are a nonprofit organization existing on our membership fees. I don't think we could finance an international body, if we cannot finance our own association first (all our work is on a voluntary basis). Rather, we could consider a human resource contribution: some of our members could be delegated as representatives in that "rotating task force" or something similar.

Essential Steps

I believe it is essential to elect or nominate a group that will organize the first conference.


Presidents Panel at AEA
Australasian Evaluation Society (AES) Position Paper

Sue Funnell

November 6, 1998

Introduction

The following paper was prepared by the 1998-99 President of the AES, Sue Funnell. The AES Board gives endorsement to the broad concepts in the paper in principle without at this time making a commitment to the detail. Such commitments would be made after the panel discussion and depending upon the likely viability of an international strategic alliance of evaluation organisations. As a means of testing the palatability to other organisations of the concepts involved in the AES position, I suggest that where "AES" appears throughout the document, others substitute the name of their own organisation and see if it fits.

The AES sees the primary goal of the panel as determining whether there is sufficient agreement amongst the various evaluation organisations concerning the potential value of and general nature of such an alliance to warrant pursuing the concept further in terms of determining the details of operation and testing its feasibility.

Hence, pending the results of this panel, the AES Board reserves its judgment concerning the value of proceeding with such a strategic alliance in any formal sense. At the same time the AES Board recognises that some benefits have already started to flow from the informal relationships that have begun to be established amongst the various organisations through the discussion leading up to this panel. Reciprocal membership arrangements amongst some organisations and contributions to each others' publications from time to time have also contributed to the development of alliances.

The AES Board supports the notion that an alliance should operate as a partnership amongst current and emerging organisations rather than as a separate organisation that would be superordinate to other existing regional- and country-based evaluation organisations. The AES does not support the notion of a Federation as a body with a governance role in relation to its member organisations or in relation to the profession of evaluation. This means that voting rights and issues of universal representation are less important than involvement of willing partners who believe that they can work together to achieve mutual interests and advance the profession of evaluation more effectively through collective efforts than through the efforts of individual organisations.

Where possible the alliance should build on existing structures within the various organisations with minimal establishment of new structures. It should be low cost with minimal administrative support. The AES recognises, however, that some coordinating processes are required to undertake joint projects and to consolidate and carry forward the relationships that have already been established amongst current office bearers of the various associations.

The development of information technology as an instrument for communication renders the creation and operation of such a partnership both timely and feasible. "Virtual" approaches to a partnership are now possible where ten years ago a partnership would have needed to operate as a physical structure with a physical location. That said, we do need to keep our feet on the ground about the issues involved in setting up such a partnership or we really will disappear into the ether.

Over the past several months the presidents have (by an electronic discussion list) tossed around various concepts stemming from the original proposal that a "federation" be created. The original concept seemed to invoke concerns about possible domination by one or more of the organisations. Accordingly, our thinking has shifted toward the idea of a world community of evaluators.

However, a world community of evaluators does not simply happen. It needs some catalytic people and processes. A partnership amongst evaluation organisations may be one instrument for achieving such a world community.

Domestically, the AES is strongly committed to valuing and learning from diversity. This principle is enshrined in our statement of "Values and Principles to Guide Board Decision-Making." The AES would want to ensure that any partnership of international organisations shows similar commitment to valuing and learning from diversity rather than promoting compliance with dominant models or interests. Indeed, the greatest value that can probably be added by an international partnership will be the learning that can come from diversity. That diversity is what will set an international partnership apart from existing mechanisms for learning.

In addition to the above concepts of a low cost partnership that values and learns from diversity, the major prerequisites for AES participation are that the partnership be substantially valuable, politically viable, and operationally feasible.

Interest in information products

The AES Board has an interest in the benefits that an international partnership might provide in relation to enhancing the operation of the AES as an organisation and reciprocating in relation to other evaluation organisations (both established and emerging). It also has an interest in the wider advocacy role that an international partnership might play by virtue of its higher status and greater strength. However, such benefits are unlikely to be of immediate interest to the broader membership and would be an insufficient justification for the establishment of an international partnership of evaluation organisations using members' funds.

There is much work to be done to identify the financial requirements and implications of developing, sustaining, and enhancing an international partnership. There are also many issues that need to be resolved in terms of identifying principles for membership of the partnership such as whether the partnership should be open to any evaluation organisation that wishes to join and how regions without a formally recognised organisation but with strong interests in promoting evaluation can become involved. The next steps should be for a working group to go beyond general principles and identify a practical blueprint for operation of the partnership.

This paper is structured around the six questions posed by Arnold Love as the facilitator. Answers to the first three questions, and more importantly the relationships amongst those answers, are portrayed in the figure included in this paper.

1. What is the AES vision of a greater world evaluation community?

Here, the term vision is used to refer to our idea of what the world evaluation community would look like:

Our vision is of a world community of evaluators who learn through the diversity of practices across the world. This world community of evaluators would be one of the products of an international partnership of evaluation organisations.

Why would one want to create a partnership of evaluation organisations and a world community of evaluators? What would be the mission and goals of such a partnership, and who would be expected to benefit from it?

2. What will be the mission and goals of such a world evaluation community?

The mission of such a partnership would be

The beneficiaries of such a partnership would be evaluators, evaluation users and theorists, evaluation organisations, and the world community benefiting from the contributions made by better and more pertinent evaluations.

The goals through which the mission and the vision would be achieved would relate to

Each of these goals would be served by a series of enabling objectives (intermediate outcomes) that derive more directly from the roles performed by the partnership. There are interdependencies amongst the goals.

3. What are the major options for achieving your vision, mission, and goals?

In response to this question, I have identified four major areas in which decisions need to be made:

Roles and Strategies

The AES Board takes the position that at this stage it is more appropriate to agree on broad roles and strategies rather than specific activities for implementing those strategies. It has not discussed in any detail the desirability of specific activities such as the production of a journal or international conferences. However, I can say that opinion has been divided about active AES involvement in international conferences when discussions of the matter have arisen in the past outside the present discussion of a more enduring international partnership. The main reservations have related to logistics.

Revisiting the listserv discussion of presidents over the year, I have used three roles and the strategies described below to categorise the various activities that have been suggested on the listserv and in the course of various other discussions with AES members. The categorisation is at attachment 1.

The AES Board gives, in principle, support to the following three roles of a partnership and possible strategies for each role. For illustrative purposes some examples are provided. Further examples of each role are included in the attachment to this paper:

At this stage, the Board has not discussed the relative importance of the three roles in any detail. However, the principle of providing value to members of the AES and concurrence with AES corporate objectives suggests that the first of the three roles would be a sine qua non of involvement in an international partnership.

Our preferred position is that membership consist of organisations rather than individuals. These organisations could comprise

- Established evaluation organisations that would be invited to join the partnership. Each would have a single representative.

- A representative(s) of nations or regions with developing organisations. As an interim arrangement this could be the chair/cochair or nominated representative of AEA's I&CCE TIG on behalf of nations with developing organisations. It may be possible to establish a "chapter" of the partnership for residents of countries where no association currently exists. An elected representative from this group could then become the representative of nations or regions with developing organisations, replacing the interim representative.

Alternatively, there could be a number of such nominated representatives, with each representing a particular region, e.g., one for Africa, one for Southeast Asia.

The AES Board was of the view that if such a chapter were to exist, it should be funded by fees from the members of the chapter. The AES would not be in a position to fund the chapter although it would be happy to explore ways of giving in-kind support to assist with the establishment and maintenance of such a chapter.

There will need to be an ongoing mechanism for governance. This mechanism may be complemented by time limited, action-oriented, project-based groups. In the AES we distinguish between standing committees and project-based task forces, and I would see a similar approach as being possible for a partnership.

The ongoing mechanism could be a coordinating board or committee consisting of representatives from all organisations (e.g., vice president, or president elect, or immediate past president--someone with a good understanding of his/her own organisation and a strategic perspective) plus basic minimal administrative assistance. This could be provided through the administration of one of the larger or more established organisations supported by funding from each participating organisation.

Leadership of the ongoing governance mechanism could be on a rotational basis. A less messy alternative would be to have the leader located with whatever organisation provides the administrative assistance, but this may raise concerns about dominance by that organisation.

The time limited project groups could be mixed working committees coming from the different organisations for particular projects, depending upon which organisations chose to participate in particular projects.

The modus operandi should be designed to keep costs and amount of time committed by participants low. To this end I would recommend that

- Communication be mostly by email with defined targets for responding

- The Board could also have real time on-line conferences (like a chat room, but structured and facilitated by the leader)

- There be an annual face-to-face meeting either at an annual conference of the participating organisations on a rotational basis or at an international conference sponsored by the partnership

The "real work" would be done by the project task forces established by the governing board.

4. What are the important issues that need to be resolved before making a decision to seriously pursue the idea of building a world evaluation community?

As already indicated, the issues relate to whether the roles served by the partnership and the results achieved will be substantially valuable, politically sustainable, and operationally administratively feasible.

More specifically,

Substantially valuable?

Politically sustainable?

Operationally feasible?

5. What contributions--financial and other types of contributions--would the AES be willing to make?

The board considers the following order of magnitude of contributions to be appropriate, but would want to reassess in the light of other decisions about the partnership:

Much of the work of substance could occur through working groups of representatives from existing committees of the participating organisations working on joint projects. Existing committees would be encouraged by their respective boards to look for opportunities to initiate and participate in partnership projects. Project budgets would be reflected in their budgets following agreement to participate in a particular project.

6. If a decision is taken to move ahead, what are the essential steps needed to manage the process?

Conclusion

In conclusion, it is useful to reflect on the fact that the Joint Committee that eventually produced the Joint Committee Standards for Educational Evaluation has been suggested as a possible model for the partnership of evaluation organisations. The Joint Committee, consisting of 12 organisations, began its work in 1975 and produced the Standards in 1981. After six years of operation, the Committee was incorporated as a continuing nonprofit organisation. In 1989 the Joint Committee's process for developing standards was accredited by the American National Standards Institute and as such became available worldwide.

This example shows two things:

We must, therefore, at the one time set ourselves some objectives that will deliver tangible, short-to medium-term benefits to the members of all of our organisations while at the same time being realistic about what will be required to bring the partnership to a point at which it can work constructively as a unit to achieve those benefits. Let's choose at least some objectives that can be met in considerably less than 6 years. If we cannot deliver before then, I have no doubt we will lose whatever support we will have garnered from our respective members in the meantime. Without that grass-roots support, the partnership will undoubtedly founder.

Finally, the AES is keen to give the concept of an international partnership a proper trial. It is a concept for which we have considerable enthusiasm, but we are very much aware that the value of the concept may be easily depreciated by poor implementation strategies. We consider that the greatest challenge of an emerging partnership is to set objectives that are both valuable and realistically achievable given our capacity to design and carry through an appropriate implementation strategy.

Attachment 1

Possible Types of Activities for an International Partnership

Italicized items show my preferences; although the AES Board has seen these lists of activities, they have not discussed them in detail.

1. (Brokering) or providing intellectual capital in evaluation

- web-based connections, e.g., international WWW page with links to national societies; coordinated development of web sites to enable better exchange of information

- reciprocal membership arrangements giving access to printed materials

- production of special issue publications on international evaluation issues and case studies

- identification of particular strengths of particular nations with respect to evaluation and special projects to disseminate that information: filling the gaps by learning from each other

- key information managers whose responsibility would be to filter questions and information on an international basis that relate to a particular area of theory, practice or use; publishing guides of abstracts to the international evaluation literature in relation to particular topics

- system for traveling evaluators to notify local evaluation societies of their plans so they can participate in any scheduled activities or be invited as guest speakers

- international conferences, e.g., sponsored by participating organisations on a rotational basis, say once every 3 to 5 years

- identifying/developing/sharing information about standards and principles for evaluation

- conduct joint comparative analyses of government policies on evaluation and use the report to advocate for greater and more efficient use of evaluation (I have some doubts about the capacity of outsiders to a region to substantially affect government policy)

- subsidising exchange of personnel at conferences (responsibility for this should rest with individual organisations)

- international journal (the general consensus on the Presidents' listserv seemed to be that this wasn't needed and might split the market too much). The journal Evaluation markets itself as the international journal of theory, research, and practice.

• access to training, skills, personnel, and technologies

- facilitating exchange arrangements with institutions from different regions/nations seconding evaluators to each other for varying periods of time

- providing training in areas where particular nations have strengths for benefits of those nations that have weaknesses/needs in those areas, e.g., by distance education with possibility of mentoring and brief residential component

- facilitating access by developed nations to less costly provision of some evaluation- related services, e.g., data entry

- assisting national agencies in developing countries to evaluate development assistance projects

- identifying skilled personnel and facilitating access to them, e.g.,international job bank for vacancies that allow international recruitment

- international scholarships

- supporting evaluation of multilateral assistance programs in developing countries

• information exchange with related professions at an international level

- identifying international organisations with related interests

- using key information managers to tap into networks of related professions that might be unevenly scattered across the world and identify areas of interest to evaluators: journals, conferences, themes

2. (Brokering) or providing international exchange of information about issues of world importance

• Analysis and synthesis of processes and results of evaluations across the world in areas of knowledge

- meta-analysis and evaluation synthesis of international efforts in particular areas of knowledge (health, education, etc.) and publication of international reports on results

- metaevaluation of approaches used by different nations to evaluate particular types of projects and publication of international methodology reports

3. Initiation of studies to fill gaps

- identify topic priorities for multicountry evaluation efforts and promotion of comparative research at systems level

- develop a coordinated response to progress reviews from U.N. sponsored international initiatives in the Year 2000 (Human rights, HFA 2000) and hold a press conference to disseminate conclusions of discussions

Supporting Evaluation Organisations

- identifying and fostering relevant points of contact amongst the various national organisations; e.g., establishing a network of those members of the various boards that are involved with awards, with ethics, with professional training and development, and encouraging joint projects

- identifying and documenting what appears to work well for members

- documenting the most important characteristics of an effective evaluation association

- translating mission statements of existing societies into several languages and comparing similarities/differences

- coordinating approaches to World Bank and other international organisations: OECD, Rotary, etc.

- identifying and acting on opportunities for international advocacy for evaluation; promoting recognition of evaluation

- endorsing efforts by particular nations to seek additional support for evaluation from within their own nation

- giving permission to developing organisations to distribute AES publications where it won't compromise benefits and entitlements of AES members

- encouraging AES members who are visiting regions with developing organisations to give some services gratis, e.g., addressing a seminar on work they are doing, AES activities

- identifying opportunities for keynote speakers from regions with developing organisations to address AES conferences



An "International Organisation of Evaluation"
Position Paper of the Canadian Evaluation Society (CES)
presented to the Panel of Presidents

Jean-René Bibeau

American Evaluation Association Annual Meeting

Chicago, 1998

française

Introduction

The Canadian Evaluation Society (CES) recognizes the unique value of international collaboration among evaluators and their national organizations in the increasingly "global village" of evaluation. In recognition of a need for further collaboration, the CES is initiating this position paper on the issues and principles associated with the concept of an international evaluation organization. It will participate in the panel discussions at the AEA meetings in Chicago, in the hope of devising a framework to more fully explore possible avenues of international collaboration.

The Society recognizes that evaluators and their organizations operate within, and are influenced by, a very diversified social and cultural milieu. The CES board thinks that further collaboration is necessary for the benefit of its members and that it is essential for the Society to play an important role in the development and promotion of evaluation internationally while recognizing our own diversity. CES has already demonstrated leadership in international collaboration and will continue to pursue that objective.

The potential of modern technology as a means of information exchange and communication between evaluators creates exciting international opportunities. Some form of international organization, moreover, creates additional opportunities for evaluation and evaluators. CES board members think that there are numerous options to help fulfill a global evaluation mission. These options should be assessed in the interests of both the members of our respective national organizations and the international community of evaluators. It is with great interest, enthusiasm, and caution that the CES enters the discussion of international collaboration.

Background

The CES believes that it is important to clarify the context of the debate. The forthcoming discussions on an international evaluation organization originated as a proposal that first appeared on EVALTALK in 1997. Several members of the AEA suggested that the name of their association be changed to the International Evaluation Association as a means of recognizing that many of its members were not from the United States of America.

Further discussions on the topic indicated that the term "international" had both many meanings and implications. At the same time, evaluators in other national organizations insisted that the concept/issue should be discussed in a much broader context. Hence, the concept of an International Federation of Evaluation (IFE) was created.

In the spring of 1998, the International and Cross-Cultural Evaluation Topical Interest Group (I&CCE) thought it should resume the previous discussion of an international organization on XC-EVAL and it was subsequently proposed that a panel of national presidents would discuss the issues associated with the concept of a federation at the 1998 AEA meetings in Chicago. In addition, an electronic forum (EVAL-PRES) was established for presidents to discuss some of the issues in preparation for the meeting.

Some Initial Principles

In pursuit of a framework for the discussions of international collaboration and some form of organization, the CES strongly suggests that the following principles be an essential part of the context and deliberations:

1. Only an assembly of a nation's organization can sanction both involvement and commitment to an international organization.

2. An assembly of duly sanctioned representatives from our national organizations is required to legitimize and to obtain the commitment for an international organization.

3. Any future form of international cooperation requires an identification and assessment of the international/national needs it is to address.

4. Clear identification of beneficiaries of an organization's activities is key to any commitment.

5. The CES considers a Code of Ethics and the principle of equity to be essential components of our organization and essential features of an international endeavor.

6. Involvement in an international organization shall be considered an "investment" by the CES on behalf of its members, and those benefits should be defined and assessed by its members.

7. A wide range of cooperative activities allow for the pursuit of international initiatives.

Needs and Benefits

The CES recognizes that the potential benefits of international collaboration and organization are numerous for both the field of evaluation and evaluators. We also recognize, however, that the benefits and beneficiaries should be identified in a precise manner. The activities of an international organization must be preceded by a thorough identification of the needs of evaluation and evaluators in an international sense, while recognizing that the needs of members of national organizations also have to be addressed to justify an "investment" in an international venture.

In theory, many can benefit from international cooperation and/or organization. Practical experience tells us that this is not typically the case. Some members will have more opportunities to promote and/or practice evaluation. The CES recognizes that opportunities will obviously gravitate toward some international members such as academics and consultants, but not most members. Although we recognize and salute these opportunities for some, our organization feels that the benefits to some interest groups should be of secondary importance to the needs and interests of our prime beneficiaries, the general membership.

The CES feels that international endeavors create new and exciting opportunities for the members of our national organizations whose interests are varied and often conflicting. We should openly address this fact and pursue international endeavors with a clear assessment of both the nature of needs and the likely beneficiaries of our endeavors. Consequently, goals and priorities can be more readily defined and activities/operations more easily devised and assessed.

Constraints and Priorities

The CES believes that a decision to venture into any form of an international evaluation organization requires both sound organizational and financial analyses. There are numerous, some unforeseen, financial implications to any concept of an international organization. Financial commitments are required for both a constant and contingent flow of funds to ensure the success of our endeavors. Adequate financial assessment and planning will be required to ensure that an international organization is not only viable, but capable of sustainable growth.

As we examine the plethora of international goals that we may entertain, we should also be aware of the necessary structures and operations needed to adequately attain these goals. In our deliberations, we should assess the experience of other international organizations that have pursued a variety of organizational options in an attempt to realize their goals. We must also recognize that some goals may be more aptly pursued by the enhanced activity of a national organization in selected areas--an international division of labor of sorts.

We also believe that the marriage of the evaluator needs and those of the international evaluation can be served through a variety of means. The activities may take the form of conferences and networking. More global needs, such as advocacy and joint mission, may require more complex and organized strategies. All strategies, as previously stated, should first reflect the majority of needs in our sister organizations. Also, our choice of endeavor(s) should be based on a multiplicity of strategies and appraised options for each strategy. A large international organization, in essence, is not necessary to expedite forms of international collaboration.

Any form of international collaboration, formalized or not, has to reflect a principle of equity not only in benefits, but also in costs and responsibility. The success of our endeavors, especially one of this nature, will require clearly identifiable benefits for member organizations and clearly distinct responsibilities for our national organizations. Both actions will allow members of our national organizations to fairly assess the required levels of commitment and participation. Equity has to be one of the underlying values of assessment.

With recent and future developments in technology, CES is very excited and interested in pursuing innovative means of international collaboration. Some of these means require little or no formal intervention. Other means of collaboration require highly organized activities, some of which can overshadow our national organizations and their members. We must pursue these endeavors while preserving the strength and richness inherited from our diverse characters in both context and approach.

In the CES, for example, most members are practitioners. The majority are employed in the public sector. To many, most forms of internationalization, especially in evaluation, may not be perceived as beneficial. Likewise, they work and live in diverse social and cultural settings. An international effort has to address this diversity of experience and need. It can do so by creating new and exciting opportunities in learning, teaching and practicing evaluation which reflect our diversity and respect our unique organizational cultures.

Concluding Statement

The Canadian Evaluation Society sees great promise in both promoting evaluation and further developing practice in the global village. Ours is a vision of an increasingly mature and continually evolving profession in all nations. A profession recognized for its expertise and transcending cultural difference.

We foresee an international community of professionals with both diverse backgrounds and experience whose basis for interacting with each other is a common code of ethics, a common understanding of their mission, an acknowledgment of common goals, and a confidence in exchanging expertise, approaches and information. Most importantly, we foresee an international collectivity who choose to enrich their knowledge and practice through a mix of cultural experience.

Since the world of information exchange has accelerated the ease of and reduced the cost of communication, international collaboration does not necessarily imply additional organizational structures nor their costs. If the roles of evaluators as an international community can be best enhanced by some organizational form(s), we will do a disservice to our members by not enthusiastically pursuing an international entity.

The CES will further commit to an international entity when its assembly of members are satisfactorily convinced that there are sufficient and equitable benefits for the costs and efforts required by such a venture. Moreover, there should be a reasonable expectation of attaining outcomes for the betterment of evaluation and members of our society.

The Society suggests that the following elements be included in the Terms of Reference for the Committee:

1. A definition and statement of principles

2. A mission(s) statement

3. An identification of the international needs of evaluation and evaluators

4. A statement of plausible goals

5. An identification of probable structures and alternative means

6. An estimation of resource requirements and the division of costs

7. An identification of a means of assembly ratification

8. A scheme for representation and executive succession

9. An analysis of alternative means of communication for members

10. A sample of initiatives and possible outcomes

11. A recommended scheme for evaluation of international collaboration, initiatives and an organizational entity.

The Canadian Evaluation Society hopes that the first meeting of presidents will establish future goals for their endeavors based upon a consensus for achieving and sustaining a vision of international collaboration among evaluators. Accordingly, the CES commits itself to enthusiastically pursue this endeavor and then submit the concept and its alternatives to our members for their consideration. Lastly, we wish the forthcoming panel success and offer our thanks in pursuing this necessary but complex set of tasks.


La reconnaissance et le respect de la diversité
sont des conditions essentielles
à l'épanouissement harmonieux de la personne
au sein des valeurs universelles du village global.


Une "Organisation internationale d'évaluation"
Exposé de principes de la Société canadienne d'évaluation (SCÉ)
présenté à la table de discussion des présidents

Congrès annuel de l'Association américaine d'évaluation

Chicago, 1998

Introduction

La Société canadienne d'évaluation (SCÉ) reconnaît la valeur unique de la collaboration entre les évaluateurs et entre leurs organisations sur le plan international, dans un contexte où la profession devient de plus en plus un "village global". La reconnaissance d'un besoin de collaboration accrue amène la SCÉ à prendre position sur les questions et les principes liés au concept d'une organisation internationale de l'évaluation. La SCÉ prendra part à la table de discussion lors du congrès de l'AEA à Chicago, dans l'espoir de produire un cadre de travail qui permettra d'explorer pleinement les possibilités d'une collaboration internationale.

La Société reconnaît que les évaluateurs et leurs organisations oeuvrent dans un milieu social et culturel très diversifié et sont influencés par celui-ci. Elle croit aussi qu'une plus grande collaboration est requise pour le bénéfice de ses membres et qu'il est essentiel pour elle de jouer un rôle dans le développement et la promotion de l'évaluation au niveau international tout en respectant sa propre diversité. La SCÉ a déjà fait preuve d'un leadership en termes de collaboration internationale et continuera dans cette voie.

Le potentiel de la technologie moderne comme outil d'échange d'information et de communication entre les évaluateurs offre des possibilités excitantes. De plus, certaines formes d'organisations internationales offrent des possibilités accrues pour l'évaluation et pour les évaluateurs. La SCÉ croit que plusieurs voies sont possibles pour aider à réaliser la mission globale de l'évaluation. Ces voies doivent être examinées à la fois dans l'intérêt des membres de nos organisations respectives et dans celui des évaluateurs de la communauté internationale. C'est avec un grand intérêt, avec enthousiasme et aussi avec prudence que la SCÉ entreprend de débattre la collaboration internationale en évaluation.

Contexte

La SCÉ considère qu'il est important de bien situer l'origine du présent débat. Les discussions à venir concernant une organisation internationale de l'évaluation prennent racine dans une proposition faite sur EVALTALK en 1997. Plusieurs membres de la l'AEA avaient alors suggéré d'appeler leur association "Association internationale de l'évaluation" dans le but de refléter le fait que plusieurs membres provenaient de l'extérieur des États-Unis d'Amérique.

Les discussions qui s'ensuivirent révélèrent que le terme "international" avaient plusieurs significations et plusieurs implications. En même temps, les évaluateurs d'autres organisations nationales insistèrent pour élargir le débat sur la question. D'où le concept d'une Fédération internationale de l'évaluation (FIE).

Au printemps de 1998, le groupe d'intérêt international et inter-culturel en évaluation (I&CCE) a senti le besoin de reprendre les discussions à propos d'une organisation internationale sur la liste XC-EVAL. Il a été par la suite proposé que les présidents nationaux discutent en groupe des questions inhérentes au concept de fédération, au congrès annuel de 1998 de l'AEA, à Chicago. De plus, un forum électronique d'échanges (EVALPRES) a été créé pour permettre aux présidents d'approfondir certaines questions et de préparer leur rencontre.

Quelques principes fondamentaux

En vue de l'établissement d'un cadre de discussion au sujet d'une collaboration et d'une éventuelle organisation internationale, la SCÉ insiste pour que le débat tienne compte des principes suivants

1. Seule l'assemblée d'une organisation nationale peut autoriser toute implication et tout engagement dans une organisation internationale.

2. Une assemblée de représentant(e)s dûment mandaté(e)s par nos organisations nationales est nécessaire pour rendre légitime et obtenir tout engagement en faveur d'une organisation internationale.

3. Toute forme de coopération à venir requiert une identification et une évaluation des besoins internationaux et nationaux à satisfaire.

4. Une identification claire des bénéficiaires des activités d'une organisation est une condition clé de tout engagement.

5. La SCÉ considère que son code de conduite et le principe d'équité sont des composantes essentielles de son organisation et le serait tout autant dans la perspective d'un effort international.

6. La SCÉ considérera tout engagement dans une organisation internationale comme un investissement pour ses membres, dont les bénéfices devraient être définis et évalués par ceux-ci.

7. Un large éventail d'activités de coopération permettent de réaliser des projets internationaux.

Besoins et bénéfices

La SCÉ reconnaît que les bénéfices potentiels d'une collaboration et d'une organisation internationales sont nombreux, pour l'évaluation et pour les évaluateurs. Elle reconnaît aussi, cependant, que les bénéficiaires et les bénéfices devraient être identifiés d'une manière précise. Les activités d'une organisation internationale doivent être précédées d'une identification détaillée des besoins de l'évaluation et des évaluateurs dans une perspective internationale, tout en reconnaissant à la fois que l'on doit répondre aux besoins des membres des organisations nationales de manière à justifier l'investissement dans un effort international.

En principe, plusieurs évaluateurs peuvent bénéficier d'une coopération et d'une organisation internationales. En pratique, l'expérience nous indique que ce n'est pas toujours le cas. Certains membres auront plus de chances de promouvoir leurs services et de faire de l'évaluation. La SCÉ croit que les occasions se présenteront pour certains membres internationaux, comme les universitaires et les consultants, mais pas à la majorité des membres. Tout en reconnaissant et en saluant ces occasions, notre organisation croit que les bénéfices de certains groupes d'intérêt devraient être d'une importance secondaire par rapport aux besoins et aux intérêts de nos premiers bénéficiaires, soit les membres en général.

La SCÉ croit que les tentatives de collaboration internationale offrent des occasions nouvelles et palpitantes pour les membres de nos organisations, dont les intérêts varient et sont souvent conflictuels. Nous devrions discuter franchement de cette réalité et poursuivre nos efforts internationaux, en évaluant clairement la nature des besoins et les bénéficiaires possibles de ces efforts. En ce sens, les buts et les priorités seraient plus facilement définis, et les activités et les opérations seraient plus facilement conçues et évaluées.

Contraintes et priorités

La SCÉ croit que l'essai de toute forme d'organisation internationale d'évaluation requiert une analyse organisationnelle et financière solide. Il y a de nombreuses et imprévisibles implications financières. Le succès d'un tel effort nécessite un apport de fonds constant et donc un engagement financier. Une évaluation et une planification financière seront requises dans le but d'assurer que l'organisation internationale soit viable et qu'elle puisse aussi se développer.

Devant le grand nombre de buts que nous pourrions poursuivre sur le plan international, il nous faut aussi être conscients des structures et des opérations requises pour les atteindre. Nous devrions considérer dans notre analyse l'expérience des autres organisations internationales qui ont recherché une diversité de possibilités pour atteindre leurs buts. Nous devons aussi être conscients que certains buts pourraient être poursuivis plus adéquatement en rehaussant les activités d'une organisation nationale, dans un secteur donné; ceci donnant lieu à une sorte de division du travail.

Nous croyons aussi que le mariage des besoins des évaluateurs et de ceux de l'évaluation internationale peut se faire de bien des façons. Les activités peuvent prendre la forme de conférences et de réseautage. Les besoins plus globaux, comme la promotion et des projets conjoints, peuvent requérir des stratégies plus complexes. Comme il a été dit précédemment, toutes les stratégies devraient correspondre aux besoins de la majorité des membres de nos organisations respectives. Aussi, le choix de nos efforts devraient se faire sur la base d'une multitude de stratégies et d'une appréciation des choix pour chaque stratégie. Dans le fond, il n'est pas requis d'avoir une grosse organisation internationale pour réaliser une collaboration internationale.

Le principe d'équité, sur le plan des bénéfices mais aussi en regard des coûts et des responsabilités, doit faire partie intégrante de toute forme de collaboration internationale, qu'elle soit formelle ou non. Le succès de nos efforts dépendra d'une identification claire des bénéfices pour les organisations membres et de l'attribution de responsabilités distinctes pour chacune des organisations nationales. Ces deux conditions permettront à nos membres d'évaluer correctement leur niveau d'engagement et de participation. L'équité devra être une valeur sous-jacente de cette évaluation.

Dans le contexte des développements actuels et futurs de la technologie, la SCÉ est enthousiaste et très intéressée à rechercher des moyens innovateurs de collaboration internationale. Certains de ces moyens ne nécessitent que peu ou pas d'intervention formelle. D'autres moyens de collaboration requerront des activités hautement structurées; certaines d'entre elles pourraient éclipser nos organisations nationales et leurs membres. Nous devons donc nous engager dans ces efforts en prenant soin de préserver la force et la richesse de nos caractéristiques propres.

Par exemple, la plupart des membres de la SCÉ sont des praticiens. La majorité est à l'emploi du secteur public. Plusieurs d'entre eux jugent que toutes formes de collaboration internationale, particulièrement en évaluation, ne seraient pas utiles. De plus, ils travaillent et vivent dans des contextes sociaux et culturels diversifiés. Tout effort international doit être compatible avec cette diversité et ces besoins. Cet effort peut l'être en créant de nouvelles occasions d'apprentissage, de formation et de pratique de l'évaluation qui refléteraient et respecteraient la diversité de nos cultures organisationnelles spécifiques.

Conclusion

La Société canadienne d'évaluation place un grand espoir dans la promotion et la pratique de l'évaluation à l'échelle du village global. Notre vision est celle d'une profession de plus en plus mature et en développement continu dans toutes les nations. Une profession reconnue pour son expertise et qui transcende les différences culturelles.

Nous entrevoyons une communauté internationale formée de professionnel(le)s qui ont une formation et une expérience diversifiées et qui communiqueraient en référant à un code d'éthique commun, à une mission commune, à des buts communs, et en ayant l'assurance de pouvoir partager leur expertise, leurs approches et leurs informations. Surtout, nous voyons une collectivité internationale qui cherche à enrichir ses connaissances et à exercer son métier dans un contexte culturel varié.

Les communications sont devenues plus faciles et leurs coûts ont été réduits. De ce fait, il n'est plus toujours nécessaire d'ajouter des structures organisationnelles. Si toutefois l'accomplissement des rôles des évaluateurs au sein d'une communauté internationale requiert une forme quelconque d'organisation, nous ne rendrions pas service à nos membres en refusant de participer à la création d'une entité internationale.

La SCÉ s'engagera davantage dans la réalisation d'une entité internationale lorsque ses membres seront convaincus que les bénéfices à en retirer sont équitables et suffisants, en considérant les coûts et les efforts requis par une telle entreprise. De surcroît, la SCÉ s'attend à ce qu'on vise minimalement des buts qui permettraient d'améliorer la fonction d'évaluation et la situation de ses membres.

La SCÉ propose que d'éventuels termes de référence incluent les éléments suivants:

1. Une définition et une déclaration de principes

2. Un énoncé de mission

3. L'identification des besoins internationaux de l'évaluation et des évaluateurs

4. Un énoncé de buts réalistes

5. L'identification de structures possibles et de moyens à choisir

6. Un estimé des coûts et de leur partage

7. Une identification des moyens de ratification par les membres

8. Une proposition de forme de représentation et d'élection des dirigeants

9. Une analyse des divers moyens de communication entre les membres

10. Une proposition sommaire de projets et de buts à poursuivre

11. Une proposition de collaboration internationale en évaluation et d'une forme d'organisation

La Société canadienne d'évaluation espère que la première rencontre des présidents précisera les buts à venir de leurs efforts en visant l'établissement d'un consensus sur la manière de réaliser et de maintenir une collaboration internationale entre les évaluateurs. En ce sens, la SCÉ s'engage à poursuivre ce but avec enthousiasme et à soumettre ce concept et les diverses possibilités offertes à ses membres. Enfin, la SCÉ souhaite que la rencontre des présidents soit un succès et remercie ceux-ci des efforts qu'ils mettent à réaliser ce projet complexe.

La reconnaissance et le respect de la diversité
sont des conditions essentielles
à l'épanouissement harmonieux de la personne
au sein des valeurs universelles du village global.



European Evaluation Society (EES)
Position Statement

Hellmut Wollman

The position unanimously agreed upon by the 10 members board of EES is the following:

1. that we are in favour of a general increase in international exchange, while we acknowledge that there are already many different international evaluation networks in which many of us are already engaged

2. that some more formal arrangements for liaison between different evaluation societies would be useful

3. that we do not favour a federation or setting up a separate organisation with its own budget

Let me add some of the arguments which were made and that underlay the above-mentioned position.

a. It was pointed out that the European evaluation community is still emergent with an increasing number of national evaluation societies either recently established or in the making. During the Rome conference we had a meeting of chairpersons and representatives of national evaluation associations either already existent or in the making.

As you know, there are national evaluation societies in existence in Europe already (in the order of "seniority") in the U.K., Switzerland, Italy, and Germany. The foundation of new national associations appears to be imminent in France, Belgium, and Poland. Reference also should be made to the recently founded evaluation society in Israel. Its members attended the Rome conference and also the gathering of national evaluation societies held at this occasion.

Against this background and vis-a-vis the limited time, organisational, financial and other resources of the persons and parties concerned, EES sees its top priority in further strengthening and consolidating the EUROPEAN network, including cooperation with the national evaluation societies (the cooperation with the Associazione Italiana di Valutazione in organizing the Rome conference was hailed and praised as an excellent case in point). Much remains to be done on this score, particularly with regard to promoting and assisting the emergence of the evaluation community in Central Eastern European countries. Pertinent activities were discussed in Rome and will be pursued.

b. There was also some concern about the financial implications of a new international organization with a self-standing budget. Living under a very tight budgetary regime, EES does not see the possibility of making a financial contribution to such a new organization in the foreseeable future. It might be useful to mention that the European national associations are not institutional members of EES and, thus, they do not pay any contributions to EES (and there are no considerations to change this in view of the extreme tight financial situation of the national associations).

As expressed in the above-mentioned points, EES welcomes your initiative insofar as it aims at deepening and strengthening the international discourse and exchange in and on evaluation by creating some institutional arrangement (which, however, stays below the level of some federation or a separate organization with its own budget).

In this sense EES, including myself, is interested and prepared to remain in contact with and further engage in the debate which you have the merit of having initiated.

Hellmut Wollmann

President of EES




Malaysian Evaluation Society (MES)
Position Paper

Aru Rasappan

Foreword

In preparation for the Presidents Panel session during the AEA conference in Chicago, the Interim Committee of the soon-to-be-formed Malaysian Evaluation Society (MES) would like to put forth its views on several issues related to the proposed formation of an international evaluation body. These opinions are not that of the Malaysian Evaluation Society, since MES is in the process of being officially registered. These views were compiled from discussions with members of the Protem Committee set up to establish MES and should not be construed to be the views of MES itself. This should be applied wherever the name "MES" is used in this paper.

The final version of this position paper was compiled by the Chairman of the MES Protem Committee.

A Little Background

The idea of an international evaluation body to represent the interests of all evaluation entities in all countries is both timely and crucial. This idea was first mooted at the International Evaluation Conference (IEC) in 1995 in Vancouver. The year 1995 will remain a watershed year for evaluation due to the first IEC ever held with representatives from many countries outside of the U.S. and Canada. It also marked the first time that two important global electronic groups (GOVTEVAL and EVALTALK) were set up and launched. The IEC, the work of the Cross-Cultural TIG, and the above two global evaluation discussion forums have made a significant difference to the world of evaluation.

Though Malaysia is relatively new to the field of program evaluation, evaluation itself has been practiced in the country for several decades. This was particularly in the form of project evaluations within the country's development initiatives. Most of these were developed through and dependent on the evaluation requirements imposed by development funding bodies (such as the World Bank, WHO, UNICEF, UNIDO, and UNDP). However, program evaluation per se is relatively new in the Malaysian context and particularly in the public sector, having been first introduced in 1969 but seldom seriously implemented over the years until more recently.

As a result of the IEC in 1995 and the tremendous interest in promoting evaluation on an international basis, various attempts have been made to try to bring together the evaluation communities worldwide onto a common international platform. This was not only to further promote evaluation globally, but also to assist those countries that were newcomers to evaluation and that needed both technical and financial assistance.

The global discussion groups of GOVTEVAL and EVALTALK, while playing an important role in bringing together evaluators and others associated with evaluation onto a virtual "meeting place," have limited utility on an international basis. It is thus imperative that a more formal body be formed on an international basis to represent the interests and future of all evaluation communities or those associated with it.

The MES Overall Views

The general view of MES is that we need a more formal international institution to represent all those who are involved in evaluation in so many ways. Such an international institution will prove to be an immense asset, both to those who are already established and mature in evaluation, as well as to those countries that are only now getting onto the evaluation journey.

However, though the idea of an international evaluation body is both timely and extremely useful, MES would also like to caution a thorough review and agreement on the proposed form, structures, functions, role, and powers of such a body. This is to help ensure that such a body will indeed prove useful to all members as well as to ensure that this body is sustainable in the long run. MES is particularly conscious that the formation of such an international body should result in a win-win situation for all countries, based on a sharing and learning concept and premised on the principles of equity and sustainable development well into the 21st century.

It is not possible to always bring all national evaluation entities together due to physical and fiscal reasons. In this respect, MES hopes that with modern technology and innovations the proposed international evaluation body will have a virtual presence in all countries. Distance and other resource constraints should not prevent the full integration and enjoyment of the benefits to be derived by all countries from the formation of this international body.

In line with the general framework for discussion at the Presidents Panel, MES presents below its views and suggestions on the predetermined discussion topics.

Vision for a Greater World Evaluation Community. This is a bit longish but sounded like a good vision. Our vision for the ultimate world evaluation community is "to have a truly international evaluation entity representing the interests and concerns of all parties globally involved in evaluation activities and committed to the pursuit and sustainable development of evaluation and all its subfields."

Mission and Goals Essential for Vision Achievement. There are several goals and objectives that are applicable for the effective achievement of the vision. These can be summarized through the following considerations:

Options for Achieving Goals

There are several options that could be considered for achieving the desired goals and objectives of the proposed international evaluation body. These are outlined or discussed at some length below.

1. Establishment of the international evaluation entity as a first step

2. Establishment of various collaborative working committees to address evaluation issues

a. Joint and collaborative research efforts on evaluation issues

b. Regular conferences or other forums on a rotational basis to present, review, and discuss evaluation issues, innovations, initiatives, and findings

c. Regular working seminars and workshops in various member countries to address issue areas in evaluation with implications for members/countries. These initiatives could be on a collaborative approach.

d. Special IES publications (journals, newsletters, electronic forum, etc.) for disseminating and sharing the research, developmental efforts, and other problem-solving initiatives with members globally

Important Issues for Deliberation/Resolution before Formation of the IES

There are several issue areas that need to be addressed and resolved before the proposed international evaluation body is formed. These are outlined or discussed briefly below.

1. The form and structure of the proposed body

2. The membership categories, options, and related issues

3. The operational aspects and mechanisms of the IES

4. Forms of cooperation and collaboration between member evaluation societies

5. The form, nature, and focus of collaborative efforts in the future

6. The strategic directions of the international body in the near future

7. Establishment of a common language and understanding of evaluation and related terms/issues for all member countries

8. Distinguishing the focus areas for future efforts by the international body and perhaps distinguishing between different evaluation fields and concentrations, including distinction and approach between public and private sector evaluation issues

9. Due to the implications of evaluation work on both policy and management matters, it may be best to define policy and operational boundaries for the IES in its work, including a detailed terms of reference

10. Seeking out and reviewing the views and constraints likely to be faced by member countries in relation to the formation and future functions of the IES

11. Reviewing and determining the future scope, depth, and focus of evaluation efforts by the IES

12. Reviewing the existing disparities that exist or that may become apparent between different countries and seek out ways to address them

13. Obtaining views and seeking consensus among potential member countries on possible issues they would like to see addressed by the IES

Possible Contributions by the National Society

MES is a very young organization. It is only now in the process of being registered and becoming operational by early 1999. As such, it will need much assistance from the more advanced and older evaluation societies around the world.

However, MES and its members also bring with them certain strengths in terms of experiences and evaluation innovations, either in terms of concepts or experiences. There are many academics and practitioners in evaluation who have much that they can share with the rest of the evaluation world from other countries. MES and its members will be committed toward a sharing and learning philosophy and will seek to try and give as much as they receive from the outside evaluation community. MES, either by itself or through individual members, will try to make available and share with the rest of the world any special experiences or innovations in theory and practice that they might have over the years.

Financial assistance by MES will tend to be very limited in the beginning, as it would be in the infant stages of operation. In fact, MES may well need both financial and technical assistance from the more established evaluation societies in other countries. However, MES will play its part toward helping to raise funds both locally and internationally through various means for the betterment and promotion of the evaluation agenda and related priorities.

Much of the materials and evidence in evaluation work carried out in Malaysia may not be easily available or even accessible to the outside world. In this respect, MES will assist in obtaining or providing access to evaluation experiences and materials in Malaysia to the outside world. This includes a possible repository of local evaluation materials as well as research efforts.

Essential Steps Needed to Move Ahead

Several preparatory and prerequisite steps are necessary toward ensuring the smooth creation, implementation, and successful existence of the proposed international body. Some of the more important ones are presented below:

1. Communicate the proposal to form the international body to as wide an audience as possible with sufficient time for feedback from those countries that were unable to attend AEA.

2. Decide on a suitable time frame for the formation of the international body.

3. Decide on the form, structure, and functions of the proposed international body with due regard to the needs and considerations of various members.

4. Identify a program of promoting the international body to all countries. This could be done either through the interim committee or through a volunteer committee.

5. Make full use of global communication channels such as GOVTEVAL, EVALTALK, and other similar forums to disseminate information and seek feedback about the proposed international body.

6. Carry out a detailed survey to assess how many countries have a national evaluation society.

7. Carry out an immediate campaign to inform and seek out feedback from these existing organizations.

8. Seek out suitable country-specific representatives or contact persons/bodies that can act as a liaison between the international body (or interim committee) and the evaluation community in those countries where there are no national evaluation societies.

9. Find ways and means of encouraging the early formation of a national evaluation society in those countries where there are no national evaluation bodies.

10. In this respect, the Interim Committee can set up a volunteer working committee to work on promoting and assisting the relevant countries in their formation of a national evaluation body.

11. Review the financial considerations for the establishment of the international body. Where necessary, preliminary actions can be initiated to start sourcing for funds to run the new body and its activities. A launching grant from world bodies such as the World Bank or UNDP would be both possible and most appropriate for the above. Representations could be made from now to highlight to such bodies the utility of forming such an entity.

12. Set up an interim working committee to look into the various preparatory actions needed to get the project off the ground. The interim committee should preferably have adequate representation from various countries/regions to ensure that their views or concerns are taken into account.

13. Work out a draft constitution and detailed terms of reference for the international body. This could be done by the interim committee or perhaps a special volunteer working committee.

14. Provide guidelines on how to form a national evaluation society and assist new evaluation countries in their efforts to set up national evaluation societies.

15. Plan from now on what kinds of programs can be conducted by the international body once it is formed.

Concluding Observations and Comments

MES is very encouraged by the show of support, both technical and financial, extended to it by the Australasian Evaluation Society (AES) and some of its individual members. Wherever possible, MES will avail itself to other countries where its assistance may be needed. In this respect, its draft constitution has also been circulated to a number of countries that are only now attempting to set up their evaluation societies. While the MES is fully conscious and supportive of the need to form an international evaluation body to represent and further the cause of evaluation globally, it is also just as concerned that such an action will not disadvantage or marginalise the less developed countries and especially those that are very young in evaluation.

There is a lot of synergy between the evaluation developments and innovations in the more developed countries and the practical difficulties and experiences in the less developed countries. The proposed international body should seek to draw on these synergies toward further promoting evaluation globally in order to make evaluation truly useful and utilitarian to all countries, as well as toward the sustainable development of evaluation globally. MES, while needing much assistance from other countries, will also seek to extend whatever assistance it can render to those countries that need it. It is fully committed toward a sharing and learning approach.

MES hopes that the proposed international evaluation body will become a reality well before the turn of the century and that we will be able to launch it during a specially convened international evaluation gathering, either in 1999 or no later than the Year 2000.

Acknowledgments and Reservations

The author gratefully acknowledges the critical thoughts, views, and suggestions of the members of the MES Protem Committee pertaining to the issues presented above. The author is also appreciative of the assistance and encouragement extended by the Australasian Evaluation Society over the last few years toward making the formation of MES a reality. The views and issues raised in the paper above remain the responsibility of the author. Though the issues raised are quite detailed, they are far from being comprehensive. There may be one or more important issues that may have been left out inadvertently, and the author takes responsibility for it.

The issues raised in this paper should be taken as a starting point for further and more detailed deliberations in the near future and not as being the final and conclusive views of MES.


AEA Presidents Panel
Improving International Links
United Kingdom Evaluation Society (UKES) Position Paper

Nick Tilley

The following comprises the views of the United Kingdom Evaluation Society (UKES) Council on the questions asked by Dr. Love in connection with moves to foster greater links between national evaluation societies and those from countries where there is not yet such a society.

1. What is your vision for a greater world evaluation community?

The UKES Council sees many benefits in fostering dialogue and collaboration between evaluators in different countries. Indeed, many within the U.K. evaluation community are already involved in work with an international focus. This includes studies of initiatives conducted under the auspices of the European community. It also includes work with evaluators in other countries. Many members also belong to the European Evaluation Society. There is, thus, already a strong international dimension to much work in this country. We can see benefits in extending this. We believe that there is virtue in diverse approaches to evaluation and have some concerns about convergence toward a particular model at this stage. Our vision, therefore, for any development of further international links is of a community of mutually respecting practitioners and theorists amongst whom there are wide-ranging positions. From this there should be scope for mutual learning and enrichment. What is not wanted, and in truth slightly feared, is some form of colonisation from those in countries where formal evaluation organisations are relatively well established.

2. From your viewpoint, what mission and goals are essential to achieving your vision?

From our perspective the mission of a greater world evaluation community should be to facilitate exchanges of ideas and the fostering of dialogue across differing perspectives. In some cases this will mean allowing space for new ideas to develop without risking that they founder in the face of teething troubles. The history of science shows that new ideas are often received unsympathetically and it takes time for them to be fully formulated to the point where critical discussion is fruitful.

3. In your opinion, what are the major options for achieving your vision, mission, and goals (e.g., international conference, international federation)?

The UKES Council would favour loose, but effective arrangements whereby members of national societies can engage with those from other countries as desirable: perhaps a league of national associations rather than a federation to indicate the independence of the member societies. UKES would certainly favour international meetings, though of course national meetings are often internationally attended. Those of us who were at Vancouver thought that this was a successful meeting. Moreover, we have also been involved in small meetings with groups from other countries. A regular, fully international conference, however, has many attractions. It would, though, raise some problems. For example, travel is clearly costly and this is likely to limit the scope for poorer members to attend, so that international meetings would always tend to comprise a skewed group. Moreover, the location of international meetings will obviously affect those likely to come. We would suggest that if international meetings are to take place regularly under the auspices of a federation or league of national associations, the site should also move such that no two consecutive meetings take place on the same continent.

4. From your position, what are the important issues that need to be resolved before making a decision to pursue seriously the idea of building a world evaluation community?

The major issue for us, apart from contributions that are dealt with below, relates to ways of catering to those who live in countries where there is not yet a national society. An international organisation of some kind that tries both to provide a way in which national associations can work together for the benefit of their members would be rather different from an association that catered directly to individual members. It seems to us that this issue needs to be worked through very carefully.

A second issue relates to language. In the UKES we are fortunate to speak the commonest second language. Yet, not everybody speaks English. And it would be wrong to disadvantage those who cannot communicate in it equally. Once we contemplate using multiple languages, then costs become very high indeed.

A third issue relates to the risks the Council perceives in domination by any single country, most probably the United States.

5. What contributions--financial and other types of contributions--do you think your society would be willing to make?

It is difficult to see what material contribution could currently be made by UKES to an international grouping. Yet, we acknowledge that the more robust the international organisation and the more it tries to do, the more it will cost. This brings its own dangers. If one society provides the bulk of the funding or directly offers to manage the international organisation itself, then the unintended consequence will be that that society will come to exert undue influence. It is not unreasonable that those who do not pay do not warrant a seat at an international grouping. Yet, the exclusion of those who cannot pay will tend to give any international organisation a national bias.

6. If a decision is taken to move ahead, what are the essential steps needed to manage the process?

If the decision is made to develop some formal organisation to which national associations belong, it will be essential first to canvass the views of members. In particular, if the view is taken that a levy is necessary in order to fund such an organisation (or even development work prior to its establishment), members would need to be fully consulted. The current UKES Council would not suggest to the membership that such additional levy be accepted.

It would also be necessary to establish a steering group to take the idea further. This is not going to be easy. There will be a significant amount of work and some costs. Moreover, most active evaluators are very busy people, and it is going to be difficult to find a group willing and able to take the lead.

An international conference to launch a new international grouping would be a useful way of marking its emergence. It would also provide some tangible indication of some of the advantages that could be expected.

Finally, the new supranational organisation would need to ensure that adequate benefits were provided both to the national society and to its members. Only if there is significant added value is it likely that wide support across different countries is likely to be created and sustained.



I&CCE TIG Presidents Panel: Background Document, First Draft for Discussion
Developing Countries and their Participation in an
"International" Evaluation Community or Federation

Mahesh Patel

Nairobi, October 1998

Abstract

This paper compares and contrasts the structure, functions, activities, probable membership, and implications for developing countries of two different structures of an international community of evaluators.

Key Words

Evaluation, association, international, developing countries

Developing Country Participation in International Organizations

Developing countries have had mixed experiences in international fora. Of course, there are many different types of international communities. And perhaps the differences and contrasts between them are particularly acute for developing countries!

Let us, as a starting point, exclude overtly military alliances such as NATO. Even the many United Nations organizations themselves have very varied representational modes. The United Nations General Assembly puts China, with a population of one billion, on par with Bhutan, with one million, and awards to "one country, one vote." The Bretton Woods institutions such as the IMF are based on "one dollar, one vote" and are run by a few large economies. The U.N. Security Council basically distributed veto powers to countries who won the second "world" war. Quite a wide range of possible modalities for participation by developing countries!

In many other categories of international organizations, especially those in the private sector such as transnational corporations, developing countries hardly figure at all, even though they comprise three-quarters of the world's population. In Nairobi we have a very good "international school," which allows people from any country to attend as long as they pay the fees. But the curriculum is American, the currency used in mathematics is dollars, and geography lessons are "Americentric." This is actually an American school with international participation, mostly in the sense that students of other nationalities may enroll. But it is called an "international school." So, there are many different types of internationalism.

It is for these reasons that I am particularly pleased that the international evaluation community is giving serious thought to the participation of developing countries in their deliberations about the possible formation of an international federation of evaluation associations. It is our hope that this community will be established in such a manner that developing countries will be given a fair representation at the table, that our actual and potential contributions to the formation and evolution of this community will be well noted, and that we shall be able to call on that community for assistance in several of the critical areas discussed below.

The Mission, Goals, Structures, and Activities of an International Group Mission

Mission. The convention is to start with an agreed mission and work through to goals, structures, and activities. My personal problem is that I have a clearer vision of the useful things that could be done than of these other issues, except perhaps for the mission. The exact wording of a mission statement can be debated for a long time. But the essence is perhaps more easy to present. My suggestion is

"To Promote Internationalism in Evaluation as a Discipline
on the Eve of the New Millennium"

I think it has a nice ring without getting too specific, but then others may want something more exact, or exhaustive. That could actually be quite helpful. The trouble with this generic statement of good intentions is that it does not provide much traction in defining what such an association, federation, or good time club should look like and how it could usefully work in the sense of being helpful in satisfying the some of the needs of some of its members at least some of the time. But perhaps I can be more specific.

Structure. It may be that the range of potential structures for an international association is not so wide. Indeed, I see only two strands that have a reasonably high probability of emerging, based on analysis of the historical success of various types of existent international organizations. One is the nongovernmental model, which would lead us toward an international community of (individual) evaluators; and one is the more formal governmental model, which would lead us toward an international federation of evaluation associations. (1)

At this point, it is unclear whether we are heading for an "international community of evaluators" (ICE) or an "international federation of evaluation associations" (IFEA).(2) There are several meaningful differences, some of which do make a difference from a developing country perspective. This paper will compare and contrast these alternatives, hopefully shedding some light on the implications of the choice we shall be making together.

While an ICE might well be primarily a group of interested individuals, an IFEA would be a group of associations. These different organizational modalities would generate different memberships and missions. We see an ICE as initially comprising mostly evaluators who already have a significant degree of international exposure or an interest in gaining that exposure. In contrast, the components of an IFEA would be the national associations themselves. The ICE and the IFEA would then be quite different organizational entities!

In both cases, an international institutional framework would be required. If the ICE structure were adopted, there would be an international association with an elected president independent of any individual national evaluation associations. An association of this type might take longer to grow in membership and to gain professional legitimacy, but would hopefully in the end take on most of the tasks undertaken by national associations, though in an international context. This would include, for example, the definition of international professional standards. The IFEA structure might be a more rapid route to the same objective, as the initial starting point would include a wider range of experience.

If the IFEA were adopted, the presidency could follow the EEC model and be undertaken by the presidents of each national association in an annual or more rapid rotation. (The EEC manages well with six months--people with other commitments might not have much more time than that.)

Continuity of purpose would be essential, especially in an international context. It is proposed that, in both cases, a more permanent secretariat to service the president would be needed, and this could perhaps be provided by an existent and experienced international group in order to minimize costs.

Activities. In the IFEA structure, that international secretariat might be expected to extend a more systematic support to the international strands of national evaluation association meetings, to notify all members of all associations about those meeting schedules and venues, and to solicit papers for presentation.

A supplementary activity might be to prepare an international directory of professional evaluators, which could facilitate efforts aimed at improving international cooperation in the evaluation of International Development Assistance projects. It might also be nice to know who to ask about local meetings when on mission or even on holiday! Or which individual or group to contact to find out about evaluation work in a given country.

An ICE meeting of a group of evaluators with international interests might mostly discuss their personal experiences and differences in evaluation cultures between countries. In contrast, an IFEA meeting of national associations might have a more structured focus on institutional relations, professional standards, and institutional support to developing countries.

The choice between an ICE and an IFEA structure will probably have implications for fees. An independent ICE would have to charge fees that enabled its existence as an independent financial entity. An IFEA might exist only as a secretariat or strand of an existent national association and require only a minimal "tax" on the budget of the component national associations.(3)

While it is hard to envisage how much money an IFEA would require to function effectively, we would consider that a contribution of 1 percent of national association fees would be sufficient. The Kenya Evaluation Association has already agreed to make this contribution.

Finally, a likely output of an international group of evaluators is an "international" version of the Programme Evaluation Standards. If these are likely to be used by donors in the evaluation of Development Assistance projects, then it is essential that developing countries have a substantive formal input.

Proposals

We think that either option, the ICE or the IFEA, would be preferable to inaction. Perhaps the path toward the ICE is "icy" and that for the IFEA is "iffy?" Our proposals are aimed at minimizing the risks of inaction. Several of the more specific proposals are made for discussion purposes:

1. It is proposed that funding be sought to hold a summit meeting of presidents of national evaluation associations. The goal of this meeting would be to work out a possible agenda for an international federation by the summer of 1999. Presidents could discuss these proposals with their constituents during the remainder of 1999. The I&CCE TIG should then reconvene the Presidents Panel by the end of 1999 to present the results of their deliberations and consultations.

2. If a consensus is obtained, then we could go one step further and propose that a Year 2000 meeting of a major evaluation association be donated to some international theme and serve the dual purpose of initiating the new federation or community.

We would also like to suggest that

3. All formally constituted associations represented in the federation be represented on equal footing, regardless of their membership

4. Presidents shall be selected from the group of presidents of national associations who have not yet had a president of the IFEA by the entire group of presidents themselves. This is intended partly to be a means of avoiding the costs of an internationally coordinated election. Given the speed with which secure electronic communications are developing, it probably could be stipulated that a "proper" election be held within five years. But most of the voters might be from the U.S.A., and so proportional representation would diminish the internationalism of the results. If internationalism is the very thing the association wishes to promote, single nationality voter dominance would be counterproductive.

5. The International and Cross Cultural Technical Interest Group of the American Evaluation Association, which is the sponsor of the Presidents' Panel session, serves as the interim secretariat to the IFEA while the functional details of that latter association are being elaborated

6. Key functions of the federation secretariat, in addition to support to the president, be support to nascent evaluation associations, particularly in developing countries, and coordination of the development of a first draft of an international version of the Programme Evaluation Standards.

In Conclusion

Whatever consensus emerges, be it an IFEA or ICE, or both, or some other mode entirely, it would be fitting for some first meeting(s) to take place in the Year 2000. It is suggested that the major theme or a substantive agenda item be a review of progress--not just of the discipline, though that could be a major strand, but of humanity itself.

Most national governments are committed to a number of United-Nations-supported initiatives that mature in the Year 2000 such as "Health for All by the Year 2000," "Education for All," the World Summit "Goals for Children," "Habitat 2001," and others. A series of national reviews will be taking place in most countries during that year. If one of the national association meetings during 2000 offered substantial support to that effort by hosting an international meeting to bring together these different strands from different countries, this could be an opportunity for a major milestone in the history of evaluation.

If a suitable consensus does not emerge at some institutional level, then the developing countries group would like to participate with all individuals who believe that international solutions are the way forward and join with you in the creation of an international evaluation association by the Year 2000.

In that spirit, the Nairobi Evaluation Network has redefined itself as the "African Evaluation Association: Kenya Branch," that name to be used interchangeably with our alternate title of the "Kenya Evaluation Association." About half our members are regional focal points for our organizations and cover several African countries each. One of our missions shall be to promote the formation of evaluation associations in the countries we support as part of a network of evaluation associations in Africa.

Of course, it would be nice to get the best of both worlds! Several members of the African Evaluation Association, Kenya Branch, have suggested just that. It is perhaps too easy to insist, out of hand, that such an approach would be duplicative. It might also be synergistic. Further, if the representational modalities adopted for an international federation do not encourage developing country participation on an equal footing, then we would be particularly keen on supporting the creation of an international community of evaluators that was independent of these federal structures.

Table: Some Implications of the Choice: A Federation or a New Association?

Structure International Community of Evaluators: ICE International Federation of Evaluation Associations: IFEA
Overall Mission To Promote Internationalism in Evaluation as a Discipline on the Eve of the New Millennium
Examples of
Possible
Goals & Activities		 • International conferences

• International standards work in the long run

• Sharing of individual experiences

• International conferences and strands in national conferences

• Thematic development of international standards

• Institutional support to nascent evaluation associations

Support to Developing Countries More networking and training oriented More formal, structural, developmental, systematic
Modality of Meetings Annual dedicated international conferences and regional meetings Plus systematic development of "international strands" in national conferences
Nature of Entity A new stand-alone association A composite association
Unit of Membership Individuals National associations
Membership Directory Useful Impressive
Presidency Internationally elected president National association presidents in rotation
Secretariat Independent permanent secretariat Permanent secretariat could be a component of a national association
Additional Fees Yes, in most cases.
Some people might join an international association instead of their local one.		 A 1 percent contribution from national associations
Financial Utilization Conferences, networking, training Plus time bound subsidies to 3 or 4 developing country associations; $1K p.a., would be very helpful
Time Frame
 To be in existence and active by the Year 2000.

AcknowledgementsThanks are due to the Nairobi Evaluation Network, now the Kenya Evaluation Association, for two discussions of drafts. I think it is fair to say that we are broadly in agreement on the contents.

Disclaimer

The views expressed in this paper are entirely those of the author and not necessarily those of UNICEF, his employer.

 

1. The word "federation" is confusing a lot of people, so perhaps I should not use it. But I prefer the acronym IFEA to IAEA (or International Association of Evaluation Associations) for three reasons. Firstly, the IAEA takes the concept of association to what are, for me, dizzying heights by stipulating an "association of associations." Secondly, the term IAEA is too close to AEA, which refers to either the African or the American or the Australian evaluation association, not to mention the associations of economists, epidemiologists and electricians! I would prefer not to further increase my own confusion. Thirdly, while those more linguistically gifted than I may have no difficulty pronouncing the word IAEA, I much prefer to say "ifea." So, once we have a pronounceable acronym, then we can start to have fun debating what it actually means, ifea catch my drift.

2. There are certainly other modalities as well. I have chosen these two to simplify some of the arguments to be presented below and do not purport to be presenting an exhaustive list of options.