JMDE

JMDE

Journal of MultiDisciplinary Evaluation

Number 2, February 2005

Part I

Editors

E. Jane Davidson & Michael Scriven

Associate Editors

Chris L. S. Coryn & Daniela C. Schröter

Assistant Editors

Thomaz Chianca

Nadini Persaud

John S. Risley

Regina Switalski Schinker

Lori Wingate

Brandon W. Youker

Webmaster

Dale Farland

Mission

—The news and thinking

of the profession and discipline of evaluation

in the world, for the world—

A peer-reviewed journal published in association with

The Interdisciplinary Doctoral Program in Evaluation

The Evaluation Center, Western Michigan University

Editorial Board

Katrina Bledsoe	Shawn Kana'iaupuni
Nicole Bowman	Ana Carolina Letichevsky
Robert Brinkerhoff	Mel Mark
Tina Christie	Masafumi Nagao
J. Bradley Cousins	Michael Quinn Patton
Lois-Ellen Datta	Patricia Rogers
Stewart Donaldson	Nick Smith
Gene Glass	Robert Stake
Richard Hake	James Stronge
John Hattie	Dan Stufflebeam
Rodney Hopson	Helen Timperley
Iraj Imam	Bob Williams

Table of Contents

PART I

Editorial

In this Issue: JMDE(2) 1

Marketing Evaluation as a Profession and a Discipline. 3

E. J. Davidson

Articles

Monitoring and Evaluation for Cost-Effectiveness in Development Management 11

Paul Clements

Network Evaluation as a Complex Learning Process. 38

Susanne Weber

Practical Ethics for Program Evaluation

Client Impropriety. 71

Chris L. S. Coryn, Daniela C. Schröter, & Pamela A. Zeller

Ideas to Consider

Managing Extreme Evaluation Anxiety Though Nonverbal Communication. 75

Regina Switalski Schinker

Is Cost Analysis Underutilized in Decision Making?. 80

Nadini Persaud

Is E-Learning Up to the Mark?. 82

Oliver Haas

The Problem of Free Will in Program Evaluation. 101

Michael Scriven

Marketing Evaluation as a Profession and a Discipline

E. Jane Davidson

Davidson Consulting Limited, Aotearoa/New Zealand

It can be a bit like pushing sand uphill with a pointy stick, as they say here in New Zealand. One of the great challenges in developing evaluation as a discipline is getting it recognised as being distinct from the various other disciplines to which it applies. In this piece, I offer a few reflections on the challenges with this, recount a story where a group of practitioners from outside the discipline actually sat up and took notice, and propose some possible solutions for moving us forward.

It’s all very well for us to come together in our evaluation communities around the world and talk to each other about our unique profession. Not that there isn’t a lot to talk about. After all, we are still working on building a shared understanding of what it is exactly that makes evaluation distinct from related activities such as applied research and organisational development. But with a little more application, I am hopeful we can persuade enough of a critical mass to call it a reasonable consensus.

Meanwhile, it seems to me that a more difficult yet equally important task is to articulate clearly to the outside world—to clients and to other disciplines—what it is that makes evaluation unique.

Right across the social sciences and in many other disciplines where evaluation is relevant in more than just its intradisciplinary application, it seems that the vast majority of practitioners consider it to be part of their own toolkit already, albeit often under a different name. Most of these practitioners consider evaluators delusional when we suggest that evaluation is sufficiently distinct to call a profession, let alone an autonomous discipline.

Here’s a fairly typical response, in this case from an industrial/organisational psychologist:

[A] discipline evaluation is not. Disciplines are systematic, coherent, founded more often than not on sound theory, and offered as programs in accredited colleges, universities, and professional schools. Evaluation, without detracting in the least from its multitude of contributions and creative authors and practitioners, is not systematic, coherent, theory-driven, and offered—oh perhaps with an exception here and there—as a program of study at institutions of higher learning. Evaluation is a helter-skelter mishmash, a stew of hit-or-miss procedures, notwithstanding the fact that it is a stew that has produced useful studies and results in a variety of fields, including education, mental health, and community development enterprises.[1]

Industrial and organisational psychology is a relatively young discipline itself, but obviously not quite young enough for its practitioners to recall the struggles they must have had in the late 1800s and early 1900s. Industrial psychology, which focuses primarily on personnel selection and ergonomics/human factors, grew out of a blend of industrial engineering and experimental psychology. The first doctoral degrees in industrial psychology did not emerge until about the 1920s.

It seems likely to me that the fledgling discipline of industrial psychology had its share of critics in those days. Perhaps it was even called a “helter-skelter mishmash.” There was probably a lot of dissent in the ranks as to whether it really was any different from industrial engineering, measurement, experimental psychology, and a host of other disciplines. And I am sure there were furious debates about the definition of industrial psychology itself.

Was it (or would it have been) reasonable to declare industrial psychology a discipline even though most insiders didn’t agree on its definition, underlying logic, or the soundness of its theories? How much shared understanding constitutes a critical mass?

There’s something of a “chicken and egg” argument here. It seems to me that little progress in theory or practice can be made beyond a certain point without first declaring evaluation to be a discipline and then seeing what develops. Sure, not everyone will buy the idea initially, but there’s no point being put off by those who like to throw their hands in the air and declare the whole exercise impossible. These things take time, open minds, thinking and rethinking.

Whether or not we have the courage and conviction to declare ourselves a discipline at this point, I think it’s fair to say we have a critical mass who are quite clear that evaluation is at least a professional practice with a unique skill set that is honed with reflective practice and other forms of learning. The challenge here is convincing non-evaluators (such as the I/O psychologist quoted earlier) of this.

Consultants are a particularly hard nut to crack. Often trained to the graduate level in business and/or the social sciences, the almost universally held perception is that all one needs to do evaluation is some content expertise and perhaps a few measurement skills (and accounting skills).

What could possibly make seasoned professionals such as management consultants sit up and take notice of evaluation?

Let me set the scene. A client organisation had put out an RFP asking for an independent evaluation of a leadership initiative. Interestingly, the RFP specifically stated that the client was looking for an evaluation expert with content expertise rather a content expert (e.g., a management consulting or industrial/organisational psychologist) with evaluation experience. This is very unusual in the evaluation of leadership initiatives. Most clients are unaware that there is such a thing as evaluation expertise, as distinct from the applied research skills a well-qualified management consultant or organisational psychologist might possess.

Of 22 initial expressions of interest in the contract, just two (yes, 2!) of these were from people who identified as evaluators and participated actively as members of the [national and international] evaluation community. This was despite unusual efforts on the part of the client to attract expressions of interest from evaluators. Rather than simply posting the RFP on the usual electronic bulletin boards, which they had heard good evaluators do not usually respond to, they also sent out direct emails to evaluators who had been recommended by other evaluators and had the notice posted on an evaluation listserv.

[It is interesting to note that the process used by the client to specifically target evaluators closely mirrors best practice for the recruitment of top-notch job candidates, especially underrepresented groups—don’t just use the regular channels that yield the same old candidate pools; go to where you know the right people are and personally encourage them to apply.]

The client in this case, under the guidance of an evaluator not bidding on the job, used a creative and unusual process to select the contractor. Rather than asking shortlisted bidders to submit the usual 20-page proposal, the selection team invited them to a face-to-face meeting where they could present their thoughts on the evaluation. This was because credibility was a key element of the evaluation, which the client felt couldn’t accurately be gauged without meeting the evaluator face to face.

An added benefit of the face-to-face interview approach was that it increased the odds of both attracting and identifying a “real” evaluator. In a small community such as New Zealand, the vast majority of evaluators are solo practitioners who often partner with others for particular pieces of work. As such, they have inadequate resources to devote to compiling lengthy, slickly presented proposals that have less than an even chance of being successful. In contrast, larger consulting firms who do not have evaluation as their primary function are far more likely to have an extensive library of proposal templates and a number of junior staff trained in writing proposals. Therefore, the standard written proposal solicitation process is far more likely to yield bids from content experts than from evaluators.

Prospective contractors were asked to submit a number of supporting documents for the interview, including an outline of their “quality assurance procedures.” The proposed quality assurance procedures turned out to be one of the more telling pieces of information. After all, what better way to understand an evaluator’s grasp of his or her profession than to ask how his or her work should itself be evaluated? One case in point was a large, multinational business consulting firm (“Firm X”) whose quality assurance procedure consisted of appointing one of their independent auditors to oversee the evaluation.

In the final round, only the two “actual” evaluators passed the interview process and made it onto the final short-shortlist for being awarded the contract. When the final decision was made, the runner-up was told the background and qualifications of the successful bidder—and immediately recognised who the competitor was (New Zealand being a small evaluation community). By chance, the two met up a few days later and had a chuckle when they finally connected the dots.

In contrast, Firm X was, by all accounts, extremely surprised not to make even the final short-shortlist of two. To their credit, they did send a junior employee to see the client to get feedback about why their bid had been unsuccessful. They were even more surprised to be told that the main reason was because they were not evaluators. And no, “audits” and “reviews” of the type they were well versed in were not the same as high-quality evaluations. The consultants from Firm X were flummoxed!

Firm X asked who had been awarded the contract to evaluate the leadership initiative, and were told. They then asked who the runner-up was. The client quietly pointed out that, if they really were evaluators (as they claimed to be), they would already have found that out through their extensive evaluation networks—in the same way as the top two contenders had found out about each other.

There is a wonderful lesson here for evaluation as it strives for recognition as a distinct profession and as a discipline. I think we’ve all tried convincing the colleagues in our content disciplines that what we do is unique, complex, more than just measuring a couple of variables of interest, and something worth paying attention to. And every now and then we get a breakthrough with our evaluation evangelising. But the reality is that evaluation-savvy clients will likely sell us more converts among this audience than we could possibly manage for ourselves. There is nothing quite like being denied a contract for not being an actual evaluator!

What are some of the strategies we can use to educate clients? The simplest one that comes to mind is to highlight in our work what it is we are doing that is unique to evaluation. This might be serious and systematic attention to utilisation issues, the application of evaluation-specific methodologies not known to our non-evaluator colleagues, or the use of frameworks and models that have been developed specifically for evaluation. Whatever it is, we should be sure to highlight it in a way that makes it easy for a client to tell a “real” evaluation from the rest.

A second client education strategy is to seek opportunities to help with the development of evaluation RFPs. This was the case in the organisation I described, and it made a very substantial difference to how well the task was outlined, the selection criteria, the quality of the selection process, and client satisfaction with the outcome. Although the organisation was constrained by regulations about how an RFP process could be managed, good evaluative thinking allowed individuals within the organisation to generate a creative solution that led to the right result.

The third strategy for spreading the word about evaluation would be to follow the example of the Society for Industrial and Organizational Psychology (SIOP) in the States. Like us, I/O psychologists also have trouble getting the general public (especially managers in organisations) to understand what it is they are particularly skilled to do. In response to this need, SIOP has developed an extremely simple and straightforward leaflet, which it sends to members for distribution to managers they know. The goal was to have each member distribute the leaflet to five managers. A copy of the leaflet may be viewed online at http://siop.org/visibilitybrochure/siopbrochure.htm

It is likely that by directing our educational efforts outwards toward clients, we will have the side effect of creating some better clarity within the evaluation profession, which will in turn let us make better sense to the outside world.

Monitoring and Evaluation for Cost-Effectiveness in Development Management

Paul Clements[2]

1. Development Assistance Requires a High Analytic Standard

In the Malawi Infrastructure Project, the World Bank planned to rehabilitate 1500 rural boreholes at a cost of $4.4 million, with an estimated economic rate of return of 20%. At the project’s Midterm Review, two years later, the rate of return was reduced to 14%, but the reasons for the reduction were not clear. The plan had anticipated that 85% of project benefits would come from the value of time the villagers saved that they would have spent collecting water, and 10% from the incremental water consumed. The Midterm Review estimated 31% of benefits from time savings, however, and 56% from incremental water consumed.[3] No reason was given for reducing the estimate for time savings or for increasing the value for water consumption.

The World Bank’s Fourth Population Project in Kenya aimed to decrease Kenya’s total fertility rate to six births per woman by improving family planning services. The project was approved in 1990, and Kenya’s total fertility rate fell from 6.4 in 1989 to 5.4 in 1993. The project’s Implementation Summary Reports consistently indicated that “All development objectives are expected to be substantially achieved,” and a 1995 supervision report asserted that “The project development objectives have been fully met.”[4] Project activities, however, mainly supporting the National Council for Population and Development, were largely unsuccessful, and in 1994 a large part of the project budget was reallocated to the fight against AIDS. There were many other development agencies with family planning projects in Kenya, some with much stronger performance. Documents for the Fourth Population Project do not explain how its development objectives were related to the activities it funded.

The World Bank’s Water Supply and Sanitation Rehabilitation Project in Uganda aimed to rehabilitate the water and sewerage system in Kampala, the capital city, and in six other major towns. Its plan calculated a 20% economic rate of return based on incremental annual water sales of $5.5 million from 1988 to 2014. The completion report estimated actual returns at 18% because water production in 1991 was 10-20% below expectations.[5] The project had indeed achieved its construction goals, but its efforts to strengthen the National Water and Sewerage Corporation (NWSC) had been undermined by the government’s failure to raise water rates amidst hyperinflation and late payments on its water bills. The NWSC would have been unable to maintain the system without ongoing support, and indeed by 1993, even with a major new project supporting the water company, it was once more operating in the red.[6]

These examples come from a blind selection of four World Bank projects that I studied for my doctoral dissertation.[7] What is remarkable about these inconsistencies – an economic analysis in a midterm review that does not follow from the one in the project plan, development objectives that do not reflect project activities, an economic rate of return that anticipates 23 additional years of water sales based only on the current state of the infrastructure – is that even though at least the second two are at face value analytically incorrect, they are presented as routine reporting information, with no attempt to hide them such as in obfuscating language. Indeed they reflect common analytic practice in the international development community, and this common practice reflects a structural problem of accountability.

I would like to argue that the tasks undertaken by the large multilateral and bilateral donor agencies require a particularly high analytic standard, but several incentives that influence development practice – political incentives for donor and recipient governments, organizational incentives for development agencies, and personal incentives for managers – have led to positive bias and analytic compromise. These incentives are “structural” in that they result from the pattern of the flow of resources inherent in development assistance. The problem therefore requires a structural solution, and this paper proposes a possible solution involving a dramatic improvement in the quality and consistency of project evaluations. We can be confident that such an improvement is possible, first, because the evaluation problem facing development agencies has determinate features with specific analytic implications, and second, because a similar structural problem has already been addressed in the management of public corporations.

Sooner or later, development assistance comes down to designing investments and managing projects. Unlike private sector investments, development projects aim not to make a profit, but to improve conditions for a beneficiary population—to reduce poverty, or to contribute to economic growth. There is no automatic feedback such as in sales figures, and no profit incentive to keep managers on task. Typically one needs to strengthen existing institutions or to build new ones, and/or to encourage beneficiaries to adopt new behaviors and to take on new challenges. Yet in the project environment there is likely to be weaker infrastructure, a less well-educated population, and more risk and uncertainty than in the environments facing most for-profit enterprises. Furthermore, in places that need development assistance one cannot assume that institutional partners will be competent and mission-oriented. These conditions in combination place particular demands on development managers. Project managers need to maintain a unified conception of the project, its unfolding activities, and its relations with its various stakeholders, a conception grounded in a view of its likely impacts. Donor agency officials need a conception of the relative merits of many actual and potential projects, and an analysis that turns problems on the horizon for developing countries into programmatic opportunities.

The central challenge in the management of development assistance is to maintain this kind of consciousness—this analytic perspective—among the corps of professional staff. Some might like to think that development can be achieved by getting governments to liberalize markets or by getting local participation in project management, and these may well be important tactics. Intuition suggests and experience teaches, however, that there can be no formula for successful development. Each investment presents a unique design and management challenge. There are two problems in maintaining the will and the capacity to address this challenge: an incentive problem and one we can call intellectual or cognitive. The key to solving both problems, or so I will argue, is strong evaluation.

2. But Accountability in Development Assistance is Weak

2.1 Donor agencies are responsible for the success of their projects

According to the World Bank’s procurement guidelines, “The responsibility for the execution of the project, and therefore for the award and administration of contracts under the project, rests with the Borrower.”[8] One might think that a development loan to a government is like a business loan to an entrepreneur. The donor agency makes the loan, but it is entirely the responsibility of the borrower government to spend the money. Whether a government manages its projects well or poorly, one might imagine, is primarily its own affair, with the donor providing technical assistance upon request. We know, of course, that this image is incorrect—donor agencies typically have the predominant influence over project design, and substantial influence over project administration—but it is useful to recall why this is so. One reason is parallel to a private bank’s prudential interest in the management of its loans. As the World Bank’s Articles of Agreement state,

The Bank shall make arrangements to ensure that the proceeds of any loan are used only for the purposes for which the loan was granted, with due attention to considerations of economy and efficiency and without regard to political or other non-economic influences or considerations.[9]

The Bank wants to be repaid, and it also has an interest in promoting economic growth and enhancing well-being in borrower countries, so it may take pains to see that its loans are well spent. Many loans are to governments with limited bureaucratic capacity in countries with inconsistent management standards, so the Bank must retain enough control to ensure that the projects it supports are properly administered.

By this logic we would expect relationships with bureaucratically stronger governments to be closer to the private sector model, and indeed some governments with coherent industrial strategies (consider South Korea in the 1970s) have succeeded in using Bank loans very much for their own purposes.[10] Many development loans, however, are for projects at the edge of the borrower’s frontier of technological competence, and the Bank (like other donor agencies) is a repository of expertise in the sectors it supports. The Bank also has demanding requirements for project proposals, and many governments have been unable independently to prepare proposals that the Bank could accept, particularly in earlier years when patterns of Bank-borrower relations were established. Therefore the Bank has generally taken primary responsibility for designing the projects it funds,[11] and the responsibility that comes with authorship cannot be lightly abandoned during implementation.

A second reason that donor agencies take an interest in how their funds are spent is that donor funds come from (or are guaranteed by) governments, and, the Bank’s Articles of Agreement notwithstanding, governments do not release funds without taking an interest in their disposition. On one hand this political logic reinforces the prudential logic discussed above. Donor governments want their funds to contribute to the borrower’s development, so they insist that donor agencies take responsibility for project results. Foreign aid is also, on the other hand, enmeshed in donor governments’ general promotion of their foreign policy agendas.[12] It matters that the United States is not indifferent as to whether and when the World Bank will make loans to Cuba, and World Bank loans to Côte d’Ivoire have been subject to particular influence from France, the country’s former colonial master.[13] Bilateral aid is even more closely linked to donor government interests than aid through multilateral institutions. Not only from the donor side but from that of recipient governments too, the parameters of development spending cannot be understood merely in terms of the requirements for maximizing development impacts.

As intermediaries between donor and recipient governments, donor agencies are required to take more responsibility than private banks for managing the loans they make. The analogy with the private sector breaks down even further, however, when we consider the incentives governing a donor agency’s management of its portfolio. The main cause for different incentive structures between donor agencies and private banks, of course, arises from differential exposure to financial risk. With private loans, the borrower and often the lender suffer a financial loss if the investment fails. With most development projects, by contrast, neither the donor nor the implementing agency faces a financial risk if impacts are disappointing. For projects funded by loans it is the borrower government, typically the treasury, that is responsible for payments. But the treasury seldom has control over individual development projects.

2.2 The usual watchdogs are not there to hold donor agencies accountable

The structural conditions of development assistance, therefore, create an accountability problem. Donor agencies have control over development monies but they face no financial liability for poor results (and no financial gain when impacts are strong). In this context their orientation to their task will depend largely on the demands and constraints routinely placed on them by other agents in their organizational environment, on the individual and corporate interests of their leaders and employees, and on the mechanisms of accountability that are institutionally (“artificially”) established.

In regard to external agents, Wenar notes that there has been a “historical deficiency in external accountability” for donor agencies.

Aid organizations have evolved to a great extent unchecked by the four major checking mechanisms on bureaucratic organizations. These four mechanisms are democratic politics, regulatory oversight, press scrutiny, and academic review.[14]

The electorates in donor countries want to believe that aid is helping poor people, but democratic politics also leads to pressures on donor agencies to support the agendas of well-organized interest groups.[15] Some promote humanitarian and progressive agendas, but others have aims that create tensions with development goals. Generally, since the intended beneficiaries of aid cannot vote in donor country elections, the reliability of democratic politics as a source of accountability is limited. There has been significant regulatory oversight aiming to ensure that aid funds are not fraudulently spent, but external oversight of project effectiveness faces major practical hurdles. Aid projects are so widely dispersed, and the period between when monies are spent and when their results transpire is typically so substantial that effective oversight would require major bureaucratic capacity. Responsibility for project evaluation, however, has normally rested with the donor agencies themselves. This clearly leads to conflicts of interest, and it is the aim of this paper to suggest how these conflicts could be, if not removed, at least substantially ameliorated. Donor agencies have not, in any case, been subject to significant external accountability by way of regulatory oversight. Press scrutiny and particularly academic review, in contrast, have been significant sources of accountability, and academic studies have contributed to many foreign aid reforms. Given the strength of the political and bureaucratic interests that drive the programming of aid, however, and the above-noted dispersal of aid projects, scholars and journalists can only be expected to hold aid agencies accountable in a limited and inconsistent manner. Also, they are largely dependent, for information on aid operations on the donor agencies themselves.

Few who have spent much time with development agency personnel can doubt their generally admirable commitment to development goals, and the reforms this paper will propose depend heavily on the personnel’s sustained interest in professionalism and effectiveness. Their behavior is also influenced, however, by their individual and corporate interests, and these interests take shape in the specific task environments that they face in their home offices and in the field. There are two aspects of the way their interests come to be constructed that are particularly relevant to the problem of accountability. First and most obviously, while institutional norms require donor agencies to maintain the appearance of a coherent system of responsibility for results, their institutional relationships require them to maintain the appearance that their operations are generally successful. They must evaluate, but it serves their individual and corporate interests if evaluation results are generally positive (or at least not often terribly negative). Since donor agencies have generally controlled their own evaluation systems, they have had the opportunity to design these systems in such a way that they would tend to reflect positively on the agencies themselves. Second, due in part to the long time span between the commitment of funds and the evaluation of results, internal personnel evaluations have tended to focus on variables only loosely correlated with good results, and sometimes on variables that conflict with good practice.

2.3 Lacking secure accountability for results, other less relevant criteria inform resource allocation decisions

We will later consider some of the approaches donor agencies have taken to evaluation below. For the purposes of understanding the accountability problem in development assistance, it is enough for now to note that donor agencies have controlled their own evaluation systems. In the context of the general deficiency in external accountability, the priorities that have been enforced within donor agencies take on particular significance. Perhaps the most longstanding and sustained critique of donor agencies’ internal operations involves the imperative to “move money.”

The classic account of the “money-moving syndrome” is Tendler’s Inside Foreign Aid.[16] Focusing on the U.S. Agency for International Development (USAID) and the World Bank, Tendler identifies a “pressure to commit resources that is exerted on a donor organization from within and without,” and finds that “standards of individual employee performance … place high priority on the ability to move money.”[17] In the context of her organizational analysis, she gives several examples of aid officials knowingly supporting weak projects in order to reach spending targets.[18] Tendler also finds, reinforcing the present argument about evaluation, that in a political environment often hostile to foreign assistance, aid officials learned to self-censor reports that could provide ammunition for critics.

For writing what he considered a straightforward description of a problem or a balanced evaluation of a project, an AID technician might be remonstrated with, “What would Congress or the GAO [General Accounting Office] say if they got hold of that!?” … Words were toned down, thoughts were twisted, and arguments were left out, all in order to alleviate the uncomfortable feeling of responsibility for possible betrayal. … Such a situation must have resulted in a certain atrophy of the capacity for written communication – and, inevitably, for all communication through language.[19]

The World Bank typically required economic analysis of proposed projects, but Tendler found that many ostensibly economic projects were selected by non-economic criteria.[20] Much of the economic analysis that was carried out amounted to a “post hoc rationalization of decisions already taken.”[21]

While Tendler offers several political and organizational reasons to explain the money-moving imperative, I would like to emphasize what is absent from the organizational culture she describes. We do not find a sustained effort to consider how development funds can be employed to maximize their contribution to development. In such an environment we might expect well-intentioned professionals, once they win some organizational power, to act like policy entrepreneurs, promoting their individual conception of a good development agenda in large measure despite the prevailing incentives. We might expect segments of a donor agency that have strong external allies to develop coherent agendas that they can implement themselves, as I believe reproductive health professionals at USAID have done. What we cannot expect, however, is that organizational decisions will routinely be taken on the basis of expected impacts.

The World Bank’s “project approval culture” was recognized in its internal 1992 study, “Effective Implementation: Key to Development Impact” (popularly called the Wapenhans Report). The report cites a “pervasive preoccupation with new lending,”[22] in part because “signals from senior management are consistently seen by staff to focus on lending targets rather than results on the ground,”[23] noting also that “[t]he methodology for project performance rating is deficient; it lacks objective criteria and transparency.”[24] Although the report describes the Bank’s evaluation system as “independent and robust,” it finds that “[l]ittle is done to ascertain the actual flow of benefits or to evaluate the sustainability of projects during their operational phase.”[25]

Since the appearance of the Wapenhans Report, the Bank has moved increasingly to spending modalities that further dilute accountability for results. The two kinds of programs that have become most central to Bank strategies particularly in lower income countries are adjustment loans of various kinds (structural, sectoral) and Poverty Reduction Strategy Papers (PRSPs).[26] Adjustment loans require borrowers to adopt free market reforms in order to better align economic incentives with development goals. They tend to operate on a wider scale than traditional projects, with more diffuse impacts. There is often a feeling that they are imposed, as the government receives the loan for policy changes it presumably otherwise would not have made, and they are often implemented only partially and inconsistently. These factors make them harder to evaluate. Poverty Reduction Strategy Papers typically push a larger part of the responsibility for evaluation onto the borrower government, and it seems that their more participatory approach to policy formation and implementation is intended to substitute, to some extent, for rigorous agency evaluation. They ask the government, as part of the process of generating a poverty reduction strategy, to identify a set of indicators for measuring the strategy’s impacts. If the World Bank has had such a hard time ascertaining the level and sustainability of impacts from its own portfolio, however, it is questionable whether governments of low-income countries will be able to do much better.

3. Independent and Consistent Evaluation Can Improve Accountability and Learning in Development Assistance

3.1 The basic idea of the proposed evaluation approach

The problems discussed above present formidable obstacles to maintaining accountability in foreign assistance on the basis of program and project results. We should recall, however, what is at stake. In the absence of meaningful accountability there is little to counter-balance the pressures for aid resources to support political interests of donor and recipient governments, organizational interests of donor and implementing agencies, and personal interests of management stakeholders. The inconsistency and mixed reliability of evaluations have also undermined learning from experience, so the aid community has been slower than it would otherwise have been to identify successful strategies and to modify or abandon weak ones.[27]

One way to address the historical deficit of external accountability, for example to push the focus of management attention forward from moving money to achieving results, and to improve the incentive and the capacity to manage for impacts, is to institute independent and consistent evaluations of the impacts and cost-effectiveness of donor-funded projects.[28] This