<incom> Evaluation says key World Bank research ‘not remotely reliable’

[Soenke Zehle]

"research was not supposed to offend NGOs, nor to provide them with 
material they could use to criticise the Bank" ... World Development 
Reports "were a prime example of research where the conclusions are 
either predetermined or negotiated in advance," Soenke

Knowledge Bank-rupted: Evaluation says key World Bank research ‘not 
remotely reliable’
News|Bretton Woods Project|31st January 2007|update 54|url
Illustration by Robin Heighway-Bury/Thorogood.net

<http://www.brettonwoodsproject.org/art.shtml?x=549070>

An evaluation by a panel of self-described 'academic superstars' has 
cast doubt over the independence and reliability of World Bank research. 
However, in failing to address fundamental problems in the Bank's role 
in development research, the report misses an opportunity to provide a 
clear signal for donors to shift support to developing country research 
institutions.

The evaluation covers Bank research - conducted in the development 
economics vice-presidency (DEC), regional and thematic departments, the 
World Bank Institute, and that by consultants - in the period 1998 to 
2005. Chaired by Angus Deaton, professor of economics at Princeton 
University, the evaluation says that, overall, the Bank's researchers 
"have done a creditable job of delivering on the many, potentially 
inconsistent, demands made of them". "Much of what we read was of very 
high quality", say the report's authors.

But this praise is scant recompense for what follows. The report finds 
that "there is a great deal of research that is undistinguished and not 
well-directed either to academic or policy concerns". Amongst this 
'undistinguished' work, the evaluators found research which is 
"technically flawed and in some cases strong policy positions have been 
supported by such (non) evidence". In some cases, "the Bank proselytised 
selected new work in major policy speeches and publications, without 
appropriate caveats on its reliability". The ensuing litany of critiques 
is withering:

* New research methods have been promoted "without adequate evaluation";

* Many Bank researchers try to prove causality by tinkering with 
economic models; in the words of the evaluators, they "appear to think 
that the attribution of causality can be solved by technical means";

* In some cases, "the degree of self-reference rises almost to the level 
of parody"; and

* There is "remarkably little work co-authored by non-Bank researchers 
from developing countries".

The evaluators posit a number of reasons to explain the breakdowns. Most 
damning for Bank senior management is their finding that researchers are 
"under pressure from the Bank presidency and elsewhere not to say things 
that go directly against the broad policy line that the Bank is 
espousing." Researchers are chosen by country teams who are looking for 
"a particular answer or a particular researcher who they know and like 
working with, or perhaps someone known for not rocking the boat." The 
researchers themselves say it is not unusual to be told that "we should 
do an evaluation to prove that X programme works".

This evidence is backed up by interviews with former Bank research heads 
who complain that "there was an enormous amount of interference by the 
public relations people, especially after Wolfensohn became president; 
research was not supposed to offend NGOs, nor to provide them with 
material they could use to criticise the Bank." Former chief economist 
Joseph Stiglitz added that during the Asian financial crisis "the belief 
that certain policies always worked meant that the more relevant 
questions of when the policies worked were not addressed."
Bank attempts to silence critics

Update 53 carried an abridged version of American University professor 
Robin Broad's paper Research, knowledge and the art of 'paradigm 
maintenance': the World Bank's Development Economics Vice-Presidency, 
which originally ran in the August 2006 issue of the Review of 
International Political Economy. Before it was published, Daniel 
Lederman and Martin Ravallion of DEC wrote to the board of the journal 
calling into question the quality and accuracy of Broad's article, 
labelling it as "shoddy scholarship." The journal's editorial board, 
however, defended its decision to publish Broad's piece and offered the 
Bank a chance to debate Broad in the pages of the journal. With the 
impending release of the official evaluation, perhaps it is not 
surprising they turned down the offer.

"Paradigm Maintenance", or why we can't trust the World Bank's research

D-day for Dollar

The examples given of the best of Bank research include several 
controversial choices (teacher absenteeism, project evaluation using 
randomised trials, and the Investment Climate and Doing Business 
surveys) as well as elemental statistical work such as the World 
Development Indicators.

When asked to point to the most flawed examples, the evaluators chose 
some of the highest-profile research conducted by the Bank over the past 
decade. Of Dollar and Burnside's paper Aid, policies and growth, which 
the Bank has cited repeatedly to argue for increased support for 
countries which it considers to have 'good policies': "We think that the 
Bank was unwise to place so much weight on one paper whose evidence is 
so unconvincing." The implications for the Bank, say the evaluators, are 
alarming: "once the evidence is chosen selectively without supporting 
argument, and empirical scepticism selectively suspended, the 
credibility and utility of the Bank's research is threatened."

Heavily slated is Dollar and Kraay's research providing succour to Bank 
arguments that trade liberalising countries have seen greater poverty 
reduction: "Much of this line of research appears to have such deep 
flaws that, at present, the results cannot be regarded as remotely 
reliable." International trade director Uri Dadush will undoubtedly be 
looking over his shoulder in the new year, as the evaluators find that, 
in addition to the misplaced advocacy, Bank trade work more generally 
has "insufficiently addressed the effects of trade on poverty", and has 
been dominated by arcane computable general equilibrium models. 
Dissenting research conducted within the Bank, such as that by Branko 
Milanovic, has been routinely "ignored" (see Update 30). This repeats 
criticisms that were made of the Bank's trade work last year in an 
evaluation conducted by the Independent Evaluation Group (see Update 50).

Other research which comes in for heavy criticism includes that on:

* Pensions and insurance: "The analytical errors referred to are those 
that would be well understood by a first-year graduate student in 
economics." Criticism of the pro-privatisation bias of the Bank's 
pension reform team echoes similar disapproval expressed in two earlier 
internal Bank reports: Keeping the Promise of Old Age Income Security, 
by the Bank's Latin America department and last year's evaluation of 
Bank assistance for pension reform by the IEG. Trade union opposition to 
the Bank's pensions policy was initially rebuffed, but a conference has 
now been scheduled in 2008 for the Bank to review its policy.;

* Poverty mapping: "Our recommendation is that this work be put on hold 
until the statistical problems are resolved."; and

* Civil war: Paul Collier's work "lacks an appropriate conceptual and 
empirical framework".

World Development Reports escape the knife

The survival of the Bank's flagship annual report, the World Development 
Report (WDR), is brought into question by the evaluators' comments. 
Interviews with former heads of Bank research confirmed Bank critics' 
worst suspicions that WDRs "were a prime example of research where the 
conclusions are either predetermined or negotiated in advance."

While the evaluators said that WDRs are valued for providing a 
comprehensive literature summary and for their "breadth of scholarship", 
the enormously costly annual exercises tend to downplay critical policy 
trade-offs, overlap hugely from year to year, and are marred by poor 
editing and presentation. Of the 2005 WDR on the investment climate (see 
Update 42 , the evaluation finds it "is almost a caricature of the view 
that everything is important". "If there are priorities, they are vague 
and constantly changing", and the report notes that "virtually every 
conceivable aspect of a country's social, political and economic 
institutions affects its investment climate."

The evaluators note with concern the absence of an evaluation of the 
impact or usefulness of the WDRs. In their recommendations they consider 
going to publication every two years, but in the end, defer to the 
importance of the WDR as an annual high-profile vehicle for Bank 
research. Of the Bank's other high-profile reports, the evaluation says 
"the Bank produces too many long (and sometimes unreadable) book-length 
reports that are ostensibly directed at policymakers, but seem very 
unlikely to be read by them."

A missed opportunity?

Astonishing considering the analysis which precedes it, is the 
evaluators' recommendation to increase support to Bank research through 
the creation of a research department. They recommend the establishment 
of a "research endowment fund" taken from Bank's retained earnings - 
that is, taken from developing countries' interest payments - to pay for 
it. Other more useful recommendations include:

* Increasing the presence of developing country researchers and support 
for institution-based research in developing countries;

* The establishment of a peer review mechanism for research output; and

* Improved cost accounting for research - cutting back on too many thick 
volume flagship reports and improving those that remain.

Not surprisingly, suggestions that there should be more funding for 
research have been warmly received at the Bank. Bank chief economist 
François Bourguignon refuted the evaluation's description of Bank 
research as "undistinguished". The Bank has said it agrees with the 
recommendation to build research capacity in developing countries and 
will 'increase their efforts'. There was no comment on the 
recommendation to begin peer review of Bank research, and on the 
suggestion to cut its major outputs, the Bank will "explore approaches 
to reduce the volume of reports where possible".

But while the evaluation has shed light on a number of concerns, other 
more serious problems are left unaddressed. Two DEC staff, economist 
Vijayendra Rao and sociologist Michael Woolcock, stress that "economics 
should not have (as it currently does at the Bank) a near-monopoly on 
determining the content and validity of development research". Rao and 
Woolcock criticise the evaluators themselves over their methodology and 
for the complete absence of non-economists in the team. They urge the 
Bank to "engage with alternatives to the dominant views in development 
research" and expand the number of staff with non-economic training.

Even more fundamental is the belief of David Ellerman, visiting scholar 
at the University of California and advisor to former chief economist 
Joseph Stiglitz, that the root of the problem lies in the fact that the 
Bank taking a stand itself on the issues contradicts the institution 
posing as an unbiased research organisation: "If any university took an 
official stand on certain issues in physics, biology, or the social 
sciences, then it would quickly distort the future research by faculty 
members since any contrary findings would 'embarrass' the university."

Which harkens back to the question of whether or not the Bank should 
play the role of global development 'knowledge bank'. On the basis of 
similar findings to those of the official evaluation, but derived from 
independent research, American University professor Robin Broad (see 
Update 53) concludes: Bank backers "would do far better to support 
independent research institutions that are stimulating a more diverse 
development debate".