The Place of Randomized Controlled Trials in Development

Source

A recent article in The Economist was devoted to randomized controlled trials (RCTs):

(I)n 2003, RCTs were regarded as wacky. Critics said that doing a trial was like putting people in a cage and experimenting on them. They pointed out that you cannot conduct randomised trials for big macroeconomic questions (“What happens if we devalue the currency by 50%?”) because there can be no control group. They conceded RCTs might generate useful nuggets of evidence (raising teachers’ wages in India, for example, did surprisingly little to improve learning). But they argued that evidence from such trials would always remain small-scale, tied to a specific context and not be useful beyond it. 

Ten years on, few of those criticisms have stood up to scrutiny. RCTs have entered the mainstream.

The idea of a RCT is to test a treatment by applying it to one group and not to a second group, and seeing which group does better. They are seen as the gold standard in testing new drugs for safety and efficacy. The two groups should be as similar in characteristics as possible, and thus random assignment of members is a key characteristic of RCTs.  In medical trials, the control group may receive a placebo if there is no accepted drug for the condition, or the best alternative if one does exist. Ideally, RCTs when possible are blinded, so that preconceptions of those delivering the treatment, receiving it, or evaluating results do not bias the results.

Of course, RCTs were used in the context of developing nations more than ten years ago. Drug trials for preventive and curative treatments of tropical diseases specific to developing countries go back many, many years. What is new is the extension of the RCT approach to other fields. Thus The Economist lists

• whether boosting teachers’ pay improved educational outcomes
• whether identity cards would improve the delivery of subsidized rice to the poor
• whether job-training encouraged employment growth overall or just boosted the prospects of trainees at the expense of the untrained
• the effect of remedial classes
• what happens if you double the number of teachers
• what are the effects on micro loans on recipients entrepreneurship and well being
• what are the effects of small charges in acceptance of bednets treated to combat malaria or water purification tablets

The characteristic of all these applications of RCT is that a single "thing" is tested to see if it is better than the alternative, which is previous practice. That is all to the good.

The Limitations of RCT

Do we always need to do a full scale RCT to figure the better alternative. I doubt it. When a vaccine or antibiotic has already been proven safe, efficacious, and effective in many countries, it may be applied in a new country on the basis of the existing foreign data (with some careful monitoring).

Moreover, a lot of improvements are made step by step, improving what is being done by small, conservative innovations. Workers improve the use of equipment on the shop floor; Managers improve the allocation of staff to functions or business processes. Buyers and sellers figure ways to improve the function of the market in which they transact their business. Farmers switch from a less to a more profitable crop. Teachers find a better way to help students to learn. It happens all the time! And it is a key to development. RCT doesn't work well for monitoring such incremental change.

And of course RCT doesn't work on all interventions. You can't try an approach to reduce global warming on 50 planets with 50 more as controls. You can't ethically impose sanctions on 50 countries for stockpiling WMDs and not on 50 others committing the same infraction, and wait for wars to break out to see which group kills fewer people. But that does not mean that they are not a useful tool where they work.

Beyond RCT

Fundamentally, development is about systematically doing things better than they were done before.   The fundamental objective of development is to establish a widely effective, indigenous process of rapid improvement in the way things are done.

Many years ago I took a course in community development from Jack Donoghue. He told us of an experience he had in which community leaders had invited him to help their small town community which was not functioning at all well. He spent a few days in the town, talking to people and came to the conclusion that the problem was simply that they were so divided that they would not negotiate constructively among themselves to solve local problems. His solution? He gathered the town leaders together and refused the job, explaining that they were acting like jerks; if they continued to act as they were, he couldn't do any good, and if they changed their behavior and acted more reasonably they didn't need him. The shock treatment worked, and behavior changed. Something similar seems recently to have happened with the House of Representatives; members received a storm of criticism when they shut down the government in October, leading them to agree to an appropriations bill last week. These may be seen as examples of changing the culture to achieve a development goal.

Steve Jobs is credited with turning Apple around when he returned to the company as CEO in the late 1990s. The company stock price increased, attesting to his success. Was what he did suitable for analysis using RCT approaches? I doubt it. I suspect that he changed the culture of the organization so that it began to do many things better, especially the development of new products.

We have been celebrating Nelson Mandela this month. South Africa was a mess in the early 1990s. People were rioting and the Apartheid government was reacting violently -- violence begat violence! The GDP was declining; South Africa was becoming a pariah state. Mandela, representing the ANC, and de Klerk, the President of South Africa, agreed that democratic elections would be held. When they were, majority rule was established in South Africa with Mandela as President, apartheid policies were ended, violence greatly reduced, and growth was restored to the economy. Under the truth and reconciliation policies, racial tensions were reduced. A single event, combined with great leadership, led to a cultural change which greatly enhanced national social and economic development.

More generally, I doubt that we will often find "magic bullets" to change cultures in ways the significantly enhance the rate at which improvements are made in the ways things are done. I suspect development is more like herding cats, putting one thing after another back on track. Thus one might make a country's government a little more capable, a little more responsive to the people. One might then help the farmers to be a little more productive, and the agricultural research and extension service to be a little more efficient and effective. That might be followed by making some improvements in economic policy and strengthening some domestic markets. The process might go on and on until the culture changed and the process of development maintained by the people, their own policies, and their institutions.

RCTs would have a place, but a relatively small place, in such a process of development. One might also evaluate the efficiency and impact of each separate initiative, but especially the sustainability of each effort after the external stimulus was applied. Perhaps more important still would be a big picture evaluation of whether the culture was changing in the right direction, and whether social and economic indicators were suggesting progress.