A collation of my work and writing on randomised control trials (RCTs)

The single topic that I have written on most extensively to date is the use of randomised control trials (RCTs) in economics to identify causal effects, generalise those findings and make policy claims. Much of this was done before this approach was awarded the 2019 Nobel Memorial Prize in Economics: I started work on RCTs in 2010 for my PhD in economics.

The purpose of this page is to collate links to all of that work in one place. I have ordered the publications based on how some interested readers might want to go through them (which is why the link to my 200+ page PhD thesis comes at the end!).

Some of the academic articles are, unfortunately, gated – I put 🔐 symbols next to those. Feel free to contact me if you’d like a copy of any of them.

The 2019 Nobel Memorial Prize in Economics was awarded to three scholars for the methodological approach that has been the focus of critique in my work. In a short paper in a special issue of the journal World Development on the 2019 Nobel, The implications of a fundamental contradiction in advocating randomized trials for policy” (🔐), I aim to provide a succinct version of my argument against that ‘randomista’ approach.

Two articles with co-authors (Grieve Chelwa and Nimi Hoffmann) in The Conversation aimed at a more general audience also respond to the 2019 Nobel award. The first, How randomised trials became big in development economics, provides some background. The second, Randomised trials in economics: what the critics have to say, explains some of the criticisms – including my own.

In a special issue of the CODESRIA Bulletin on Randomised Control Trials and Development Research in Africa, I argue that RCTs are “A Dead-End for African Development”. In other words, I argue that the likely outcome of the emphasis on such methods will be to retard development – in sharp contrast to what proponents claim.

One example I discuss in that working paper is the use of RCTs in the context of education policy debates in South Africa. In a seminar given as a visiting fellow at the Johannesburg Institute for Advanced Study, “The new colonial missionaries: basic education policy and randomised trials in South Africa”, I explain how academics with a ‘missionary zeal’ (Bardhan) have sought, with some success, to inappropriately dominate basic education policy debates in South Africa. (I also wrote a fairly lengthy blog post on related matters in 2016, “Some thoughts on Taylor and Watson’s (2015) RCT on the impact of study guides on school-leaving results in South Africa”).

The crux of the formal (technical/econometric) argument I have made against the ‘randomista’ use of randomised trials in economics and for public policy was published well before the Nobel was awarded, in an article in the World Bank Economic Review, Causal Interaction and External Validity: Obstacles to the Policy Relevance of Randomized Evaluations”.

Some of the limitations of using RCTs for policy, and insisting on them as the only truly credible basis for decision-making, have been revealed during the Covid-19 pandemic. I wrote a short paper on that for a special issue of History and Philosophy of the Life Sciences, Masks, mechanisms and Covid-19: the limitations of randomized trials in pandemic policymaking”.

The misuse of an RCT to distort a public policy process in a manner that facilitated private sector rent-seeking – rather than the policy objective of reducing youth unemployment – was the focus of my detailed analysis of South Africa’s ‘Youth Employment Tax Incentive’ (YETI) published in Development and Change, Evidence for a YETI? A Cautionary Tale from South Africa’s Youth Employment Tax Incentive (🔐).

In a recent chapter in the Edward Elgar compilation A Modern Guide to Philosophy of Economics, Randomised trials in economics(🔐), I provide my most detailed assessment of these issues. A notable additional contribution of this chapter is that it examines the strategies advocates of these methods are using in an attempt to counter criticisms and explains why those are unconvincing and cannot succeed.

In another chapter forthcoming in the Routledge volume The Positive and the Normative in Economic Thought, The Unacknowledged Normative Content of Randomised Control Trials in Economics and Its Dangers(🔐), I explain how normative factors (biases, prejudices, ideologies, etc) enter a process that is typically represented as ‘objective’, ‘scientific’ and ‘neutral’. This develops a point I alluded to in earlier work.

All of this work began with the research conducted for my PhD in economics, “The external validity of treatment effects: an investigation of educational production”, which I started at the beginning of 2010 and completed in 2014 – under the supervision of Martin Wittenberg, examined by Gary Solon, Jeff Smith and Steve Koch.

Higher education funding in South Africa

Recently, student protests have again erupted at higher education institutions in South Africa. When the original #FeesMustFall protests began in 2015 I was working at the Parliamentary Budget Office and trying to advise members of the finance and appropriations committees as best I could on the proposals being hurriedly drafted by the government. Subsequently, after moving back into academia, I wrote a number of pieces on the issues raised both by the students, the associated public debate, and actual or proposed policy decisions:

“Free higher education in South Africa: cutting through the lies and statistics” https://theconversation.com/free-higher-education-in-south-africa-cutting-through-the-lies-and-statistics-90474

Options on the table as South Africa wrestles with funding higher educationhttps://theconversation.com/options-on-the-table-as-south-africa-wrestles-with-funding-higher-education-87688

On the recent protests and apparent policy decisions, these two interviews (radio and television) provide my initial assessment:

Radio: https://iono.fm/e/1008788

Television: https://www.youtube.com/watch?v=CcoOAS03zuY

At present, I am conducting research on some related matters with colleagues from UCT and we hope through that to contribute to deeper understanding of the issues. Our report should be completed by the end of 2021.

Some thoughts on Taylor and Watson’s (2015) RCT on the impact of study guides on school-leaving results in South Africa

Since 2010 most of my time spent on academic research has focused on two particular areas:

  1. The use of randomised control trials (RCTs) to support inappropriate, or overly strong, policy claims or recommendations
  2. Empirical examples of how this has manifested in the economics of education.

I was therefore somewhat frustrated when I attended a presentation at the Economic Society of South Africa conference in 2013 to find some rather strong policy claims being made on the basis of what is very weak evidence (even by the standards of practitioners favouring RCTs). I raised my concerns with the relevant author, but I see that the recently-published working paper contains the same problems.

It therefore seems appropriate to summarise my concerns with this work: partly so that interested parties can understand its flaws, but mainly to provide an illustration of how the new fad for RCT-based policy is often oversold.[1] That’s important, because despite seemingly ample evidence I often get economists saying: “Oh but no-one really uses RCT results in that way”.

Continue reading “Some thoughts on Taylor and Watson’s (2015) RCT on the impact of study guides on school-leaving results in South Africa”