Category: causality

A collation of my work and writing on randomised control trials (RCTs)

The single topic that I have written on most extensively to date is the use of randomised control trials (RCTs) in economics to identify causal effects, generalise those findings and make policy claims. Much of this was done before this approach was awarded the 2019 Nobel Memorial Prize in Economics: I started work on RCTs in 2010 for my PhD in economics.

The purpose of this page is to collate links to all of that work in one place. I have ordered the publications based on how some interested readers might want to go through them (which is why the link to my 200+ page PhD thesis comes at the end!).

Some of the academic articles are, unfortunately, gated – I put 🔐 symbols next to those. Feel free to contact me if you’d like a copy of any of them.

The 2019 Nobel Memorial Prize in Economics was awarded to three scholars for the methodological approach that has been the focus of critique in my work. In a short paper in a special issue of the journal World Development on the 2019 Nobel, “The implications of a fundamental contradiction in advocating randomized trials for policy” (🔐), I aim to provide a succinct version of my argument against that ‘randomista’ approach.

Two articles with co-authors (Grieve Chelwa and Nimi Hoffmann) in The Conversation aimed at a more general audience also respond to the 2019 Nobel award. The first, “How randomised trials became big in development economics”, provides some background. The second, “Randomised trials in economics: what the critics have to say”, explains some of the criticisms – including my own.

In a special issue of the CODESRIA Bulletin on Randomised Control Trials and Development Research in Africa, I argue that RCTs are “A Dead-End for African Development”. In other words, I argue that the likely outcome of the emphasis on such methods will be to retard development – in sharp contrast to what proponents claim.

One example I discuss in that working paper is the use of RCTs in the context of education policy debates in South Africa. In a seminar given as a visiting fellow at the Johannesburg Institute for Advanced Study, “The new colonial missionaries: basic education policy and randomised trials in South Africa”, I explain how academics with a ‘missionary zeal’ (Bardhan) have sought, with some success, to inappropriately dominate basic education policy debates in South Africa. (I also wrote a fairly lengthy blog post on related matters in 2016, “Some thoughts on Taylor and Watson’s (2015) RCT on the impact of study guides on school-leaving results in South Africa”).

The crux of the formal (technical/econometric) argument I have made against the ‘randomista’ use of randomised trials in economics and for public policy was published well before the Nobel was awarded, in an article in the World Bank Economic Review, “Causal Interaction and External Validity: Obstacles to the Policy Relevance of Randomized Evaluations”.

Some of the limitations of using RCTs for policy, and insisting on them as the only truly credible basis for decision-making, have been revealed during the Covid-19 pandemic. I wrote a short paper on that for a special issue of History and Philosophy of the Life Sciences, “Masks, mechanisms and Covid-19: the limitations of randomized trials in pandemic policymaking”.

The misuse of an RCT to distort a public policy process in a manner that facilitated private sector rent-seeking – rather than the policy objective of reducing youth unemployment – was the focus of my detailed analysis of South Africa’s ‘Youth Employment Tax Incentive’ (YETI) published in Development and Change, “Evidence for a YETI? A Cautionary Tale from South Africa’s Youth Employment Tax Incentive” (🔐).

In a recent chapter in the Edward Elgar compilation A Modern Guide to Philosophy of Economics, “Randomised trials in economics“(🔐), I provide my most detailed assessment of these issues. A notable additional contribution of this chapter is that it examines the strategies advocates of these methods are using in an attempt to counter criticisms and explains why those are unconvincing and cannot succeed.

In another chapter forthcoming in the Routledge volume The Positive and the Normative in Economic Thought, “The Unacknowledged Normative Content of Randomised Control Trials in Economics and Its Dangers”(🔐), I explain how normative factors (biases, prejudices, ideologies, etc) enter a process that is typically represented as ‘objective’, ‘scientific’ and ‘neutral’. This develops a point I alluded to in earlier work.

All of this work began with the research conducted for my PhD in economics, “The external validity of treatment effects: an investigation of educational production”, which I started at the beginning of 2010 and completed in 2014 – under the supervision of Martin Wittenberg, examined by Gary Solon, Jeff Smith and Steve Koch.

Causal inference, alcohol bans and Covid-19 in South Africa: a short comment

As in other countries, South Africa has used various forms of restrictions on societal activity in an attempt to slow or prevent the spread of SARS-CoV-2 (‘Covid’). One measure that is relatively unusual is the limit on alcohol sales, which has varied in severity from a complete ban on any sales or transporting of alcohol to less severe variations on that such as banning only sales for off-site consumption, or limiting such sales to particular days and hours.

Such measures have drawn some vehement criticism, not least from the alcohol industry itself. One large player in that industry, Distell, commissioned a piece of research which argued that there was no defensible basis for these measures. That in turn was widely cited in the media, and at least one editor claimed that it showed: “There’s no way the alcohol bans in SA have been based on credible science. They’re based on prejudice.”

In the same month (April 2021) I was contacted by a civil society organisation for an expert opinion on that report. I wrote a short assessment, which takes a dim view of the approach and claims of the report – with corresponding implications for associated assertions that use it as a ‘scientific’ basis for opposing alcohol restrictions.

Alcohol_trauma_SMM_FINAL

As I indicate in my comments, this is ultimately an empirical question on which I have no prior views. The claim that a reduction in access to alcohol does significantly reduce the demand for hospital resources that are needed for critical Covid-19 cases is plausible. Whether it is true remains to be seen. A number of papers have been published on the subject, see:

https://onlinelibrary.wiley.com/doi/full/10.1111/dar.13310

https://www.sciencedirect.com/science/article/pii/S2211419X20301464

https://journals.co.za/doi/abs/10.4102/phcfm.v12i1.2528

https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7719204/

I leave thoughts on those, and others which are likely to be out soon, for later work.

On the issue of economic impact, which the alcohol industry emphasises, there are certainly also concerns. However, it is useful to remember that industry estimates of economic harm from limiting their activities are often exaggerations of the net economic impact. Reductions in consumption also have a significant negative impact on government revenue from excise duties, though these are arguably quite small when compared to the broader economic and fiscal harm of ‘lockdown’ measures.

Elsewhere I have outlined in detail my views on the balancing act required of decision-makers, especially for less wealthy countries, in dealing with the pandemic. I argued that contrary to the conventional wisdom in 2020, South Africa’s response was deeply flawed and caused social and economic harm without adequate benefits in terms of long-term health outcomes. That remains my view, but it does not follow that every decision is flawed: in my assessment, the restrictions on alcohol sales/consumption, even if unnecessary or ineffective, are amongst the least of the government’s failures.

A letter rejected by the South African Medical Journal

Throughout South Africa’s Covid-19 pandemic response, I have been raising concerns about the basis for the government’s decisions – starting with an op-ed when the lockdown was announced. It has been particularly concerning how uncritical academics and journalists were at the outset. I am still in the process of writing a number of academic pieces on this, but unfortunately these will only come out later in this year or next year. One effort I made in the interim was to write a cautionary letter to the South African Medical Journal, which has been responsible for publishing some concerning editorials that contribute to the problematic stance of the South African academy. Today I received notification that my correspondence was rejected as “The editors have determined that this submission is not appropriate for this journal and will not be considered for publication.” Given that the pieces criticised were journal editorials, this is perhaps not very surprising. Decide for yourself.

Unmitigated praise of government’s Covid-19 response is premature and inconsistent with available evidence

A recent editorial[1] expresses concern with statements by the Minister and Department of Health[2,3] and Medical Research Council (MRC)[4] in response to remarks attributed to Dr Glenda Gray[5,6] relating to the government’s approach to lockdown regulations and public health consequences of the lockdown. I concur with the authors on the primacy of Constitutional principles of free speech and academic freedom for members of the Ministerial Advisory Committee (MAC) on Covid-19, and the institutional independence of the MRC (which has now found no transgression[7]). However, premature, unsubstantiated statements about government’s response to Covid-19 contribute to an environment of uncritical praise that preceded, and arguably contributed to, the controversy in question.
The theoretically optimal policy response to Covid-19 remains unknown under the usual standards of academic and scientific justification, given extensive uncertainty about characteristics of the virus itself along with the dynamics of contagion, morbidity and mortality in different populations and contexts. Furthermore, thorough assessment of the efficacy and optimality of government responses can only be made on evidence that will become available after the pandemic is over. Only preliminary assessments are possible at present and cannot be exempt from basic standards of justification and evidence. The authors’ statements[1], as with others elsewhere[8], do not meet such standards and thereby undermine impartial, evidence-based criticism. For example:

Notwithstanding the concerns raised above, the Minister of Health’s management of the country’s COVID-19 pandemic, to date, is laudable

And:

SA’s response to COVID-19 has been swift and science based, and merits praise

The apparent premise, that “other governments around the world have not grounded their response to the pandemic in science and evidence”[1], is too low a bar. The British government, for example, has rightly been criticised not for failing to use science and evidence, but for doing so selectively and secretively[9]. Yet the authors fail to critically examine the composition and conduct of the MAC, asking only for “involvement of experts from academia outside of the biomedical sciences, and statutory bodies”[1].
There is evidence that contradicts this stance. As noted by others[10], and reflected in international open Covid-19 databases[11], South Africa has not been forthcoming in publishing detailed data on testing, screening, contact tracing and patient characteristics – despite international calls for transparency[9,12]. Even more concerning, while the original lockdown decision was premised on modelling[13], only limited details of the strategy and basis were provided much later[14] while current projections and model details have only recently been made public[15].

Good intent along with science- and evidence-based decision-making are not sufficient to ensure the best policy decisions are taken. Transparency in evidence, modelling, decision-making, use of expertise and balancing of societal priorities is paramount. The South African government has performed badly on some of these dimensions. Rhetoric of “unity and solidarity” in that context potentially undermines the role of dissent, rather than deference, in contributing to the public good. Unsubstantiated and premature praise may contribute to a sub-optimal response to the pandemic.

1. Singh JA. Freedom of speech and public interest, not allegiance, should underpin science advisement to government. S Afr Med J. 2020 May 26;
2. Mkhize Z. Health Minister’s statement on Prof Glenda Gray’s public attack of government based on inaccurate information [Internet]. National Department of Health; [accessed 26 May 2020]. Available from: http://www.health.gov.za/index.php/2014-03-17-09-48-36/2014-03-17-09-49-50?download=4247:statement-by-minister-mkhize-prof-glenda-gray-public-attack-of-government-20-may-2020
3. Human L, Geffen N. Health department boss calls for investigation into Glenda Gray. GroundUp [Internet]. 22 May 2020 [accessed 26 May 2020]; Available from: https://www.groundup.org.za/article/health-department-boss-calls-investigation-glenda-gray/
4. Herman P. SAMRC board apologises for Prof Gray’s comments, bars staff from speaking to media. News24 [Internet]. 25 May 2020; Available from: https://www.news24.com/SouthAfrica/News/breaking-samrc-board-apologises-for-glenda-grays-comments-bars-staff-from-speaking-to-media-20200525
5. Karrim A, Evans S. Unscientific and nonsensical: Top scientist slams government’s lockdown strategy. News24 [Internet]. 16 May 2020 [accessed 16 May 2020]; Available from: https://www.news24.com/SouthAfrica/News/unscientific-and-nonsensical-top-scientific-adviser-slams-governments-lockdown-strategy-20200516
6. Karrim A. I didn’t criticise the lockdown, but the regulations ‒ Prof Glenda Gray after Mkhize slams criticism. News24. 21 May 2020
7. SAMRC. Media statement from the SAMRC Board [Internet]. 26 May 2020 [accessed 26 May 2020]. Available from: https://www.samrc.ac.za/media-release/media-statement-samrc-board
8. Academy of Science of South Africa (ASSAf). Public Statement on COVID-19. 18 May 2020.
9. Alwan NA, Bhopal R, Burgess RA, Colburn T, Cuevas LE, Smith GD, et al. Evidence informing the UK’s COVID-19 public health response must be transparent. Lancet. 2020 Mar;395(10229):1036–7.
10. Marivate V, Combrink HM. Use of Available Data To Inform The COVID-19 Outbreak in South Africa: A Case Study. Data Science Journal. 6 May 2020; 19(1):19.
11. Xu B, Kraemer MUG, Xu B, Gutierrez B, Mekaru S, Sewalk K, et al. Open access epidemiological data from the COVID-19 outbreak. The Lancet Infectious Diseases. 2020; 20(5):534.
12. Barton CM, Alberti M, Ames D, Atkinson J-A, Bales J, Burke E, et al. Call for transparency of COVID-19 models. Sills J, editor. Science. 2020; 368(6490):482.2-483.
13. Republic of South Africa. President Cyril Ramaphosa: Escalation of measures to combat Coronavirus COVID-19 pandemic [Internet]. 23 March 2020 [accessed 25 May 2020]. Available from: https://www.gov.za/speeches/president-cyril-ramaphosa-escalation-measures-combat-coronavirus-covid-19-pandemic-23-mar
14. Abdool Karim SS. SA’s Covid-19 epidemic: Trends & Next steps. Presentation for the Minister of Health; 13 April 2020. [accessed 16 May 2020].
15. Silal S, Pulliam J, Meyer-Rath G, Nichols B, Jamieson L, Moultrie H. Estimating cases for COVID-19 in South Africa Update: 19 May 2020. South African COVID-19 Modelling Consortium; 19 May 2020.

Economics: scientists and plumbers, or bullshit and mathiness?

On the 6th of January 2017 the Annual American Economic Association conference is scheduled to host a plenary address entitled The Economist as Plumber: Large Scale Experiments to Inform the Details of Policy Making. The speaker is the academic economist Esther Duflo, widely-acclaimed for popularising the use of randomised control trials (RCTs).

Given my PhD work in economics on external validity of RCTs and implications for policy, and parallel work in philosophy, I have a few thoughts on this subject. In a draft paper (first presented in 2015) entitled When is Economics Bullshit? I argue that practitioners promoting RCTs have systematically overstated the policy-relevance of results and thereby produced ‘bullshit’ (as defined in the famous essay by philosopher Harry Frankfurt).

A consistent problem in critiquing so-called ‘randomistas’ is that the goalposts have been constantly shifted. Early advocacy for RCTs within economics reflected a ‘missionary zeal’ (Bardhan). It has been suggested that experimental methods have led to a ‘credibility revolution‘: giving credibility to applied microeconomic work that apparently did not exist before. One recipient of the Bates Clarke medal argued that the introduction of RCTs indisputably rendered economics ‘a science’. In the policy domain I, along with other economists, have come across much grander and/or more extreme claims. But when challenged, proselytisers scale back the claims and deny ever overclaiming. So from missionary zeal, revolution and science we now have plumbing….

I look forward to reading Duflo’s speech/paper, but my own view of the methodology and philosophy of economics and RCTs suggests that plumbing is a very poor analogy.

In my own paper, motivated in part by claims that RCTs render economics ‘a science’, I tackle the question of scientific status head on. Using a revival of the so-called demarcation question (basically: how do we demarcate science from non-science or pseudoscience?) in philosophy, I argue that economics cannot (yet) be classified as a science, may never be classifiable as such and in the way it is used by some economists too-often verges on pseudoscience and/or bullshit.

The similarities between this very critical view and that of Romer’s recent critique of macroeconomics (which was made public later) are interesting. Romer focuses more on the use of mathematical modelling whereas my focus is on empirical methods. I will write a detailed comment on Romer’s piece later this year; I agree with some aspects but strongly disagree with others.

In its two presentations so far, my paper on bullshit has been relatively well-received by philosophers of science but not so well-received by philosophers of economics. There is good reason for this: the paper is even more an indictment of the current trend in philosophy of economics than it is of economics itself. The paper notes that in the absence of sufficient technical training and understanding of economics, philosophers in this area have increasingly taken the safer route of becoming apologists for the discipline. In effect, they compete to provide explanations of why economists are correct in their approach. (Exceptions to this, such as Nancy Cartwright – who has collaborated with Angus Deaton in providing important and influential critiques of RCTs – arguably prove the rule: Cartwright’s reputation was already established in philosophy of physics, causality and metaphysics).

The result, unfortunately, is that philosophy of economics currently has very little to add to economists’ critical understanding of their own discipline. Some critics, such as Skidelsky, argue that economists should read more philosophy, but while I am sympathetic to his overall stance I do not think economists would find much worth reading at present. Combining the abject failure of the ‘mainstream’ of philosophy of economics with the low quality of most economists’ reflections on methodological issues leaves us with few critical insights that could move the discipline beyond parochial or self-interested debates.