I was therefore somewhat frustrated when I attended a presentation at the Economic Society of South Africa conference in 2013 to find some rather strong policy claims being made on the basis of what is very weak evidence (even by the standards of practitioners favouring RCTs). I raised my concerns with the relevant author, but I see that the recently-published working paper<\/a> contains the same problems.<\/p>\n It therefore seems appropriate to summarise my concerns with this work: partly so that interested parties can understand its flaws, but mainly to provide an illustration of how the new fad for RCT-based policy is often oversold.[1]<\/sup><\/a> That\u2019s important, because despite seemingly ample evidence I often get economists saying: \u201cOh but no-one really uses RCT results in that way\u201d.<\/p>\n <\/p>\n The authors randomly allocated exam study guides to Grade 12 pupils in a sample of schools in Mpumalanga province (South Africa), for four different subjects: accounting, economics, geography and life sciences. Of the sample, 79 schools received the guides and 239 did not. These guides were already available online, but the explicit assumption is that the schools in question would not have known that or been able to access such online resources.<\/p>\n The study focuses on students\u2019 final National Senior Certificate (\u2018matric\u2019) exam scores as the outcome of interest. The authors find that in two subjects there was no statistically significant difference in outcomes, but that for two subjects students did score two percentage points higher on average if they received study guides. This conclusion is reached by regressing average school NSC scores on a binary variable representing whether a school was selected for study guide distribution or not.<\/p>\n Based on this finding and a rough \u2018cost-benefit analysis\u2019 the authors conclude that: \u201cdistributing the geography and life science [study guides] at scale… is amongst the most cost-effective of educational interventions internationally that have been tested using randomised experiments\u201d. [For readers unfamiliar with this kind of research: that\u2019s a very strong claim!]<\/p>\n In a number of basic respects the paper can be considered a local imitation of the paper by Glewwe, Kremer and Moulin (AER, 2009) on the impact of randomly assigning textbooks to schools in Kenya.<\/p>\n The first major problem with the paper arises from the fact that its hypothesis is very crude: do study guides improve NSC scores? The hypothesis itself has little contextual foundation and no subject-specific rationale either. Such \u2018atheoretic\u2019 approaches have been on the receiving end of quite scathing critiques (such as Cartwright, Deaton, Heckman and Keane among others) and I will not repeat those here. Suffice to say that even if we agree that it is not entirely pointless to conduct such an experiment, the lack of a richer conceptual (theoretical) basis for the experiment becomes a problem when authors want to depart from this simple hypothesis.<\/p>\n In this case the appropriate answer to the basic hypothesis would appear to be: the study does not provide statistically significant evidence that study guides improve performance. I say \u2018appear\u2019 because the paper does not report this result. However, that is not the conclusion of the paper. The authors argue (having already seen the evidence) that actually there are four hypotheses we should test:<\/p>\n They then treat the findings for each subject as separate from the other subjects. This allows them to conclude that there is a statistically significant, positive and sizeable effect of sending study guides to schools for two subjects: geography and life sciences.<\/p>\n While there is a genuine methodological issue one could debate here (see one recent discussion here<\/a>), when pursued in the manner of this paper I would argue that this constitutes inappropriate data mining: authors who don\u2019t find significant main results start disaggregating their samples or treatments in order to find a significant result.[2]<\/sup><\/a><\/p>\n To be maximally fair to Taylor and Watson, much higher profile authors have engaged in similar practices in the past (albeit hidden behind more technical sophistication) and seemingly gotten away with it. I\u2019m pleased to say, though, that the proverbial worm has been turning in the last decade and this kind of approach to multiple hypothesis testing, and also treatment heterogeneity analysis, is increasingly recognised to be dubious (even by JPaL<\/a>).<\/p>\n The apparent good news for the authors in this case is that even with a simple Bonferroni adjustment<\/a> (which is often viewed as conservative), the two findings remained significant at the 5% level (though not the 1% level reported).<\/p>\n Having said that, I am sceptical about the main results reported because they are based on school-averaged scores \u201cweighted by the number of pupils that wrote that specific subject\u201d. As far as I know this is an unusual approach to school-level treatment assignment. I\u2019d be interested to see the results from using a student-level specification, as done by Glewwe et al, with one treatment and appropriate standard errors.<\/p>\n The second major problem I raised with the author was external validity. The paper now throws in this term a few times, but with little substantive engagement with the problem.[3]<\/sup><\/a><\/p>\n In my PhD research<\/a>, also published here<\/a> and here<\/a>, I argue that there is currently no rigorous basis for extrapolating results from RCTs. That being the case, basing a cost-benefit analysis and policy recommendations on assuming that all schools in the country will experience similar results to 79 schools in Mpumalanga is unjustifiable.<\/p>\n Moreover, while the authors note that they use various criteria to determine which schools were included in their final sampling frame (318 schools) they never report the total number of potentially eligible schools in the province. According to the Department\u2019s Schools Masterlist<\/a> for 2013, there were 738 secondary (or combined) schools in Mpumalanga. I.e. the study apparently excluded half the province\u2019s schools and its discussion of extrapolation entirely ignores possible implications of that.<\/p>\n The authors randomly assigned study guides to schools but have no actual information on usage of these, either at the school or pupil level. Later in the paper (but not in the results section) the authors note that this means they estimate an \u2018intent-to-treat\u2019 (ITT) effect, not the actual effect of study guide usage.<\/p>\n The correct technical approach, if the relevant data was available, would be to treat assignment as an instrumental variable. In that case, however, the aim is to estimate a local average treatment effect (LATE), which is a function of the presence of \u2018compliers\u2019 (students who use the guides if their school gets them, or student who don\u2019t use them if their school doesn\u2019t get them) and \u2018non-compliers\u2019 (students who use or don\u2019t use the guides regardless of whether their school gets them).<\/p>\n One problem with LATEs is that they are likely to be even less generalisable than Sample Average Treatment Effects, because the proportion of compliers and non-compliers may vary across populations. The authors appear largely unaware of this issue. Instead, they argue that the ITT is likely to be smaller than the treatment effect on students who use the guides and that therefore the likely benefits of an expanded programme are larger. [4]<\/sup><\/a> Unfortunately, there is no actual evidence from the study to support that claim.<\/p>\n Furthermore, the authors state that \u201cone can confidently rule out\u201d the possibility that the observed treatment effect is due to \u2018experimenter effects\u2019 (Hawthorne and John Henry effects). However, I think they fail to understand that any effect unrelated to study guide usage<\/em> would fall in that category. For example: students might work harder simply because they feel privileged relative to a neighbouring school that did not receive the guides. The authors therefore can not<\/em>, in my view, rule-out John Henry or Hawthorne effects and this is also relevant to extrapolation.<\/p>\n Given all the issues above, I would be very concerned if this kind of approach to evaluation and policy formulation took hold in the South African education sector. While I appreciate the authors\u2019 good intentions — to contribute evidence to improve basic education in South Africa — I don\u2019t think the paper actually adds much to our knowledge base. RCTs may have some uses, but they continue to be widely oversold.<\/p>\n <\/p>\n [1]<\/a> There are various other objections from structural econometricians to the standard approach to RCTs, but it is not necessary to discuss those here.<\/sup><\/p>\n [2]<\/a> I say \u2018inappropriate data mining\u2019 because data mining is increasingly recognised by econometricians as not being a problem in itself, but rather as a problem if not done according to well-founded principles and its results assessed accordingly. This issue has also been one motivation for pre-analysis plans; more on those in later posts.<\/sup><\/p>\n [3]<\/a> The authors do not reference any sources for their discussions of external validity. Given that the paper emphasises John Henry and Hawthorne effects, I suspect their preferred source of information on external validity is probably the \u2018toolkit\u2019 provided by Duflo, Glennerster and Kremer (2006)<\/a>. Having said that: I have observed a disturbing trend of some academic peers not citing critical methodological work that is inconvenient \u2013 more on that in future posts.<\/sup><\/p>\n [4]<\/a> Structural econometricians would argue that one should also think about Marginal Treatment Effects \u2013 see the work by Heckman and Vytlacil \u2013 but that\u2019s also a subject for later posts.<\/sup><\/p>\n <\/p>\nShort summary of the paper<\/h6>\n
\u00a0Problem #1: Fishing for statistical significance?<\/h6>\n
\n
Problem #2: Inappropriate extrapolation (external validity problems)<\/h6>\n
Problem #3: What is being estimated?<\/h6>\n
![\"Creative](\"https:\/\/i.creativecommons.org\/l\/by-nc-nd\/4.0\/88x31.png\")
<\/a>This work is written in my personal capacity and licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License<\/a>.<\/p>\n","protected":false},"excerpt":{"rendered":"