Uninformative trials and their causes
What’s going wrong with large-scale educational randomised controlled trials?
The last decade has seen a major change in educational research funding in the UK. The advent of the Education Endowment Foundation (EEF) means that a majority of the money spent on education research is now used to conduct randomised controlled trials (RCTs) of educational interventions.
Prior to the publication of the first EEF trials, such studies were almost unheard of: in 2012 only 3% of articles published in the eight major mathematics education journals reported RCTs. Has this change in focus been a success? With the aim of finding out, Hugo Lortie-Forgues and Matthew Inglis recently conducted a review of all RCTs commissioned by the EEF and the NCEE, a US-based funder that also commissions educational RCTs.
In total we reanalysed 141 large-scale educational RCTs commissioned by the EEF and NCEE that involved a total of 1.2m children. There were two main findings. First, most educational RCTs find small effects: the average difference between the intervention and control groups was just 0.06 standard deviations. One way of understanding this figure is to state the probability that a randomly picked member of the intervention group has a higher score than a randomly picked member of the control group. With an effect size of 0.06 the answer is 51.7%, barely above the 50% chance level.
Second, and most importantly, we found that 40% of trials were uninformative, in the technical sense that the Bayes factor associated with their primary outcome measure fell between 1/3 and 3. In other words, between a third and half of all large-scale educational trials did not permit a conclusion to be drawn about whether the intervention they were testing was effective or ineffective. This is an alarmingly high number: at £500k per trial it suggests that the EEF and NCEE have spent around £28m conducting uninformative trials.
Our continuing work aims to understand why so many trials are uninformative and what can be done about it.
Related information
- Lortie-Forgues, H. & Inglis, M. (2019). Rigorous large-scale educational RCTs are often uninformative: Should we be concerned? Educational Researcher, 48, 158-166. https://doi.org/10.3102/0013189X19832850
- Inglis, M. (2020). Uninformative educational trials and their causes. Researching Education, 1(1). https://researchingeducation.com/uninformative-educational-trials-and-their-causes/
- Inglis, M. & Crook, C. (2021). Conversation: Uninformative randomised control trials in educational research. Researching Education, 2(3). https://researchingeducation.com/minglis_rct_conversation/