Millions of pounds are being spent on running educational trials in schools that often fail to produce useful information, a new study has found.
In 2011, at a cost of £125 million, the Government set up the Education Endowment Foundation (EEF) charity, which to date has commissioned more than 190 large-scale educational randomised controlled trials (RCTs).
The organisation has run projects such as finding out whether providing free school breakfasts raises grades in Maths and English, and whether playing chess improves numeracy skills.
Each trial costs around £500,000, and since the charity’s inception more than 700,000 schoolchildren, across the UK, have been involved.
Now, a new study carried out by researchers at Loughborough and York universities has found that 40 per cent of the large-scale RCTs failed to produce either evidence the educational intervention being tested is effective or evidence that it is ineffective.
It identified three possible reasons why the current trials could be uninformative:
The interventions being tested may not be suitable for trial in the first place
Interventions may not be being correctly implemented during trials – for example, due to inadequate training of teachers in the methods being tested
The trials themselves may be poorly designed
The academics also looked at data from EEF’s transatlantic counterpart, the National Center for Education Evaluation and Regional Assistance (NCEE), based in Washington DC, and found similar results.
Professor Matthew Inglis, of Loughborough’s Mathematics Education Centre, said: “The project aimed to assess the contribution that randomised controlled trials (RCTs) are making to education research.
“In the last 10 years, there has been a large focus on RCTs in education.
“Since 2011, the Education Endowment Foundation (EEF) has since commissioned more than 190 large-scale educational RCTs. They were fairly rare in the UK before this point.
“Our goal was to evaluate this new emphasis on trials in education research.
“We found that a surprising number of RCTs, some 40%, are uninformative in the sense that they neither allow us to conclude that the educational intervention they were assessing is effective, nor to conclude that the intervention they were assessing is ineffective.
“Such a trial has not really added any information.
“This is a worry, as educational RCTs have the aim of producing information that will help teachers to improve their students’ education. If our current research efforts are not achieving this, then reform is needed.”
The authors suggest a series of changes that could make the trials more informative, including higher-standards when considering which new initiatives are trialled.
A paper, Rigorous Large-Scale Educational RCTs are Often Uninformative: Should We Be Concerned?, has been published in the journal, Educational Researcher.
The study was led by the University of York.
Lead author Dr Hugues Lortie-Forgues, from the Department of Education, at York, said: “Just like in medicine, trials of educational interventions are an important way to allow policymakers and teachers to make informed decisions about how to improve education.
“However, many of these trials are currently not fulfilling their main aim of demonstrating which interventions are effective and which are not.”