Expensive educational trials are often uninformative and fail to produce useful information

Millions of pounds are being spent on running educational trials in schools that often fail to produce useful information, a new study has found.

In 2011, at a cost of £125 million, the Government set up the Education Endowment Foundation (EEF) charity, which to date has commissioned more than 190 large-scale educational randomised controlled trials (RCTs).

The organisation has run projects such as finding out whether providing free school breakfasts raises grades in Maths and English, and whether playing chess improves numeracy skills.

Each trial costs around £500,000, and since the charity’s inception more than 700,000 schoolchildren, across the UK, have been involved.

Now, a new study carried out by researchers at Loughborough and York universities has found that 40 per cent of the large-scale RCTs failed to produce either evidence the educational intervention being tested is effective or evidence that it is ineffective.

It identified three possible reasons why the current trials could be uninformative:

The interventions being tested may not be suitable for trial in the first place

Interventions may not be being correctly implemented during trials – for example, due to inadequate training of teachers in the methods being tested

The trials themselves may be poorly designed

The academics also looked at data from EEF’s transatlantic counterpart, the National Center for Education Evaluation and Regional Assistance (NCEE), based in Washington DC, and found similar results.

Professor Matthew Inglis, of Loughborough’s Mathematics Education Centre, said: “The project aimed to assess the contribution that randomised controlled trials (RCTs) are making to education research.

“In the last 10 years, there has been a large focus on RCTs in education.

“Since 2011, the Education Endowment Foundation (EEF) has since commissioned more than 190 large-scale educational RCTs. They were fairly rare in the UK before this point.

“Our goal was to evaluate this new emphasis on trials in education research.

“We found that a surprising number of RCTs, some 40%, are uninformative in the sense that they neither allow us to conclude that the educational intervention they were assessing is effective, nor to conclude that the intervention they were assessing is ineffective.

“Such a trial has not really added any information.

“This is a worry, as educational RCTs have the aim of producing information that will help teachers to improve their students’ education. If our current research efforts are not achieving this, then reform is needed.”

The authors suggest a series of changes that could make the trials more informative, including higher-standards when considering which new initiatives are trialled.

A paper, Rigorous Large-Scale Educational RCTs are Often Uninformative: Should We Be Concerned?, has been published in the journal, Educational Researcher.

The study was led by the University of York.

Lead author Dr Hugues Lortie-Forgues, from the Department of Education, at York, said: “Just like in medicine, trials of educational interventions are an important way to allow policymakers and teachers to make informed decisions about how to improve education.

“However, many of these trials are currently not fulfilling their main aim of demonstrating which interventions are effective and which are not.”


Notes for editors

Press release reference number: 19/26

Loughborough University is equipped with a live in-house broadcast unit via the Globelynx network. To arrange an interview with one of our experts please contact the press office on 01509 223491. Bookings can be made online via www.globelynx.com

Loughborough is one of the country’s leading universities, with an international reputation for research that matters, excellence in teaching, strong links with industry, and unrivalled achievement in sport and its underpinning academic disciplines.

It has been awarded five stars in the independent QS Stars university rating scheme, named the best university in the world for sports-related subjects in the 2018 QS World University Rankings, top in the country for its student experience in the 2018 THE Student Experience Survey and named University of the Year by The Times and Sunday Times University Guide 2019 and the Whatuni Student Choice Awards 2018.

Loughborough is in the top 10 of every national league table, being ranked 4th in the Guardian University League Table 2019, 5th in the Times and Sunday Times Good University Guide 2019 and 7th in The UK Complete University Guide 2019. It was also named Sports University of the Year by The Times and Sunday Times Good University Guide 2017.

Loughborough is consistently ranked in the top twenty of UK universities in the Times Higher Education’s ‘table of tables’ and is in the top 10 in England for research intensity. In recognition of its contribution to the sector, Loughborough has been awarded seven Queen's Anniversary Prizes.

The Loughborough University London campus is based on the Queen Elizabeth Olympic Park and offers postgraduate and executive-level education, as well as research and enterprise opportunities.

It is home to influential thought leaders, pioneering researchers and creative innovators who provide students with the highest quality of teaching and the very latest in modern thinking.