Measuring mathematical understanding

We start with open-ended tests that ask students to write and draw their understanding of a concept. An example is given in the figure. We assume that experts, such as mathematics teachers, know good understanding when they see it. The experts are shown pairs of student responses and have to decide which evidences the ‘better understanding’ of the concept of interest. Their pairwise decisions are statistically modelled to produce a score of mathematical understanding for each student.

How does this simple process based on subjective decisions produce a robust measure? The answer is what psychologists call the Law of Comparative Judgement: when people compare one thing to another they are highly consistent. We have used our approach to construct demonstrably robust measures of mathematical understanding across topics from primary school to university. We have also applied comparative judgement to measure things other than student understanding, such as the beauty of mathematical equations, the ‘explanatoriness’ of mathematics textbook pages, and standards in mathematics qualifications.

We continue to develop the comparative judgement method and apply it to new research areas.