Annual Output Review Framework (2025-2028)

NOTE: This version of the Annual Output Review (AOR) Framework is a temporary measure covering the period 2025–2028. Its purpose is to provide support for the University in the lead-up to REF 2029. This framework is intended to operate only for the current REF cycle. Following REF 2029, AORs will revert to a business-as-usual process, with oversight and management by ADRIs.

1.    Responsibility for Annual Output Reviews 

1.1.    Deans of Schools hold ultimate responsibility of the AOR process. They are accountable for ensuring its completeness and timeliness within their School.

1.2.    Unit of Assessment (UoA) Leads are responsible for coordinating with respective Deans and ensuring delivery of the AOR process within their UoA.

1.2.1.    UoA Leads must ensure that all eligible staff are contacted to participate, including those based in Schools other than their own.

1.2.2.    Eligible staff will include those on Research, Teaching and Enterprise (RTE) contracts; independent research-only (RO) contracts (which may include those on Specialist and Supporting Academic Research (SSAR) and Technician contracts); or any other individual contributing to research with an eligible employment relationship with the University (as defined by the REF guidance).

1.2.3.    UoA Leads are also required to ensure that outputs from recent leavers, such as staff who have moved institutions or retired, are identified and reviewed. This could be as part of an AOR alongside current staff, or through a follow-up process with the same requirements detailed in this framework.

1.2.4.    In all cases, the Research and Innovation Office (RIO) and Planning Office can provide lists and guidance to help UoA Leads identify eligible individuals whose research aligns to a UoA.

1.3.    UoA Leads may delegate the responsibility of running the process to colleagues or coordinate it jointly with other UoAs (for example, at School level). ADRIs can support and help coordinate these arrangements. In doing so, however, UoA Leads must:

1.3.1.    Retain oversight of the process and be accountable for the quality of the reviews.

1.3.2.    Inform the Research and Innovation Office (RIO) team of the arrangements in place, including who is the main point of contact.

2.    Purpose 

2.1.    The purposes of AORs are as follows: 

2.1.1.    To provide colleagues with feedback on either draft or published outputs to support an ongoing commitment to improve the quality of Loughborough’s research. 

2.1.2.    To identify on an ongoing basis, outputs that might be considered for submission to the next REF.

2.1.3.    To help Deans identify, in conjunction with other qualitative and quantitative sources, individuals who may benefit from additional support to produce such high-quality outputs.

2.1.4.    To provide quantitative and qualitative output feedback that might be used, at the researchers’ own discretion, in other internal assessment processes such as promotion, probation and PDR. 

2.2.    For the avoidance of doubt, output reviews will not be used systematically by the institution other than for predicting REF output profiles at UoA level and planning the REF submission, i.e., not for performance management purposes.

 
3.    Timeframe 

3.1.    UoA Leads must agree the timing of AORs in advance with their respective Schools and the RIO team, choosing at least one of the following deadlines: 20th December, 28th February, 30th April or 21st June.

3.2.    Final output scores for all AORs, including moderated scores where required, must be provided to the RIO team by the selected deadline. 

3.3.    UoA Leads should determine an appropriate timeline for each AOR cycle, taking into account disciplinary practices, relevant training provision (see Section 4), reviewer availability, and other local factors. As a guide, it is recommended that a minimum of three weeks is allowed for output nomination. 

3.4.    Timelines for each AOR should be communicated in advance to those participating and, wherever possible, designed to allow flexibility for colleagues to plan and undertake their activities around their other responsibilities.

3.5.    UoA Leads may also organise ad hoc or additional review activity outside these timelines, as appropriate.

4.    Training requirements

4.1.    A diverse panel of suitable reviewers should be identified to undertake the reviews. Within the pool of suitable reviewers, considerations should be given to ensuring that, where possible, those selected are representative of the population they are assessing with regards to gender, ethnicity, discipline and/or career stage. Panel membership may rotate annually.

4.2.    To mitigate bias, all internal reviewers must participate in a training process that involves:

4.2.1.    Unconscious bias training

4.2.2.    A calibration exercise supported where possible by former REF panel members, to enable them to confidently assess the originality, significance and rigour dimensions of outputs at different REF ‘star’ levels.

4.3.    All reviewees should receive appropriate guidance and, where needed, training to help them understand the REF criteria and confidently identify the originality, significance, and rigour of their outputs, including how these relate to REF star levels.

4.4.    UoA Leads are encouraged to involve a broad range of colleagues in review and review-related activities (e.g. mock reviews) to provide development opportunities, help diversify the pool of suitable reviewers and support workload distribution.

4.5.    The RIO team will work with UoA Leads to ensure consistent training provision across the University.

4.6.    External reviewers may be engaged where no suitable internal reviewers are available. Advice and support can be provided by the RIO team.

5.    Use of LUPIN

5.1.    AORs must be completed using LUPIN (Loughborough University Publication INformation). This requires staff to nominate outputs by selecting from those eligible in their LUPIN records, and internal reviewers to provide individual scores and comments directly within LUPIN.

5.1.1.    Exceptions may apply in specific circumstances, such as when reviews are conducted in panels or where external reviewers are used who cannot be granted access. All exceptions must be approved by the RIO team, who can advise in these instances.

5.2.    To initiate the process, the RIO team will create a School or UoA level AOR exercise in the appropriate LUPIN module and add eligible users to the exercise (as outlined in Section 1.2.2) 

5.3.    The RIO team will provide practical guidance for users on nominating and submitting outputs in LUPIN, and for reviewers on how to input and submit reviews within the system.

5.4.    UoA Leads (or those acting on their behalf) may be granted oversight access within LUPIN to monitor progress and support coordination.

6.    Output nomination & selection

6.1.    All eligible staff are required to nominate a minimum of one output per year (pro rata for part-time staff) for review in an AOR, which represents their strongest produced during the current REF period, plus any additional outputs they believe may be rated 3* or 4* that have not previously been reviewed.

6.1.1.    Staff are encouraged to nominate a diverse range of output types for review.

6.1.2.    Exceptions on the ‘one output minimum’ can be made for new starters, colleagues in roles without a routine expectation of publishing outputs, colleagues who are on or have had periods of prolonged leave, or other extenuating circumstances. These decisions should be mutually agreed between UoA Leads and the individuals and communicated to the RIO team for record.

6.2.    Nominated outputs should not have previously been reviewed in an AOR, unless as a draft or an ongoing large-scale project (e.g., monographs/portfolios).

6.2.1.    The RIO team can support by providing lists of previously reviewed outputs for UoA Leads and/or staff.

6.3.    When nominating published outputs, these must have been published, or otherwise made first publicly available, while the staff member was employed at Loughborough University.

6.3.1.    Exceptions can be made for longform and/or long process outputs only. More details on this will be provided following updates to REF’s Contributions to Knowledge and Understanding (CKU) guidance.

6.4.    Where unpublished outputs are nominated, individuals should create a provisional record in LUPIN for them using the appropriate status (e.g. Unpublished, In Preparation, or Submitted). Mandatory fields such as Journal Name and Publisher may be entered as ‘TBC’ if not yet confirmed. 

6.4.1.    Those submitting drafts for review should state when they expect them to be published. 

6.4.2.    Draft outputs should not be deposited in the University repository. Instead, UoA Leads should ensure that these are securely stored whilst under review and deleted immediately after the scores and feedback have been generated. 

6.4.3.    Where staff are unable to supply draft outputs due to, for example, reasons of national security, contractual obligations or external collaborator concerns, staff should explain their reasons and provide a date by which they expect the output might be available.

6.4.4.     Further guidance on how scores for outputs reviewed in draft form should be managed once those outputs are published or otherwise enter the public domain is provided in Section 10.

6.5.    During the nomination process via LUPIN, staff may be asked to provide a self-assessment of their outputs using the established 13-point scale (see 7.4), and/or submit a concise narrative that explains and contextualizes the significance, originality, and rigour of their outputs.

6.5.1.    Colleagues may find it helpful to refer to the University’s guidance on Evidence-informed Output Narratives to support writing such a statement. More REF-focused guidance may be provided later.

6.6.    Staff should contact the relevant UoA Lead if they have interdisciplinary outputs that may be suitable for submission to more than one UoA, or if they wish to nominate an output that may be better aligned with a different UoA, and that has previously not been reviewed. 

6.7.    Once staff have selected outputs in LUPIN, RIO team will cross-check to ensure they have not been previously reviewed, and flag REF 2029 eligibility concerns, prior to providing UoA Leads with a list of nominated outputs for reviewer allocation. 

6.8.    UoA Leads may suggest nominated outputs for consideration within another UoA, where appropriate. This may occur in addition to, or instead of, review within their own UoA, meaning some outputs may be reviewed under more than one UoA as part of the internal AOR process to inform potential REF submission. UoA Leads should coordinate directly to ensure that such reviews are arranged and managed appropriately and inform the RIO team.

6.8.1.    Whether an output could be treated as ‘interdisciplinary’ should be determined by consensus among the reviewers from the relevant UoAs and confirmed by the UoA Leads/REF Planning Group.

6.9.    UoA Leads should identify outputs that constitute substantial bodies of work, such as monographs or portfolios, and determine whether these are suitable for double‑weighting, consulting with authors and reviewers as appropriate. All double‑weighting decisions must be agreed by the REF Planning Group.

6.10.    The RIO team is available to provide bibliometric data and analysis to support UoA Leads in identifying additional outputs.

7.    Review & scoring requirements 

7.1.    Outputs should be reviewed by at least two independent reviewers, at least one of which should be from the disciplinary area.

7.1.1.    For the avoidance of doubt, a researcher’s self-assessment of an output cannot count as an independent review.

7.2.    UoA Leads (or those acting on their behalf) should provide the RIO team with a list with reviewer allocations for each output. The RIO team will configure the reviewer assignments in LUPIN.

7.3.    Reviewers should provide both qualitative feedback about the output for the reviewee, as well as a quantitative assessment of the quality of the output, using the most recent REF criteria (currently REF 2021, but switch to REF 2029 when available) of originality, significance and rigour.

7.3.1.    If a reviewer feels that an output falls significantly outside their expertise, aligns more closely with another UoA, or may be ineligible for assessment, they should raise this with the UoA Lead directly instead of scoring it.

7.4. The quantitative assessment should use a 13-point scale as follows: 

7.5.    The RIO team will collate the review scores and feedback from LUPIN, and provide UoA Leads, ADRIs and Deans and whoever they may designate with a report.

7.6.    Where reviewer scores differ by three points or more, the output must be moderated as follows: 

7.6.1.    Where the output has been reviewed by the minimum of two reviewers, it will be sent to a third reviewer, who may be the UoA Lead or School ADRI if appropriate. 

7.6.2.    Where the output has already been reviewed by three or more reviewers, moderation may instead be achieved through discussion among the existing reviewers.

7.6.3.    In both cases, all reviewers should then agree a final score through discussion with reference to the REF criteria.

7.6.4.    If a decision cannot be made via this approach, the output may be submitted to RIO to be considered as part of the External Validation Process (see Section 8). 

7.7.    Where reviewer scores differ by less than three points, the output will be handled as follows: 

7.7.1.    The Unit may choose to moderate the score through discussion among reviewers, with reference to the REF criteria.  

7.7.2.    Alternatively, where the Unit does not undertake moderation through discussion, the final score will be the average of the reviewers’ scores (rounded down).

7.8.    The reviews and scores may be used by the researcher to whom they pertain in any internal process that uses output assessments, for example as part of their Evidence-informed Output Narrative proposed for promotion and probation.

7.9.    Reviews and scores for any output may be shared with all authors and co‑authors of that output, irrespective of who nominated it, and may also be shared with the current and former institutions of those authors and co‑authors, provided that they agree to use the information only in ways that are consistent with the purposes of the Annual Output Review exercise as set out in Section 2.

8.    External Validation Process

8.1.    RIO will coordinate an ongoing external, independent, validation process of outputs internally assessed at different star levels and across star boundaries.

8.2.    UoA Leads and ADRIs will be invited to nominate external reviewers for this purpose.

8.3.    The outcome of the validation process will be used by the RIO team to propose adjustments to the predicted Unit level quality profile, where considered necessary. The proposed approach to adjustment will be approved by the REF Leadership Group and communicated to REF Planning Group and ADRIs to support future reviewer training.

9.    Reporting Requirements

9.1.    As most review scores and comments should collected centrally via LUPIN, the only additional requirements for reporting to the RIO team are as follows:

9.1.1.    Where external reviews have been used (independently of any centrally administered External Validation Process), UoA Leads must provide the reviewers’ scores (0–12), comments, and the LUPIN ID for each output via an Excel spreadsheet.

9.1.2.    Where moderation has taken place, UoA Leads must confirm the outputs’ final scores, provide a brief explanation of how these were determined, and include the LUPIN ID for each output.

9.2.    UoA Leads are responsible for ensuring that scores and qualitative feedback is shared with reviewees for development purposes. Shared feedback should be constructive, appropriate, and aligned with the aims of the review process.

9.3.    Upon receipt of the final scores, the RIO team upload the final score data to the central Loughborough AOR Exercise to support REF 2029 planning and submission.

9.4.    The RIO team can provide other reports, for example, detailing diversity of output types and the REF Open Access compliance status of reviewed outputs.

10.    Management of Scores for Outputs Previously Reviewed as Drafts

10.1.    Decisions on how to handle scores for outputs first reviewed in draft form, and later published or otherwise made public, should be made on a case‑by‑case basis by the UoA Lead, in consultation with the author(s) and other relevant colleagues. Such decisions may be made, and any additional review activity undertaken, outside the timeline originally agreed for the Unit’s AOR where appropriate.

10.2.    Depending on the extent to which the published version differs materially from the draft originally reviewed, these decisions may be to:

10.2.1.    retain the original score;

10.2.2.    arrange new reviews as per Section 7, undertaken by the original and/or new reviewers.

10.3.    Decisions on approach and final scores will be shared with RIO team to be recorded in LUPIN and used in updated reporting.

11.    Appeals Process

11.1.    The purpose of the appeals process is to support fair outcomes for all authors and the institution’s confidence in the reliability, consistency and integrity of assessment processes and review scores.

11.2.    As the AOR seeks to provide both developmental feedback for authors and an indication of how an output may perform against the specific assessment criteria used in the REF, a lower score does not necessarily imply that the research lacks quality or significance. Some high-quality outputs may not map neatly onto REF’s assessment framework, and authors should not automatically interpret lower scores as criticism of the value of the work.

11.3.    Authors are also reminded while they may choose to use the qualitative comments or the review scores to evidence the quality of their work in other internal assessments such as promotion applications, this is entirely at their discretion; they are under no obligation to use either and may present alternative evidence or outputs in support of their application.

11.4.    Authors may appeal a review from the AOR where they believe there has been:

11.4.1.    a significant factual error in the review that would affect the assessment;
11.4.2.    a procedural irregularity in how the output was reviewed;
11.4.3.    evidence of bias or prejudice in the process.

11.5.    Appeals must be submitted via email to the School’s ADRI, within one month of receiving the reviews; appeals received after this point may also be considered but only at the School’s discretion. Appeals should include the details of the output reviewed, the reviews and scores received, and a short statement outlining the grounds for appeal. Only one appeal per output will be considered.

11.5.1.    If the ADRI was involved in the original review, the appeal should instead be submitted to the Associate Pro-Vice Chancellor Research & Innovation – Focus and Excellence.

11.6.    The ADRI (or equivalent designated alternative) will review the case within one month and may consult with the authors, the UoA Lead(s) and/or reviewers as appropriate.

11.6.1.    If the original reviews are upheld, the ADRI (or equivalent) will explain the reasons to the author.

11.6.2.    If the appeal is accepted, the ADRI (or equivalent) will arrange an independent review of the output using new reviewers, which may include external reviewers where appropriate. RIO can support this.

11.7.    Details of all appeals and their outcomes should be shared with the RIO team for recording.

V13.2: APPROVED following R&I Committee in June 2026 by Chair’s Action
Dr Stuart King
Research & Innovation Office