BU Research Blog

Welcome to the REF section of the BU Research Blog!

The Research Excellence Framework (REF) is the exercise for assessing the volume and quality of research in UK HEIs. As with its predecessor (the RAE), the results of the REF are used by Research England (previously HEFCE) to determine the annual quality-related research (QR) grant distributed from Research England to HEIs in England.

For REF 2014, institutions made their submissions in autumn 2013 and the results were published in December 2014. Following a seven-year cycle, the latest exercise, REF 2021 was submitted in March 2021 and the results were published in May 2022. It has now been confirmed that the next REF will be in 2029.

The REF assesses research excellence through a process of expert review, informed by indicators where appropriate. It is based on HEIs submitting evidence of their research activity and outcomes to discipline-based units of assessment (UOAs) to be assessed by expert panels.

We are not quite sure yet what any future research assessment exercise may look like . In June 2023, the REF team published initial decisions confirming there will be three elements:

People, Culture and Environment
Contribution to Knowledge and Understanding
Engagement and Impact

Many other aspects are still unknown with a number of consultations with the sector on going, up to date information can be found here www.ref.ac.uk

BU’s REF 2029 preparation

If you would like to learn more about BU’s preparations for REF2029, principles for assessment and mock timelines can be found on the RKE SharePoint site . You can also speak to your unit of assessment (UoA) teams.

BU’s REF 2021 submission

We submitted to 13 Units of Assessment for REF 2021, compared with eight in REF 2014. The number of staff submitted to REF 2021 more than tripled compared to REF 2014, reflecting our inclusive approach.

94% of BU research was found to be internationally-recognised or above, with 19% found to be world-leading in quality.

95.7% of our research was found to be delivering considerable impact or above, with 31.5% achieving an outstanding impact score.

Find out more about our submission on the Bournemouth University website. You can also view the full REF 2021 results at www.ref.ac.uk

About REF 2021

Research England, on behalf of the four UK funding bodies, consulted with the sector in early 2017 as to the shape of the next REF. This consultation was informed by Lord Stern’s independent review of the REF, published in July 2016. The consultation closed on 17 March 2017 and over 300 responses were received. The finding bodies published the initial decisions on REF 2021 on 1 September 2017, alongside another consultation with the sector about submitting staff and outputs (HEFCE Circular letter 33/2017). The final decisions on REF 2021 were published on 21 November 2017.

Assessment elements: These were the same as REF 2014 but with slightly different weightings:

Outputs (60%)
Environment (15%)
Impact (25%)

The role of metrics in research assessment: Research England conducted an independent review to ascertain the extent to which metrics could be used in the assessment and management of research. This was reported in The Metric Tide report that concluded that metrics are not yet robust enough to replace peer review. The consultation proposes that metrics are used to inform peer review in some UoAs (similar to REF 2014). The Forum for Responsible Metrics, established in 2016, produced advice in 2017 for HEFCE (and the other HE funding bodies) on the use of quantitative indicators in the assessment of outputs in REF 2021 (with further discussion planned later for assessment of impact and environments).

Submitting staff:

Before REF2021, previous REF/RAE exercises asked institutions to select staff for submission. The Stern Review in 2016 recognised how divisive this practice was and instead recommended that all research-active staff be returned to the next REF. Research England implemented this recommendation by expecting all staff with a ‘significant responsibility for research’ to be submitted, provided they were ‘independent researchers’. What did these terms mean in practice? The REF definition is:

“those for whom explicit time and resources are made available to engage actively in independent research, and that is an expectation of their job role. Research is defined as a process of investigation leading to new insights, effectively shared. Staff engaged exclusively in scholarship would not be considered to have a significant responsibility for research.”

Working with the REF 2021 main panels, Research England provided institutions with further guidance on identifying staff with significant responsibility. In the REF Guidance on Submissions and Panel Criteria and Working Methods published in January 2019.

Recognising that there were staff who had more significant responsibility for other activities, Research England implemented an approach whereby institutions, working with their staff and with guidelines, identified who was in scope for submission among staff meeting core eligibility criteria. Research England therefore, defined the core eligibility criteria as:

‘Category A eligible’:

academic staff with a contract of employment of ≥0.2 FTE
on the payroll of the submitting institution on the census date (31 July 2020)
whose primary employment function is to undertake either ‘research only’ (independent researchers only) or ‘teaching and research’
have a substantive connection with the submitting institution (i.e. BU)
for staff on ‘research only’ contracts, the eligible pool should only include those who are independent researchers, not research assistants

‘Category A submitted’ described the staff from the ‘Category A eligible’ pool who had been identified as having significant responsibility for research on the census date.

Where the ‘Category A eligible’ staff definition accurately identified all staff in the submitting unit with significant responsibility for research, the unit should submit 100% of staff. Where it did not accurately identify all staff in the submitting unit who had significant responsibility for research, institutions needed to implement processes to determine this and document this in its code of practice.

Submitting outputs:

The average number of outputs required per submitted FTE was 2.5
A minimum of one output was required for each staff member employed on the census date.
A maximum of five outputs may be attributed to individual staff members (including those who have left)
Data on the distribution of outputs across staff in the unit, including staff who have left, were provided to the sub-panels for consideration in relation to the assessment of the environment.

Output portability: A transitional approach was adopted whereby outputs may be submitted by both the institution employing the staff member on the census date and the originating institution where the staff member was previously employed when the output was demonstrably generated. ‘Demonstrably generated’ will be determined by the date when the output was first made publicly available. This applied to the whole REF 2021 period.

Open access: The REF Open Access policy outlined in the Guidance on Submission was implemented. This required outputs within the scope of the policy (journal articles and some conference proceedings) to be deposited as soon after the point of acceptance as possible, and no later than three months after this date from 1 April 2018. Due to concerns around deposit on acceptance, a deposit exemption was introduced from 1 April 2018 and remained in place for the rest of the REF 2021 publication period. This allowed outputs unable to meet this deposit timescale to remain compliant if they were deposited up to three months after the date of publication.

Research England also confirmed that extra credit may be awarded via the environment section to institutions that could demonstrate:

that they had taken steps towards making other outputs types available open access on their repositories (such as book chapters)
that outputs were presented in a form that allowed re-use of the work, including via text-mining

Citation data: This was provided to some sub-panels in REF 2014 to inform the peer review process. Anecdotal evidence suggests reviewers are using these to judge borderline papers or to compare papers, with a high citation count tipping the score into the higher classification. It was proposed in the consultation that citations would be used in REF 2021 in a similar way to REF 2014 (i.e. used to inform peer review in some UOAs).

Environment:

The assessment of the environment part of REF 2014 was measured by a narrative plus metrics (PGR completions and research income awarded). Assessment was based on the vitality and sustainability of the research environment (e.g. culture, staff development opportunities, career breaks, targeted recruitment, etc.). This was the same in REF 2021 but with more reliance on metrics alongside the narrative.

Impact:

The impact element of REF 2021 was measured via impact case studies (the impact strategy will be incorporated into the environment section). For the purposes of the REF 2014, impact was defined as an effect on, change or benefit to the economy, society, culture, public policy or services, health, the environment or quality of life, beyond academia. Impact included, but was not limited to, an effect on, change or benefit to:

the activity, attitude, awareness, behaviour, capacity, opportunity, performance, policy, practice, process or understanding
of an audience, beneficiary, community, constituency, organisation or individuals
in any geographic location whether locally, regionally, nationally or internationally.

Impact included the reduction or prevention of harm, risk, cost or other negative effects.

For the purposes of the impact element of the REF 2014:

Impacts on research or the advancement of academic knowledge within the higher education sector (whether in the UK or internationally) were excluded.
Impacts on students, teaching or other activities within the submitting HEI were excluded.
Other impacts within the higher education sector, including on teaching or students, were included where they extend significantly beyond the submitting HEI.

For REF 2021 the funding bodies provided greater clarity on the definition and scope of impact.

Impact evaluation: Research England commissioned RAND Europe to undertake a comprehensive evaluation of the impact assessment element of REF 2014. Their remit was to:

Look at the benefit and burden associated with impact submission preparation
Recommend possible refinements of the process for future exercises
Identify good practice across the sector

Further information and the final report can be found here: http://www.rand.org/randeurope/research/projects/hefce-ref2014-impact.html

Impact evidence: The consultation proposed that the impact assessment period (i.e. the period in which the claimed impact(s) occurred) ran from 1 August 2013 to 31 July 2020. It’s important that you keep the impact of your research in mind and get into the habit of collecting evidence and other corroborating information as impacts are realised, which will be of great help in developing case studies for the next REF.

Number of impact case studies required: Submissions included a total of one case study, plus one further case study per up to 15 FTE staff submitted, for the first 105 FTE staff returned (with a reduced requirement above this of one additional case study per 50 FTE staff). Submissions therefore included a minimum of two case studies.

How was preparation for REF 2021 supported at BU?

BU governance: Prof Tim McIntyre-Bhatty (Deputy Vice-Chancellor) established a REF Steering Group to strategically oversee the university’s REF 2021 submission. Reporting into the steering group is the REF Committee (all UOA Leaders plus colleagues from Professional Services as appropriate).