Welcome to the REF section of the BU Research Blog!
The Research Excellence Framework (REF) is the exercise for assessing the volume and quality of research in UK HEIs. As with its predecessor (the RAE), the results of the REF are used by HEFCE to determine the annual quality-related research (QR) grant distributed from HEFCE to HEIs in England.
The last exercise was REF 2014. Institutions made their submissions in autumn 2013 and the results were published in December 2014. The next REF exercise has been confirmed as REF 2021 meaning that the submission will be in autumn 2020; this will be a seven year assessment period.
The REF assesses research excellence through a process of expert review, informed by indicators where appropriate. It is based on HEIs submitting evidence of their research activity and outcomes to discipline-based units of assessment (UOAs) to be assessed by expert panels.
What do we know about REF 2021?
HEFCE, on behalf of the four UK funding bodies, consulted with the sector in early 2017 as to the shape of the next REF. This consultation was informed by Lord Stern’s independent review of the REF, published in July 2016. The consultation closed on 17 March 2017 and over 300 responses were received.
Although no firm decisions have yet been announced, HEFCE held a webinar in July 2017 during which they shared some possible decisions with the sector (the webinar and the slides are available here on the HEFCE website). The key suggestions were:
- 100% of academics with a “significant responsibility” to undertake research are likely to be included. It is unclear at this stage what “significant responsibility” means in practice, although it is anticipated this will be based on there being an expectation for an academic member of staff to undertake research.
- Staff without a significant responsibility for research may be exempt from inclusion. Auditable documentation would be required which explicitly evidences there is not an expectation of them to undertake research (examples given were workload models or career frameworks linked to the individual).
- Everyone submitted is likely to need a minimum of 1 output. The average and maximum outputs per FTE are to be determined – in the consultation it was proposed these were an average of 2 outputs per submitted FTE and a maximum of six outputs per submitted individual.
- Portability of outputs (i.e. which HEI can submit the outputs authored by a member of academic staff who moves from one institution to another during the REF period) – HEFCE proposed two options:
- Simple model whereby both old and new institutions can submit the outputs produced by the academic member of staff when he/she was employed at the old institution (this would, some might say unfortunately, result in double counting of outputs but this can probably be tolerated as it happens already in some cases, for example, where co-authors at different HEIs submit the same output).
- Complex model whereby a census date and employment range date are used to determine which outputs can be submitted by which institution.
The funding bodies will continue to engage with the sector on these proposals and their consequences. We are anticipating announcements at the following times:
- August 2017 – initial decisions on all proposals, other than those relating to staff and outputs.
- Autumn 2017 – initial decisions on matters relating to staff and outputs.
- 2018 – REF 2021 guidance to be published.
Further information about the consultation is available here: HEFCE consultation on REF2021 – what you need to know.
BU governance: Prof John Fletcher, Pro Vice Chancellor, Research and Innovation, has established a REF Committee (all UOA Leaders plus colleagues from Professional Services as appropriate) as a sub-committee of URKEC. The REF Committee met for the first time in September 2014 and meets at least three times per year.
RKEO REF team: The preparation for BU’s REF submission is managed by RKEO.
|Head of RKE – Julie Northam||Responsible for overseeing the management of the REF preparation, submission and results, to include policies, procedures and processes; stocktake exercises; and responsibility for the environment part of the submission.|
|Research Outputs Adviser – Shelly Anne Stringer||Two major roles: i) lead for the outputs part of the submission; ii) linchpin of the REF preparations for the whole submission, including: member on all committees; first point of contact for internal and external REF queries; owner for all REF IT systems; REF outputs and metrics expert.|
|Project Delivery Manager – Paul Lynch||Lead on the data and metrics for the environment part of the submission and to provide support to the Research Outputs Adviser, for example, with the REF IT systems and the outputs part of the submission.|
|KE and Impact Manager – Rebecca Edwards||Lead on the impact part of the submission, to include the impact case studies and impact strategies, as well as the impact tracker. Responsible for supporting UOA Teams with setting and achieving impact strategies, to include identifying where support from other team members would accelerate impact.|
|Engagement and Impact Facilitator – Genna West||Responsible for providing events and engagement advice and support to specific impact case studies, and for advising UOA Teams on engagement strategies to accelerate impact.|
|Research Communications Manager – Rachel Bowen||Responsible for providing communications advice and support, to include proactive media placement, to specific impact case studies. To identify, develop and draft the impact case studies and verify and evidence claims made.|
Assessment elements: These are not yet confirmed, however, the consultation proposes these to be predominantly the same as REF 2014 (outputs, environment, impact), but with the addition of new categories (institutional environment and impact) (see the diagram below).
The role of metrics in research assessment: HEFCE conducted an independent review to ascertain the extent to which metrics could be used in the assessment and management of research. This was reported in The Metric Tide report that concluded that metrics are not yet robust enough to replace peer review. The consultation proposes that metrics are used to inform peer review in some UoAs (similar to REF 2014). The Forum for Responsible Metrics, established in 2016, produced advice in 2017 for HEFCE (and the other HE funding bodies) on the use of quantitative indicators in the assessment of outputs in REF 2021 (with further discussion planned later for assessment of impact and environments).
REF 2014 permitted the submission of research outputs in any form providing they embodied original research. It is likely this will be the same in REF 2021.
Open access: HEFCE have confirmed that all journal papers and conference proceedings published from April 2016 will have to have been made freely available in an institutional (such as BURO) and/or subject repository at the time of acceptance in order to be eligible for submission to the next REF. Read the policy here: http://www.hefce.ac.uk/whatwedo/rsrch/rinfrastruct/oa/policy/. You can read about how to use BRIAN to add your outputs to BURO and ensure you are compliant with HEFCE policy here: Adding outputs to BRIAN.
Extra points available: HEFCE have confirmed that extra points may be awarded via the environment section to institutions that can demonstrate:
- that they have taken steps towards making other outputs types available open access on their repositories (such as book chapters)
- that outputs are presented in a form that allows re-use of the work, including via text-mining
Citations: Citation data was provided to some sub-panels in REF 2014 to inform the peer review process. Anecdotal evidence suggests reviewers are using these to judge borderline papers or to compare papers, with a high citation count tipping the score into the higher classification. It is proposed in the consultation that citations will be used in REF 2021 in a similar way to REF 2014 (i.e. used to inform peer review in some UOAs).
The assessment of the environment part of REF 2014 was measured by a narrative plus metrics (PGR completions and research income awarded). Assessment was based on the vitality and sustainability of the research environment (e.g. culture, staff development opportunities, career breaks, targeted recruitment, etc.). The consultation proposes to retain the environment element but to split it between an institutional environment narrative (7.5% weighting) and a UOA-level environment narrative (7.5% weighting).
The impact element of REF 2014 was measured via impact case studies and a strategy narrative. For the purposes of the REF 2014, impact was defined as an effect on, change or benefit to the economy, society, culture, public policy or services, health, the environment or quality of life, beyond academia. Impact included, but was not limited to, an effect on, change or benefit to:
- the activity, attitude, awareness, behaviour, capacity, opportunity, performance, policy, practice, process or understanding
- of an audience, beneficiary, community, constituency, organisation or individuals
- in any geographic location whether locally, regionally, nationally or internationally.
Impact included the reduction or prevention of harm, risk, cost or other negative effects.
For the purposes of the impact element of the REF 2014:
- Impacts on research or the advancement of academic knowledge within the higher education sector (whether in the UK or internationally) were excluded.
- Impacts on students, teaching or other activities within the submitting HEI were excluded.
- Other impacts within the higher education sector, including on teaching or students, were included where they extend significantly beyond the submitting HEI.
The consultation on REF 2021 invited feedback from the sector on whether the definition and scope of impact should be broadened.
Impact evaluation: HEFCE commissioned RAND Europe to undertake a comprehensive evaluation of the impact assessment element of REF 2014. Their remit was to:
- Look at the benefit and burden associated with impact submission preparation
- Recommend possible refinements of the process for future exercises
- Identify good practice across the sector
Further information and the final report can be found here: http://www.rand.org/randeurope/research/projects/hefce-ref2014-impact.html
Impact evidence: The consultation proposes that the impact assessment period (i.e. the period in which the claimed impact(s) occurred) runs from 1 August 2013 to 31 July 2020. It’s important that you keep the impact of your research in mind and get into the habit of collecting evidence and other corroborating information as impacts are realised, which will be of great help in developing case studies for the next REF.