Welcome to the REF section of the BU Research Blog!
The Research Excellence Framework (REF) is the exercise for assessing the volume and quality of research in UK HEIs. As with its predecessor (the RAE), the results of the REF are used by HEFCE to determine the annual quality-related research (QR) grant distributed from HEFCE to HEIs in England.
The last exercise was REF 2014. Institutions made their submissions in autumn 2013 and the results were published in December 2014. The next REF exercise has been confirmed as REF 2021 meaning that the submission will be in autumn 2020; this will be a seven year assessment period.
The REF assesses research excellence through a process of expert review, informed by indicators where appropriate. It is based on HEIs submitting evidence of their research activity and outcomes to discipline-based units of assessment (UOAs) to be assessed by expert panels.
How will REF 2021 work?
HEFCE, on behalf of the four UK funding bodies, consulted with the sector in early 2017 as to the shape of the next REF. This consultation was informed by Lord Stern’s independent review of the REF, published in July 2016. The consultation closed on 17 March 2017 and over 300 responses were received. The finding bodies published the initial decisions on REF 2021 on 1 September 2017, alongside another consultation with the sector about submitting staff and outputs (HEFCE Circular letter 33/2017). The final decisions on REF 2021 were published on 21 November 2017.
Assessment elements: These will be the same as REF 2014 but with slightly different weightings:
- Outputs (60%)
- Environment (15%)
- Impact (25%)
The role of metrics in research assessment: HEFCE conducted an independent review to ascertain the extent to which metrics could be used in the assessment and management of research. This was reported in The Metric Tide report that concluded that metrics are not yet robust enough to replace peer review. The consultation proposes that metrics are used to inform peer review in some UoAs (similar to REF 2014). The Forum for Responsible Metrics, established in 2016, produced advice in 2017 for HEFCE (and the other HE funding bodies) on the use of quantitative indicators in the assessment of outputs in REF 2021 (with further discussion planned later for assessment of impact and environments).
Previous REF/RAE exercises asked institutions to select staff for submission. The Stern Review in 2016 recognised how divisive this practice was and instead recommended that all research-active staff be returned to the next REF. HEFCE are implementing this recommendation by expecting all staff with a ‘significant responsibility for research’ to be submitted, provided they are ‘independent researchers’. What do these terms mean in practice? The HEFCE definition is:
“those for whom explicit time and resources are made available to engage actively in independent research, and that is an expectation of their job role. Research is defined as a process of investigation leading to new insights, effectively shared. Staff engaged exclusively in scholarship would not be considered to have a significant responsibility for research.”
Working with the REF 2021 main panels, HEFCE will provide further guidance on identifying staff with significant responsibility. This will be published in the guidance on submissions and panel criteria. This guidance will not prescribe a fixed set of criteria that all staff would be required to meet, but will set out a ‘menu’ of what HEFCE would consider may be appropriate indicators of significant responsibility.
Recognizing that there are staff who have more significant responsibility for other activities, HEFCE will implement an approach whereby institutions, working with their staff and with guidelines, identify who is in scope for submission among staff meeting core eligibility criteria. HEFCE has defined the core eligibility criteria as:
‘Category A eligible’:
- academic staff with a contract of employment of ≥0.2 FTE
- on the payroll of the submitting institution on the census date (31 July 2020)
- whose primary employment function is to undertake either ‘research only’ (independent researchers only) or ‘teaching and research’
- have a substantive connection with the submitting institution (i.e. BU)
- for staff on ‘research only’ contracts, the eligible pool should only include those who are independent researchers, not research assistants
‘Category A submitted’ describes the staff from the ‘Category A eligible’ pool who have been identified as having significant responsibility for research on the census date.
Where the ‘Category A eligible’ staff definition accurately identifies all staff in the submitting unit with significant responsibility for research, the unit should submit 100% of staff. Where it does not accurately identify all staff in the submitting unit who have significant responsibility for research, institutions will need to implement processes to determine this and document this in a code of practice, approved by the relevant funding body with advice from the Equality and Diversity Advisory Panel (EDAP).
- The average number of outputs required per submitted FTE will be 2.5 (up from 2 outputs as previously suggested by HEFCE).
- A minimum of one output will be required for each staff member employed on the census date (as expected).
- A maximum of five outputs may be attributed to individual staff members (including those who have left) (down from 6 outputs as previously suggested by HEFCE).
- Data on the distribution of outputs across staff in the unit, including staff who have left, will be provided to the sub-panels for consideration in relation to the assessment of the environment.
Output portability: A transitional approach is being adopted whereby outputs may be submitted by both the institution employing the staff member on the census date and the originating institution where the staff member was previously employed when the output was demonstrably generated. ‘Demonstrably generated’ will be determined by the date when the output was first made publicly available. This applies to the whole REF 2021 period.
Open access: The REF Open Access policy will be implemented as previous set out. This requires outputs within the scope of the policy (journal articles and some conference proceedings) to be deposited as soon after the point of acceptance as possible, and no later than three months after this date from 1 April 2018. Due to concerns around deposit on acceptance, a deposit exemption will be introduced from 1 April 2018 and remain in place for the rest of the REF 2021 publication period. This will allow outputs unable to meet this deposit timescale to remain compliant if they are deposited up to three months after the date of publication. You can read about how to use BRIAN to add your outputs to BURO and ensure you are compliant with HEFCE policy here: Adding outputs to BRIAN.
HEFCE have confirmed that extra points may be awarded via the environment section to institutions that can demonstrate:
- that they have taken steps towards making other outputs types available open access on their repositories (such as book chapters)
- that outputs are presented in a form that allows re-use of the work, including via text-mining
Citation data: This was provided to some sub-panels in REF 2014 to inform the peer review process. Anecdotal evidence suggests reviewers are using these to judge borderline papers or to compare papers, with a high citation count tipping the score into the higher classification. It is proposed in the consultation that citations will be used in REF 2021 in a similar way to REF 2014 (i.e. used to inform peer review in some UOAs).
The assessment of the environment part of REF 2014 was measured by a narrative plus metrics (PGR completions and research income awarded). Assessment was based on the vitality and sustainability of the research environment (e.g. culture, staff development opportunities, career breaks, targeted recruitment, etc.). This is expected to be the same in REF 2021 but with more reliance on metrics alongside the narrative.
The impact element of REF 2021 will be measured via impact case studies (the impact strategy will be incorporated into the environment section). For the purposes of the REF 2014, impact was defined as an effect on, change or benefit to the economy, society, culture, public policy or services, health, the environment or quality of life, beyond academia. Impact included, but was not limited to, an effect on, change or benefit to:
- the activity, attitude, awareness, behaviour, capacity, opportunity, performance, policy, practice, process or understanding
- of an audience, beneficiary, community, constituency, organisation or individuals
- in any geographic location whether locally, regionally, nationally or internationally.
Impact included the reduction or prevention of harm, risk, cost or other negative effects.
For the purposes of the impact element of the REF 2014:
- Impacts on research or the advancement of academic knowledge within the higher education sector (whether in the UK or internationally) were excluded.
- Impacts on students, teaching or other activities within the submitting HEI were excluded.
- Other impacts within the higher education sector, including on teaching or students, were included where they extend significantly beyond the submitting HEI.
For REF 2021 the funding bodies will provide greater clarity on the definition and scope of impact. This is expected to be published in 2018.
Impact evaluation: HEFCE commissioned RAND Europe to undertake a comprehensive evaluation of the impact assessment element of REF 2014. Their remit was to:
- Look at the benefit and burden associated with impact submission preparation
- Recommend possible refinements of the process for future exercises
- Identify good practice across the sector
Further information and the final report can be found here: http://www.rand.org/randeurope/research/projects/hefce-ref2014-impact.html
Impact evidence: The consultation proposes that the impact assessment period (i.e. the period in which the claimed impact(s) occurred) runs from 1 August 2013 to 31 July 2020. It’s important that you keep the impact of your research in mind and get into the habit of collecting evidence and other corroborating information as impacts are realised, which will be of great help in developing case studies for the next REF.
Number of impact case studies required: Submissions will include a total of one case study, plus one further case study per up to 15 FTE staff submitted, for the first 105 FTE staff returned (with a reduced requirement above this of one additional case study per 50 FTE staff). Submissions will therefore include a minimum of two case studies.
How is preparation for REF 2021 supported at BU?
BU governance: Prof John Fletcher, Pro Vice Chancellor, Research and Innovation, has established a REF Committee (all UOA Leaders plus colleagues from Professional Services as appropriate) as a sub-committee of URKEC. The REF Committee met for the first time in September 2014 and meets at least three times per year.
RKEO REF team: The preparation for BU’s REF submission is managed by RKEO:
|Head of RKE – Julie Northam||Responsible for overseeing the management of the REF preparation, submission and results, to include policies, procedures and processes; stocktake exercises; and responsibility for the environment part of the submission.|
|Research Outputs Adviser – Shelly Anne Stringer||Two major roles: i) lead for the outputs part of the submission; ii) linchpin of the REF preparations for the whole submission, including: member on all committees; first point of contact for internal and external REF queries; owner for all REF IT systems; REF outputs and metrics expert.|
|Project Delivery Manager – Paul Lynch||Lead on the data and metrics for the environment part of the submission and to provide support to the Research Outputs Adviser, for example, with the REF IT systems and the outputs part of the submission.|
|KE and Impact Manager – Rebecca Edwards||Lead on the impact part of the submission, to include the impact case studies and impact strategies, as well as the impact tracker. Responsible for supporting UOA Teams with setting and achieving impact strategies, to include identifying where support from other team members would accelerate impact.|
|Engagement and Impact Facilitator – Jane Kavanagh-Lauridsen||Responsible for providing events and engagement advice and support to specific impact case studies, and for advising UOA Teams on engagement strategies to accelerate impact.|
|Research Communications Manager – Rachel Bowen||Responsible for providing communications advice and support, to include proactive media placement, to specific impact case studies. To identify, develop and draft the impact case studies and verify and evidence claims made.|