Tagged / research assessment

Call for EoIs: Unit of Assessment (UOA) Leaders for UOA 2 and UOA 3 to drive REF 2021 preparations

BU is preparing submissions for units of assessment (UOAs) for REF 2021. Preparation for each UOA is led by a UOA Leader who is supported by an Impact Champion and an Output Champion. From March 2018, UOA Leaders are recruited via an open and transparent process. All academic staff have the opportunity to put themselves forward for UOA Leader roles. The roles are until December 2020.

We are currently seeking expressions of interest (EoIs) from academic staff interested in leading preparations for two UOAs:

  • Public health, health services and primary care
  • Allied health professions, dentistry, nursing and pharmacy

UOA Leaders serve a term up to December 2020, although they can choose to step down during this time. The UOA Leader undertakes a vital role in driving and delivering BU’s REF submission, influencing the University’s preparations, shaping optimal submissions for each UOA and ultimately having a significant effect on BU’s REF 2021 results.

Key responsibilities of the UOA Leader role include:

  • Providing leadership, advice and support on all issues relating to research planning, impact, performance metrics and published guidance relating to the UOA
  • Considering the widest available staff pool for the UOA and present these options to the REF Committee (being mindful of where this potentially impacts upon other UOAs)
  • Having an institutional outlook for the REF, i.e. aiming to optimise BU’s overall REF performance
  • Optimising the UOA submission and that of related UOAs by working to mitigate weaknesses and to highlight strengths across all aspects of the submission
  • Ensuring that outputs undergo rigorous review, internally and externally in order to assess quality prior to inclusion for REF
  • Working with Impact champions and the Impact Working Group to understand the interrelationship of case study quality, selection, placement and staff numbers for the UOA
  • Leading on REF communications within departments represented in the UOA and be the key point of contact and advice with regard to the UOA for Heads of research entities, DDRPPs and Executive Deans
  • Working closely with RKEO who are managing the central REF preparation and submission process
  • Attend the REF Committee meetings

Being a UOA Leader is a big commitment and is recognised accordingly. UOA Leaders are given time to attend meetings and take responsibility for tasks. As such potential applicants should discuss their workload balance with their Head of Department before applying.

 

Application process:

To apply for either role, please submit a short statement (suggested length 300 words) stating which role you are interested in and explaining your interest in the role and what you could bring to it. This should be sent by email to Julie Northam by 5pm on Thursday 12 April 2018.

The EoIs will be reviewed by a gender balanced panel comprising a DDRPP and a member of the professoriate. Applicants successful at this stage will be invited to an interview with the same panel.

The selection criteria used at EoI and interview stage are outlined below. Each criterion carries a total possible score of 5. The role will be offered to the highest scoring applicant. A member of the panel will provide feedback to all applicants.

  • Commitment, motivation and enthusiasm (scored out of 5): Being a UOA Leader is a big commitment. UOA Leaders need to be willing and able to make this commitment. They need to be enthusiastic about the REF and boosting research performance.
  • Skills and knowledge (scored out of 5): UOA Leaders should bring with them skills and knowledge to optimise BU’s REF preparations and submission (e.g. knowledge of the REF process, expertise in research metrics, leadership experience, knowledge about impact, experience of writing and delivering research strategies, etc).
  • Plans for preparing the UOA submission and awareness of the potential challenges and opportunities UOA Leaders are responsible for driving and delivering the UOA’s submission to REF 2021 whilst also maintaining an institutional outlook to optimise BU’s overall REF performance. They should have ideas for how they will do this and the potential challenges and opportunities of this, specific to the UOA.

 

Questions:

Questions regarding the process should be directed to Julie Northam (Head of RKEO).

UOA-specific questions should be directed to Prof Vanora Hundley (Deputy Dean for Research in HSS) or to Prof Tiantian Zhang (Deputy Dean for Research in FST)

REF 2021 workshops – what makes a 2*, 3* or 4* output?

 

We have a series of externally-facilitated REF outputs workshops scheduled to take place in early 2018 as part of the RKE Development Framework. Each session is led by REF 2014 sub-panel member who will explain how the panel interpreted and applied the REF 2014 guidance when assessing the quality of outputs. The workshops are open to all academic staff to attend.

The expected learning outcomes from the workshops are for attendees to:

  • Gain insight into how the REF panels applied the REF criteria when considering the significance, rigour and originality of outputs;
  • Understand the differences between outputs scored 4*, 3*, 2*, 1* and Unclassified;
  • Gain insight into what is meant by ‘world leading’ and ‘internationally excellent’;
  • Understand how scores borderline cases were agreed and what the tipping points were to either break the ceiling into the higher star level or to hold an output back a star level;
  • Understand how panels used other information such as metrics, markers of journal quality or prior knowledge in output assessment;
  • Gain insight into how future outputs could be strengthened for REF20

Workshops scheduled so far are:

  • UOA 2/3 – Prof Dame Jill Macleod Clark – 15 March 2018
  • UOA 11 – Prof Iain Stewart – 29 January 2018
  • UoA 14 – Prof Jon Sadler – 2 February 2018
  • UOA 15 – Prof Graeme Barker – 7 February 2018
  • UOA 17 – Prof Terry Williams – 17 January 2018
  • UOA 18 – TBC
  • UOA 23 – Prof Jane Seale – 26 January 2018
  • UOA 24 (Sport & Exercise Science) – Professor John Horne –date TBC
  • UoA 24 (Leisure and Tourism) – Prof Gareth Shaw – date TBC
  • UOA 11/32 (Computer Animation) – Prof Anthony Steed – 31 January 2018
  • UOA 32/34 (Practice-based) – Prof Stephen Partridge – date TBC
  • UOA 34 – Prof Peter Lunt – date TBC

Please see the full range of REF-related workshops on the staff intranet.

REF 2021 workshops – what makes a 2*, 3* or 4* output?

We have a series of externally-facilitated REF outputs workshops scheduled to take place in early 2018 as part of the RKE Development Framework. Each session is led by REF 2014 sub-panel member who will explain how the panel interpreted and applied the REF 2014 guidance when assessing the quality of outputs. The workshops are open to all academic staff to attend.

The expected learning outcomes from the workshops are for attendees to:

  • Gain insight into how the REF panels applied the REF criteria when considering the significance, rigour and originality of outputs;
  • Understand the differences between outputs scored 4*, 3*, 2*, 1* and Unclassified;
  • Gain insight into what is meant by ‘world leading’ and ‘internationally excellent’;
  • Understand how scores borderline cases were agreed and what the tipping points were to either break the ceiling into the higher star level or to hold an output back a star level;
  • Understand how panels used other information such as metrics, markers of journal quality or prior knowledge in output assessment;
  • Gain insight into how future outputs could be strengthened for REF2021.

 

Workshops scheduled so far are:

  • UOA 2/3 – Prof Dame Jill Macleod Clark – 15 March 2018
  • UOA 4 – Prof Marion Hetherington – 10 January 2018
  • UOA 11 – Prof Iain Stewart – 29 January 2018
  • UOA 12 – Prof Chris Chatwin – 8 January 2018
  • UOA 14 – Prof Jon Sadler – 11 January 2018
  • UOA 15 – Prof Graeme Barker – 7 February 2018
  • UOA 17 – Prof Terry Williams – 17 January 2018
  • UOA 18 – tbc
  • UOA 20/21 – Prof Imogen Taylor – 15 January 2018
  • UOA 23 – Prof Jane Seale – 26 January 2018
  • UOA 24 – tbc
  • UOA 27 – Prof Pat Waugh – 16 January 2018
  • UOA 11/32 (computer animation) – Prof Anthony Steed – 31 January 2018
  • UOA 32/34 (practice-based) – Prof Stephen Partridge – date tbc
  • UOA 36 – Prof Peter Lunt – date tbc

Bookings for these can be made via the Staff Intranet: https://staffintranet.bournemouth.ac.uk/workingatbu/staffdevelopmentandengagement/fusiondevelopment/fusionprogrammesandevents/rkedevelopmentframework/researchexcellenceframework/

REF 2021 – final decisions published

HEFCE kept their word and published the final decisions on REF 2021 in the autumn. Having issued the initial decisions on the Research Excellence Framework 2021 in September, the final decisions were published this week. HEFCE released its further decisions on staff and outputs on 21 November 2017.  These decisions have been informed by responses to key questions relating to staff and outputs and a survey of staff in scope for submission. This blog post provides a summary of the key decisions.

 

Submitting staff:

Previous REF/RAE exercises asked institutions to select staff for submission. The Stern Review in 2016 recognised how divisive this practice was and instead recommended that all research-active staff be returned to the next REF. HEFCE are implementing this recommendation by expecting all staff with a ‘significant responsibility for research’ to be submitted, provided they are ‘independent researchers’. What do these terms mean in practice? The HEFCE definition is:

“those for whom explicit time and resources are made available to engage actively in independent research, and that is an expectation of their job role. Research is defined as a process of investigation leading to new insights, effectively shared. Staff engaged exclusively in scholarship would not be considered to have a significant responsibility for research.”

Working with the REF 2021 main panels, HEFCE will provide further guidance on identifying staff with significant responsibility. This will be published in the guidance on submissions and panel criteria. This guidance will not prescribe a fixed set of criteria that all staff would be required to meet, but will set out a ‘menu’ of what HEFCE would consider may be appropriate indicators of significant responsibility.

 

Recognising that there are staff who have more significant responsibility for other activities, HEFCE will implement an approach whereby institutions, working with their staff and with guidelines, identify who is in scope for submission among staff meeting core eligibility criteria. HEFCE has defined the core eligibility criteria as:

Category A eligible’

  • academic staff with a contract of employment of ≥0.2 FTE
  • on the payroll of the submitting institution on the census date (31 July 2020)
  • whose primary employment function is to undertake either ‘research only’ (independent researchers only) or ‘teaching and research’
  • have a substantive connection with the submitting institution (i.e. BU)
  • for staff on ‘research only’ contracts, the eligible pool should only include those who are independent researchers, not research assistants

‘Category A submitted’ describes the staff from the ‘Category A eligible’ pool who have been identified as having significant responsibility for research on the census date.

Where the ‘Category A eligible’ staff definition accurately identifies all staff in the submitting unit with significant responsibility for research, the unit should submit 100% of staff. Where it does not accurately identify all staff in the submitting unit who have significant responsibility for research, institutions will need to implement processes to determine this and document this in a code of practice, approved by the relevant funding body with advice from the Equality and Diversity Advisory Panel (EDAP).

 

Submitting outputs:

  • The average number of outputs required per submitted FTE will be 2.5 (up from 2 outputs as previously suggested by HEFCE).
  • A minimum of one output will be required for each staff member employed on the census date (as expected).
  • A maximum of five outputs may be attributed to individual staff members (including those who have left) (down from 6 outputs as previously suggested by HEFCE).
  • Data on the distribution of outputs across staff in the unit, including staff who have left, will be provided to the sub-panels for consideration in relation to the assessment of the environment.

Output portability: A transitional approach is being adopted whereby outputs may be submitted by both the institution employing the staff member on the census date and the originating institution where the staff member was previously employed when the output was demonstrably generated. ‘Demonstrably generated’ will be determined by the date when the output was first made publicly available. This applies to the whole REF 2021 period.

Open access: The REF Open Access policy will be implemented as previous set out. This requires outputs within the scope of the policy (journal articles and some conference proceedings) to be deposited as soon after the point of acceptance as possible, and no later than three months after this date from 1 April 2018. Due to concerns around deposit on acceptance, a deposit exemption will be introduced from 1 April 2018 and remain in place for the rest of the REF 2021 publication period. This will allow outputs unable to meet this deposit timescale to remain compliant if they are deposited up to three months after the date of publication.

 

Number of impact case studies required

Submissions will include a total of one case study, plus one further case study per up to 15 FTE staff submitted, for the first 105 FTE staff returned (with a reduced requirement above this of one additional case study per 50 FTE staff). Submissions will therefore include a minimum of two case studies.

 

For the latest information, see our REF 2021 webpage.

REF 2021 workshops – what makes a 2*, 3* or 4* output?

We have a series of externally-facilitated REF outputs workshops scheduled to take place in early 2018 as part of the RKE Development Framework. Each session is led by REF 2014 sub-panel member who will explain how the panel interpreted and applied the REF 2014 guidance when assessing the quality of outputs. The workshops are open to all academic staff to attend.

The expected learning outcomes from the workshops are for attendees to:

  • Gain insight into how the REF panels applied the REF criteria when considering the significance, rigour and originality of outputs;
  • Understand the differences between outputs scored 4*, 3*, 2*, 1* and Unclassified;
  • Gain insight into what is meant by ‘world leading’ and ‘internationally excellent’;
  • Understand how scores borderline cases were agreed and what the tipping points were to either break the ceiling into the higher star level or to hold an output back a star level;
  • Understand how panels used other information such as metrics, markers of journal quality or prior knowledge in output assessment;
  • Gain insight into how future outputs could be strengthened for REF2021.

 

We’ve got dates for half of the UOAs so far:

  • UOA 2/3 – Prof Dame Jill Macleod Clark – date tbc (likely to be mid to late February 2018)
  • UOA 4 – Prof Marion Hetherington – 10 January 2018
  • UOA 11 – Prof Iain Stewart – 29 January 2018
  • UOA 12 – Prof Chris Chatwin – 8 January 2018
  • UOA 14 – Prof Jon Sadler – date tbc
  • UOA 15 – Prof Graeme Barker – date tbc
  • UOA 17 – Prof Terry Williams – 17 January 2018
  • UOA 18 – tbc
  • UOA 20/21 – Prof Imogen Taylor – 15 January 2018
  • UOA 23 – Prof Jane Seale – 26 January 2018
  • UOA 24 – tbc
  • UOA 27 – Prof Pat Waugh – 16 January 2018
  • UOA 32 – Prof Stephen Partridge – date tbc
  • UOA 36 – Prof Peter Lunt – date tbc

Bookings for these can be made via the Staff Intranet: https://staffintranet.bournemouth.ac.uk/workingatbu/staffdevelopmentandengagement/fusiondevelopment/fusionprogrammesandevents/rkedevelopmentframework/researchexcellenceframework/

REF2021 – initial decisions finally published

On Friday there was an exciting update from the REF Team based at HEFCE – they published the initial decisions on REF 2021. Whilst this does not include decisions regarding submitting staff, output portability or the eligibility of institutions to participate in the REF, it does include key decisions regarding the UOA structure, institution-level assessment, and the assessment weightings.

The decisions published on Friday are summarised below:

 

OVERALL:

Assessment weightings:

  • Outputs 60% (down from 65%)
  • Impact 25% (up from 20%)
  • Environment 15% (same but now includes impact strategy)

The move of the impact template from the impact assessment to the environment assessment means impact will actually contribute to more than 25% of the weighting (see impact section).

Assessment will continue to use the five-point REF 2014 scale (1*-4* and Unclassified).

UOA structure:

  • Total UOAs reduced from 36 to 34
  • Engineering will be a single UOA – UOA 12
  • REF 2014 UOA 17 will be restructured to form UOA 14: Geography and Environmental Studies and UOA 15: Archaeology
  • ‘Film and Screen Studies’ will be located and included in the name of UOA 33: Music, Drama, Dance, Performing Arts, Film and Screen Studies
  • HEFCE will continue consulting with the subject communities for forensic science and criminology to consider concerns raised about visibility. A decision is expected this autumn.

HESA cost centres will not be used to allocate staff to UOAs. Responsibility for mapping staff into UOAs will therefore remain with institutions.

 

TIMETABLE:

Impact:

  • Underpinning research must have been produced between 1 Jan 2000 – 31 Dec 2020.
  • Impacts must have occurred between 1 Aug 2013 – 31 Jul 2020.

Environment:

  • Environment data (such as income and doctoral completions) will be considered for the period 1 Aug 2013 – 31 Jul 2020.

Outputs:

  • The assessment period for the publication of outputs will be 1 Jan 2014 – 31 Dec 2020.

The draft REF 2021 guidance will be published in summer/autumn 2018 and the final guidance will be published in winter 2018-19. The submission will be in autumn 2020.

 

OUTPUTS:

Interdisciplinary research:

  • Each sub-panel will have at least one appointed member to oversee and participate in the assessment of interdisciplinary research submitted in that UOA.
  • There will be an interdisciplinary research identifier for outputs in the REF submission system (not mandatory).
  • There will be a discrete section in the environment template for the unit’s structures in support of interdisciplinary research.

Outputs due for publication after the submission date:

A reserve output may be submitted in these cases.

Assessment metrics:

Quantitative metrics may be used to inform output assessment. This will be determined by the sub-panels. Data will be provided by HEFCE.

 

IMPACT:

  • Impact will have a greater weighting in REF 2021 (25% overall plus impact included in the environment template and therefore weighting).
  • Harmonised definitions of academic and wider impact will be developed between HEFCE and the UK Research Councils.
  • Academic impacts will be assessed as part of the ‘significance’ assessment of the outputs and therefore not in the impact assessment.
  • Further guidance will be provided on the criteria for reach and significance and impacts arising from public engagement.
  • The guidance on submitting impacts on teaching will be widened to include impacts within, and beyond, the submitting institution.
  • Impacts remain eligible for submission by the institution in which the associated research was conducted. They must be underpinned by excellent research (at least REF 2*).
  • Impact case study template will have mandatory fields for recording standardised information, such as research funder, etc.
  • The number of case studies required – still not confirmed – HEFCE are exploring this in relation to the rules on staff submission and the number of outputs.
  • Case studies submitted to REF 2014 can be resubmitted to REF 2021, providing they meet the REF 2021 eligibility requirements.
  • The relationship between the underpinning research and impact will be broadened from individual outputs to include a wider body of work or research activity.

 Institutional-level assessment (impact case studies):

  • HEFCE will pilot this assessment in 2018 but it will not be included in REF 2021.

 

ENVIRONMENT:

The UOA-level environment template will be more structured, including the use of more quantitative data to evidence narrative content:

  • It will include explicit sections on the unit’s approach to:
    • supporting collaboration with organisations beyond HE
    • enabling impact – akin to the impact template in REF 2014
    • supporting equality and diversity
    • structures to support interdisciplinary research
    • open research, including the unit’s open access strategy and where this goes beyond the REF open access policy requirements

Institutional-level assessment (environment):

  • Institution-level information will be included in the UOA-level environment template, assessed by the relevant sub-panel.
  • HEFCE will pilot the standalone assessment of institution-level environment information as part of REF 2021, but this will not form part of the REF 2021 assessment. The outcomes will inform post-REF 2021 assessment exercises.

 

PANEL RECRUITMENT:

  • The sub-panel chair application process is now open (details available via the link).
  • The document sets out the plan for the recruitment of panel members (a multi-stage approach)

 

OUTSTANDING DECISIONS:

The announcement does not include decisions regarding submitting staff, output portability or the eligibility of institutions to participate in the REF. There is ongoing dialogue between HEFCE (on behalf of the funding councils) and the sector regarding this. The letter (accessed via the link above) sets out HEFCE’s current thoughts on these points and invites the sector to feedback by 29 September 2017.  BU will be providing feedback so if you have a view on this then please email me (jnortham@bournemouth.ac.uk).

 

SUMMARIES AVAILABLE:

I’m an academic at BU. Will I be submitted to REF 2021?

Good question and, although no firm decisions have yet been announced by HEFCE, it is looking increasingly likely that all academic staff at BU will be included in the REF 2021 submission, each with at least one output published between 2014-2020.

In the midst of the sector waiting with baited breath for the initial decisions from the UK funding bodies on this, and other REF questions, HEFCE held a webinar in July. During this webinar they shared some possible decisions with the sector (the webinar and the slides are available here on the HEFCE website). The key suggestions were:

  • 100% of academics with a “significant responsibility” to undertake research are likely to be included. It is unclear at this stage what “significant responsibility” means in practice, although it is anticipated this will be based on there being an expectation for an academic member of staff to undertake research.
  • Staff without a significant responsibility for research may be exempt from inclusion but auditable documentation would be required. This would need to explicitly evidence there is not an expectation of the individual to undertake research (examples given were workload models or career frameworks linked to the individual).
  • Everyone submitted is likely to need a minimum of 1 output. The average and maximum outputs per FTE are to be determined – in the consultation it was proposed these were an average of 2 outputs per submitted FTE and a maximum of six outputs per person.
  • There is likely to be a hybrid model for output portability (i.e. which HEI can submit the outputs authored by a member of academic staff who moves from one institution to another during the REF period) – HEFCE proposed two options:
    • Simple model whereby both old and new institutions can submit the outputs produced by the academic member of staff when he/she was employed at the old institution (this would, some might say unfortunately, result in double counting of outputs but this can probably be tolerated as it happens already in some cases, for example, where co-authors at different HEIs submit the same output).
    • Complex model whereby a census date and employment range date are used to determine which outputs can be submitted by which institution.

Whilst these are not yet firm decisions (these are expected in two communications – one on staff and outputs in the autumn and one on everything else later this month), these are the clearest indications yet that all academic staff at BU will be included in REF 2021, each with at least one output.

For further information on REF 2021, see BU’s REF 2021 webpage. If you have any queries, please contact Julie Northam or Shelly Anne Stringer.

REF 2021 – stocktake exercises

With the publication of the Stern Review last summer and the funding bodies’ Second Consultation on the REF earlier this year, there’s been a lot of discussion at BU and across the sector around REF 2021 lately. Despite this, and indeed because of this, we’re still none the wiser as to what the next REF will look like. Like many other universities, we are progressing with our internal preparations whilst we await the publication of the initial decisions from the funding bodies’ in response to the feedback to their consultation (predicted to be later this year).

One of the ways BU is preparing is by running a stocktake exercise to see what outputs academic staff have published since 1 January 2014 and what potential impact BU research is having. Not only will this provide a summary of progress c. half way through the REF assessment period, it will also enable resources to be allocated to support further high-quality outputs and to accelerate research impact.

The stocktake exercise is being run in two cohorts:

  • Cohort 1 takes place this summer and involves UOAs – 2, 3, 4, 12, 22/23, 25, 34 and 36.
  • Cohort 2 takes place this autumn and involves UOAs – 11, 17 (archaeology), 17 (geography and environmental studies), 19, 20, 26 and 29.

The process will be the same for each cohort. On the outputs side, we are changing from individuals self-nominating for their inclusion in the exercise to a model where all academic staff (with a research-only or a teaching and research contract) are automatically included. This ensures the exercise is fully inclusive whilst reducing the burden on individual academic staff. In terms of impact, we are changing from colleagues writing impact case studies to inviting them to attend a meeting and deliver a short informal presentation of their research, its impact and their plans for generating further impact, followed by a discussion with the panel. This is linked to the launch of the new impact tracker in BRIAN.

The stocktake exercises are designed to be fully inclusive, positive and developmental. Further information about the REF is available on the Research Blog’s REF webpage.

HEFCE consultation on REF2021 – what you need to know

ref-logoIt hardly feels like the dust has settled on the results from the last Research Excellence Framework, yet here we are again – almost half way through another research assessment cycle with HEFCE proposing potential changes to the next exercise. It feels like I’ve been transported back to 2009 when HEFCE were running bibliometrics and impact pilots to inform REF2014. I have to remind myself that it is in fact 2017 and things have moved on. We are now talking about REF2021 (how time flies!) and HEFCE have launched a consultation on how this exercise might work. Whilst citations and impact are still on the agenda for discussion, they are joined by other substantial changes that, if adopted, will transform the shape and potentially outcome of the next exercise.

The proposed changes have been informed by Lord Stern’s independent review of the REF which was published last July. This made 12 recommendations on how the next REF could be strengthened, whilst reducing the bureaucracy and cost of running the exercise (read Jane Forster’s overview of the recommendations – Stern review of the REF: what next?). These recommendations have now been reviewed by the four UK HE funding bodies and, prior to Christmas, HEFCE published proposals to amend the REF to incorporate the recommendations. HEFCE’s proposals are now open for consultation with the sector.

The document itself is fairly lengthy and dense, and over 40 questions are put forward for consideration. The key proposals are:

  • All research-active staff to be submitted
  • The decoupling of staff from outputs
  • Outputs will no longer be portable across institutions
  • All outputs must be available in open access form (with some exceptions)
  • Impact will have a broader definition
  • Institutional environment narratives and case studies to be submitted

There are some excellent summaries available online, many of which provide thoughts on what the proposals could mean in practice. The following summaries are particularly informative and worth reading: HEFCE launches consultation on REF2021, Soft Stern or Hard Stern and Implementing REF2021.

BU will be submitting an institutional response to the consultation before the deadline of 17th March. As these are significant proposals, all staff will have the opportunity to contribute to the response. As a number of the key questions would benefit from a discipline-level response, UoA teams (UoA leaders, impact champions and output champions) are calling meetings within Faculties to hold discussions related to their own UoA. These will take place over the next month or so. Contact a member of your UoA team for further information. Staff not based in Faculties are also invited to comment on the proposals and can do so via their Director/Head of Professional Service.

The are various REF-focused conferences and events taking place over the next few months, organised by HEFCE, ARMA, etc. Interesting news from these will be posted to the Research Blog. HEFCE are anticipating significant sector-wide engagement with the proposals and have committed to reading all feedback received. It is anticipated that, following the consultation, the initial decisions regarding the shape of REF2021 will be published in summer 2017.

REF review to be led by Lord Stern of Brentford

Lord SternSummary from press release:

The HE green paper (Fulfilling our potential: teaching excellence, social mobility and student choice) and the outcome of the Comprehensive Spending Review, both released in November 2015, mentioned that there would be a review UK research funding and the REF as a mechanism for allocating funding. Further information about this review was released on 16 December 2015 in a press release from Jo Johnson, Universities and Science Minister (Government launches review to improve university research funding). The aim of the review is to reduce the burden and cost to universities and government that has become associated with preparing for, and submitting to, the REF, therefore ensuring the government gets the most return for its investment.

The review will be chaired by the President of the British Academy and former World Bank Chief Economist Lord Nicholas Stern. He will be assisted by a high-level steering group of academic experts, including the Vice-Chancellor of Aston University, Professor Julia King, and the Past President of the Academy of Medical Sciences, Professor Sir John Tooke.

Lord Stern commented: “Research assessment should not unwittingly introduce incentives for perverse behaviour, nor should it be overly burdensome. Excellence, properly defined, must remain the central basis for allocating support and funding for research. We will explore ways in which a simpler, lighter-touch, system for the REF might be developed.”

ref-logoThe review aims to:

1. Investigate different approaches to the evaluation of UK higher education research performance which can encourage and strengthen the emphasis on delivering excellent research and impact, while simplifying and reducing the administrative burden on the HE sector.

2. Draw on the evidence from the evaluation of REF2014 and consider other models of research performance assessment, which could provide robust means of informing future research funding allocations.
3. Provide options for future iterations of the REF focusing on a simpler, lighter-touch method of research assessment, that more effectively uses data and metrics while retaining the benefits of peer review. The review should ensure that a future process identifies and supports excellent research across the UK, including dynamic changes in research quality and emerging areas of research excellence, retains the frequency of approach of the current REF arrangements (at 5-6 year cycle) and secures the confidence of the HE/academic sector.

The full terms of reference for the review, including membership of the steering group, are available here: REF review terms of reference.

The outcome is due to be delivered in summer 2016.

 

Comments from the sector:

russell group logoUnsurprisingly many of the criticisms of the review have focused on the membership of the nine-member steering committee which includes seven academics and vice-chancellors from UK universities, all but one from members of the Russell Group. There are also no representatives from Wales, Northern Ireland or the funding councils and only one university in Scotland. Lord Stern defended the committee membership, saying he had looked for panel members from a “range of subjects” and for “people who are outstanding”. “Outstanding people are necessary to recognise excellence,” he said. “They are of the highest intellectual quality” and had experience in “running things”. The panel was not meant to be “a parliament of universities”, he said, but added that “we want to hear from everybody” about the future of the REF.

There have also been murmurings that the REF already offers a good return on investment, with costs estimated to only be 2.4% of the total funds its results will inform. If this is the case then is it even posible or desirable to reduce costs further?

The THE labelled Lord Stern as a ‘REF sceptic’, quoting a joint letter he wrote with Sir Paul Nurse in which he asked: “Have criteria of quality become too narrow and formulaic in some subjects? Are researchers feeling pressured to adopt short-term horizons and a narrow focus, and chasing publication rather than following their own judgements on which are the most fruitful avenues for research and most likely to yield major outcomes?” He also questioned whether the impact element of the REF was “insufficiently deep and broad”. The letter added: “is the REF incentivisation of universities to hire stars in the closing months, like an imminent transfer deadline in the Premier League, really a way to build a long-term scholarly department?”.

The suggestion of a more metrics-based approach is concerning, especially taking into account the recommendations in the Metric Tide report (July 2015) that concluded that metrics are not yet sufficiently robust to replace peer review or to be relied upon any more than they were in REF 2014.

Lord Stern defended the review, saying there was no “foregone conclusion” about the results, despite the HE green paper and CSR presuming it would lead to changes to the REF system. Dr Wendy Piatt, Director General of the Russell Group said that the Russell Group was supportive of the review but would be concerned if it resulted in any dilution of the REF’s rigour and international reputation.

 

Further reading:

THE – Lord Stern review: no ‘foregone conclusions’ about future of REF, 22 December 2015

THE – REF sceptic to lead review into research assessment, 16 December 2015

Russell Group – Stern review of research funding, 16 December 2015

Times – Prove research is useful or lose funds, universities are told, 16 December 2015

University Alliance – University Alliance responds to the announcement of Stern Review of university research funding, 16 December 2015

REF update: HEFCE’s REFlections event, 25 March 2015

I went to HEFCE’s (rather cleverly named) REFlections event on Wednesday to hear about the review of REF 2014 and plans for the future of research assessment.

The key points were:

  • Collaboration and multi-/interdisciplinary research are likely to be important for the next REF
  • HEFCE have commissioned Elsevier to undertake a project on measuring multidisciplinary research to inform the next REF
  • The REF impact case studies database went live yesterday and is an excellent resource
  • Dual support system is likely to stay
  • Impact case studies are likely to stay, however, the impact template may change/become obsolete
  • Peer review will stay, informed by metrics in some disciplines (akin to REF 2014)
  • Metrics are not yet robust enough to have a metrics-driven REF. In particular, this is not yet possible for the assessment of outputs or impact. It is possible, however, to rely more heavily on metrics for the environment assessment and there could be changes to this part for the next REF.

 

HEFCE plan to consult with the sector on future plans for the REF this coming autumn.

 

Further information:

MS Society looking for members to join their Grant Review Panels

The MS Society is looking for members to join their Grant Review Panels. For the Grant Review Panel for Care and Services Research they are also recruiting a new Chair.

The closing date for expressions of interest is 12 noon on Thursday 11 December. Please email research@mssociety.org.uk  with your CV and a brief cover letter outlining your experience and how that will positively contribute to the work of the grant review panels.

 

Areas of Expertise Required

Members of the Panels would normally serve for a period of three years. They are looking for people from a variety of backgrounds, covering research, health and social care expertise.

If you possess the necessary skills, knowledge, experience and motivation to help ensure the MS Society funds high quality, relevant research then contact them ASAP. This is also a great development opportunity for you, as knowing how a review panel works can inform how you write and present bids.

For the GRP1 panel, they are encouraging applications from those with an expertise in the following areas:

–       Immunology

–       Stem Cells

–       Animal Models

–       Genetics

 For the GRP2 panel, they are encouraging applications from those with an expertise in the following areas:

–       Psychology

–       Statistics

–       Health economics

About the grant review panels

MS research work is driven by a strategy agreed with the Board of Trustees and which reflects the perspectives of the whole MS community – people affected by MS, clinicians, research scientists and others.

To help guide their research work they work with the advice and support of a Research Strategy Committee (RSC) and two Grant Review Panels (GRP). The RSC looks at the strategic picture, providing advice on broad areas of research, setting priorities and scrutinizing the larger, ongoing research programmes. The RSC does not decide which specific projects we should or should not fund. That more detailed work is carried out by our Grant Review Panels, one for Biomedical Research (GRP1) and one for Care and Services Research (GRP2).

HEFCE are looking for views on a potential international REF in future…

HEFCE has published a survey inviting views on an internationalised system of research assessment.

This survey forms part of a project exploring the benefits and challenges of expanding the UK’s research assessment system, the Research Excellence Framework (REF), on an international basis. At the broadest level, this means an extension of the assessment to incorporate submissions from universities overseas.

This follows an invitation earlier this year from the then Minister of State for Universities and Science, David Willetts, for HEFCE to provide an opinion on the feasibility of an international REF. The project belongs in a wider context of international interest in the exercise, on which HEFCE frequently provides information and advice to higher education policymakers and university senior management from overseas.

The THE ran a story about this in April 2014: HEFCE looks at overseas links for research excellence 

Responses are invited from any organisation or individual with an interest in higher education research or its assessment. The survey will be open until Wednesday 12 November 2014.

The survey only has four questions –

1. What do you think the key benefits would be of expanding the REF internationally?

2. What do you think the key challenges would be in expanding the REF internationally?

3. In view of the potential benefits and challenges overall, how supportive would you be of further work to explore the issues in more depth?

4. Have you got any further comments relating to internationalisation of REF?

To complete the survey visit: https://www.surveymonkey.com/r/refinternationalisation

REF 2020 update

In approximately 7 weeks we will know the outcome of the REF 2014 exercise. It is hard to believe that it is almost a year since we submitted to the exercise and that the results are round the corner. Whilst the expert panels have been assessing the submissions this past year, HEFCE have been working hard to design and shape the post-2014 REF, currently being referred to as REF 2020. They are currently midway through a review of the role of metrics in research assessment to ascertain the extent to which metrics could be used in the assessment and management of research. They have commissioned RAND Europe to undertake an assessment of the impact element of REF 2014, part of which will include recommendations to the assessment of impact in future REF exercises. They are currently consulting on whether an international REF exercise, rather than a national one, is the way forward. And, arguably the most important announcement to date, they have confirmed their open access policy for the next REF which stipulates that in order to be eligible for submission to the next REF from April 2016 all journal papers and conference proceedings have to be made freely available in an institutional and/or subject repository at the time of acceptance. Outputs not made freely available in a repository at the time of acceptance after April 2016 will be exempt from inclusion.

The Research Blog’s REF pages have recently been updated and you can read more of what we know about REF 2020 here: http://blogs.bournemouth.ac.uk/research/ref/

As soon as we know more we will post it on the Blog. Until then we wait with anticipation for the REF 2014 results!

HEFCE are looking for views on a potential international REF in future…

HEFCE has published a survey inviting views on an internationalised system of research assessment.

This survey forms part of a project exploring the benefits and challenges of expanding the UK’s research assessment system, the Research Excellence Framework (REF), on an international basis. At the broadest level, this means an extension of the assessment to incorporate submissions from universities overseas.

This follows an invitation earlier this year from the then Minister of State for Universities and Science, David Willetts, for HEFCE to provide an opinion on the feasibility of an international REF. The project belongs in a wider context of international interest in the exercise, on which HEFCE frequently provides information and advice to higher education policymakers and university senior management from overseas.

The THE ran a story about this in April 2014: HEFCE looks at overseas links for research excellence 

Responses are invited from any organisation or individual with an interest in higher education research or its assessment. The survey will be open until Wednesday 12 November 2014.

The survey only has four questions –

1. What do you think the key benefits would be of expanding the REF internationally?

2. What do you think the key challenges would be in expanding the REF internationally?

3. In view of the potential benefits and challenges overall, how supportive would you be of further work to explore the issues in more depth?

4. Have you got any further comments relating to internationalisation of REF?

To complete the survey visit: https://www.surveymonkey.com/r/refinternationalisation

Should metrics be used more widely in the next REF?

Back in 2008, after the dust was settling from the REF 2008 submission, HEFCE initiated a series of exercises to investigate whether bibliometric indicators of research quality (such as citation counts) could be used as part of the assessment for REF 2014. BU was one of 22 institutions that took part in the bibliometrics pilot, the result of which was that HEFCE concluded that citation information was not sufficiently robust enough to be used formulaically or as a primary indicator of quality but that there might be scope for it to inform and enhance processes of expert review in some disciplines. The REF 2014 guidelines stated that citation data would be provided for outputs submitted to all sub-panels in Main Panel A and some sub-panels in Main Panel B.

In April 2014, the Minister for Universities and Science asked HEFCE to undertake a fresh review of the role of metrics in determining quality, impact and other key characteristics of research undertaken in the HE sector. The review is being chaired by Professor James Wilsdon, Professor of Science and Democracy at the Science Policy Institute, University of Sussex.

HEFCE have launched a sector-wide call for evidence about research metrics and BU will be making an institutional response. BU colleagues are therefore invited to send feedback to me so that it can be considered as part of BU’s response. Colleagues are also invited to send individual responses to HEFCE.

Thinking back to 2008-09, I remember research metrics being an emotive subject and many researchers, both at BU and  across the sector, were extremely skeptical of their use in research assessment. Although bibliometrics have moved on a long way since then I think that there will still be concern as to whether metrics are robust enough to be used formulaically, particularly in the arts, humanities and social sciences.

HEFCE have asked that responses focus on the following issues:

1. Identifying useful metrics for research assessment.

2. How should metrics be used in research assessment?

3. ‘Gaming’ and strategic use of metrics.

4. International perspective.

Further information about the call for evidence is available here: http://www.hefce.ac.uk/whatwedo/rsrch/howfundr/metrics/

It is anticipated that the outcome of the review will inform the framework for the next REF assessment so it is vitally important that HEFCE receive a high quality and quantity of feedback from all disciplines.

If you would like to contribute to the BU institutional response, please add your comments to this response form and email it to me (jnortham@bournemouth.ac.uk) by Friday 30th May.

Seatbelts, sleepless nights, REF babies and a big yellow button: my reflections on REF 2014

Last week we pushed the Big Red Button (actually many big, yellow buttons) after many years of hard work preparing and finessing every last bit of the University’s submission to REF 2014.  I first got involved with REF in late 2008 which seems like a distant memory now.  HEFCE were consulting with the sector on bibliometrics and the role that citations should play in the REF which at that point still didn’t have a submission date.  BU was lucky enough to be one of 22 institutions taking part in the bibliometrics pilot to test the reliability and validity of citations and ways of identifying authors and papers in the large publication databases, Scopus and Web of Science.  There were a lot of meetings in London and so one of my first introductions to REF was standing on a cold, dark train platform at 6am with Anita Somner, waiting to get the train to one of the REF events!  In December 2008 the RAE 2008 results came out and that prompted a series of RAE/REF roadshows – in essence Prof Nick Petford, the then PVC, and I visiting all Schools to talk about the RAE results and introducing people to REF and how it was likely to be different.

Then a lot of the central REF drive died down which on reflection was a shame but also inevitable.  In 2009 all we knew about the next REF was that it would comprise outputs, environment and impact, but the key information about the assessment was still unknown.  For example, we didn’t yet know what the role of citations would be and in which UOAs this would apply, we didn’t know how the impact element would be assessed, the weightings weren’t agreed, we didn’t know what the environment template would look like or what information would be required.  With so much unknown it was very difficult to prepare anything other than for the outputs element and so the message given out was to concentrate on publications, getting them in the strongest outlet possible (we didn’t even know at this stage whether the number of outputs required per individual would be 3, 4 or 5!), and much of this work was driven from within the Schools.

Fast forward to 2010.  We had a change of senior leadership at BU with Prof John Vinney becoming VC and Prof Matthew Bennett taking the strategic lead for RKE (officially becoming PVC in January 2011).  There was still a lot undecided about REF; HEFCE had finished the bibliometrics pilot and were currently undertaking an impact pilot to test how best to assess this part of the REF.  At BU the new leadership provided by John Vinney and Matthew Bennett kickstarted our central REF preparations.  John established the REF Academic Steering Group in summer 2010 with a remit to take the strategic lead of BU’s preparations for REF.  UOA Leaders were identified in the then 12 subject areas in which we were considering submissions and they formed the REF Academic Leadership Team.  In the absence of templates or guidance from HEFCE we started work on the first drafts of the environment narratives and did the first trawl of outputs, inviting staff to submit up to four outputs for a light-touch review exercise that winter (282 individuals submitted a total of c. 1,128 outputs).  In autumn 2010 HEFCE published the results from the impact pilot, including some good practice examples of what we then knew to be one of the submission templates – the impact case studies.  Armed with this new information we undertook impact training with all of the Schools, using the infamous seatbelt example of how research can be undertaken and disseminated to achieve interim and then final impact and being informed by HSC that a reduction of the number of people killed in car crashes was actually a negative impact as there were less organs available for donation!  I have provided this lovely linear example of impact for posterity.  Around this time we also started to write up the first impact case studies, some of which evolved into the ones submitted last week.  In 2010 HEFCE confirmed what the UOAs would be and released the first list of who would be on the sub-panels.

2011 was a good REF year!  HEFCE confirmed in March how impact would be assessed in the REF and the official guidance document was finally released in June, providing us with something concrete on which to base our REF preparations.  It was the year we employed Sally Gates as the Research Communications Manager, focusing specifically on REF with the remit of working with colleagues to write the impact case studies in earnest.  We held the first of our HEFCE-supported REF events at BU attracting over 150 delegates from 39 institutions and speakers including the Deputy REF Manager Chris Taylor (HEFCE) and key academics involved in the impact pilot, including Prof James Goodwin (Age UK), Prof Peter Taylor-Gooby (University of Kent) and Prof Jim Griffiths (University of Plymouth).  This not only gave us an insight as to what was required for this still-very-unknown impact element, but also raised the profile of BU as a research university.  We held another mock exercise in winter 2011-12 to ask a selection of external reviewers to assess the draft environment narratives and impact statements.

I think it is fair to say that 2012 was a somewhat bizarre and full-on year, primarily due to the significant changeover in staffing but also because we were only a year from submission.  The year started with HEFCE releasing the Panel Working Methods and Criteria – key documentation detailing what the panels expected to see in the submissions.  This, along with the Guidance on Submissions published the previous year, became a lot of people’s bedtime reading for the next 23 months.  Despite swearing not to do so again (!) in February we held another HEFCE-supported REF event, this time focusing on how each of the Main Panels will assess research, and attracted over 150 delegates from 32 institutions, again putting BU on the research map.  We submitted the BU REF Code of Practice to HEFCE to the first of two possible opportunties and were pleased when it was approved first time (this wasn;t the case for a lot of other institutions).  Becca Edwards joined us in April as Public Engagement Officer and was immediately keen to be involved in the impact element of the REF, showing how public engagement could be a route to impact.  In the summer we held another mock exercise – this one focusing solely on outputs (265 individuals and c. 1,325 outputs).  Then bizarrely the three key REF staff in the R&KEO had babies between September and December 2012, resulting in a changeover of staff supporting the preparations – Pengpeng Hatch replacing Anita Somner, Becca Edwards replacing Sally Gates and Rita Dugan replacing me.  In December we responded to HEFCE’s Survey of Submission Intentions, a rough approximation of which UOAs we might submit to, how many staff, and the areas of the impact case studies.  This information was used to determine whether additional expertise was required on the Sub-Panels.

And then we came to 2013 – the year of submission.  Did we feel prepared?  Kind of, although there was still an awful lot to do; in fact I would go as far to say that REF probably dominated most waking minutes of those closely involved on an increasing basis as the year progressed.  The final mock exercise was held in spring 2013 (322 individuals and c. 1,610 outputs) and was a full exercise including assessment of outputs, environment narratives, impact statements and impact case studies.  This was a huge amount of data to pull together and it was essential that it was undertaken well as the results, along with those from the exercise the previous year, would be used to determine staff selection decisions.  I came back from maternity in the midst of the review meetings that followed the mock exercise, very much a baptism of fire back into the REF.  Wherever possible we tried to emulate how the panels might assess the work in the real thing, for example, with a panel of research users assessing the impact case studies.  After these meetings RASG met with UOA Leaders in a series of gruelling meetings in a very stuffy room in Christchurch House to go through each individual member of staff and their output scores, determining who’s outputs would be included in the final submission.  These decisions were ratified by the VC in July, decisions were then relayed to staff and an appeals period ran during the summer.  During this period we continued to get new outputs externally reviewed and considered, and had c. 100 new outputs assessed during this time.  The summer saw the return of Sally Gates from maternity leave and she took on the responsibility for rewriting and finalising the impact case studies, by all accounts doing a fantastic job.  In the autumn a huge amount of work went into finalising and finessing the environment narratives and impact statements, primarily involving the UOA Leaders, Profs Matthew Bennett and John Fletcher, Becca Edwards and myself.  I know from personal experience that a lot of sleep was lost during this time and there were many iterations of the narratives written.

Tuesday 26th November, 9:30am – button pushing time.  After checking, rechecking and checking the data again it was time to submit.  The #ref2014 hashtag on Twitter had gone crazy with institutions posting that they had made their submissions and now it was BU’s turn.  There were lots of submit buttons and they were all yellow – submit, validate, submit, declare and submit, submit, are you sure you want to submit?!  We pressed them all and that was it – BU’s REF 2014 submission was sent to HEFCE only to be acknolwedged with a rather bland, system generated email confirming receipt of our submission.  Sadly there were no fireworks or party poppers or massive thanks from HEFCE for all the hard work that went into preparing every last part of the submission.  REF has involved hundreds of people from across BU – from the UOA Leaders to those academics who have produced outputs, won grant funding or supervised doctoral students, from the REF Circs Board to the REF Appeals Panel, from the RKE Ops and Graduate School staff who support research activity and checked the REF data, to the members of RASG – the REF submission is the culmination of everyone’s hard work over the past few years and for that reason I am immensely proud to have been involved.  Roll on the results in December 2014!

And you may be surprise, amazed or even horrified to know that we have already started planning for REF2020…!

The Big Red Button

Actually it was yellow and there were several!  And yes we submitted our REF submission this morning; something of an anti-climax to be honest after three years of preparation and a huge amount of work by a large number of people especially over the last few weeks.  In terms of statistics we have:

  • submitted in eight units, notably for the first time in Psychology as well as in Leisure and Tourism;
  • 33% of eligible academic staff have been returned, up by 10% on RAE-2008 with a growth of 15% in eligible staff over the same period;
  • just over 40% of eligible staff were considered for selection;
  • our biggest submission, just short of 30 FTE, and is Geography/Archaeology;
  • we have submitted 22 Impact Case Studies and prepared many more. 

These numbers and statistics do not reflect the huge amount of work done by our UoA Leaders and their advisors, or the academics who have contributed the outputs to be returned and we salute you all for your work.  But in truth this is not the work of a few but a collective endeavour – academic and non-academic colleagues – a tribute to us all.  Without the selfless work of academics covering teaching while others have focused on research, without others generating RKE income, or supervising PGR students our collective success would not have been possible.  As such it is something that we should all feel proud of since we have all contributed whether returned with outputs or not.  As such we should be proud, whatever the outcome next December, of what we have achieved together.  The blood, sweat and toil is still written large in a few peoples’ mind at the moment; but boy will it be worth it and thank you!