Category / Research assessment

I’m an academic at BU. Will I be submitted to REF 2021?

Good question and, although no firm decisions have yet been announced by HEFCE, it is looking increasingly likely that all academic staff at BU will be included in the REF 2021 submission, each with at least one output published between 2014-2020.

In the midst of the sector waiting with baited breath for the initial decisions from the UK funding bodies on this, and other REF questions, HEFCE held a webinar in July. During this webinar they shared some possible decisions with the sector (the webinar and the slides are available here on the HEFCE website). The key suggestions were:

  • 100% of academics with a “significant responsibility” to undertake research are likely to be included. It is unclear at this stage what “significant responsibility” means in practice, although it is anticipated this will be based on there being an expectation for an academic member of staff to undertake research.
  • Staff without a significant responsibility for research may be exempt from inclusion but auditable documentation would be required. This would need to explicitly evidence there is not an expectation of the individual to undertake research (examples given were workload models or career frameworks linked to the individual).
  • Everyone submitted is likely to need a minimum of 1 output. The average and maximum outputs per FTE are to be determined – in the consultation it was proposed these were an average of 2 outputs per submitted FTE and a maximum of six outputs per person.
  • There is likely to be a hybrid model for output portability (i.e. which HEI can submit the outputs authored by a member of academic staff who moves from one institution to another during the REF period) – HEFCE proposed two options:
    • Simple model whereby both old and new institutions can submit the outputs produced by the academic member of staff when he/she was employed at the old institution (this would, some might say unfortunately, result in double counting of outputs but this can probably be tolerated as it happens already in some cases, for example, where co-authors at different HEIs submit the same output).
    • Complex model whereby a census date and employment range date are used to determine which outputs can be submitted by which institution.

Whilst these are not yet firm decisions (these are expected in two communications – one on staff and outputs in the autumn and one on everything else later this month), these are the clearest indications yet that all academic staff at BU will be included in REF 2021, each with at least one output.

For further information on REF 2021, see BU’s REF 2021 webpage. If you have any queries, please contact Julie Northam or Shelly Anne Stringer.

REF Main Panel Chairs announced

The main panels will provide leadership and guidance to the sub-panels that undertake the REF assessment. As chairs designate, the appointees will at first advise the funding bodies on the initial decisions and on the further development of the framework. They will take up their roles as chairs later in the year*, once the outcomes of the ‘consultation on the second REF’ are announced and further appointments to the REF panels have been made.

The Main Panel Chairs (designate) for each of the four main panel areas are:

Biographies for the Main Panel Chairs are available here: Biographies

*Interesting to note that HEFCE have reaffirmed their previous commitment to announce the outcomes of the consultation later this year, despite rumours this would either be delayed or result in a second technical consultation.

REF 2021 – stocktake exercises

With the publication of the Stern Review last summer and the funding bodies’ Second Consultation on the REF earlier this year, there’s been a lot of discussion at BU and across the sector around REF 2021 lately. Despite this, and indeed because of this, we’re still none the wiser as to what the next REF will look like. Like many other universities, we are progressing with our internal preparations whilst we await the publication of the initial decisions from the funding bodies’ in response to the feedback to their consultation (predicted to be later this year).

One of the ways BU is preparing is by running a stocktake exercise to see what outputs academic staff have published since 1 January 2014 and what potential impact BU research is having. Not only will this provide a summary of progress c. half way through the REF assessment period, it will also enable resources to be allocated to support further high-quality outputs and to accelerate research impact.

The stocktake exercise is being run in two cohorts:

  • Cohort 1 takes place this summer and involves UOAs – 2, 3, 4, 12, 22/23, 25, 34 and 36.
  • Cohort 2 takes place this autumn and involves UOAs – 11, 17 (archaeology), 17 (geography and environmental studies), 19, 20, 26 and 29.

The process will be the same for each cohort. On the outputs side, we are changing from individuals self-nominating for their inclusion in the exercise to a model where all academic staff (with a research-only or a teaching and research contract) are automatically included. This ensures the exercise is fully inclusive whilst reducing the burden on individual academic staff. In terms of impact, we are changing from colleagues writing impact case studies to inviting them to attend a meeting and deliver a short informal presentation of their research, its impact and their plans for generating further impact, followed by a discussion with the panel. This is linked to the launch of the new impact tracker in BRIAN.

The stocktake exercises are designed to be fully inclusive, positive and developmental. Further information about the REF is available on the Research Blog’s REF webpage.

SciVal’s Field weighted citation impact: Sample size matters!

There’s been a buzz on social media recently about Field weighted Citation Impact (FWCI) particularly around the recent leak from the University of Manchester that the FWCI is one of the measures suggested by which to assess academics most at risk of redundancy:

In his recent blog on The Bibliomagician Blog  (reposted here with permission) Iain Rowlands a Research Information & Intelligence Specialist at King’s College London and a member of the LIS-Bibliometrics committee questions the stability of the FWCI indicator for sets of fewer than 10,000 documents. Ian invites others to use his methodology to further test his theory…

SciVal’s field-weighted citation impact (FWCI) is an article-level metric that takes the form of a simple ratio: actual citations to a given output divided by the expected rate for outputs of similar age, subject and publication type.  FWCI has the dual merits of simplicity and ease of interpretation: a value of 2 indicates that an output has achieved twice the expected impact relative to the world literature.  It is a really useful addition to the benchmarking toolkit.

The trouble is that, typically, the distribution of citations to outputs is highly skewed, with most outputs achieving minimal impact at one end and a small number of extreme statistical outliers at the other.  Applying the arithmetic mean to data distributed like this, as does FWCI, is not ideal because the outliers can exert a strong leveraging effect, “inflating” the average for the whole set.  This effect is likely to be more marked the smaller the sample size.

I explored this effect in a simple experiment.  I downloaded SciVal FWCI values for 52,118 King’s College London papers published up until 2014.  I then calculated mean FWCI and 95% confidence (or stability) intervals for the whole sample using the bootstrapping[1] feature in SPSS.  Then I took progressively smaller random samples (99%, 98%, and so on to 1%, then 0.1%), recalculating mean FWCI and stability intervals each time.

The findings shows how mean FWCI becomes less stable as sample size decreases.  Highly cited outliers are relatively uncommon, but their chance inclusion or exclusion makes a big difference, especially as the number of outputs decreases.  In this experiment, FWCI values range across four orders of magnitude, from 0.03 to 398.28.

FWCI chart_black

What does this mean for interpreting FWCI, especially when benchmarking? The table below offers some guidance.  It shows typical stability intervals around FWCI at different scales.  The final column assumes that SciVal spits out a value of 2.20 and shows how that figure should be interpreted in terms of its stability.

FWCI Table

It’s pretty clear from this analysis that you need to know when it’s time to stop when you are drilling down in SciVal!  Another implication is that there is no sensible justification for quoting FWCI to two let alone three decimal places of precision.  I’ve kept the second decimal place above simply for purposes of demonstration.

I am well aware that the guidance above is based on data from just one institution, and may not travel well. If you would like to replicate this experiment using your own data, I’m happy to share my SPSS Syntax file.  It automates the whole thing, so you just have to load and go off on a short holiday! Just drop me an email.

Ian Rowlands is a Research Information & Intelligence Specialist at King’s College London and a member of the LIS-Bibliometrics committee.

ian.rowlands@kcl.ac.uk

BRIAN OFFLINE – 2nd & 3rd May 2017

BRIAN is being upgraded and will be unavailable for use on Tuesday 2nd and Wednesday 3rd May.

 

 

The main improvements from this upgrade include:

  • New Impact Tracking Module
  • New Homepage
  • More User Friendly Navigation

The new and improved features will make BRIAN easier and simplier to use for everyone, whilst also providing a valuable tool to academics helping them record the impact of their research

All relevant guidance notes and video guides on the Staff Intranet will be updated in due course. If you need any help using the new system or if you encounter any problems after the upgrade, please do send an email to BRIAN@bournemouth.ac.uk and a member of staff will be able to assist you.

BRIAN training sessions are also available:

  • Thursday 15th June 2017

With further dates planned. If you are interested to book on to any of these training sessions, please click here to book on!

In the meantime, if you do have queries relating to the upgrade, please get in touch with BRIAN@bournemouth.ac.uk

FHSS – New Impact Champion

I just wanted to take this opportunity to introduce myself as the new Impact Champion for the Faculty of Health and Social Sciences. When I began my research journey several years ago I knew I wanted my research to make a difference. It is this on-going desire to make a difference which led me to pursue this role of Impact Champion.

Part of my role is to take the lead in supporting academics to develop, maintain and ensure that research impact aligns with our core principle of ‘helping to make people’s lives better’. I will also be involved in developing impact case studies, and proactively accelerating impact in relation to REF.

Please don’t hesitate to get in touch if there is anything I can do to support you in your research impact.

Clare Killingback: ckillingback@bournemouth.ac.uk

Writing Days

Imagine… dedicated time and space to write away from every day distractions.

Sounds great? Then you’ll be glad to hear that as part of the Writing Academy we are busy scheduling a number of Writing Days for BU authors to book on to.

Watch. This. Space.

HE policy update w/e17th March 2017

Brexit:

  • Research Professional illustrates the Brexit threats to research positioning and job losses by highlighting the difficulties facing an EU astronomy consortium. The consortium represents seven countries, led by the UK, but will move headquarters to an EU member state from January 2021. The move means the UK will lose the project’s leadership and the 12 UK universities may not continue post-Brexit. Research Professional notes that while access to research infrastructures is available to non-EU states, the EU membership plays a significant role in decisions on where to locate facilities. Gerry Gilmore (the consortium leader, from University of Cambridge) stated:
    The UK will lose substantial scientific leadership and influence in the EU. There is going to be bad news all around. I don’t think people realise how many new jobs and new opportunities have just been destroyed.”
  • The EU (Notification of Withdrawal) Bill has survived the parliamentary process and received Royal Assent on 16th March (BBC). This bill allows the Prime Minister to notify the EU of the UK’s intention to withdraw from the EU. The Lords made two amendments to the Bill – one relating to Parliament having a “meaningful vote” on the final arrangements and one requiring a guarantee for EU citizens to remain in the UK. The bill was approved by the House of Commons, which rejected the Lords bill and then went back to the Lords under what is called “ping-pong”. The Lords voted again on both issues but the House of Lords majority backed down and the bill was passed. The PM is expected to trigger article 50 later in March.
  • 2018/19 EU student and staff guarantees: During oral questions in the Lords Baroness Royall of Blaisdon pressed the government spokesperson (Viscount Younger of Leckie) when announcements would be made regarding fees and access to loans for 2018/19 EU student starters. Leckie gave a side stepping response: “The noble Baroness makes the important point that there are uncertainties arising from Brexit, but the Government have moved rapidly to give assurances to this sector… “We have also provided similar assurances that EU nationals starting courses in 2016-17 and 2017-18 remain eligible for Research Council postgraduate support. As I have said, we will ensure that students starting in 2018-19 have the information well in advance

International students:

  • The debate over the inclusion of international students in the long-term migrant numbers continues. Even senior ministers are rebelling – Boris Johnson, Phillip Hammond and Liam Fox have all protested, although Jo Johnson continues to toe the party line backing the PM’s stance to include international students within the original immigration statistics. Liam Fox spoke out this week about the value of overseas campuses.
  • On Monday the House of Lords defeated the government on the Higher Education and Research Bill (HERB), approving an amendment to prevent international students being counted as long-term migrants. The government have responded that “the proposed amendment would create a situation where we were potentially unable to apply basic visa checks, or impose conditions on a student visa. It would also mean that fresh primary legislation were needed just to make minor, technical changes to immigration rules.” (Wonkhe)
  • HERB is scheduled to have its third reading in the Lords on 22 March 2017 and then will go back to the Commons. The PM’s stance on international students seems rock solid (Financial Times) and Theresa May is not expected to waiver – the parliamentary ping pong regarding international students will surely make headlines over the coming weeks.
  • Meanwhile there are worries about student recruitment. Politics Home quotes an Office for National Statistics release stating the number of students coming to the UK dropped by 41,000 in 2016.

Higher Education and Research Bill:

  • The HE and Research Bill has finished its third reading in the House of Lords (although it will have to go back if the House of Commons makes any changes, as seems likely).  The report stage in the Lords is on 22nd March – usually only technical or minor amendments are made at this stage.  The current version of the bill as amended by the Lords is here.
  • The surprise amendment on international students is referred to above.
  • The government won the final vote on the proposed amendment that would have required UKRI and OfS to jointly revoke research degree awarding powers, the amendment was defeated. Wonkhe report that Lord Mackay made an impassioned speech noting that it was “extraordinary” that the OfS was not required to have any expertise or experience regarding research, and yet had the unilateral power to revoke research degree awarding powers, but to no avail. The Bill continues to say that research degree-awarding powers should be made by the OfS with advice from UKRI.

With long debates, late nights and a large number of amendments, it is fair to say that HERB has received an excellent level of scrutiny within the Lords. Lord Prior of Brampton notes: “Everyone who has contributed [to the Bill debates] can take some credit for having improved it considerably. For me, it is a good example of the value this House can bring to a Bill of this kind.”

HEFCE 2017/18 funding to universities: The grant letter details the overall funding to the sector for 2017/18. It includes doubled funding for the National Collaborative Outreach Programme (£60m pa), an additional £17m increase for mainstream quality-related research, a reduction of £40m for teaching (including a reduction in PGT FTE funding rate), maintaining the disabled students premium at the 2016/17 level, the inclusion of nursing, midwifery and allied health professions (£32m), cuts to the student premium budget for full time UG of £20m (part time UG funding remains static). Institutions will receive individual allocations in April although with a publication embargo in force until May. Capital allocations will be announced in March.

Student Loans Sale: A parliamentary question tabled by Steve McCabe requested publication of the ‘in-depth market testing exercise associated with the same of the student loan book. Jo Johnson has responded: “The Government ran a market testing process with a cross-section of potential investors in the student loan book from the end of September into November 2016. This sought feedback on potential sale structures and key features of the transaction and informed the design of the sale. This was a commercial rather than a public process and was conducted under non-disclosure agreements. We do not intend to publish a report of the details. Protecting the details of the conclusions of market testing will help the ongoing sale process achieve value for money for taxpayers.

Student Fees: On Thursday 16th the Petitions Committee released its latest decisions regarding recent petitions with a high number of signatures. This included a petition to government to change the University fees from £9250 back to the £3000 fee. The Committee agreed to wait for the Higher Education and Research Bill to complete its passage through Parliament before deciding whether to schedule a debate – effectively this was a dismissal of the petition.

Research Excellence Framework  The responses to the REF2021 consultation were due in by midday on 17th March.

  • There has been a lot of focus on one area, the definition of “research active staff” for the returns – there are some interesting views:
  • HEFCE blog (and BU’s reply) – HEFCE are proposing a negotiated definition for each university, BU is proposing all staff should be returned, including teaching only
  • Royal Society blog on Research Professional – they say staff shouldn’t be returned at all, it should be institutional
  • The PVC (Research and Enterprise) from Hertfordshire says on Times Higher Education that the solution is flawed and that clarity is needed

There are many other issues in the REF consultation, including the portability of outputs, which will have important consequences for institutions and their staff. The HEFCE REF consultation on the implementation of the REF 2021 closed on 17 March 2017.  You can read BU’s response here.

Blog by the Vice-Chancellor – what next for the Teaching Excellence Framework

The BBC 2 series “Meet the Lords” could not have been better timed. The House of Lords has flexed its muscles on the Article 50 Bill and this week’s episode coincided with them passing an amendment to the Higher Education and Research Bill (HE Bill) that breaks the link between the Teaching Excellence Framework (TEF) and fees. Since then another amendment has been passed that would change the nature of the TEF, and bring it under Parliamentary scrutiny.

It would be easy to dismiss these (as some have done) as acts of rebellion by a non-elected chamber that is in the case of the HE Bill, representing vested interests in the face of a genuine government attempt to reform a sector that is badly in need of it. The Department for Education could be forgiven if they had thought that the HE Bill was nearly home and dry. They had published a long list of amendments which had been largely welcomed by the sector. The TEF does not require Parliamentary approval. Universities UK and GuildHE, amongst others, had expressed support for the HE bill as amended and expressed support for the TEF – opposing the addition of more detail as it would reduce flexibility in future negotiations on the detail. But the House of Lords did not agree – they have not sought to add more detail in the TEF, but to change its nature completely. Reading the debates, it is clear that members of the House of Lords, like most of the sector, generally support the objectives of TEF in bringing focus on the quality of education and student outcomes. They support the provision of more and better information about universities for applicants and others. They, like many in the sector, also generally support an inflationary increase in fees.

In the latest amendment, the provisions for the TEF in clause 26 have been removed and the new clause instead requires the Secretary of State to bring forward a scheme to identify whether an institution meets or fails to meet expectations based on quality standards but it “must not be used to create a single composite ranking of English higher education providers”.  The arguments are neatly summarised by Lord Lucas: “Bronze will be seen as failing because these universities will be marked out as the bottom 20%. This is just not necessary. We have succeeded, in our research rankings, in producing a measure of sufficient detail and sophistication for people to read it in detail. It produces quite marked differences between institutions, but nobody reads it as a mark of a failing institution. It is information, not ranking…”.

An earlier amendment removed the differentiation between fees based on different ratings. The speeches in the House of Lords demonstrate that they are opposed to this link for different reasons, for example:

  • Baroness Deech “If we detach fees from gold, silver and bronze, we stand a chance of increasing social mobility under the amendment. If we do not, social mobility will be frozen and ghettoisation will increase.”
  • Baroness Wolf of Dulwich: “I want to cite three groups of academics ….all of which feel, as do students, that in their current state the TEF metrics are not up to the job of determining fee levels and that, until we are sure that we have valid and reliable measures, we should not do this.”
  • Lord Lipsey : “… what seems knocking on bizarre is to plough on with bringing in this link between fees and the TEF before we have got the TEF right….The Government would give themselves the best chance of proving themselves right and the sceptics wrong if they gave time for the TEF to settle down before they brought in the fees link.”
  • Lord Kerslake: “My second reason for not making the link is that the TEF rating will relate to the university, not the subject or course. We will not see subject-level ratings until 2020 and yet we know that it is perfectly possible to have a mediocre course in an otherwise excellent university, and indeed vice versa. It can be argued that the TEF ranking gives an indication of the overall ​student experience at a particular institution, but the variation which so obviously exists within institutions makes that argument quite unconvincing.”

Except for the subject level fee point (which has not become a topic of debate yet), these are all arguments that were made by the sector in responding to the Green Paper and the TEF consultation. These are all things that we have continued to raise as we discuss the implications of subject-level TEF.

So as it stands, the TEF has lost both of its “incentives” – aka its carrot and its stick, which were both in the form of the impact on fees and reputation. It is not at all clear what will happen next – some ideas are given in this Wonkhe blog. In blogs on the Times Higher Education, Maddaleine Ansell of the University Alliance and Sorana Vieru gave very different perspectives.

So what compromise could there be to address all the concerns and yet still preserve the positive aspects of the TEF – i.e. the increasing focus on education and outcomes? I go back to BU’s response to the Green Paper, when we said that the TEF should model itself on the REF.. It should celebrate excellence wherever it is found, there should not be a link with tuition fees and there should be no forced ranking. To achieve that now, a remodelled TEF could include the following features:

  • no link to fees
  • have two rather than three levels of award – perhaps indicating good and outstanding. The last category is those who fail their quality assessment and don’t qualify for TEF.
  • take a different approach to benchmarking that does not force differentiation
  • include a place for commendations

I am not convinced by the argument that no-one would participate in the TEF without the direct financial incentive. That does not hold true for the REF. The REF has increased the focus on impact and had a beneficial impact on research. (We have some reservations about the changes proposed in the latest REF consultation, but that is a separate issue.) The concerns about the TEF would be mitigated substantially if the Olympic rating system and the link to fees were dropped. The sector would be able to engage in a much more constructive debate about subject-level TEF.

The TEF does not need to be thrown out completely – but this is an opportunity to go back to where this started from and ensure that the TEF brings focus on the quality of education and student outcomes.

HEFCE consultation on REF2021 – what you need to know

ref-logoIt hardly feels like the dust has settled on the results from the last Research Excellence Framework, yet here we are again – almost half way through another research assessment cycle with HEFCE proposing potential changes to the next exercise. It feels like I’ve been transported back to 2009 when HEFCE were running bibliometrics and impact pilots to inform REF2014. I have to remind myself that it is in fact 2017 and things have moved on. We are now talking about REF2021 (how time flies!) and HEFCE have launched a consultation on how this exercise might work. Whilst citations and impact are still on the agenda for discussion, they are joined by other substantial changes that, if adopted, will transform the shape and potentially outcome of the next exercise.

The proposed changes have been informed by Lord Stern’s independent review of the REF which was published last July. This made 12 recommendations on how the next REF could be strengthened, whilst reducing the bureaucracy and cost of running the exercise (read Jane Forster’s overview of the recommendations – Stern review of the REF: what next?). These recommendations have now been reviewed by the four UK HE funding bodies and, prior to Christmas, HEFCE published proposals to amend the REF to incorporate the recommendations. HEFCE’s proposals are now open for consultation with the sector.

The document itself is fairly lengthy and dense, and over 40 questions are put forward for consideration. The key proposals are:

  • All research-active staff to be submitted
  • The decoupling of staff from outputs
  • Outputs will no longer be portable across institutions
  • All outputs must be available in open access form (with some exceptions)
  • Impact will have a broader definition
  • Institutional environment narratives and case studies to be submitted

There are some excellent summaries available online, many of which provide thoughts on what the proposals could mean in practice. The following summaries are particularly informative and worth reading: HEFCE launches consultation on REF2021, Soft Stern or Hard Stern and Implementing REF2021.

BU will be submitting an institutional response to the consultation before the deadline of 17th March. As these are significant proposals, all staff will have the opportunity to contribute to the response. As a number of the key questions would benefit from a discipline-level response, UoA teams (UoA leaders, impact champions and output champions) are calling meetings within Faculties to hold discussions related to their own UoA. These will take place over the next month or so. Contact a member of your UoA team for further information. Staff not based in Faculties are also invited to comment on the proposals and can do so via their Director/Head of Professional Service.

The are various REF-focused conferences and events taking place over the next few months, organised by HEFCE, ARMA, etc. Interesting news from these will be posted to the Research Blog. HEFCE are anticipating significant sector-wide engagement with the proposals and have committed to reading all feedback received. It is anticipated that, following the consultation, the initial decisions regarding the shape of REF2021 will be published in summer 2017.