Category / Research assessment

REF 2021 workshops – what makes a 2*, 3* or 4* output?

We have a series of externally-facilitated REF outputs workshops scheduled to take place in early 2018 as part of the RKE Development Framework. Each session is led by REF 2014 sub-panel member who will explain how the panel interpreted and applied the REF 2014 guidance when assessing the quality of outputs. The workshops are open to all academic staff to attend.

The expected learning outcomes from the workshops are for attendees to:

  • Gain insight into how the REF panels applied the REF criteria when considering the significance, rigour and originality of outputs;
  • Understand the differences between outputs scored 4*, 3*, 2*, 1* and Unclassified;
  • Gain insight into what is meant by ‘world leading’ and ‘internationally excellent’;
  • Understand how scores borderline cases were agreed and what the tipping points were to either break the ceiling into the higher star level or to hold an output back a star level;
  • Understand how panels used other information such as metrics, markers of journal quality or prior knowledge in output assessment;
  • Gain insight into how future outputs could be strengthened for REF2021.

 

We’ve got dates for half of the UOAs so far:

  • UOA 2/3 – Prof Dame Jill Macleod Clark – date tbc (likely to be mid to late February 2018)
  • UOA 4 – Prof Marion Hetherington – 10 January 2018
  • UOA 11 – Prof Iain Stewart – 29 January 2018
  • UOA 12 – Prof Chris Chatwin – 8 January 2018
  • UOA 14 – Prof Jon Sadler – date tbc
  • UOA 15 – Prof Graeme Barker – date tbc
  • UOA 17 – Prof Terry Williams – 17 January 2018
  • UOA 18 – tbc
  • UOA 20/21 – Prof Imogen Taylor – 15 January 2018
  • UOA 23 – Prof Jane Seale – 26 January 2018
  • UOA 24 – tbc
  • UOA 27 – Prof Pat Waugh – 16 January 2018
  • UOA 32 – Prof Stephen Partridge – date tbc
  • UOA 36 – Prof Peter Lunt – date tbc

Bookings for these can be made via the Staff Intranet: https://staffintranet.bournemouth.ac.uk/workingatbu/staffdevelopmentandengagement/fusiondevelopment/fusionprogrammesandevents/rkedevelopmentframework/researchexcellenceframework/

REF2021 – initial decisions finally published

On Friday there was an exciting update from the REF Team based at HEFCE – they published the initial decisions on REF 2021. Whilst this does not include decisions regarding submitting staff, output portability or the eligibility of institutions to participate in the REF, it does include key decisions regarding the UOA structure, institution-level assessment, and the assessment weightings.

The decisions published on Friday are summarised below:

 

OVERALL:

Assessment weightings:

  • Outputs 60% (down from 65%)
  • Impact 25% (up from 20%)
  • Environment 15% (same but now includes impact strategy)

The move of the impact template from the impact assessment to the environment assessment means impact will actually contribute to more than 25% of the weighting (see impact section).

Assessment will continue to use the five-point REF 2014 scale (1*-4* and Unclassified).

UOA structure:

  • Total UOAs reduced from 36 to 34
  • Engineering will be a single UOA – UOA 12
  • REF 2014 UOA 17 will be restructured to form UOA 14: Geography and Environmental Studies and UOA 15: Archaeology
  • ‘Film and Screen Studies’ will be located and included in the name of UOA 33: Music, Drama, Dance, Performing Arts, Film and Screen Studies
  • HEFCE will continue consulting with the subject communities for forensic science and criminology to consider concerns raised about visibility. A decision is expected this autumn.

HESA cost centres will not be used to allocate staff to UOAs. Responsibility for mapping staff into UOAs will therefore remain with institutions.

 

TIMETABLE:

Impact:

  • Underpinning research must have been produced between 1 Jan 2000 – 31 Dec 2020.
  • Impacts must have occurred between 1 Aug 2013 – 31 Jul 2020.

Environment:

  • Environment data (such as income and doctoral completions) will be considered for the period 1 Aug 2013 – 31 Jul 2020.

Outputs:

  • The assessment period for the publication of outputs will be 1 Jan 2014 – 31 Dec 2020.

The draft REF 2021 guidance will be published in summer/autumn 2018 and the final guidance will be published in winter 2018-19. The submission will be in autumn 2020.

 

OUTPUTS:

Interdisciplinary research:

  • Each sub-panel will have at least one appointed member to oversee and participate in the assessment of interdisciplinary research submitted in that UOA.
  • There will be an interdisciplinary research identifier for outputs in the REF submission system (not mandatory).
  • There will be a discrete section in the environment template for the unit’s structures in support of interdisciplinary research.

Outputs due for publication after the submission date:

A reserve output may be submitted in these cases.

Assessment metrics:

Quantitative metrics may be used to inform output assessment. This will be determined by the sub-panels. Data will be provided by HEFCE.

 

IMPACT:

  • Impact will have a greater weighting in REF 2021 (25% overall plus impact included in the environment template and therefore weighting).
  • Harmonised definitions of academic and wider impact will be developed between HEFCE and the UK Research Councils.
  • Academic impacts will be assessed as part of the ‘significance’ assessment of the outputs and therefore not in the impact assessment.
  • Further guidance will be provided on the criteria for reach and significance and impacts arising from public engagement.
  • The guidance on submitting impacts on teaching will be widened to include impacts within, and beyond, the submitting institution.
  • Impacts remain eligible for submission by the institution in which the associated research was conducted. They must be underpinned by excellent research (at least REF 2*).
  • Impact case study template will have mandatory fields for recording standardised information, such as research funder, etc.
  • The number of case studies required – still not confirmed – HEFCE are exploring this in relation to the rules on staff submission and the number of outputs.
  • Case studies submitted to REF 2014 can be resubmitted to REF 2021, providing they meet the REF 2021 eligibility requirements.
  • The relationship between the underpinning research and impact will be broadened from individual outputs to include a wider body of work or research activity.

 Institutional-level assessment (impact case studies):

  • HEFCE will pilot this assessment in 2018 but it will not be included in REF 2021.

 

ENVIRONMENT:

The UOA-level environment template will be more structured, including the use of more quantitative data to evidence narrative content:

  • It will include explicit sections on the unit’s approach to:
    • supporting collaboration with organisations beyond HE
    • enabling impact – akin to the impact template in REF 2014
    • supporting equality and diversity
    • structures to support interdisciplinary research
    • open research, including the unit’s open access strategy and where this goes beyond the REF open access policy requirements

Institutional-level assessment (environment):

  • Institution-level information will be included in the UOA-level environment template, assessed by the relevant sub-panel.
  • HEFCE will pilot the standalone assessment of institution-level environment information as part of REF 2021, but this will not form part of the REF 2021 assessment. The outcomes will inform post-REF 2021 assessment exercises.

 

PANEL RECRUITMENT:

  • The sub-panel chair application process is now open (details available via the link).
  • The document sets out the plan for the recruitment of panel members (a multi-stage approach)

 

OUTSTANDING DECISIONS:

The announcement does not include decisions regarding submitting staff, output portability or the eligibility of institutions to participate in the REF. There is ongoing dialogue between HEFCE (on behalf of the funding councils) and the sector regarding this. The letter (accessed via the link above) sets out HEFCE’s current thoughts on these points and invites the sector to feedback by 29 September 2017.  BU will be providing feedback so if you have a view on this then please email me (jnortham@bournemouth.ac.uk).

 

SUMMARIES AVAILABLE:

REF & TEF: the connections – 11th October 2017

The outcomes of this year’s Teaching Excellence Framework (TEF) and the direction for the Research Excellence Framework (REF) as set out in the 2017 consultation response are likely to have significant implications for the higher education sector.  The links between research and teaching are likely to become ever more important, but set against the context of increasing emphasis on student experience, how should the sector respond and where should it focus?

REF & TEF: the connections will be hosted at Bournemouth University and will bring together some of the leading experts in higher education in both research and teaching policy.  During the morning, attendees will have the opportunity hear from experts from across the higher education sector, as they share their insights into the importance of the links between teaching and research.  The afternoon will feature a number of case studies with speakers from universities with a particularly good record of linking research and  teaching.

Speakers confirmed to date include Kim Hackett, REF Manager and Head of Research Assessment, HEFCE and John Vinney Bournemouth University, William Locke University College London, Professor Sally Brown Higher Education Academy.

For more information or to book on visit: https://reftef.eventbrite.co.uk

I’m an academic at BU. Will I be submitted to REF 2021?

Good question and, although no firm decisions have yet been announced by HEFCE, it is looking increasingly likely that all academic staff at BU will be included in the REF 2021 submission, each with at least one output published between 2014-2020.

In the midst of the sector waiting with baited breath for the initial decisions from the UK funding bodies on this, and other REF questions, HEFCE held a webinar in July. During this webinar they shared some possible decisions with the sector (the webinar and the slides are available here on the HEFCE website). The key suggestions were:

  • 100% of academics with a “significant responsibility” to undertake research are likely to be included. It is unclear at this stage what “significant responsibility” means in practice, although it is anticipated this will be based on there being an expectation for an academic member of staff to undertake research.
  • Staff without a significant responsibility for research may be exempt from inclusion but auditable documentation would be required. This would need to explicitly evidence there is not an expectation of the individual to undertake research (examples given were workload models or career frameworks linked to the individual).
  • Everyone submitted is likely to need a minimum of 1 output. The average and maximum outputs per FTE are to be determined – in the consultation it was proposed these were an average of 2 outputs per submitted FTE and a maximum of six outputs per person.
  • There is likely to be a hybrid model for output portability (i.e. which HEI can submit the outputs authored by a member of academic staff who moves from one institution to another during the REF period) – HEFCE proposed two options:
    • Simple model whereby both old and new institutions can submit the outputs produced by the academic member of staff when he/she was employed at the old institution (this would, some might say unfortunately, result in double counting of outputs but this can probably be tolerated as it happens already in some cases, for example, where co-authors at different HEIs submit the same output).
    • Complex model whereby a census date and employment range date are used to determine which outputs can be submitted by which institution.

Whilst these are not yet firm decisions (these are expected in two communications – one on staff and outputs in the autumn and one on everything else later this month), these are the clearest indications yet that all academic staff at BU will be included in REF 2021, each with at least one output.

For further information on REF 2021, see BU’s REF 2021 webpage. If you have any queries, please contact Julie Northam or Shelly Anne Stringer.

REF Main Panel Chairs announced

The main panels will provide leadership and guidance to the sub-panels that undertake the REF assessment. As chairs designate, the appointees will at first advise the funding bodies on the initial decisions and on the further development of the framework. They will take up their roles as chairs later in the year*, once the outcomes of the ‘consultation on the second REF’ are announced and further appointments to the REF panels have been made.

The Main Panel Chairs (designate) for each of the four main panel areas are:

Biographies for the Main Panel Chairs are available here: Biographies

*Interesting to note that HEFCE have reaffirmed their previous commitment to announce the outcomes of the consultation later this year, despite rumours this would either be delayed or result in a second technical consultation.

REF 2021 – stocktake exercises

With the publication of the Stern Review last summer and the funding bodies’ Second Consultation on the REF earlier this year, there’s been a lot of discussion at BU and across the sector around REF 2021 lately. Despite this, and indeed because of this, we’re still none the wiser as to what the next REF will look like. Like many other universities, we are progressing with our internal preparations whilst we await the publication of the initial decisions from the funding bodies’ in response to the feedback to their consultation (predicted to be later this year).

One of the ways BU is preparing is by running a stocktake exercise to see what outputs academic staff have published since 1 January 2014 and what potential impact BU research is having. Not only will this provide a summary of progress c. half way through the REF assessment period, it will also enable resources to be allocated to support further high-quality outputs and to accelerate research impact.

The stocktake exercise is being run in two cohorts:

  • Cohort 1 takes place this summer and involves UOAs – 2, 3, 4, 12, 22/23, 25, 34 and 36.
  • Cohort 2 takes place this autumn and involves UOAs – 11, 17 (archaeology), 17 (geography and environmental studies), 19, 20, 26 and 29.

The process will be the same for each cohort. On the outputs side, we are changing from individuals self-nominating for their inclusion in the exercise to a model where all academic staff (with a research-only or a teaching and research contract) are automatically included. This ensures the exercise is fully inclusive whilst reducing the burden on individual academic staff. In terms of impact, we are changing from colleagues writing impact case studies to inviting them to attend a meeting and deliver a short informal presentation of their research, its impact and their plans for generating further impact, followed by a discussion with the panel. This is linked to the launch of the new impact tracker in BRIAN.

The stocktake exercises are designed to be fully inclusive, positive and developmental. Further information about the REF is available on the Research Blog’s REF webpage.

SciVal’s Field weighted citation impact: Sample size matters!

There’s been a buzz on social media recently about Field weighted Citation Impact (FWCI) particularly around the recent leak from the University of Manchester that the FWCI is one of the measures suggested by which to assess academics most at risk of redundancy:

In his recent blog on The Bibliomagician Blog  (reposted here with permission) Iain Rowlands a Research Information & Intelligence Specialist at King’s College London and a member of the LIS-Bibliometrics committee questions the stability of the FWCI indicator for sets of fewer than 10,000 documents. Ian invites others to use his methodology to further test his theory…

SciVal’s field-weighted citation impact (FWCI) is an article-level metric that takes the form of a simple ratio: actual citations to a given output divided by the expected rate for outputs of similar age, subject and publication type.  FWCI has the dual merits of simplicity and ease of interpretation: a value of 2 indicates that an output has achieved twice the expected impact relative to the world literature.  It is a really useful addition to the benchmarking toolkit.

The trouble is that, typically, the distribution of citations to outputs is highly skewed, with most outputs achieving minimal impact at one end and a small number of extreme statistical outliers at the other.  Applying the arithmetic mean to data distributed like this, as does FWCI, is not ideal because the outliers can exert a strong leveraging effect, “inflating” the average for the whole set.  This effect is likely to be more marked the smaller the sample size.

I explored this effect in a simple experiment.  I downloaded SciVal FWCI values for 52,118 King’s College London papers published up until 2014.  I then calculated mean FWCI and 95% confidence (or stability) intervals for the whole sample using the bootstrapping[1] feature in SPSS.  Then I took progressively smaller random samples (99%, 98%, and so on to 1%, then 0.1%), recalculating mean FWCI and stability intervals each time.

The findings shows how mean FWCI becomes less stable as sample size decreases.  Highly cited outliers are relatively uncommon, but their chance inclusion or exclusion makes a big difference, especially as the number of outputs decreases.  In this experiment, FWCI values range across four orders of magnitude, from 0.03 to 398.28.

FWCI chart_black

What does this mean for interpreting FWCI, especially when benchmarking? The table below offers some guidance.  It shows typical stability intervals around FWCI at different scales.  The final column assumes that SciVal spits out a value of 2.20 and shows how that figure should be interpreted in terms of its stability.

FWCI Table

It’s pretty clear from this analysis that you need to know when it’s time to stop when you are drilling down in SciVal!  Another implication is that there is no sensible justification for quoting FWCI to two let alone three decimal places of precision.  I’ve kept the second decimal place above simply for purposes of demonstration.

I am well aware that the guidance above is based on data from just one institution, and may not travel well. If you would like to replicate this experiment using your own data, I’m happy to share my SPSS Syntax file.  It automates the whole thing, so you just have to load and go off on a short holiday! Just drop me an email.

Ian Rowlands is a Research Information & Intelligence Specialist at King’s College London and a member of the LIS-Bibliometrics committee.

ian.rowlands@kcl.ac.uk

BRIAN OFFLINE – 2nd & 3rd May 2017

BRIAN is being upgraded and will be unavailable for use on Tuesday 2nd and Wednesday 3rd May.

 

 

The main improvements from this upgrade include:

  • New Impact Tracking Module
  • New Homepage
  • More User Friendly Navigation

The new and improved features will make BRIAN easier and simplier to use for everyone, whilst also providing a valuable tool to academics helping them record the impact of their research

All relevant guidance notes and video guides on the Staff Intranet will be updated in due course. If you need any help using the new system or if you encounter any problems after the upgrade, please do send an email to BRIAN@bournemouth.ac.uk and a member of staff will be able to assist you.

BRIAN training sessions are also available:

  • Thursday 15th June 2017

With further dates planned. If you are interested to book on to any of these training sessions, please click here to book on!

In the meantime, if you do have queries relating to the upgrade, please get in touch with BRIAN@bournemouth.ac.uk

FHSS – New Impact Champion

I just wanted to take this opportunity to introduce myself as the new Impact Champion for the Faculty of Health and Social Sciences. When I began my research journey several years ago I knew I wanted my research to make a difference. It is this on-going desire to make a difference which led me to pursue this role of Impact Champion.

Part of my role is to take the lead in supporting academics to develop, maintain and ensure that research impact aligns with our core principle of ‘helping to make people’s lives better’. I will also be involved in developing impact case studies, and proactively accelerating impact in relation to REF.

Please don’t hesitate to get in touch if there is anything I can do to support you in your research impact.

Clare Killingback: ckillingback@bournemouth.ac.uk

Writing Days

Imagine… dedicated time and space to write away from every day distractions.

Sounds great? Then you’ll be glad to hear that as part of the Writing Academy we are busy scheduling a number of Writing Days for BU authors to book on to.

Watch. This. Space.

HE policy update w/e17th March 2017

Brexit:

  • Research Professional illustrates the Brexit threats to research positioning and job losses by highlighting the difficulties facing an EU astronomy consortium. The consortium represents seven countries, led by the UK, but will move headquarters to an EU member state from January 2021. The move means the UK will lose the project’s leadership and the 12 UK universities may not continue post-Brexit. Research Professional notes that while access to research infrastructures is available to non-EU states, the EU membership plays a significant role in decisions on where to locate facilities. Gerry Gilmore (the consortium leader, from University of Cambridge) stated:
    The UK will lose substantial scientific leadership and influence in the EU. There is going to be bad news all around. I don’t think people realise how many new jobs and new opportunities have just been destroyed.”
  • The EU (Notification of Withdrawal) Bill has survived the parliamentary process and received Royal Assent on 16th March (BBC). This bill allows the Prime Minister to notify the EU of the UK’s intention to withdraw from the EU. The Lords made two amendments to the Bill – one relating to Parliament having a “meaningful vote” on the final arrangements and one requiring a guarantee for EU citizens to remain in the UK. The bill was approved by the House of Commons, which rejected the Lords bill and then went back to the Lords under what is called “ping-pong”. The Lords voted again on both issues but the House of Lords majority backed down and the bill was passed. The PM is expected to trigger article 50 later in March.
  • 2018/19 EU student and staff guarantees: During oral questions in the Lords Baroness Royall of Blaisdon pressed the government spokesperson (Viscount Younger of Leckie) when announcements would be made regarding fees and access to loans for 2018/19 EU student starters. Leckie gave a side stepping response: “The noble Baroness makes the important point that there are uncertainties arising from Brexit, but the Government have moved rapidly to give assurances to this sector… “We have also provided similar assurances that EU nationals starting courses in 2016-17 and 2017-18 remain eligible for Research Council postgraduate support. As I have said, we will ensure that students starting in 2018-19 have the information well in advance

International students:

  • The debate over the inclusion of international students in the long-term migrant numbers continues. Even senior ministers are rebelling – Boris Johnson, Phillip Hammond and Liam Fox have all protested, although Jo Johnson continues to toe the party line backing the PM’s stance to include international students within the original immigration statistics. Liam Fox spoke out this week about the value of overseas campuses.
  • On Monday the House of Lords defeated the government on the Higher Education and Research Bill (HERB), approving an amendment to prevent international students being counted as long-term migrants. The government have responded that “the proposed amendment would create a situation where we were potentially unable to apply basic visa checks, or impose conditions on a student visa. It would also mean that fresh primary legislation were needed just to make minor, technical changes to immigration rules.” (Wonkhe)
  • HERB is scheduled to have its third reading in the Lords on 22 March 2017 and then will go back to the Commons. The PM’s stance on international students seems rock solid (Financial Times) and Theresa May is not expected to waiver – the parliamentary ping pong regarding international students will surely make headlines over the coming weeks.
  • Meanwhile there are worries about student recruitment. Politics Home quotes an Office for National Statistics release stating the number of students coming to the UK dropped by 41,000 in 2016.

Higher Education and Research Bill:

  • The HE and Research Bill has finished its third reading in the House of Lords (although it will have to go back if the House of Commons makes any changes, as seems likely).  The report stage in the Lords is on 22nd March – usually only technical or minor amendments are made at this stage.  The current version of the bill as amended by the Lords is here.
  • The surprise amendment on international students is referred to above.
  • The government won the final vote on the proposed amendment that would have required UKRI and OfS to jointly revoke research degree awarding powers, the amendment was defeated. Wonkhe report that Lord Mackay made an impassioned speech noting that it was “extraordinary” that the OfS was not required to have any expertise or experience regarding research, and yet had the unilateral power to revoke research degree awarding powers, but to no avail. The Bill continues to say that research degree-awarding powers should be made by the OfS with advice from UKRI.

With long debates, late nights and a large number of amendments, it is fair to say that HERB has received an excellent level of scrutiny within the Lords. Lord Prior of Brampton notes: “Everyone who has contributed [to the Bill debates] can take some credit for having improved it considerably. For me, it is a good example of the value this House can bring to a Bill of this kind.”

HEFCE 2017/18 funding to universities: The grant letter details the overall funding to the sector for 2017/18. It includes doubled funding for the National Collaborative Outreach Programme (£60m pa), an additional £17m increase for mainstream quality-related research, a reduction of £40m for teaching (including a reduction in PGT FTE funding rate), maintaining the disabled students premium at the 2016/17 level, the inclusion of nursing, midwifery and allied health professions (£32m), cuts to the student premium budget for full time UG of £20m (part time UG funding remains static). Institutions will receive individual allocations in April although with a publication embargo in force until May. Capital allocations will be announced in March.

Student Loans Sale: A parliamentary question tabled by Steve McCabe requested publication of the ‘in-depth market testing exercise associated with the same of the student loan book. Jo Johnson has responded: “The Government ran a market testing process with a cross-section of potential investors in the student loan book from the end of September into November 2016. This sought feedback on potential sale structures and key features of the transaction and informed the design of the sale. This was a commercial rather than a public process and was conducted under non-disclosure agreements. We do not intend to publish a report of the details. Protecting the details of the conclusions of market testing will help the ongoing sale process achieve value for money for taxpayers.

Student Fees: On Thursday 16th the Petitions Committee released its latest decisions regarding recent petitions with a high number of signatures. This included a petition to government to change the University fees from £9250 back to the £3000 fee. The Committee agreed to wait for the Higher Education and Research Bill to complete its passage through Parliament before deciding whether to schedule a debate – effectively this was a dismissal of the petition.

Research Excellence Framework  The responses to the REF2021 consultation were due in by midday on 17th March.

  • There has been a lot of focus on one area, the definition of “research active staff” for the returns – there are some interesting views:
  • HEFCE blog (and BU’s reply) – HEFCE are proposing a negotiated definition for each university, BU is proposing all staff should be returned, including teaching only
  • Royal Society blog on Research Professional – they say staff shouldn’t be returned at all, it should be institutional
  • The PVC (Research and Enterprise) from Hertfordshire says on Times Higher Education that the solution is flawed and that clarity is needed

There are many other issues in the REF consultation, including the portability of outputs, which will have important consequences for institutions and their staff. The HEFCE REF consultation on the implementation of the REF 2021 closed on 17 March 2017.  You can read BU’s response here.

Blog by the Vice-Chancellor – what next for the Teaching Excellence Framework

The BBC 2 series “Meet the Lords” could not have been better timed. The House of Lords has flexed its muscles on the Article 50 Bill and this week’s episode coincided with them passing an amendment to the Higher Education and Research Bill (HE Bill) that breaks the link between the Teaching Excellence Framework (TEF) and fees. Since then another amendment has been passed that would change the nature of the TEF, and bring it under Parliamentary scrutiny.

It would be easy to dismiss these (as some have done) as acts of rebellion by a non-elected chamber that is in the case of the HE Bill, representing vested interests in the face of a genuine government attempt to reform a sector that is badly in need of it. The Department for Education could be forgiven if they had thought that the HE Bill was nearly home and dry. They had published a long list of amendments which had been largely welcomed by the sector. The TEF does not require Parliamentary approval. Universities UK and GuildHE, amongst others, had expressed support for the HE bill as amended and expressed support for the TEF – opposing the addition of more detail as it would reduce flexibility in future negotiations on the detail. But the House of Lords did not agree – they have not sought to add more detail in the TEF, but to change its nature completely. Reading the debates, it is clear that members of the House of Lords, like most of the sector, generally support the objectives of TEF in bringing focus on the quality of education and student outcomes. They support the provision of more and better information about universities for applicants and others. They, like many in the sector, also generally support an inflationary increase in fees.

In the latest amendment, the provisions for the TEF in clause 26 have been removed and the new clause instead requires the Secretary of State to bring forward a scheme to identify whether an institution meets or fails to meet expectations based on quality standards but it “must not be used to create a single composite ranking of English higher education providers”.  The arguments are neatly summarised by Lord Lucas: “Bronze will be seen as failing because these universities will be marked out as the bottom 20%. This is just not necessary. We have succeeded, in our research rankings, in producing a measure of sufficient detail and sophistication for people to read it in detail. It produces quite marked differences between institutions, but nobody reads it as a mark of a failing institution. It is information, not ranking…”.

An earlier amendment removed the differentiation between fees based on different ratings. The speeches in the House of Lords demonstrate that they are opposed to this link for different reasons, for example:

  • Baroness Deech “If we detach fees from gold, silver and bronze, we stand a chance of increasing social mobility under the amendment. If we do not, social mobility will be frozen and ghettoisation will increase.”
  • Baroness Wolf of Dulwich: “I want to cite three groups of academics ….all of which feel, as do students, that in their current state the TEF metrics are not up to the job of determining fee levels and that, until we are sure that we have valid and reliable measures, we should not do this.”
  • Lord Lipsey : “… what seems knocking on bizarre is to plough on with bringing in this link between fees and the TEF before we have got the TEF right….The Government would give themselves the best chance of proving themselves right and the sceptics wrong if they gave time for the TEF to settle down before they brought in the fees link.”
  • Lord Kerslake: “My second reason for not making the link is that the TEF rating will relate to the university, not the subject or course. We will not see subject-level ratings until 2020 and yet we know that it is perfectly possible to have a mediocre course in an otherwise excellent university, and indeed vice versa. It can be argued that the TEF ranking gives an indication of the overall ​student experience at a particular institution, but the variation which so obviously exists within institutions makes that argument quite unconvincing.”

Except for the subject level fee point (which has not become a topic of debate yet), these are all arguments that were made by the sector in responding to the Green Paper and the TEF consultation. These are all things that we have continued to raise as we discuss the implications of subject-level TEF.

So as it stands, the TEF has lost both of its “incentives” – aka its carrot and its stick, which were both in the form of the impact on fees and reputation. It is not at all clear what will happen next – some ideas are given in this Wonkhe blog. In blogs on the Times Higher Education, Maddaleine Ansell of the University Alliance and Sorana Vieru gave very different perspectives.

So what compromise could there be to address all the concerns and yet still preserve the positive aspects of the TEF – i.e. the increasing focus on education and outcomes? I go back to BU’s response to the Green Paper, when we said that the TEF should model itself on the REF.. It should celebrate excellence wherever it is found, there should not be a link with tuition fees and there should be no forced ranking. To achieve that now, a remodelled TEF could include the following features:

  • no link to fees
  • have two rather than three levels of award – perhaps indicating good and outstanding. The last category is those who fail their quality assessment and don’t qualify for TEF.
  • take a different approach to benchmarking that does not force differentiation
  • include a place for commendations

I am not convinced by the argument that no-one would participate in the TEF without the direct financial incentive. That does not hold true for the REF. The REF has increased the focus on impact and had a beneficial impact on research. (We have some reservations about the changes proposed in the latest REF consultation, but that is a separate issue.) The concerns about the TEF would be mitigated substantially if the Olympic rating system and the link to fees were dropped. The sector would be able to engage in a much more constructive debate about subject-level TEF.

The TEF does not need to be thrown out completely – but this is an opportunity to go back to where this started from and ensure that the TEF brings focus on the quality of education and student outcomes.