BU Research Blog

We continue our series of summer updates focussing on themes rather than news with a look at learning gain.  We have updates on the Industrial Strategy Bell review of Life Sciences, and an update on the TEF from UUK.

Learning Gain

Learning gain has become a potential hot topic for universities over the last year – could it be the magic bullet for problems with TEF metrics?  Why is it a policy issue and what are the implications of the policy context for universities and students?  Wonkhe recently published a helpful summary in July by Dr Camille B. Kandiko Howson, from Kings College.

Background – TEF – The Teaching Excellence Framework (TEF) includes learning gain alongside student outcomes more generally as one of its three main criteria for assessing teaching excellence (the others are teaching quality and learning environment).  The relevant TEF criteria are:

Further definition was given in the “Aspects of Quality” guidance (see the TEF guidance issued by HEFCE):

“Student Outcomes and Learning Gain is focused on the achievement of positive outcomes. Positive outcomes are taken to include:

• acquisition of attributes such as lifelong learning skills and others that allow a graduate to make a strong contribution to society, economy and the environment,
• progression to further study, acquisition of knowledge, skills and attributes necessary to compete for a graduate level job that requires the high level of skills arising from higher education

The extent to which positive outcomes are achieved for all students, including those from disadvantaged backgrounds, is a key feature. The distance travelled by students (‘learning gain’) is included”.

And it goes on:

• “Work across the sector to develop new measures of learning gain is in progress. Until new measures become available and are robust and applicable for all types of providers and students, we anticipate providers will refer to their own approaches to identifying and assessing students’ learning gain – this aspect is not prescriptive about what those measures might be.”

The TEF guidance issued by HEFCE included examples of the sorts of evidence that universities might want to consider including (amongst a much longer list):

• Learning gain and distance-travelled by all students including those entering higher education part-way through their professional lives
• Evidence and impact of initiatives aimed at preparing students for further study and research
• Use and effectiveness of initiatives used to help measure and record student progress, such as Grade Point Average (GPA)
• Impact of initiatives aimed at closing gaps in development, attainment and progression for students from different backgrounds, in particular those from disadvantaged backgrounds or those who are at greater risk of not achieving positive outcomes.

TEF Assessment – If you have been following the debates about the TEF in Year 2 (results now published), you will be aware that the assessment of institutions against these criteria was done in two ways – by looking at metrics (with benchmarking and subdivision into various sub-sets), and by review of a written provider assessment.

• The metrics that were used in TEF Year 2 for Student Outcomes and Learning Gain were from the Destination of Leavers from Higher Education survey (DLHE), specifically the DLHE declared activity 6 months after graduation – were they in employment of further study, and if in employment, was it “highly skilled” as defined by SOC groups 1-3 (managerial and professional).
• So the metrics used in Year 2 of TEF do not cover learning gain at all. In fact they only really relate to SO1 above, are of limited use in terms of employability for SO2. DLHE doesn’t measure employability, only employment. Of course, DLHE is being replaced, after major consultations by HESA throughout 2016 and 2017 with the new Graduate Outcomes survey, which will take a longer-term view and look at a broader range of outcomes. (read more in our Policy Update of 30^th June 2017).
• So for the TEF year 2, any assessment of learning gain was done through the written submissions – and as noted above there are no measures for this, it was left to providers to “refer to their own approaches to identifying and assessing students’ learning gain”.

Universities UK have published their review of Year 2 of the TEF (see next section below) which includes a strong endorsement from UUK members for a comparative learning gain metric in future iterations of the TEF.

Measuring Learning Gain – As referred to above, there is a HEFCE project to look at ways of measuring learning gain.

They are running 13 pilot projects:

• careers registration and employability initiatives – this  uses surveys and is linked most closely to SO2 – employability
• critical-thinking ‘CLA+’ standardised assessment tool – also uses the UK Engagement Survey (UKES). CLA+ is a US assessment that is done on-line and asks students to assess data and evidence and decide on a course of action or propose a solution. As such, it measures general skills but is not subject specific.
• self-efficacy across a range of disciplines
• skills and self-assessment of confidence measures
• a self-assessment skills audit and a situational judgement test
• HE in FE
• A multi-strand one: standardising entry and exit qualifications, new measures of critical skills and modelling change in learning outcomes
• a project that will analyse the Affective-Behaviour-Cognition (ABC) model data for previous years
• research skills in 6 disciplines
• psychometric testing
• learning gain from work-based learning and work placements
• a project evaluating a range of methodologies including degree classifications, UKES, NSS, Student Wellbeing survey and CLA+
• employability and subject specific learning across a range of methods – includes a project to understand the dimensions of learning gain and develop a way to measure them, one to look at R2 Strengths, one to look at career adaptability and one looking at international experience.

These are long term (3 year) projects – HEFCE published a year 1 report on 6^th July 2017 – you can read more on our 14^th July policy update – this flags a couple of challenges including how to get students to complete surveys and tests that are not relevant to their degree (a problem also encountered by the UKES). The report suggests embedding measurement “in the standard administrative procedures or formal curriculum” – which means a survey or test through enrolment and as part of our assessment programme.

To become a core TEF metric there would need to be a national standard measure that was implemented across the sector. That means that have to be mass testing (like SATs for university students) or another national survey alongside NSS and the new Graduate Outcomes survey (the replacement for DLHE) – with surveys on enrolment and at other points across the lifecycle.

Some BU staff are taking a different approach – instead of looking at generic measures for generic skills they have been looking at measuring specific learning gain against the defined learning outcomes for cohorts of students on a particular course. This is a much more customised approach but the team have set some basic parameters for the questions that they have asked which could be applied to other courses. The methodology was a survey. (Dr Martyn Polkinghorne, Dr Gelareh Roushan, Dr Julia Taylor) (see also a more detailed explanation, March 2017)

Pros, cons and alternatives

In January 2016, HEPI published a speech delivered in December 2015 by Andreas Schleicher, Director for Education and Skills, and Special Advisor on Education Policy to the Secretary-General at the Organisation for Economic Co-operation and Development (OECD) in Paris. In the speech, the author argues strongly for institutions worldwide to measure and use learning gain data. He supports the use of specific measures for disciplines although points out the difficulties with this – not least in getting comparable data. So he also focuses on generic skills – but he doesn’t suggest a specific methodology.

An HEA presentation from December 2016 mentions a number of inputs that “predict both student performance and learning gains” – including contact hours, class size (and a host of other things including the extent and timing of feedback on assignments).

It is worth looking quickly at GPA (Grade Point Average) as this is also mentioned in the TEF specification as noted above. The HEA are looking at degree standards for HEFCE now, having done a pilot project on GPA in 2013-14.  The report notes that “potential capacity to increase granularity of awards, transparency in award calculations, international recognition and student engagement in their programmes”. The summary says, “The importance to stakeholders of a nationally-agreed, common scale is a key finding of the pilot and is considered crucial for the acceptance and success of GPA in the UK”, and that “The pilot providers considered that the development of widespread stakeholder understanding and commitment would require clear communication to be sustained over a number of years.”

Wonkhe have a round up on the background to the GPA debate from June 2016,

Although the big focus for the TEF was on outputs not inputs, the Department for Education has announced that it will start to look at including some of the inputs. See our HE policy update for 21^st July where we look at the new teaching intensity measure that will be part of the subject level TEF pilots. You can read more about this in a THE article from 2^nd August:

• “The pilot “will measure teaching intensity using a method that weights the number of hours taught by the student-staff ratio of each taught hour,” explains the pilot’s specification, published by the Department for Education“. Put simply, this model would value each of these at the same level: two hours spent in a group of 10 students with one member of staff, two hours spent in a group of 20 with two members of staff, one hour spent in a group of five students with one member of staff,” it explains. Once contact hours are weighted by class sizes, and aggregated up to subject level, those running the pilot will be able to calculate a “gross teaching quotient” score, which would be an “easily interpretable number” and used as a “supplementary metric” to inform subject-level assessments”.

The contact hours debate is very political – tied up with concerns about value for money and linked to the very topical debate on fees (speech on 20^th July by Jo Johnson .and see our HE Policy Update for 21^st July 2017)

This is all very interesting when, as mentioned above, the TEF specification for year two put so much emphasis on measuring outcome and not just inputs: “The emphasis in the provider submission should be on demonstrating the impact and effectiveness of teaching on the student experience and outcomes they achieve. The submission should therefore avoid focusing on descriptions of strategies or approach but instead should focus on impact. Wherever possible, impact should be demonstrated empirically. “

Experts and evidence – There will be a real push from the sector for evidence that the new teaching intensity measure and reporting of contact hours and other things really does make a difference to students before it is included in the TEF. The HEA’s position on this (2016) is a helpful summary of the debate about contact hours.

There is an interesting article in the HEPI collection of responses to the Green Paper in January 2016  from Graham Gibbs, former Professor at the University of Winchester and Director of the Oxford Learning Institute, University of Oxford, and author of Dimensions of quality and Implications of ‘Dimensions of quality’ in a market environment. He supports the use of learning gain metrics as a useful tool. He points out that “cohort size is a strong negative predictor of both student performance and learning gains”. He also adds “Russell Group Universities have comparatively larger cohorts and larger class sizes, and their small group teaching is less likely to be undertaken by academics, all of which save money but reduce learning gains”. He does not accept that contact hours, or institutional reputation (linked to high tariff entry and research reputation) impact learning gain.

There is an interesting article on the Higher Education Policy Institute (HEPI) website here written by the authors of an article that looked at class size.

Impact so far – So what happened in the TEF – a very quick and incomplete look at TEF submissions suggests that not many institutions included much about learning gain (or GPA) and those that did seem to fall into two categories – those participating in the pilot who mention the pilot, and some who mention it in the context of the TEF core data – e.g. Birmingham mention their access project and learning gain (but don’t really evidence it except through employment and retention). Huddersfield talk about it in the context of placements and work experience but again linked to employment outcomes, although they also mention assessment improvement.

Teaching Excellence Framework (TEF) – year 2 review

Universities UK have published their review of Year 2 of the TEF following a survey that UUK did of their members.

The key findings from the report are:

• There appears to be general confidence that overall process was fair, notwithstanding the outcomes of individual appeals. Judgements were the result of an intensive and discursive process of deliberation by the assessment panel. There was a slight correlation between TEF results, entry tariff and league table rankings.
• It is estimated that the cost of participating in the TEF for 134 higher education institutions was approximately £4 million. This was driven by the volume of staff engagement, particularly senior staff.
• Further consideration will need to be given to how the TEF accounts for the diversity of the student body, particularly part-time students, and how the TEF defines and measures excellence. [UUK also raises a concern about judgements possibly being skewed by prior attainment]
• If subject-level TEF is to provide students with reliable information it must address the impact of increased metric suppression [this relates to metrics which could not be used because of small numbers, particularly for part-time students and for the ethnicity splits], how judgments are made in the absence of data [particularly an issue for those institutions affected by the NSS boycott], the comparability of subject groupings and the increase in cost and complexity of submissions and assessment.

[To address the issue with suppression, the report noted that the splits for ethnicity will be reduced from 6 to 3 for subject level TEF (p35)]

These findings also suggest that if the TEF is to make an effective contribution to the ongoing success of the whole UK sector, the following issues would merit consideration as part of the independent review:

• How the TEF defines and measures excellence in a diverse sector and supports development of teaching and learning practice.
• The role that the TEF plays across the student decision-making process and the relationship with the wider student information landscape.
• The process for the future development of the TEF and the role of the sector, including students and devolved nations.
• The relationship between the TEF and quality assessment, including regulatory baselines and the Quality Code.

Figure 4 shows the data benchmarking flags received by providers at each award level – these two charts are interesting because they show that providers with negative flags still received gold (and silver).

The survey also asked about future developments for the TEF with learning gain being a clear leader – ahead of teaching intensity. HEFCE is running learning gain pilots, as discussed above, and teaching intensity will be the subject of a pilot alongside subject level TEF. Interestingly, on p 33 a chart shows that nearly 70% of respondents believed that “there is no proportionate approach for producing a robust subject level TEF judgement which will be useful for students”.

Industrial Strategy

Following our update on the Industrial Strategy last week there are a couple of updates. Innovate UK has announced funding for businesses to work on innovative technologies, future products and services. The categories link closely to the Industrial Strategy priorities including digital technologies, robotics, creative economy and design and space applications as well as emerging technologies and electronics.

There was also an announcement about funding for innovative medicines manufacturing solutions.

Sir John Bell has published his report for the government on Life Sciences and the Industrial Strategy. There are seven main recommendations under 4 themes, which are summarised below. You can read a longer summary on the BU Research Blog.

Some interesting comments:

• “The key UK attribute driving success in life sciences is the great strength in university-based research. Strong research-based universities underpin most of the public sector research success in the UK, as they do in the USA and in Scandinavia. National research systems based around institutes rather than universities, as seen in Germany, France and China, do not achieve the same productivity in life sciences as seen in university-focussed systems.” (p22)
• “The decline in funding of indirect costs for charity research is coupled to an increasing tendency for Research Councils to construct approaches that avoid paying indirect Full Economic Costs (FEC). Together, these are having a significant impact on the viability of research in universities and have led to the institutions raising industrial overhead costs to fill the gap. This is unhelpful.” (p24)
• “It is also recommended, that the funding agencies, in partnership with major charities, create a high-level recruitment fund that would pay the real cost of bringing successful scientists from abroad to work in major UK university institutions.” (see the proposal to attract international scientists below).
• On clusters “Life sciences clusters are nearly always located around a university or other research institute and in the UK include elements of NHS infrastructure. However, evidence and experience suggests that governments cannot seed technology clusters and their success is usually driven by the underpinning assets of universities and companies, and also by the cultural features of networking and recycling of entrepreneurs and capital.” And “Regions should make the most of existing opportunities locally to grow clusters and build resilience by working in partnership across local Government, LEPs (in England), universities and research institutes, NHS, AHSNs, local businesses and support organisations, to identify and coalesce the local vision for life sciences. Science & Innovation Audits, Local Growth Funds and Growth Hubs (in England), Enterprise Zones and local rates and planning flexibilities can all be utilised to support a vision for life sciences. “ (see the proposal on clusters under “Growth and Infrastructure” – this was a big theme in the Industrial strategy and something we also covered in our Green Paper response)
• On skills: “ The flow of multidisciplinary students at Masters and PhD level should be increased by providing incentives through the Higher Education Funding Council for England.2 and  “Universities and research funders should embed core competencies at degree and PhD level, for example data, statistical and analytical skills, commercial acumen and translational skills, and management and entrepreneurship training (which could be delivered in partnership with business schools). They should support exposure to, and collaboration with, strategically important disciplines including computer and data science, engineering, chemistry, physics, mathematics and material science.”

Health Advanced Research Programme (HARP) proposal – with the goal to create 2-3 entirely new industries over the next 10 years.

Reinforcing the UK science offer 

• Sustain and increase funding for basic science to match our international competition – the goal is that the UK should attract 2000 new discovery scientists from around the globe
  • The UK should aim to be in the upper quartile of OECD R&D spending and sustain and increase the funding for basic science, to match our international competitors, particularly in university settings, encouraging discovery science to co-locate.
  • Capitalise on UKRI to increase interdisciplinary research, work more effectively with industry and support high-risk science.
  • Use Government and charitable funding to attract up to 100 world-class scientists to the UK, with support for their recruitment and their science over the next ten years.
• Further improve UK clinical trial capabilities to support a 50% increase in the number of clinical trials over the next 5 years and a growing proportion of change of practice and trials with novel methodology over the next 5 years.

Growth and infrastructure – the goal is to create four UK companies valued at >£20 billion market cap in the next ten years.

NHS collaboration – the Accelerated Access Review should be adopted with national routes to market streamlined and clarified, including for digital products. There are two stated goals:

• NHS should engage in fifty collaborative programmes in the next 5 years in late-stage clinical trials, real world data collection, or in the evaluation of diagnostics or devices.
• The UK should be in the top quartile of comparator countries, both for the speed of adoption and the overall uptake of innovative, cost-effective products, to the benefit of all UK patients by the end of 2023.

Data – Establish two to five Digital Innovation Hubs providing data across regions of three to five million people.

• Create a forum for researchers across academia, charities and industry to engage with all national health data programmes.
• Establish a new regulatory, Health Technology Assessment and commercial framework to capture for the UK the value in algorithms generated using NHS data.
• Two to five digital innovation hubs providing data across regions of three to five million people should be set up as part of a national approach and building towards full population coverage, to rapidly enable researchers to engage with a meaningful dataset. One or more of these should focus on medtech.
• The UK could host 4-6 centres of excellence that provide support for specific medtech themes, focussing on research capability in a single medtech domain such as orthopaedics, cardiac, digital health or molecular diagnostics.
• National registries of therapy-area-specific data across the whole of the NHS in England should be created and aligned with the relevant charity.

Skills

• A migration system should be established that allows recruitment and retention of highly skilled workers from the EU and beyond, and does not impede intra-company transfers.
• Develop and deliver a reinforced skills action plan across the NHS, commercial and third sectors based on a gap analysis of key skills for science.
  • Create an apprenticeship scheme that focuses on data sciences, as well as skills across the life sciences sector, and trains an entirely new cadre of technologists, healthcare workers and scientists at the cutting-edge of digital health.
  • Establish Institutes of Technology that would provide opportunity for technical training, particularly in digital and advanced manufacturing areas.
  • There should be support for entrepreneur training at all levels, incentivising varied careers and migration of academic scientists into industry and back to academia.
  • A fund should be established supporting convergent science activities including cross-disciplinary sabbaticals, joint appointments, funding for cross-sectoral partnerships and exchanges across industry and the NHS, including for management trainees.
  • High quality STEM education should be provided for all, and the government should evaluate and implement additional steps to increase the number of students studying maths to level 3 and beyond.

JANE FORSTER                                                             |                               SARAH CARTER

Policy Advisor                                                                                               Policy & Public Affairs Officer

65111                                                                                                              65070

Follow: @PolicyBU on Twitter                      |                               policy@bournemouth.ac.uk