Category / Research assessment

REF week on the Blog! What were the HEFCE REF pilots?

Research assessment

This week is REF Week on the Blog! Each day we will be explaining a different element of the Research Excellence Framework (REF) as a quick reference guide to help you prepare for the forthcoming REF exercise – REF2014.

What were the HEFCE REF pilots? – HEFCE ran two pilot exercises with HEIs in the sector during the development of the REF. The first exercise was a bibliometrics pilot, and the second was an impact pilot.

Bibliometrics pilot – HEFCE ran a pilot exercise in the construction of bibliometric indicators of research quality in 2008-09, using Scopus and the Web of Scienceas the test databases. BU was chosen as one of 22 institutions to be part of phase one of the pilot exercise. This involved the provision of publication details to HEFCE, and cross-checking BU information on Web of Science and Scopus. Where possible this was completed using BU’s institutional repository, BURO. The outcome of the bibliometrics pilot was that bibliometric indicators are not yet sufficiently robust enough in all disciplines to be used formulaically or as a primary indicator of research quality. However HEFCE agreed that there was scope for bibliometrics to inform the process of expert review in some units of assessment. These findings resulted in the decision that some UOA sub-panels will receive citation data (the number of times an output has been cited, calculate via Scopus) as additional information about the academic significance of the outputs.

Impact pilot – During 2009-10, HEFCE ran a second pilot exercise, this time with the aim of developing proposals for how to assess research impact in the REF. The impact pilot involved 29 HEIs submitting evidence of impact (case studies and statements) which were assessed by pilot expert panels in five units of assessment:

  • Clinical Medicine
  • Physics
  • Earth Systems and Environmental Sciences
  • Social Work and Social Policy & Administration
  • English Language and Literature

The impact pilot completed in autumn 2010 and the final report (including recommendations and findings) was published on 11 November 2010. The full report can be accessed on the HEFCE website. For a brief summary of the report, please download the Impact Pilot Summary. You can also read our REF Impact FAQs.

You can access the latest presentation about the REF, written by the REF team, here: REF slide pack Sep 2011

Check out the posts appearing on the Blog every day this week as part of REF Week!

REF week on the Blog! Introduction to the Research Excellence Framework

Research assessment

Next week is REF Week on the Blog! Each day we will be explaining a different element of the Research Excellence Framework (REF) as a quick reference guide to help you prepare for the forthcoming REF exercise – REF2014.

What is the REF? – The Research Excellence Framework (REF) has replaced the Research Assessment Exercise (RAE) as the new process for assessing the volume and quality of research in UK HEIs. As with the RAE, the results of the REF will determine the annual quality-related research (QR) grant distributed from HEFCE to HEIs in England.

The REF will assess research excellence through a process of expert review, informed by indicators where appropriate. It will be based on HEIs submitting evidence of their research activity and outcomes, to be assessed by expert panels.

The REF will focus on assessing three elements, which together reflect the key characteristics of research excellence (weightings for REF2014 in brackets):

  • The quality of research outputs (65%)
  • The reach and significance of the impact of research (20%)
  • The vitality of the research environment (15%)

Each of these elements will be assessed against appropriate criteria for excellence, and rated by expert panels on a five-point scale ranging from 4* (excellent, world-leading) to Unclassified.

The REF assessment period started on 1st January 2008 and the first REF submission will take place in Autumn 2013, with the results published in December 2014.

You can access the latest presentation about the REF, written by the REF team, here: REF slide pack Sep 2011

Check out the posts appearing on the Blog every day next week as part of REF Week!

Have you been involved with an event designed for the external community?

Guidance, Research assessment

Then we want to hear from you! 🙂

The University is currently compiling the data for the annual Higher Education – Business & Community Interaction survey (HE-BCI) due to be submitted to HESA in December.

We are asked to submit details of social, cultural and community events designed for the external community (to include both free and chargeable events) which took place between 1 August 2010 and 31 July 2011.

Event types that should be returned include:

  • public lectures
  • performance arts (dance, drama, music, etc)
  • exhibitions
  • museum education
  • events for schools and community groups
  • business breakfasts

We cannot return events such as open days, Student Union activity, commercial conferences, etc.

If you have been involved with an event which could be returned, please could you let your contact (see below) know the event name and date, whether it was free or chargeable, and the estimated number of attendees:

  • if you are in a School – your Director of Operations or Deputy Dean (R&E) / equivalent
  • if you are in a Professional Service – please contact Julie Northam in the Research Development Unit

The data returned is used by HEFCE to allocate the HEIF funding so it is important that we return as accurate a picture as possible.

Southern Universities REF event at BU – 22 February 2012 – SAVE THE DATE!

Research assessment, Research news

REF logoBU will be hosting a half day Research Excellence Framework (REF) event for southern universities, supported by the REF Team, on 22 February 2012 to which all staff are invited to attend.

This event follows hot on the heels of the first REF Team-supported event held at BU on 19 May 2011, to which over 150 delegates from 39 institutions attended (see our previous blog post – The excellent HEFCE REF event at BU!).

The sector-wide consultation on the proposed REF panel criteria closed earlier this month and the final documents are due to be published in January 2012. This event will provide an update on the current developments with the REF and the confirmed REF panel documentation, focusing specifically on the assessment of impact within each of the four Main Panels.

The event will be open to BU staff and external delegates and the provisional programme is shown below.

Provisional programme:

09:30 – 10:00    Coffee and registration

10:00 – 10:30    REF Team overview of the assessment framework

Chris Taylor, Deputy REF Project Manager, REF Team

10:30 – 11:00    Similarities between the 4 Main Panel criteria

Professor Stephen Holgate, Chair of Main Panel A

11:00 – 11:30    Morning break

11:30 – 12:15    Panel specifics, differences and impact assessment

There will be concurrent sessions, one for each of the four Main Panels. Attendees choose which one to attend.

Main Panel A:    Professor Stephen Holgate, Chair of Main Panel A

Main Panel B:    Professor Philip Nelson, Chair of sub-panel 15 (General Engineering)

Main Panel C:    Professor John Scott, Chair of sub-panel 23 (Sociology)

Main Panel D:    Professor Bruce Brown, Chair of Main Panel D

12:15 – 13:00    Panel Q&A session with all participants

13:00 – 14:00   Lunch, networking and close

 

The event is free to attend but booking is essential. Booking will open very soon – further details to follow!
 

REF update from the VC and PVC

Research assessment, Research news

In the latest Vice-Chancellor’s email, Prof John Vinney gave an update on the Research Excellence Framework (REF) and BU’s preparations for the exercise. The email also introduced this month’s VC video in which he and Prof Matthew Bennett discuss the REF in more detail – and what it means for BU – with Sue Eccles from the Media School.

You can watch the video here:

httpv://youtu.be/7s8RTlOOPnU.

Overview of the REF draft panel criteria – what are the subtleties between panels?

Guidance, Research assessment, Research news

At the end of July the REF team released the draft panel working methods and criteria documentation (see our previous blog post for access to the documents).

We’ve spent the week wading through the four main panel documents and have produced a very brief overview of the subtleties between the panels on key criteria (such as the use of citation data, co-authored outputs, additional environment data, etc) in a tabular format.

You can access the overview table here: REF – draft panel criteria comparison table

Unfortunately this is no substitute for reading the actual documentation (sorry!) but does highlight the key points and differences between panels.

These documents are currently open to sector-wide consultation until 5 October 2011. BU will be submitting a single institutional response coordinated by the Research Development Unit. BU staff are invited to submit feedback for consideration as part of this response. Please email all comments to Anita Somner by 20 September 2011.

Latest journal impact factors

BU research, Research assessment

Following the release of the latest Journal Citation Reports® on the Thomson Reuters ISI Web of Science database, we have compiled a list of the top ranking journals in various fields related to BU research. BU staff can access these lists by going to the designated folder on the collaborative I-drive: I:\R&KEO\Public\RDU\Journal Impact Factors 2011. If there are any additional subject areas that you would like to see included, do send me an email.

Related blog posts that may be of interest:
Journal impact factors explained

REF draft panel criteria is now available

Research assessment, Research news

Two important REF documents are now available on the REF website:

1. The draft panel criteria and working methods for consultation. A summary of this will be added to the blog next week.

2. The analysis of panel membership is now available.

Panel criteria and working methods consultation – the consultation runs from now until 5 October. BU will be submitting a single institutional response coordinated by the Research Development Unit. BU staff are invited to submit feedback for consideration as part of this response. Please email all comments to Anita Somner by 20 September 2011.

The REF Guidance on Submissions document was released on 14 June, a summary of which can be accessed on the I-drive from: I:\CRKT\Public\RDU\REF.

Continuing importance of ‘Impact’ highlighted by HEFCE and RCUK

Research assessment, Research news

RCUK logoThe Higher Education Funding Council for England (HEFCE), Research Councils UK (RCUK) and Universities UK (UUK) have issued a joint statement highlighting the continuing importance of the impact of research and making a commitment to developing common frameworks.  Both the REF (undertaken by HEFCE) and the Research Councils place a strong emphasis on the impact of research.

Their key aims are:

  • ensuring that their approaches to supporting knowledge exchange are joined up
  • securing greater impact from publicly funded research whilst ensuring their funding policies are complementary and work coherently together
  • ensuring that their monitoring, assessment and evaluation as well as the collection of statistical data from HEIs is proportionate.
  • supporting the training of new researchers, at both PhD and post-doctoral level, to ensure that they acquire the skills and expertise needed to maximise the impact of their own research.

The full statement is available on the RCUK website.

REF Guidance on Submissions document released

Guidance, Research assessment, Research news

The REF2014 Guidance on Submissions document was released on Thursday and can be accessed on the HEFCE website here: REF Guidance on Submissions

We have prepared a summary document of the key points that can be accessed on the I drive: I:\CRKT\Public\RDU\REF

At the end of July the REF team will publish the draft panel working methods and criteria documents which will be open to consultation until the autumn. The Research Development Unit will be coordinating the BU response to the consultation – further details will be available once the documentation is released.

Investigating academic impact at the London School of Economics: Blogs, Twitter and bumblebees!

BU research, Research assessment


The 'Current Thinking in Assessing Impact’ panel discussion during the LSE impact event on 12 June.

As someone who is still getting to grips with exactly how impact might be defined and operationalised for the REF, I went along to the Investigating Academic Impact Conference at LSE on the 12th June looking forward to learning more about precisely how we could create more effective impact case studies for the REF.  The day was opened by Patrick Dunleavy from the Impact of Social Sciences Project at LSE with the challenging statement that we needed to think about impact as a long-term, integral part of our research work and that simply trying to maximise impact for the REF was a short-term strategy.

What followed were sessions on how to use blogging, Wikipedia and Twitter to help enhance your electronic footprint and to engage with the public in new ways.  Following their own advice, all the presentations are now available, along with blogs and tweets, at http://blogs.lse.ac.uk/impactofsocialsciences/presentations/.  On the site there is a comprehensive (200-page) handbook detailing exactly how to increase your citations and how to achieve external impacts and, for those with a shorter attention span, there are some short how-to guides.  These include standard information about citation tools (such as ISI Web of Science and Scopus) as well as more esoteric measures of citation impact (such as the G-index and H scores).

There are also simple tips on how to get more widely read:

  • make sure your titles are informative
  • work on cross-disciplinary projects
  • build dissemination plans
  • have a distinctive name (many thanks to my parents on this one!).

The Impact of Social Sciences project at LSE has created a great resource which means that if you didn’t attend the day it doesn’t matter – the information is there for you to browse and look at anyway.

In the unexpected way that often happens at conferences, there are single pieces of information that are particularly memorable.  For this one it was the importance of the bumblebees!  At both the recent BU Research Impact Event and the LSE conference, one particular case study from the REF impact pilot exercise was singled out for particular praise. This was an elegant case study submitted by the University of Stirling on the conservation of bumblebees which was able to show tangible and far-reaching impact (for further details see http://www.hefce.ac.uk/research/ref/impact/ under Earth Systems and Environmental Sciences).  This is a great concrete example of how good impact case studies might be formulated and from which those still struggling with impact might be able to gain insights.

Perhaps one final message from the day was that, of course, if you wished to have academic impact then the best starting point of all is to have good research to talk about!

Siné McDougall

 

REF Impact Observations

Research assessment

Shortly we will learn the REF panel criteria and guidance submissions due to be published in July.  At BU we have engaged with some excellent initiatives such as the recent REF event to help focus our approach to the impact case studies. The measurement of benefit that UK research has on society is an opportunity to understand the value of our work. Already it is well documented that the concept of research impact has for the moment protected some sources of external funding at least for the next few years. As academics concerned with the REF we now need to consider delivering the evidence of our research impact from 2008 resulting from good BU research. This research could have happened almost as far back as 1993.

Impact is a difficult concept and not easy to measure in terms of the REF. From an engineering viewpoint impact is difficult to quantify as like the REF it involves knowledge of time in addition to the force. Engineering impact force increases as time decreases. This view of time relates to the general research impact debate. We now have the opportunity to look back and review BU research almost over the lifetime of the University.

Investigating Academic Impact event at LSE on 13 June

Research assessment, Research news

The LSE Public Policy Group is running a free one day event on evidencing the impact of research.

Date: Monday 13 June 2011 
Time: 10-5pm 
Venue:  New Academic Building, LSE, London

Academics are increasingly being pressed to provide evidence of impact from their research on the world outside academia. And universities will have to provide evidence of impact as part of the new Research Excellence Framework. But there is confusion about the different definitions of impact that exist amongst funding bodies and research councils, and also about methods of measuring impact.

This one day conference will look at a range of issues surrounding the impact of academic work on government, business, communities and public debate. We will discuss what impact is, how impacts happen and innovative ways that academics can communicate their work. Practical sessions will look at how academic work has impact among policymaking and business communities. Also how academic communication can be improved and how individual academics can easily start to asses their own impact.

PANELS:
Research Impact and the REF
Professor Rick Rylance (Chief Executive, Arts and Humanities Research Council)
David Sweeney (Director of Research, Innovation and Skills, HEFCE)
Professor Paul Wiles (Panel Chair, social work and social policy panel, REF impact pilot)

Current Thinking in Assessing Impact
Professor Patrick Dunleavy (Impact of Social Sciences project, London School of Economics)
Professor Alan Hughes (Centre for Business Research, University of Cambridge)
Tomas Ulrichsen (Public and Corporate Economic Associates)

Innovative Methods for Impact and Engagement
Professor Stephen Curry (blogger, Imperial College London)
Martyn Lawrence (Senior Publisher, Emerald Insight)
Paul Manners (Director, National Coordinating Centre for Public Engagement, UWE)
Mike Peel (Jodrell Bank Centre for Astrophysics / Wikimedia UK)

BREAKOUT SESSIONS:
Academic impact on policy-making
Maria O’Beirne (Analysis and Innovation Directorate, Department for Communities and Local Government)
Jill Rutter (Better Policy Making Programme Director, Institute for Government)

Knowledge transfer and the role of research mediators
Nick Pearce (Director, IPPR)
Professor Judy Sebba (University of Sussex)

Academic impacts on industry and business
James John (Director of Strategy, director of strategy, civil government, HP)

A ‘how to’ guide to measuring your own academic impact
Jane Tinkler (Impact of Social Sciences project, London School of Economics)

Improving academic communication
Professor Patrick Dunleavy (Impact of Social Sciences project, London School of Economics)
Chris Gilson (Managing Editor, British Politics and Policy blog, London School of Economics)

This event is free and open to all but pre-registration is required. For more information phone and email the PPG team on 020 7955 6064 or 020 7955 6731 or by email on impactofsocialsciences@lse.ac.uk|. You can find more information on the Investigating Academic Impact website.

BU Research Impact event is a success!

Research assessment, Research news

Last Friday BU held an internal Research Impact event to share the success of the excellent research that has been undertaken by BU academics. The focus of the event was on how this research has had an impact outside of academia, for example an impact on society, the economy, quality of life, culture, policy, etc.

REF logoFor the forthcoming REF2014 BU will be required to include a number of research impact case studies as part of the submission. This is a new element to the REF (previously the RAE) and the HE sector has been grappling with the concept of impact for a number of years now.

The event, attended by over 75 BU staff, opened with a presentation from Prof Matthew Bennett (Pro Vice Chancellor – Research, Enterprise and Internationalisation) on BU’s future research strategy, planning for the REF, and how to develop and evidence research impact.

Part of the presentation focused on the BU Research Themes which are currently being identified and defined through academic consultation via the Research Blog. This is still in the early stages but Matthew presented the ten draft themes that are emerging. You can comment on the emerging themes here.

There were 35 impact case studies presented in total with most units of assessment (UOAs) presenting three case studies. At the end of each presentation members of the audience critiqued the case study and offered advice as to how the strengthen and maximise the impact claim.

Attendees were encouraged to go to impact case study presentations from different UOAs/Schools to find out about research that is undertaken in different areas of the University. Stronger impact case studies can also be developed with input from different disciplines.

The event was also attended by key staff from Marketing & Communications who will be working with UOA Leaders to develop and enhance impact case studies between now and the REF submission in autumn 2013.

There has been much positive feedback received from attendees and we are considering whether this should now be an annual event, celebrating the success of BU research and its benefit to society.

Many thanks to all the presenters and attendees, and everyone who supported the event and made it such a success! 😀

We are now seeking feedback on the impact case studies presented. These are all available on the I-drive (I:\CRKT\Public\RDU\REF\REF event May 2011\impact case study presentations). Please could you email your feedback to Anita Somner in the Research Development Unit by Friday 3 June. Anita will then anonymise and collate the feedback and share it with the UOA Leaders.

For further information on impact see the impact pages on the HEFCE website or our previous BU Research Blog posts on impact.

The excellent HEFCE REF event at BU!

Research assessment, Research news

Developing and Assessing Impact for the REF

Last week BU hosted a HEFCE-supported event for universities in the south of England outlining recent changes in how the quality of research in higher education is assessed.

The event, attended by over 150 delegates from 39 institutions, outlined the new Research Excellence Framework (REF) which includes a new assessment element focusing on research impact.

As Chris Taylor, Deputy REF Manager for HEFCE, explained: “REF will provide accountability for public investment in research and demonstrate its benefits.” He continued:

“Impact is defined as any contribution the research makes outside of academia. It is the higher education sector’s opportunity to shout about what it contributes to society.”

Professor Peter Taylor-Gooby (University of Kent), Professor Roy Harrison (University of Birmingham), Professor James Goodwin (Age UK), Dr Kathryn Monk (Environment Agency Wales) and Dr Mari Williams (RCUK) presented their experiences of assessing impact case studies in the REF pilot exercise. Professor Jim Griffiths (University of Plymouth) presented his experience of identifying and submitting impact case studies to the pilot exercise in the hope that others would learn from his experience.

Prevalent themes emerging from the pilot included the importance of a demonstrable chain of evidence from impact claim through to outcome, high quality research underpinning the impact claim and fostering the crucial relationship between academic and user.

Professor James Goodwin explained how research can change society for people’s benefit, stessing the importance of “converting research into a message that will influence people’s thinking”. He gave the recent removal of the default retirement age as an example of how this can influence policy.

The event closed with a Q&A session with all speakers, giving delegates the chance to obtain further clarity on the REF that will undoubtedly change the future of higher education research.

Matthew Bennett (BU’s PVC for Research, Enterprise and Internationalisation) said: “There has been sector-wide concern about how impact will be defined, collated and assessed in the REF, and this event provided excellent advice and guidance for academic staff likely to be submitting to the REF and those leading the submissions.”

The deadline for submitting submissions is November 2013 and the assesment will be made in 2014.

We will be adding further posts to the Research Blog focusing on the good practice shared at the event (such as defining impact, what makes a strong impact case study, etc) over the next few weeks.

REF event 19 & 20 May 2011 – REGISTRATION IS OPEN!!

BU research, Research assessment

REF logoBU is hosting a two-day REF event on Thursday 19 and Friday 20 May 2011. All staff are invited to attend.

The event is of interest to BU academic staff and anyone who will be involved in the BU submission to the REF.

There will be three separate sessions:

Session 1
Thursday 19 May 9am-2pm
This session will be open to BU staff and external delegates.
There will be presentations from the REF team at HEFCE, REF impact pilot panel members, and a REF impact pilot institution (University of Plymouth).

Session 2
Thursday 19 May 2pm-5pm
This session is only open to BU staff.
This session will provide BU staff with the opportunity for internal networking, followed by a demonstration of BU’s new publications management system and a presentation on preparing a publication profile for the REF.

Session 3
Friday 20 May 9:45am-4:30pm
This session is only open to BU staff.
The focus of this session is the development of the BU impact case studies. There will be presentations of the impact case studies being developed at the moment.

All sessions will take place in Kimmeridge House and Poole House, Talbot Campus.

You must register separately for each session you will be attending.

See our previous REF Event blog post for further details. The provisional programmes are available on the registration forms (see links above).

REF event 19 & 20 May 2011 – SAVE THE DATE!

BU research, Research assessment

REF logoBU will be holding a two day Research Excellence Framework (REF) event on 19 and 20 May to which all staff are invited to attend.

Day 1 (open to BU staff and external delegates)
9am-2pm – this will be an external event supported by HEFCE to which all HEIs in the South of England will be invited. The focus will be on developing and assessing impact for the REF. There will be speakers from HEFCE, an academic from one of the impact pilot institutions (University of Plymouth), and some of the impact pilot panel members. The event is aimed primarily at academics likely to be submitted to the REF and UOA Leaders. It will provide a forum for networking and discussion around preparations for the impact element of the REF.

Day 1 (open to BU staff only)
2pm-5pm – There will be an opportunity for internal networking, a demonstration of the publications management system BU will soon be implementing, and a talk by Prof Matthew Bennett on building a publication profile for the REF.

Day 2 (open to BU staff only)
9am-4:30pm – the focus of Day 2 is the development of the BU impact case studies. The day will open with a presentation by Prof Matthew Bennett on what impact actually is, followed by presentations of the impact case studies being developed at the moment (3 per Unit of Assessment). These will run in 9 concurrent sessions with 4 presentations taking place in parallel during each session. The main aims of Day 2 are to get academics thinking about the impact case studies in a structured way, to identify resource requirements to maximise potential impacts, and the engage staff from M&C with the case studies being developed. In addition this is a great opportunity to showcase the excellent research that is undertaken at BU, to meet colleagues from other Schools, and to stimulate ideas for future research collaborations.

The event is free to attend but booking is essential. Booking will open next week – further details to follow!

REF Highlight Report #8

Research assessment

REF logoThe latest REF Highlight Report is now available from the Research Intranet.

Key points include updates on:

  • progress with the UOA Action Plans
  • the REF two-day event to be held at BU in May (19th/20th)
  • the second mock exercises for UOAs 7 and 26
  • the RASG and RALT meetings held in March

You can access the full document from here: REF Highlight Report #8