Tagged / research assessment

BU REF Code of Practice available now!

Research assessment

As I mentioned in my previous post about the forthcoming outputs mock REF exercise, we have recently submitted our final draft REF Code of Practice document to the REF Team for their approval. We produced this document to help inform staff about the policies and processes around submitting to the REF in November 2013, as well as to meet the four UK higher education funding bodies’ request that each institution making a submission to the REF must develop and apply such a code when selecting staff to include in their submission.

The BU REF Code of Practice is the culmination of months of drafting and approval, and sets out the approach that will be, and has already been, taken by BU in preparing its REF2014 submission. It therefore includes important information about the schedule of mock REF exercises that you are able to take part in (Chapter 3), and about declaring any circumstances that may have affected your ability to produce the four outputs required (Chapter 5), along with other equality and diversity aspects. Staff eligibility and selection are also covered.

Throughout the document there are references to the relevant paragraphs within the key publications published by the REF Team, which are: Assessment Framework and Guidance on Submissions and Panel Criteria and Working Methods.

Because this document is still subject to the approval of the REF Team’s Equality and Diversity Advisory Panel (EDAP), the Code is currently available to BU staff via the new Staff Intranet under ‘Policies, Forms and Procedures/Research’. We will make it fully available here on the Blog once we’ve received the EDAP feedback, so watch this space!

Limber up for the next BU mock REF exercise

Research assessment

Just as the Olympic athletes are getting set for the 2012 games, so are we gearing up to launch the summer 2012 mock REF exercise for BU academic staff, this time looking at research outputs. This follows on from our previous light-touch review exercise, which was open to all academic staff and took place over the winter of 2010/2011.

This latest exercise will be the fourth in a series of different preparation exercises that have been held or are due to take place at BU ahead of the REF2014 submission deadline on 29 November 2013. Just as the athletes will have taken part in warm-up events and qualifiers to test their fitness and shine before selectors, so the mock REF exercises are a useful ‘dry run’ to give you a feel for what’s going to be required for the real thing. Equally, you’ll get valuable feedback on your outputs from external reviewers and your UOA Leader, and it will enable us to test out our data collection processes.

As in the light touch review, the outputs mock will be open to all academic staff. The relevant form will be distributed via your UOA leader in due course so it would be a good time to start thinking about which outputs you’re likely to put forward. Although REF2014 requires a maximum of four outputs, we’re giving you the opportunity to get feedback from the reviewers on up to six outputs, so make the most of this chance to really shape your outputs submission.

Also under starter’s orders – the BU REF Code of Practice will shortly be circulated to all staff, which outlines the timetable of REF preparation exercises and includes a host of other useful information related to how BU is preparing for REF2014. The final draft of this document has just been submitted to the REF Team for formal approval so watch this space for more information about this.

If you need any more information about the REF, have a look at all the previous blog posts that we’ve included here, or visit the new-look REF website. Alternatively, you can contact myself or Julie Northam in the Research Development Unit, or leave a comment below.

REF Panel Criteria summaries by UOA

Guidance, Research assessment

The final REF Panel Criteria and Working Methods documentation was released by the REF Team (based at HEFCE) at the end of January.

As a handy guide to the information the RDU has summarised the headline information for the UOA in the documents listed below:

Allied Health Professions, Dentistry, Nursing and Pharmacy – UOA 3 panel criteria summary

Psychology, Psychiatry and Neuroscience – UOA 4 panel criteria summary

Earth Systems and Environmental Sciences – UOA 7 panel criteria summary

Computer Science and Informatics – UOA 11 panel criteria summary

General Engineering – UOA 15 panel criteria summary

Geography, Environmental Studies and Archaeology – UOA 17 panel criteria summary

Business and Management Studies – UOA 19 panel criteria summary

Law – UOA 20 panel criteria summary

Social Work and Social Policy – UOA 22 panel criteria summary

Sports and Exerise Sciences, Leisure and Tourism – UOA 26 panel criteria summary

Art and Design – UOA 34 panel criteria summary

Communication, Cultural and Media Studies – UOA 36 panel criteria summary

Happy reading! If you have any questions about the REF then please do let me know.

The second brilliant external REF event at BU!

Research assessment

On Wednesday this week BU hosted a REF Team-supported event for universities in the south of England explaining the content of the recently released REF Panel Working Methods and Criteria documentation. This was the second REF event that has been hosted at BU in the past 12 months. The first event was held in May 2011 and you can read about it here: http://blogs.bournemouth.ac.uk/research/2011/05/25/the-excellent-hefce-ref-event-at-bu/

The event, attended by over 150 delegates from 32 institutions, outlined the similarities between how the four Main Panels will review submissions, as well as focusing on the differences between the panels particularly in how they will define and assess impact in the context of the REF.

Chris Taylor, Deputy REF Manager, spoke about the technical aspects of the REF, e.g. the timetable, element weightings, and institutional REF codes of practice, and then looked at each of the three elements of the REF in depth – impact, outputs and environment.

Prof Stephen Holgate, Chair of Main Panel A, then delivered a very interesting presentation on the similarities between the four Main Panels which have been vastly improved since the sector-wide consultation on the Panel Working Methods documentation last autumn. The Panels have put in a significant amount of work to ensuring their working methods will be as simple, transparent and similar as possible which is excellent news.

After break there were four concurrent sessions, each focusing on one of the Main Panels. Event attendees could choose to attend one session. The session were led by:

  • Main Panel A – Prof Stephen Holgate (Chair of Main Panel A)
  • Main Panel B – Prof Philip Nelson (Chair of the General Engineering sub-panel)
  • Main Panel C – Prof John Scott (Chair of the Sociology sub-panel)
  • Main Panel D – Prof Bruce Brown (Chair of Main Panel D)

After the concurrent sessions, all presenters took part in a Q&A session back in Kimmeridge House.

One of the key messages of the day was that the sub-panels will not make use of journal impact factors, journal ranking lists, or other journal scoring information to inform the review of outputs. Citation data will be provided by the REF Team to sub-panels:

  • Main Panel A: Sub-panels 1-6
  • Main Panel B: Sub-panels 7-11
  • Main Panel C: Sub-panel 18

Research collaboration (e.g. links with other institutions, business and industry, international collaboration, etc) was also highlighted at numerous points throughout the event as being of particular importance in the environment element of the assessment.

Regarding impact, Prof Holgate stressed that the assessment of impact was not necessarily linked to the size of the population affected but to the reach and significance of the impact – for example, a 4* impact case study could be for a drug that cured three people or 3 billion people.

Also interesting was the focus on 4* research being that which is transformative research and that this could be the synthesis of knowledge and the identification of a new way of doing things. A review paper could therefore be assessed as 4* if it meets this definition. Prof Holgate remarked: “we are in an era of transformation. We want game changing outputs to be submitted to the REF”.

The event was closed by Prof Matthew Bennett at 1pm after which point event attendees networked over lunch. Feedback from attendees so far has been very positive!

If you attended the session then we’d love to know what you thought! Let us know by adding a comment to this post.

The slides will be available shortly via the Blog.

REF draft code of practice – BU academic staff comments invited

Research assessment

Over the past 18-24 months we have been working on a code of practice to set out the approach that is being taken at BU in preparing our REF submission. The document has largely been authored by Anita Somner, Judith Wilson, Dr James Palfreman-Kay and myself with input from Prof Kate Galvin, and has been approved by the REF Academic Steering Group and REF Academic Leadership Team. It is a requirement for all institutions submitting to the REF to have a code of practice in place which has been approved by the REF Equality and Diversity Advisory Panel (EDAP), and the initial deadline for submitting draft codes to the EDAP is midday on 27 April 2012.

We are now inviting all academic staff to comment on this version of the BU REF code of practice. Comments received will help to refine the document in advance of the final version being sent by the VC to the EDAP at the end of April. A feedback form is available for completion by academic staff wishing to comment on the draft document.

The draft code of practice and the feedback form are available from the I-drive: I:\CRKT\Public\RDU\REF\COP feedback

Responses should be emailed to Anita Somner by 5pm on Friday 16 March 2012.

I am happy to discuss the document with colleagues and/or to meet as appropriate. If you have any queries, please do let me know.

Thanks

Julie

Southern Universities REF event at BU – 22 February 2012 – book your place NOW!

Research assessment

REF logo
BU is hosting a 1/2 day REF event on Wednesday 22 February 2012. All staff are invited to attend.
 
The final panel working methods and criteria documents are due to be published in January 2012. This event will provide an update on the current developments with the REF and the confirmed REF panel documentation, focusing specifically on the assessment of impact within each of the four Main Panels.
 
Each of the REF Main Panels will be represented. If you have any questions about the REF, how research will be assessed and graded, or how impact will be assessed then you should attend this event! 😀
 
The event is open to BU staff and external delegates. There are already 130 delegates registered to attend, representing 32 different universities.

The event is free to attend but booking is essential.

For further information (including the programme) and to register, visit HERE.

Southern Universities REF event at BU – 22 February 2012 – book your place NOW!

Research assessment

BU will be hosting a half day Research Excellence Framework (REF) event, supported by the REF team, on 22 February 2012 to which all staff are invited to attend.

Book your place now by completing the online registration form 

This event follows hot on the heels of the first REF event held at BU on 19 May 2011, to which over 150 delegates from 39 institutions attended (see our previous blog post – The excellent REF event at BU!).

The sector-wide consultation on the proposed REF panel criteria closed earlier this month and the final documents are due to be published in January 2012. This event will provide an update on the current developments with the REF and the confirmed REF panel documentation, focusing specifically on the assessment of impact within each of the four Main Panels.

The event will be open to BU staff and external delegates and the provisional programme is shown below.

Time Activity
09:30 – 10:00 Coffee and registration
10:00 – 10:30  REF Team overview of the assessment framework
Chris Taylor, Deputy REF Project Manager, REF Team
10:30 – 11:00 Similarities between the four Main Panel working methods and criteria
Professor Stephen Holgate, Chair of Main Panel A
11:00 – 11:30 Morning break
11:30 – 12:15 Panel specifics, differences and impact assessment
There will be concurrent sessions, one for each of the four Main Panels. Attendees choose which one to attend.

  • Main Panel A:   Professor Stephen Holgate, Chair of Main Panel A
  • Main Panel B:   Professor Philip Nelson, Chair of sub-panel 15 (General Engineering)
  • Main Panel C:   Professor John Scott, Chair of sub-panel 23 (Sociology)
  • Main Panel D:   Professor Bruce Brown, Chair of Main Panel D
12:15 – 13:00 Panel Q&A session with all participants
13:00 – 14:00 Lunch, networking and close

The event is free to attend but booking is essential.

Book your place now by completing the online registration form 

REF week on the Blog! BU REF preparation and governance

Research assessment

This week is REF Week on the Blog! Each day we will be explaining a different element of the Research Excellence Framework (REF) as a quick reference guide to help you prepare for the forthcoming REF exercise – REF2014.

Internal preparations for the BU submission to the REF have already started and you can keep up to date by reading the BU REF Highlight Reports. These are published regularly and detail all of the recent developments (internal and external) with the REF (see the link to the I-drive at the end of the post).

Two internal groups have been established to oversee preparations for the REF:

RASG – The REF Academic Steering Group (RASG) has been established as the primary decision-making body for the BU REF preparations. It first met on 14 June 2010 and normally meets on a monthly basis. The RASG Terms of Reference (including a list of members) can be accessed via the I-drive.

RALT – The REF Academic Leadership Team (RALT). RALT met for the first time on 27 September 2010 and will meet when required (currently this is on a monthly basis). Members of RALT include all of the RASG members plus the REF UOA Leaders. The list of REF UOA Leaders and the RALT Terms of Reference can be access on the I-drive.

BU is currently working on a REF Code of Practice and some FAQs around REF preparation, staff eligibility, staff selection, etc., which will be available from the BU Research Blog in due course.

The Research Development Unit (RDU) are managing and administering the internal REF process. If you have any questions then please do ask Anita Somner or Julie Northam.

You can access all of this information regarding the BU preparations for the REF via the I-drive: I:\CRKT\Public\RDU\REF

You can access the latest presentation about the REF, written by the REF team, here: REF slide pack Sep 2011

Check out the posts appearing on the Blog every day next week as part of REF Week!

REF week on the Blog! What were the HEFCE REF pilots?

Research assessment

This week is REF Week on the Blog! Each day we will be explaining a different element of the Research Excellence Framework (REF) as a quick reference guide to help you prepare for the forthcoming REF exercise – REF2014.

What were the HEFCE REF pilots? – HEFCE ran two pilot exercises with HEIs in the sector during the development of the REF. The first exercise was a bibliometrics pilot, and the second was an impact pilot.

Bibliometrics pilot – HEFCE ran a pilot exercise in the construction of bibliometric indicators of research quality in 2008-09, using Scopus and the Web of Scienceas the test databases. BU was chosen as one of 22 institutions to be part of phase one of the pilot exercise. This involved the provision of publication details to HEFCE, and cross-checking BU information on Web of Science and Scopus. Where possible this was completed using BU’s institutional repository, BURO. The outcome of the bibliometrics pilot was that bibliometric indicators are not yet sufficiently robust enough in all disciplines to be used formulaically or as a primary indicator of research quality. However HEFCE agreed that there was scope for bibliometrics to inform the process of expert review in some units of assessment. These findings resulted in the decision that some UOA sub-panels will receive citation data (the number of times an output has been cited, calculate via Scopus) as additional information about the academic significance of the outputs.

Impact pilot – During 2009-10, HEFCE ran a second pilot exercise, this time with the aim of developing proposals for how to assess research impact in the REF. The impact pilot involved 29 HEIs submitting evidence of impact (case studies and statements) which were assessed by pilot expert panels in five units of assessment:

  • Clinical Medicine
  • Physics
  • Earth Systems and Environmental Sciences
  • Social Work and Social Policy & Administration
  • English Language and Literature

The impact pilot completed in autumn 2010 and the final report (including recommendations and findings) was published on 11 November 2010. The full report can be accessed on the HEFCE website. For a brief summary of the report, please download the Impact Pilot Summary. You can also read our REF Impact FAQs.

You can access the latest presentation about the REF, written by the REF team, here: REF slide pack Sep 2011

Check out the posts appearing on the Blog every day this week as part of REF Week!

REF week on the Blog! Introduction to the Research Excellence Framework

Research assessment

Next week is REF Week on the Blog! Each day we will be explaining a different element of the Research Excellence Framework (REF) as a quick reference guide to help you prepare for the forthcoming REF exercise – REF2014.

What is the REF? – The Research Excellence Framework (REF) has replaced the Research Assessment Exercise (RAE) as the new process for assessing the volume and quality of research in UK HEIs. As with the RAE, the results of the REF will determine the annual quality-related research (QR) grant distributed from HEFCE to HEIs in England.

The REF will assess research excellence through a process of expert review, informed by indicators where appropriate. It will be based on HEIs submitting evidence of their research activity and outcomes, to be assessed by expert panels.

The REF will focus on assessing three elements, which together reflect the key characteristics of research excellence (weightings for REF2014 in brackets):

  • The quality of research outputs (65%)
  • The reach and significance of the impact of research (20%)
  • The vitality of the research environment (15%)

Each of these elements will be assessed against appropriate criteria for excellence, and rated by expert panels on a five-point scale ranging from 4* (excellent, world-leading) to Unclassified.

The REF assessment period started on 1st January 2008 and the first REF submission will take place in Autumn 2013, with the results published in December 2014.

You can access the latest presentation about the REF, written by the REF team, here: REF slide pack Sep 2011

Check out the posts appearing on the Blog every day next week as part of REF Week!

Southern Universities REF event at BU – 22 February 2012 – SAVE THE DATE!

Research assessment

REF logoBU will be hosting a half day Research Excellence Framework (REF) event for southern universities, supported by the REF Team, on 22 February 2012 to which all staff are invited to attend.

This event follows hot on the heels of the first REF Team-supported event held at BU on 19 May 2011, to which over 150 delegates from 39 institutions attended (see our previous blog post – The excellent HEFCE REF event at BU!).

The sector-wide consultation on the proposed REF panel criteria closed earlier this month and the final documents are due to be published in January 2012. This event will provide an update on the current developments with the REF and the confirmed REF panel documentation, focusing specifically on the assessment of impact within each of the four Main Panels.

The event will be open to BU staff and external delegates and the provisional programme is shown below.

Provisional programme:

09:30 – 10:00    Coffee and registration

10:00 – 10:30    REF Team overview of the assessment framework

Chris Taylor, Deputy REF Project Manager, REF Team

10:30 – 11:00    Similarities between the 4 Main Panel criteria

Professor Stephen Holgate, Chair of Main Panel A

11:00 – 11:30    Morning break

11:30 – 12:15    Panel specifics, differences and impact assessment

There will be concurrent sessions, one for each of the four Main Panels. Attendees choose which one to attend.

Main Panel A:    Professor Stephen Holgate, Chair of Main Panel A

Main Panel B:    Professor Philip Nelson, Chair of sub-panel 15 (General Engineering)

Main Panel C:    Professor John Scott, Chair of sub-panel 23 (Sociology)

Main Panel D:    Professor Bruce Brown, Chair of Main Panel D

12:15 – 13:00    Panel Q&A session with all participants

13:00 – 14:00   Lunch, networking and close

 

The event is free to attend but booking is essential. Booking will open very soon – further details to follow!
 

Overview of the REF draft panel criteria – what are the subtleties between panels?

Guidance, Research assessment

At the end of July the REF team released the draft panel working methods and criteria documentation (see our previous blog post for access to the documents).

We’ve spent the week wading through the four main panel documents and have produced a very brief overview of the subtleties between the panels on key criteria (such as the use of citation data, co-authored outputs, additional environment data, etc) in a tabular format.

You can access the overview table here: REF – draft panel criteria comparison table

Unfortunately this is no substitute for reading the actual documentation (sorry!) but does highlight the key points and differences between panels.

These documents are currently open to sector-wide consultation until 5 October 2011. BU will be submitting a single institutional response coordinated by the Research Development Unit. BU staff are invited to submit feedback for consideration as part of this response. Please email all comments to Anita Somner by 20 September 2011.

REF draft panel criteria is now available

Research assessment

Two important REF documents are now available on the REF website:

1. The draft panel criteria and working methods for consultation. A summary of this will be added to the blog next week.

2. The analysis of panel membership is now available.

Panel criteria and working methods consultation – the consultation runs from now until 5 October. BU will be submitting a single institutional response coordinated by the Research Development Unit. BU staff are invited to submit feedback for consideration as part of this response. Please email all comments to Anita Somner by 20 September 2011.

The REF Guidance on Submissions document was released on 14 June, a summary of which can be accessed on the I-drive from: I:\CRKT\Public\RDU\REF.

REF Guidance on Submissions document released

Guidance, Research assessment

The REF2014 Guidance on Submissions document was released on Thursday and can be accessed on the HEFCE website here: REF Guidance on Submissions

We have prepared a summary document of the key points that can be accessed on the I drive: I:\CRKT\Public\RDU\REF

At the end of July the REF team will publish the draft panel working methods and criteria documents which will be open to consultation until the autumn. The Research Development Unit will be coordinating the BU response to the consultation – further details will be available once the documentation is released.

Investigating Academic Impact event at LSE on 13 June

Research assessment

The LSE Public Policy Group is running a free one day event on evidencing the impact of research.

Date: Monday 13 June 2011 
Time: 10-5pm 
Venue:  New Academic Building, LSE, London

Academics are increasingly being pressed to provide evidence of impact from their research on the world outside academia. And universities will have to provide evidence of impact as part of the new Research Excellence Framework. But there is confusion about the different definitions of impact that exist amongst funding bodies and research councils, and also about methods of measuring impact.

This one day conference will look at a range of issues surrounding the impact of academic work on government, business, communities and public debate. We will discuss what impact is, how impacts happen and innovative ways that academics can communicate their work. Practical sessions will look at how academic work has impact among policymaking and business communities. Also how academic communication can be improved and how individual academics can easily start to asses their own impact.

PANELS:
Research Impact and the REF
Professor Rick Rylance (Chief Executive, Arts and Humanities Research Council)
David Sweeney (Director of Research, Innovation and Skills, HEFCE)
Professor Paul Wiles (Panel Chair, social work and social policy panel, REF impact pilot)

Current Thinking in Assessing Impact
Professor Patrick Dunleavy (Impact of Social Sciences project, London School of Economics)
Professor Alan Hughes (Centre for Business Research, University of Cambridge)
Tomas Ulrichsen (Public and Corporate Economic Associates)

Innovative Methods for Impact and Engagement
Professor Stephen Curry (blogger, Imperial College London)
Martyn Lawrence (Senior Publisher, Emerald Insight)
Paul Manners (Director, National Coordinating Centre for Public Engagement, UWE)
Mike Peel (Jodrell Bank Centre for Astrophysics / Wikimedia UK)

BREAKOUT SESSIONS:
Academic impact on policy-making
Maria O’Beirne (Analysis and Innovation Directorate, Department for Communities and Local Government)
Jill Rutter (Better Policy Making Programme Director, Institute for Government)

Knowledge transfer and the role of research mediators
Nick Pearce (Director, IPPR)
Professor Judy Sebba (University of Sussex)

Academic impacts on industry and business
James John (Director of Strategy, director of strategy, civil government, HP)

A ‘how to’ guide to measuring your own academic impact
Jane Tinkler (Impact of Social Sciences project, London School of Economics)

Improving academic communication
Professor Patrick Dunleavy (Impact of Social Sciences project, London School of Economics)
Chris Gilson (Managing Editor, British Politics and Policy blog, London School of Economics)

This event is free and open to all but pre-registration is required. For more information phone and email the PPG team on 020 7955 6064 or 020 7955 6731 or by email on impactofsocialsciences@lse.ac.uk|. You can find more information on the Investigating Academic Impact website.

The excellent HEFCE REF event at BU!

Research assessment

Developing and Assessing Impact for the REF

Last week BU hosted a HEFCE-supported event for universities in the south of England outlining recent changes in how the quality of research in higher education is assessed.

The event, attended by over 150 delegates from 39 institutions, outlined the new Research Excellence Framework (REF) which includes a new assessment element focusing on research impact.

As Chris Taylor, Deputy REF Manager for HEFCE, explained: “REF will provide accountability for public investment in research and demonstrate its benefits.” He continued:

“Impact is defined as any contribution the research makes outside of academia. It is the higher education sector’s opportunity to shout about what it contributes to society.”

Professor Peter Taylor-Gooby (University of Kent), Professor Roy Harrison (University of Birmingham), Professor James Goodwin (Age UK), Dr Kathryn Monk (Environment Agency Wales) and Dr Mari Williams (RCUK) presented their experiences of assessing impact case studies in the REF pilot exercise. Professor Jim Griffiths (University of Plymouth) presented his experience of identifying and submitting impact case studies to the pilot exercise in the hope that others would learn from his experience.

Prevalent themes emerging from the pilot included the importance of a demonstrable chain of evidence from impact claim through to outcome, high quality research underpinning the impact claim and fostering the crucial relationship between academic and user.

Professor James Goodwin explained how research can change society for people’s benefit, stessing the importance of “converting research into a message that will influence people’s thinking”. He gave the recent removal of the default retirement age as an example of how this can influence policy.

The event closed with a Q&A session with all speakers, giving delegates the chance to obtain further clarity on the REF that will undoubtedly change the future of higher education research.

Matthew Bennett (BU’s PVC for Research, Enterprise and Internationalisation) said: “There has been sector-wide concern about how impact will be defined, collated and assessed in the REF, and this event provided excellent advice and guidance for academic staff likely to be submitting to the REF and those leading the submissions.”

The deadline for submitting submissions is November 2013 and the assesment will be made in 2014.

We will be adding further posts to the Research Blog focusing on the good practice shared at the event (such as defining impact, what makes a strong impact case study, etc) over the next few weeks.