Tagged / impact

REF Highlight Report #10

Research assessment

REF logoThe latest REF Highlight Report is now available from the Research Blog.

Key points include updates on:

  • progress towards the Winter 2011 REF preparation exercise
  • publication of the final REF guidance on submissions in July 2011
  • the consultation on the panel working methods and criteria
  • the appointment of Sally Gates as the REF Communications Manager
  • the RASG and RALT meetings held between May and October

You can access the full document from the I-drive via this link: REF Highlight Report #10

If you are accessing the report from off-campus then you will need to locate the following folder on the I-drive: CRKT\Public\RDU\REF\REF preparations\REF highlight reports\#10

Southern Universities REF event at BU – 22 February 2012 – book your place NOW!

Research assessment

BU will be hosting a half day Research Excellence Framework (REF) event, supported by the REF team, on 22 February 2012 to which all staff are invited to attend.

Book your place now by completing the online registration form 

This event follows hot on the heels of the first REF event held at BU on 19 May 2011, to which over 150 delegates from 39 institutions attended (see our previous blog post – The excellent REF event at BU!).

The sector-wide consultation on the proposed REF panel criteria closed earlier this month and the final documents are due to be published in January 2012. This event will provide an update on the current developments with the REF and the confirmed REF panel documentation, focusing specifically on the assessment of impact within each of the four Main Panels.

The event will be open to BU staff and external delegates and the provisional programme is shown below.

Time Activity
09:30 – 10:00 Coffee and registration
10:00 – 10:30  REF Team overview of the assessment framework
Chris Taylor, Deputy REF Project Manager, REF Team
10:30 – 11:00 Similarities between the four Main Panel working methods and criteria
Professor Stephen Holgate, Chair of Main Panel A
11:00 – 11:30 Morning break
11:30 – 12:15 Panel specifics, differences and impact assessment
There will be concurrent sessions, one for each of the four Main Panels. Attendees choose which one to attend.

  • Main Panel A:   Professor Stephen Holgate, Chair of Main Panel A
  • Main Panel B:   Professor Philip Nelson, Chair of sub-panel 15 (General Engineering)
  • Main Panel C:   Professor John Scott, Chair of sub-panel 23 (Sociology)
  • Main Panel D:   Professor Bruce Brown, Chair of Main Panel D
12:15 – 13:00 Panel Q&A session with all participants
13:00 – 14:00 Lunch, networking and close

The event is free to attend but booking is essential.

Book your place now by completing the online registration form 

REF week on the blog! A Ramble about REF

BU research, Research assessment

It’s REF week on the Research Blog and I also have to give a talk to the BU Board on Thursday about progress with our REF preparations, but I am sitting here wondering what to write?  Does this often happen to you?  I often put these things off and turn my hand to something else, like the paper I am currently struggling to complete, or keeping up with my email correspondence rather than tackle the task in hand.  But you see, here am I avoiding starting on the piece again, so I had better get started before it gets any later.

For the Board presentation I am taking a historical view of REF and its ancestors.  It started out in 1985 as the Research Selectivity Exercise, before progressing in its third iteration to the Research Assessment Exercise (RAE) in 1992 which was the first time research funds (QR – Quality Research Income) was distributed as a consequence of the outcome.  In some ways it was only in 1997 that universities had really got their act together and begun to take the RAE seriously, and by 2001 it was a major focus of energy in higher education and was beginning to result in a progressive concentration of research funding in a few key institutions.  BU’s greatest success to date was in 2008 when we were the fourth most improved university in the country, and for the first time BU started to receive significant QR income as a result. 

Apart from dominating the lives of many researchers, you may well ask what it has done for UK research.  Well, the answer is actually a huge ton!  In the 1970s, research in UK universities was funded via a government block grant and the UK was a middle-ranking research power, complacent, inefficient and underperforming.  According to the recent BIS survey with the catchy title International Comparative Performance of the UK Research Base – 2011 the UK is now a leading research nation in the world, second only to the US.  The much quoted headlines run something like: 1% of the world’s population, 3% of R&D spend, 4% of researchers, but 6% of articles, 10% of citations, 14% of the most cited articles.  According to HEFCE, for every pound spent on research in the UK you get between four and seven pounds back.  When seen in this context, the RAE has done its job extremely well by introducing competition into the sector.  There are parallels here with the current move to introduce competition around student numbers.

Since RAE2008, goal posts have changed again as the name has changed to the Research Excellence Framework (REF) with the introduction of ‘Impact’ as an element of the assessment.  In 2014, it will contribute 20% of the overall profile that a Unit of Assessment (UOA) will receive.  At its simplest, impact is about justifying research spending from the public purse by demonstrating the societal benefit – economic, environmental, social, or cultural – from research.  It is a great concept and speaks to the heart of societal relevance which we are placing at the centre of BU’s future research strategy.  It will be assessed as part of REF via a series of case studies and each case study has to be based on a piece or body of research undertaken in the last 15 years, with an evidenced impact since 2008.  The basic idea is that impact often takes time to come to fruition, but for a youthful institution like our own this is challenging since the research belongs to the university where it was done, not to the researcher. In the last 5 years there has been a steady influx of talented researchers to BU, but in many cases their impact belongs to another university!  

The need to evidence societal benefit is also important – it’s not enough just to have changed government policy for example; one needs to demonstrate the benefit of that change to ordinary people.  The example we often use is that of seatbelts.  Professor X does some research into seat belts and convinces government to legislate with respect to their introduction.  This only counts, however, as interim impact – to complete the case study, one would have to demonstrate how that legislation has reduced road traffic accidents.  So evidencing one’s claim is critical.  I have used this example on several occasions but was somewhat challenged when an individual in the audience pointed out that this could also be construed in a negative way since seatbelts have reduced the number of organ donors!  You will no doubt be able to guess at this point that I was talking to staff in HSC at the time. 

The point is that it is all about the narrative you build from a piece of research and how you evidence that claim.  There are some challenges for us around the issue of impact, but it also offers great opportunity.  So I think it is time for me to finish here and go back to working on my Board presentation.

REF week on the Blog! What were the HEFCE REF pilots?

Research assessment

This week is REF Week on the Blog! Each day we will be explaining a different element of the Research Excellence Framework (REF) as a quick reference guide to help you prepare for the forthcoming REF exercise – REF2014.

What were the HEFCE REF pilots? – HEFCE ran two pilot exercises with HEIs in the sector during the development of the REF. The first exercise was a bibliometrics pilot, and the second was an impact pilot.

Bibliometrics pilot – HEFCE ran a pilot exercise in the construction of bibliometric indicators of research quality in 2008-09, using Scopus and the Web of Scienceas the test databases. BU was chosen as one of 22 institutions to be part of phase one of the pilot exercise. This involved the provision of publication details to HEFCE, and cross-checking BU information on Web of Science and Scopus. Where possible this was completed using BU’s institutional repository, BURO. The outcome of the bibliometrics pilot was that bibliometric indicators are not yet sufficiently robust enough in all disciplines to be used formulaically or as a primary indicator of research quality. However HEFCE agreed that there was scope for bibliometrics to inform the process of expert review in some units of assessment. These findings resulted in the decision that some UOA sub-panels will receive citation data (the number of times an output has been cited, calculate via Scopus) as additional information about the academic significance of the outputs.

Impact pilot – During 2009-10, HEFCE ran a second pilot exercise, this time with the aim of developing proposals for how to assess research impact in the REF. The impact pilot involved 29 HEIs submitting evidence of impact (case studies and statements) which were assessed by pilot expert panels in five units of assessment:

  • Clinical Medicine
  • Physics
  • Earth Systems and Environmental Sciences
  • Social Work and Social Policy & Administration
  • English Language and Literature

The impact pilot completed in autumn 2010 and the final report (including recommendations and findings) was published on 11 November 2010. The full report can be accessed on the HEFCE website. For a brief summary of the report, please download the Impact Pilot Summary. You can also read our REF Impact FAQs.

You can access the latest presentation about the REF, written by the REF team, here: REF slide pack Sep 2011

Check out the posts appearing on the Blog every day this week as part of REF Week!

REF week on the Blog! Introduction to the Research Excellence Framework

Research assessment

Next week is REF Week on the Blog! Each day we will be explaining a different element of the Research Excellence Framework (REF) as a quick reference guide to help you prepare for the forthcoming REF exercise – REF2014.

What is the REF? – The Research Excellence Framework (REF) has replaced the Research Assessment Exercise (RAE) as the new process for assessing the volume and quality of research in UK HEIs. As with the RAE, the results of the REF will determine the annual quality-related research (QR) grant distributed from HEFCE to HEIs in England.

The REF will assess research excellence through a process of expert review, informed by indicators where appropriate. It will be based on HEIs submitting evidence of their research activity and outcomes, to be assessed by expert panels.

The REF will focus on assessing three elements, which together reflect the key characteristics of research excellence (weightings for REF2014 in brackets):

  • The quality of research outputs (65%)
  • The reach and significance of the impact of research (20%)
  • The vitality of the research environment (15%)

Each of these elements will be assessed against appropriate criteria for excellence, and rated by expert panels on a five-point scale ranging from 4* (excellent, world-leading) to Unclassified.

The REF assessment period started on 1st January 2008 and the first REF submission will take place in Autumn 2013, with the results published in December 2014.

You can access the latest presentation about the REF, written by the REF team, here: REF slide pack Sep 2011

Check out the posts appearing on the Blog every day next week as part of REF Week!

Bibliometrics need not be baffling!

BU research

What are bibliometrics?

Bibliometrics are a set of methods used to study or measure text and information. Citation analysis and content analysis are the most commonly used bibliometric methods. Bibliometric methods can help you explore the academic impact of a field, a set of researchers or a particular journal paper.

What is citation analysis?

Citation analysis looks at where a document has been referenced by others since it was originally published – this information can be used when searching for materials and in analysing their merit. Undertaking citation analysis on yourself is useful for assessing your own research performance. Specialist databases such as Web of Science and Scopus provide various tools for doing this analysis.

Searching for citation information on the Web of ScienceSM

Web of ScienceSM is hosted by Thomson Reuters and consists of various databases containing information gathered from thousands of scholarly journals, books, book series, reports, conference proceedings, and more:

  • Science Citation Index Expanded (SCI-Expanded)
  • Social Sciences Citation Index (SSCI)
  • Arts & Humanities Citation Index (A&HCI)
  • Index Chemicus (IC)
  • Current Chemical Reactions (CCR-Expanded)
  • Book Citations Index – coming soon!

These databases enable you to perform a variety of tasks, such as search published literature, undertake citation analysis, track new research within a particular field, and identify chemical compounds and reactions. Data is available from around 1990, and even earlier in some cases.

By producing a Web of ScienceSM Citation Report for yourself (or for others), you can find out who is citing your work and how it is being used in other people’s publications so that you can get a feel for the overall citation activity around your outputs. Search for an author’s name and then click on ‘Create Citation Report’ from the results page.

Producing this report will give you information such as the number of items published in each year, the number of citations to those items for each year, the average number of citations per item, and your h-index based on this information. Click here for web tutorials on how to use the Web of ScienceSM.

Searching for citation information on Scopus

Scopus, part of Elsevier’s SciVerse facility, was launched in November 2004 and is an abstract and citation database containing around 19,500 titles from more than 5,000 publishers. Scopus enables researchers to track, analyse and visualise research, and has broad coverage of the scientific, technical, medical and social sciences fields and, more recently, the arts and humanities. Data is currently largely available from 1996 but it does go back further than this in some cases. For more information about Scopus, click here.

By searching for yourself (or others) on the Scopus database using the author search facility, you can use the ‘View Citation Overview’ function to get a feel for the citations activity around your outputs. The information is presented and can be analysed in a number of ways, including pie charts, graphs and tables, and shows the breakdown of citation activity over a number of years and your h-index based on this data. Various tutorials on using Scopus can be accessed here.

Scopus and the Research Excellence Framework (REF): HEFCE has announced that Elsevier have been chosen as the provider of citation data services to the REF sub-panels that have chosen to make use of citation information as part of the assessment process. Using the Scopus database, HEFCE will provide the relevant sub-panels with raw citation data (i.e. not normalised) accompanied by contextual information, which will assist those panel members in making decisions about the outputs part of the REF submissions.

What is the h-index?

The h-index was conceived by Professor Jorge Hirsch in 2005 within the field of physics and is fast becoming one of the most widely used metrics for research evaluation. It is also becoming increasingly used as a measure of research activity and academic prominence across various subject areas.

The benefit of the h-index over other citation measures is that it is not influenced by a few highly cited papers and it ignores any papers that remain uncited. It is calculated based on the number of papers by a particular author that receive h or more citations. Therefore, an h-index of 15 means that a person has at least 15 papers that have been cited 15 times or more. Fortunately, the Web of Science and Scopus both automatically calculate the h-index as part of their citation analysis functions so there is no need to work it out manually.

If you’d like to know more about the h-index, the original research document can be accessed from the Cornell University Library webpage.

What are journal impact factors?

Journal Impact Factors are published annually on the Web of Knowledge and provide a way of ranking journals based on the citation performance of articles published by those journals from the previous two years. For more information about how impact factors are calculated and how they can be used, see my previous blog post.

Other methods of ranking journals also exist, such as the ABS Academic Journal Quality Guide and the ERA journal ranking list. Journal rankings can be useful when choosing which journal to publish in, for example.

Exploring research impact

BU research

Why do I need to think about the impact of my research?

Given the current economic situation, tighter spending reviews and increasing constraints on public spending, there is more of a need than ever to demonstrate the economic, social and cultural benefits of publicly funded research to wider society. This broad definition of research impact is gradually being adopted and used in a number of ways by various funding bodies that need to be accountable for the money they distribute, such as the Research Councils UK (RCUK), the Higher Education Funding Council for England (HEFCE) and some charities and trusts.

Consequently, there is now greater impetus to involve researchers more directly in demonstrating the impact of their research. Researchers therefore need to actively think about how to demonstrate the value of their research and its wider impact, from the application stage through to project completion, dissemination and beyond.

This impact pathway is a fluid process and research impacts can occur at any stage in the research life cycle – often they can stem from unexpected or unintended outcomes as well as from planned activity. The key is to start thinking about potential beneficiaries and pathways to impact during the project planning stage and to continue to monitor the outcomes in an ongoing way. This will help you to make new connections and partnerships beyond the project itself, and to put in place resources and activities that enable you to make the most of opportunities for achieving impact when they arise. Keeping a record of activity related to a project, and gathering evidence to support impacts and outcomes achieved, is recommended to enable you to effectively fulfil any current and future reporting requirements.

What are funders looking for in terms of impact?

Many funding bodies, particularly larger ones such as RCUK, have a section on their funding application form that specifically asks you to consider the potential pathways to impact as appropriate for the nature of the research you’re proposing to conduct. This is to enable funders to support you in undertaking these activities – you’re not being asked to predict the actual outcomes that the research will achieve.

Each funder understands that there is great diversity in the kinds of impact that are possible; they also acknowledge that this diversity is a great strength of the research community in addressing such things as urgent social issues, remaining competitive in global markets and improving quality of life. It is about embracing the ways in which research-related knowledge and skills benefit individuals, organisations and nations.
In thinking about potential impacts, you might find it useful to consider the potential beneficiaries of the research – innovative and creative approaches to engaging beneficiaries and fostering impact are generally strongly encouraged by the funders. For more specific information about completing the impact sections on the RCUK application forms and for an indication of the potential range of impacts that can be generated from research, visit the RCUK impact web-pages.

Furthermore, the RCUK has just launched the Research Outcomes Project, which requires all RCUK grant holders to upload information about the various outcomes that have resulted from each of the RCUK-funded projects they are responsible for, and one of those categories is impact.

What is HEFCE looking for in terms of research impact for the Research Excellence Framework (REF)?

As part of submitting to the REF, HEFCE requires higher education institutions to provide evidence of research impact that has been realised within the assessment period but which stems from research undertaken at that institution within a number of years prior to the assessment period. Therefore, rather than looking forward to the kinds of impact that might stem from a research project, HEFCE is asking for information about impacts that are being, or have already been, achieved within a set timeframe. More information about how HEFCE is approaching impact in the REF is available from the HEFCE REF web-pages.

RCUK Research Outcomes Project is ready to launch!

BU research

Following my previous post about the development of the Research Councils UK (RCUK) Outcomes Project, the launch of the new system for collecting information about research outcomes from all RCUK grant holders is nearly here. Assuming all goes to plan with the final phase of user testing, the system will go live from 14 November 2011. Grant holders will be required to upload information about the following for each of the RCUK-funded projects they are responsible for:

  • Publications
  • Other research outputs
  • Collaboration
  • Communication
  • Exploitation
  • Recognition
  • Staff development
  • Further funding
  • Impact

Grant holders will be able to log in to the system using their Je-S login and will be responsible for maintaining the outcomes information about the grants they have been awarded, even if they move institution. RCUK have issued a list of FAQs to help answer some common queries.

Southern Universities REF event at BU – 22 February 2012 – SAVE THE DATE!

Research assessment, Research news

REF logoBU will be hosting a half day Research Excellence Framework (REF) event for southern universities, supported by the REF Team, on 22 February 2012 to which all staff are invited to attend.

This event follows hot on the heels of the first REF Team-supported event held at BU on 19 May 2011, to which over 150 delegates from 39 institutions attended (see our previous blog post – The excellent HEFCE REF event at BU!).

The sector-wide consultation on the proposed REF panel criteria closed earlier this month and the final documents are due to be published in January 2012. This event will provide an update on the current developments with the REF and the confirmed REF panel documentation, focusing specifically on the assessment of impact within each of the four Main Panels.

The event will be open to BU staff and external delegates and the provisional programme is shown below.

Provisional programme:

09:30 – 10:00    Coffee and registration

10:00 – 10:30    REF Team overview of the assessment framework

Chris Taylor, Deputy REF Project Manager, REF Team

10:30 – 11:00    Similarities between the 4 Main Panel criteria

Professor Stephen Holgate, Chair of Main Panel A

11:00 – 11:30    Morning break

11:30 – 12:15    Panel specifics, differences and impact assessment

There will be concurrent sessions, one for each of the four Main Panels. Attendees choose which one to attend.

Main Panel A:    Professor Stephen Holgate, Chair of Main Panel A

Main Panel B:    Professor Philip Nelson, Chair of sub-panel 15 (General Engineering)

Main Panel C:    Professor John Scott, Chair of sub-panel 23 (Sociology)

Main Panel D:    Professor Bruce Brown, Chair of Main Panel D

12:15 – 13:00    Panel Q&A session with all participants

13:00 – 14:00   Lunch, networking and close

 

The event is free to attend but booking is essential. Booking will open very soon – further details to follow!
 

Funding Opportunity – Assistance with Impact Analysis

Funding opportunities

JISC have recently announced a funding opportunity “Embedding impact analysis good practice in research, using BCE practitioners”.

This Call is designed to encourage skills sharing between researchers, engagement practitioners and information management specialists. It is aimed at research groups looking to bring in external expertise and support to help them develop improved capability and process to analyse the benefits and impact of their research, in order to enhance their impact and sustainability.

The work, which will be facilitated by the National Co-ordinating Centre for Public Engagement (NCCPE), focuses on the highly topical area of research impact. It invites people to register online to form partnerships to synthesise and share learning about how to embed technology-enabled good practice in impact analysis in research groups, using the expertise of Business and Community Engagement staff.

For any queries please contact David Owen.

The deadline for receipt of proposals is 31 October 2011. Call details can be found here.

What makes a good impact section?

BU research, EU, Guidance

In writing an FP7 bid the marks allocated for Impact are the same as those for Science & Technological Excellence. So, how do you make sure you score top marks?

Beta Technology (sponsored by DEFRA) are the UKs National Contact Point for three of the FP7 themes and offer a number of good tips. They’ve also provided real-life examples of a good and not so good Impact Section together with the evaluators’ mark and feedback – these are essential reading for any propective FP7 applicant! 

Impact section examples can be found on the I drive at the following address: \\Lytchett\IntraStore\CRKT\Public\Research Blog Docs\Impact Summary

If you would like more information on the impact advice from Beta Technology please contact Shelly Maskell.

REF update from the VC and PVC

Research assessment, Research news

In the latest Vice-Chancellor’s email, Prof John Vinney gave an update on the Research Excellence Framework (REF) and BU’s preparations for the exercise. The email also introduced this month’s VC video in which he and Prof Matthew Bennett discuss the REF in more detail – and what it means for BU – with Sue Eccles from the Media School.

You can watch the video here:

httpv://youtu.be/7s8RTlOOPnU.

Knowledge exchange and business links are increasingly important in academic promotion

Research news

Research commissioned by the Association for University Research and Industry Links (Auril) indicates that experience of knowledge exchange and links with business have become increasingly important to academics trying to win promotion.

Philip Graham, executive director of Auril, stated that “In some job applications, (knowledge transfer) is now rated as desirable, although not essential. Ten years ago it would not have been rated as desirable.”

The majority of survey respondents said that academic promotion depends more on academic teaching and research reputation, however, business links are becoming increasingly more important.

The government’s White Paper on higher education, published in June, says that universities should “look again at how they work with business across their teaching and research activities, to promote better teaching, employer sponsorship, innovation and enterprise.”

HEIF (Higher Education Innovation Funding) exists to support and develop a broad range of knowledge exchange activities between universities and colleges and the wider world, which result in economic and social benefit to the UK. BU submitted its HEIF-5 strategy to HEFCE in July; further information will appear on the blog in due course.

You can read the full story on the AURIL website: Build up the Business Assets if you Want to Get Ahead

BU REF preparations – next mock exercise underway

Research news

Hot on the heels of our REF light-touch review of outputs last winter, our subsequent mini-mock exercise in two UOAs, and the release of the final REF Guidance on Submissions and draft panel criteria documentation, BU’s next mock REF exercise is now underway.

The BU winter 2011 mock will concentrate solely on the impact and environment elements of the submissions (i.e. individual outputs will not be reviewed). During the autumn term BU Unit of Assessment (UOA) Leaders have been asked to produce 2-3 impact case studies, an impact statement and an environment statement for each UOA. These will be sent to the external reviewers (at least 2 per UOA) in early December, and feedback will be shared with the UOA Leaders in February 2012.

The next review of outputs is planned for spring 2012.

These mock exercises are intended to shape and craft our submissions to REF2014 to ensure we put forward the strongest submissions possible.

If you have any questions or suggestions regarding our internal preparations for the REF then send me an email 🙂

Overview of the REF draft panel criteria – what are the subtleties between panels?

Guidance, Research assessment, Research news

At the end of July the REF team released the draft panel working methods and criteria documentation (see our previous blog post for access to the documents).

We’ve spent the week wading through the four main panel documents and have produced a very brief overview of the subtleties between the panels on key criteria (such as the use of citation data, co-authored outputs, additional environment data, etc) in a tabular format.

You can access the overview table here: REF – draft panel criteria comparison table

Unfortunately this is no substitute for reading the actual documentation (sorry!) but does highlight the key points and differences between panels.

These documents are currently open to sector-wide consultation until 5 October 2011. BU will be submitting a single institutional response coordinated by the Research Development Unit. BU staff are invited to submit feedback for consideration as part of this response. Please email all comments to Anita Somner by 20 September 2011.

Latest journal impact factors

BU research, Research assessment

Following the release of the latest Journal Citation Reports® on the Thomson Reuters ISI Web of Science database, we have compiled a list of the top ranking journals in various fields related to BU research. BU staff can access these lists by going to the designated folder on the collaborative I-drive: I:\R&KEO\Public\RDU\Journal Impact Factors 2011. If there are any additional subject areas that you would like to see included, do send me an email.

Related blog posts that may be of interest:
Journal impact factors explained

Google Scholar’s new citations tracking tool

BU research

As the demand for information about the performance of research outputs increases, Google Scholar’s latest online citations tracking tool could prove to be a useful resource. Google Scholar Citations is currently being launched with a small number of users before widespread dissemination, but the early signs are that it could provide a simple way for academics to calculate key metrics and monitor them over time.

According to Google Scholar, the tool will ‘use a statistical model based on author names, bibliographic data, and article content to group articles likely written by the same author’. You can then identify your articles using these groups and Google Scholar will then automatically produce the citation metrics. Three metrics will be available: the h-index, the i-10 index (which is the number of articles with at least ten citations), and the total number of citations to your articles. Each of these metrics will be available for all outputs as well as for articles published in the last five years, and will be automatically updated as new citations are available on the web.

You will be able to update your ‘profile’ of outputs by adding missing items, merging duplicates or correcting bibliographic errors. You’ll also have the choice to make your profile public, the idea being that you’ll make it easier for colleagues globally to follow your research because your outputs will appear in Google Scholar search results.

To get a glimpse of Google Scholar Citations in action, there are some sample profiles available via the Google Scholar blog. You can also register your email address so that you will be notified when the tool is finally available to everyone.