Tagged / ref

REF week on the Blog! BU REF preparation and governance

Research assessment

This week is REF Week on the Blog! Each day we will be explaining a different element of the Research Excellence Framework (REF) as a quick reference guide to help you prepare for the forthcoming REF exercise – REF2014.

Internal preparations for the BU submission to the REF have already started and you can keep up to date by reading the BU REF Highlight Reports. These are published regularly and detail all of the recent developments (internal and external) with the REF (see the link to the I-drive at the end of the post).

Two internal groups have been established to oversee preparations for the REF:

RASG – The REF Academic Steering Group (RASG) has been established as the primary decision-making body for the BU REF preparations. It first met on 14 June 2010 and normally meets on a monthly basis. The RASG Terms of Reference (including a list of members) can be accessed via the I-drive.

RALT – The REF Academic Leadership Team (RALT). RALT met for the first time on 27 September 2010 and will meet when required (currently this is on a monthly basis). Members of RALT include all of the RASG members plus the REF UOA Leaders. The list of REF UOA Leaders and the RALT Terms of Reference can be access on the I-drive.

BU is currently working on a REF Code of Practice and some FAQs around REF preparation, staff eligibility, staff selection, etc., which will be available from the BU Research Blog in due course.

The Research Development Unit (RDU) are managing and administering the internal REF process. If you have any questions then please do ask Anita Somner or Julie Northam.

You can access all of this information regarding the BU preparations for the REF via the I-drive: I:\CRKT\Public\RDU\REF

You can access the latest presentation about the REF, written by the REF team, here: REF slide pack Sep 2011

Check out the posts appearing on the Blog every day next week as part of REF Week!

REF week on the Blog! What were the HEFCE REF pilots?

Research assessment

This week is REF Week on the Blog! Each day we will be explaining a different element of the Research Excellence Framework (REF) as a quick reference guide to help you prepare for the forthcoming REF exercise – REF2014.

What were the HEFCE REF pilots? – HEFCE ran two pilot exercises with HEIs in the sector during the development of the REF. The first exercise was a bibliometrics pilot, and the second was an impact pilot.

Bibliometrics pilot – HEFCE ran a pilot exercise in the construction of bibliometric indicators of research quality in 2008-09, using Scopus and the Web of Scienceas the test databases. BU was chosen as one of 22 institutions to be part of phase one of the pilot exercise. This involved the provision of publication details to HEFCE, and cross-checking BU information on Web of Science and Scopus. Where possible this was completed using BU’s institutional repository, BURO. The outcome of the bibliometrics pilot was that bibliometric indicators are not yet sufficiently robust enough in all disciplines to be used formulaically or as a primary indicator of research quality. However HEFCE agreed that there was scope for bibliometrics to inform the process of expert review in some units of assessment. These findings resulted in the decision that some UOA sub-panels will receive citation data (the number of times an output has been cited, calculate via Scopus) as additional information about the academic significance of the outputs.

Impact pilot – During 2009-10, HEFCE ran a second pilot exercise, this time with the aim of developing proposals for how to assess research impact in the REF. The impact pilot involved 29 HEIs submitting evidence of impact (case studies and statements) which were assessed by pilot expert panels in five units of assessment:

  • Clinical Medicine
  • Physics
  • Earth Systems and Environmental Sciences
  • Social Work and Social Policy & Administration
  • English Language and Literature

The impact pilot completed in autumn 2010 and the final report (including recommendations and findings) was published on 11 November 2010. The full report can be accessed on the HEFCE website. For a brief summary of the report, please download the Impact Pilot Summary. You can also read our REF Impact FAQs.

You can access the latest presentation about the REF, written by the REF team, here: REF slide pack Sep 2011

Check out the posts appearing on the Blog every day this week as part of REF Week!

REF week on the Blog! Introduction to the Research Excellence Framework

Research assessment

Next week is REF Week on the Blog! Each day we will be explaining a different element of the Research Excellence Framework (REF) as a quick reference guide to help you prepare for the forthcoming REF exercise – REF2014.

What is the REF? – The Research Excellence Framework (REF) has replaced the Research Assessment Exercise (RAE) as the new process for assessing the volume and quality of research in UK HEIs. As with the RAE, the results of the REF will determine the annual quality-related research (QR) grant distributed from HEFCE to HEIs in England.

The REF will assess research excellence through a process of expert review, informed by indicators where appropriate. It will be based on HEIs submitting evidence of their research activity and outcomes, to be assessed by expert panels.

The REF will focus on assessing three elements, which together reflect the key characteristics of research excellence (weightings for REF2014 in brackets):

  • The quality of research outputs (65%)
  • The reach and significance of the impact of research (20%)
  • The vitality of the research environment (15%)

Each of these elements will be assessed against appropriate criteria for excellence, and rated by expert panels on a five-point scale ranging from 4* (excellent, world-leading) to Unclassified.

The REF assessment period started on 1st January 2008 and the first REF submission will take place in Autumn 2013, with the results published in December 2014.

You can access the latest presentation about the REF, written by the REF team, here: REF slide pack Sep 2011

Check out the posts appearing on the Blog every day next week as part of REF Week!

Bibliometrics need not be baffling!

BU research

What are bibliometrics?

Bibliometrics are a set of methods used to study or measure text and information. Citation analysis and content analysis are the most commonly used bibliometric methods. Bibliometric methods can help you explore the academic impact of a field, a set of researchers or a particular journal paper.

What is citation analysis?

Citation analysis looks at where a document has been referenced by others since it was originally published – this information can be used when searching for materials and in analysing their merit. Undertaking citation analysis on yourself is useful for assessing your own research performance. Specialist databases such as Web of Science and Scopus provide various tools for doing this analysis.

Searching for citation information on the Web of ScienceSM

Web of ScienceSM is hosted by Thomson Reuters and consists of various databases containing information gathered from thousands of scholarly journals, books, book series, reports, conference proceedings, and more:

  • Science Citation Index Expanded (SCI-Expanded)
  • Social Sciences Citation Index (SSCI)
  • Arts & Humanities Citation Index (A&HCI)
  • Index Chemicus (IC)
  • Current Chemical Reactions (CCR-Expanded)
  • Book Citations Index – coming soon!

These databases enable you to perform a variety of tasks, such as search published literature, undertake citation analysis, track new research within a particular field, and identify chemical compounds and reactions. Data is available from around 1990, and even earlier in some cases.

By producing a Web of ScienceSM Citation Report for yourself (or for others), you can find out who is citing your work and how it is being used in other people’s publications so that you can get a feel for the overall citation activity around your outputs. Search for an author’s name and then click on ‘Create Citation Report’ from the results page.

Producing this report will give you information such as the number of items published in each year, the number of citations to those items for each year, the average number of citations per item, and your h-index based on this information. Click here for web tutorials on how to use the Web of ScienceSM.

Searching for citation information on Scopus

Scopus, part of Elsevier’s SciVerse facility, was launched in November 2004 and is an abstract and citation database containing around 19,500 titles from more than 5,000 publishers. Scopus enables researchers to track, analyse and visualise research, and has broad coverage of the scientific, technical, medical and social sciences fields and, more recently, the arts and humanities. Data is currently largely available from 1996 but it does go back further than this in some cases. For more information about Scopus, click here.

By searching for yourself (or others) on the Scopus database using the author search facility, you can use the ‘View Citation Overview’ function to get a feel for the citations activity around your outputs. The information is presented and can be analysed in a number of ways, including pie charts, graphs and tables, and shows the breakdown of citation activity over a number of years and your h-index based on this data. Various tutorials on using Scopus can be accessed here.

Scopus and the Research Excellence Framework (REF): HEFCE has announced that Elsevier have been chosen as the provider of citation data services to the REF sub-panels that have chosen to make use of citation information as part of the assessment process. Using the Scopus database, HEFCE will provide the relevant sub-panels with raw citation data (i.e. not normalised) accompanied by contextual information, which will assist those panel members in making decisions about the outputs part of the REF submissions.

What is the h-index?

The h-index was conceived by Professor Jorge Hirsch in 2005 within the field of physics and is fast becoming one of the most widely used metrics for research evaluation. It is also becoming increasingly used as a measure of research activity and academic prominence across various subject areas.

The benefit of the h-index over other citation measures is that it is not influenced by a few highly cited papers and it ignores any papers that remain uncited. It is calculated based on the number of papers by a particular author that receive h or more citations. Therefore, an h-index of 15 means that a person has at least 15 papers that have been cited 15 times or more. Fortunately, the Web of Science and Scopus both automatically calculate the h-index as part of their citation analysis functions so there is no need to work it out manually.

If you’d like to know more about the h-index, the original research document can be accessed from the Cornell University Library webpage.

What are journal impact factors?

Journal Impact Factors are published annually on the Web of Knowledge and provide a way of ranking journals based on the citation performance of articles published by those journals from the previous two years. For more information about how impact factors are calculated and how they can be used, see my previous blog post.

Other methods of ranking journals also exist, such as the ABS Academic Journal Quality Guide and the ERA journal ranking list. Journal rankings can be useful when choosing which journal to publish in, for example.

Exploring research impact

BU research

Why do I need to think about the impact of my research?

Given the current economic situation, tighter spending reviews and increasing constraints on public spending, there is more of a need than ever to demonstrate the economic, social and cultural benefits of publicly funded research to wider society. This broad definition of research impact is gradually being adopted and used in a number of ways by various funding bodies that need to be accountable for the money they distribute, such as the Research Councils UK (RCUK), the Higher Education Funding Council for England (HEFCE) and some charities and trusts.

Consequently, there is now greater impetus to involve researchers more directly in demonstrating the impact of their research. Researchers therefore need to actively think about how to demonstrate the value of their research and its wider impact, from the application stage through to project completion, dissemination and beyond.

This impact pathway is a fluid process and research impacts can occur at any stage in the research life cycle – often they can stem from unexpected or unintended outcomes as well as from planned activity. The key is to start thinking about potential beneficiaries and pathways to impact during the project planning stage and to continue to monitor the outcomes in an ongoing way. This will help you to make new connections and partnerships beyond the project itself, and to put in place resources and activities that enable you to make the most of opportunities for achieving impact when they arise. Keeping a record of activity related to a project, and gathering evidence to support impacts and outcomes achieved, is recommended to enable you to effectively fulfil any current and future reporting requirements.

What are funders looking for in terms of impact?

Many funding bodies, particularly larger ones such as RCUK, have a section on their funding application form that specifically asks you to consider the potential pathways to impact as appropriate for the nature of the research you’re proposing to conduct. This is to enable funders to support you in undertaking these activities – you’re not being asked to predict the actual outcomes that the research will achieve.

Each funder understands that there is great diversity in the kinds of impact that are possible; they also acknowledge that this diversity is a great strength of the research community in addressing such things as urgent social issues, remaining competitive in global markets and improving quality of life. It is about embracing the ways in which research-related knowledge and skills benefit individuals, organisations and nations.
In thinking about potential impacts, you might find it useful to consider the potential beneficiaries of the research – innovative and creative approaches to engaging beneficiaries and fostering impact are generally strongly encouraged by the funders. For more specific information about completing the impact sections on the RCUK application forms and for an indication of the potential range of impacts that can be generated from research, visit the RCUK impact web-pages.

Furthermore, the RCUK has just launched the Research Outcomes Project, which requires all RCUK grant holders to upload information about the various outcomes that have resulted from each of the RCUK-funded projects they are responsible for, and one of those categories is impact.

What is HEFCE looking for in terms of research impact for the Research Excellence Framework (REF)?

As part of submitting to the REF, HEFCE requires higher education institutions to provide evidence of research impact that has been realised within the assessment period but which stems from research undertaken at that institution within a number of years prior to the assessment period. Therefore, rather than looking forward to the kinds of impact that might stem from a research project, HEFCE is asking for information about impacts that are being, or have already been, achieved within a set timeframe. More information about how HEFCE is approaching impact in the REF is available from the HEFCE REF web-pages.

Need to publish a paper quickly for the REF?

Research news

open access logo, Public Library of ScienceThen come to our free Open Access event this Wednesday in the EBC!

Publishing via an open access route often results in much faster publication times, meaning your research is out in the public domain much more quickly than traditional subscription journal routes.

BU fully supports open access publishing and has recently launched a central Open Access Publication Fund to enable academic staff to access funds to easily publish via open access outlets.

We’re holding an open access publishing event this Wednesday between 10am-12:30pm in the EBC (7th floor).

You can access the programme here: BU Open Access event programme

It is free for BU staff and students to attend. Refreshments and lunch will be provided.

To reserve a place at the event please contact Anita Somner by email.

We look forward to seeing you there! 😀

Southern Universities REF event at BU – 22 February 2012 – SAVE THE DATE!

Research assessment, Research news

REF logoBU will be hosting a half day Research Excellence Framework (REF) event for southern universities, supported by the REF Team, on 22 February 2012 to which all staff are invited to attend.

This event follows hot on the heels of the first REF Team-supported event held at BU on 19 May 2011, to which over 150 delegates from 39 institutions attended (see our previous blog post – The excellent HEFCE REF event at BU!).

The sector-wide consultation on the proposed REF panel criteria closed earlier this month and the final documents are due to be published in January 2012. This event will provide an update on the current developments with the REF and the confirmed REF panel documentation, focusing specifically on the assessment of impact within each of the four Main Panels.

The event will be open to BU staff and external delegates and the provisional programme is shown below.

Provisional programme:

09:30 – 10:00    Coffee and registration

10:00 – 10:30    REF Team overview of the assessment framework

Chris Taylor, Deputy REF Project Manager, REF Team

10:30 – 11:00    Similarities between the 4 Main Panel criteria

Professor Stephen Holgate, Chair of Main Panel A

11:00 – 11:30    Morning break

11:30 – 12:15    Panel specifics, differences and impact assessment

There will be concurrent sessions, one for each of the four Main Panels. Attendees choose which one to attend.

Main Panel A:    Professor Stephen Holgate, Chair of Main Panel A

Main Panel B:    Professor Philip Nelson, Chair of sub-panel 15 (General Engineering)

Main Panel C:    Professor John Scott, Chair of sub-panel 23 (Sociology)

Main Panel D:    Professor Bruce Brown, Chair of Main Panel D

12:15 – 13:00    Panel Q&A session with all participants

13:00 – 14:00   Lunch, networking and close

 

The event is free to attend but booking is essential. Booking will open very soon – further details to follow!
 

Tourism comes of age…

BU research

Although a major contributor to life at BU, the study of Tourism is often wrongly maligned as being a niche subject on the periphery of more established areas of study such as Business & Management and Geography. Well, in the UK alone over 100 institutions offer HE courses at undergraduate level including “top tier” universities such as Exeter, Surrey, Strathclyde and Stirling with many more competing for students and staff across Europe and beyond with major concentrations of activity in North America, the Middle East, South East Asia and Australia and New Zealand where tourism is not only a significant area of academic interest but also of valuable income, foreign exchange earnings and employment.

Returning to the UK one of the most significant “coming of age” moments has been the explicit inclusion of Tourism for the very first time in a Unit of Assessment in the forthcoming Research Excellence Framework. Unit 26, Sport and Exercise Sciences, Leisure and Tourism is one of only a few new units in the REF, a fact which clearly reflects its growing maturity as an area of academic investigation and the widespread positive recognition with which it is now held across the sector. This recognition really took hold 2 to 3 years ago when the ESRC awarded colleagues at the University of Exeter £1.5 million to set up its research cluster in Sport, Leisure and Tourism, an award which would have been unthinkable only a few years before. Since then, staff from the School of Tourism at BU have been attracting funds from the ESRC, the European Union and the United Nations World Tourism Organization and others while the significant award recently won by colleagues from the School from the EPSRC on sustainable patterns of travel demonstrates the collaborative and inter-disciplinary opportunities offered by Tourism. This latter point was again highlighted recently with the inclusion in the RCUK publication Big Ideas for the Future of a project looking at the fusion between public health and tourism policies at the local level. This was BU’s only entry in this prestigious publication, testament if it were ever needed that the industry that is widely acclaimed as the world’s largest has now also come of age in the academic arena!

REF update from the VC and PVC

Research assessment, Research news

In the latest Vice-Chancellor’s email, Prof John Vinney gave an update on the Research Excellence Framework (REF) and BU’s preparations for the exercise. The email also introduced this month’s VC video in which he and Prof Matthew Bennett discuss the REF in more detail – and what it means for BU – with Sue Eccles from the Media School.

You can watch the video here:

httpv://youtu.be/7s8RTlOOPnU.

BU REF preparations – next mock exercise underway

Research news

Hot on the heels of our REF light-touch review of outputs last winter, our subsequent mini-mock exercise in two UOAs, and the release of the final REF Guidance on Submissions and draft panel criteria documentation, BU’s next mock REF exercise is now underway.

The BU winter 2011 mock will concentrate solely on the impact and environment elements of the submissions (i.e. individual outputs will not be reviewed). During the autumn term BU Unit of Assessment (UOA) Leaders have been asked to produce 2-3 impact case studies, an impact statement and an environment statement for each UOA. These will be sent to the external reviewers (at least 2 per UOA) in early December, and feedback will be shared with the UOA Leaders in February 2012.

The next review of outputs is planned for spring 2012.

These mock exercises are intended to shape and craft our submissions to REF2014 to ensure we put forward the strongest submissions possible.

If you have any questions or suggestions regarding our internal preparations for the REF then send me an email 🙂

Overview of the REF draft panel criteria – what are the subtleties between panels?

Guidance, Research assessment, Research news

At the end of July the REF team released the draft panel working methods and criteria documentation (see our previous blog post for access to the documents).

We’ve spent the week wading through the four main panel documents and have produced a very brief overview of the subtleties between the panels on key criteria (such as the use of citation data, co-authored outputs, additional environment data, etc) in a tabular format.

You can access the overview table here: REF – draft panel criteria comparison table

Unfortunately this is no substitute for reading the actual documentation (sorry!) but does highlight the key points and differences between panels.

These documents are currently open to sector-wide consultation until 5 October 2011. BU will be submitting a single institutional response coordinated by the Research Development Unit. BU staff are invited to submit feedback for consideration as part of this response. Please email all comments to Anita Somner by 20 September 2011.

Latest journal impact factors

BU research, Research assessment

Following the release of the latest Journal Citation Reports® on the Thomson Reuters ISI Web of Science database, we have compiled a list of the top ranking journals in various fields related to BU research. BU staff can access these lists by going to the designated folder on the collaborative I-drive: I:\R&KEO\Public\RDU\Journal Impact Factors 2011. If there are any additional subject areas that you would like to see included, do send me an email.

Related blog posts that may be of interest:
Journal impact factors explained

Google Scholar’s new citations tracking tool

BU research

As the demand for information about the performance of research outputs increases, Google Scholar’s latest online citations tracking tool could prove to be a useful resource. Google Scholar Citations is currently being launched with a small number of users before widespread dissemination, but the early signs are that it could provide a simple way for academics to calculate key metrics and monitor them over time.

According to Google Scholar, the tool will ‘use a statistical model based on author names, bibliographic data, and article content to group articles likely written by the same author’. You can then identify your articles using these groups and Google Scholar will then automatically produce the citation metrics. Three metrics will be available: the h-index, the i-10 index (which is the number of articles with at least ten citations), and the total number of citations to your articles. Each of these metrics will be available for all outputs as well as for articles published in the last five years, and will be automatically updated as new citations are available on the web.

You will be able to update your ‘profile’ of outputs by adding missing items, merging duplicates or correcting bibliographic errors. You’ll also have the choice to make your profile public, the idea being that you’ll make it easier for colleagues globally to follow your research because your outputs will appear in Google Scholar search results.

To get a glimpse of Google Scholar Citations in action, there are some sample profiles available via the Google Scholar blog. You can also register your email address so that you will be notified when the tool is finally available to everyone.

REF draft panel criteria is now available

Research assessment, Research news

Two important REF documents are now available on the REF website:

1. The draft panel criteria and working methods for consultation. A summary of this will be added to the blog next week.

2. The analysis of panel membership is now available.

Panel criteria and working methods consultation – the consultation runs from now until 5 October. BU will be submitting a single institutional response coordinated by the Research Development Unit. BU staff are invited to submit feedback for consideration as part of this response. Please email all comments to Anita Somner by 20 September 2011.

The REF Guidance on Submissions document was released on 14 June, a summary of which can be accessed on the I-drive from: I:\CRKT\Public\RDU\REF.

Symplectic Elements: BU’s new research management system

Research news

I’d like to introduce you to Symplectic Elements – our soon-to-be new research management system. You might have heard colleagues talking about Symplectic Elements for a while now – the supplier (Symplectic Ltd) first visited BU to demonstrate the application in August 2009. I’m pleased to announced that we have now signed the contract with Symplectic and are in discussions with the supplier to determine the implementation plan. The aim is to have the system up and running this autumn.

Symplectic Elements is already used by lots of other UK universities, including Imperial College, Oxford, Exeter, Cambridge, UCL and Plymouth.

So what is Symplectic Elements and what benefits will it bring to BU? Symplectic Elements is a research management system. It will not replace any of our existing BU systems (such as BURO or RED) but it will link to them and join them together, sharing data between the systems. This means that BU staff will be able to add information to Symplectic Elements and it will be used in multiple systems. You will also be able to access research information from a single place. A single point of data entry will enable research information (such as publications data) to be automatically formatted and reused in other forums, such as in BU’s open access repository (BURO) and the BU staff profile webpages, without the need for duplicate or additional data entry. You will also be able to query data that appears to be missing or incorrect.

Symplectic Elements will provide academics with a simple ‘dashboard’ from which to view and manage their research information. This will also help when BU begins compiling data to meet the requirements of the REF.

How will Symplectic Elements link with the existing systems?

  • Symplectic Elements will link to our Research and Enterprise Database (RED) so you can see your current bids and projects. From within Symplectic you will also be able to link yourself to the PGR students you supervise.
  • It will link to BURO so that your research outputs are entered into the repository.
  • It will also link to a number of external publication databases (including Web of Science and Scopus) and automatically search these for your publications. When it identifies a paper it thinks might be yours it will send you an email and ask you to confirm it is your paper. If it is then all you will need to do is to tick ‘yes’ and Symplectic will create a record of your publication automatically. You will simply need to add a full-text copy of the paper (copyright permitting) and it will be uploaded into BURO.
  • Symplectic Elements will link to the new content management system and you will be able to choose which information is used on your staff profile webpage.

Timeframe for implementation: A broad steering group of representatives from across BU’s Schools and Professional Services has been formed to help feed into and guide the system’s implementation. A core project management group is currently overseeing the day-to-day tasks and will manage the initial data integration and checking. A meeting with the suppliers is scheduled for the end of the month. The system is due to go live in autumn 2011.

Further updates about Symplectic Elements will be posted on the BU Research Blog in due course!

REF Guidance on Submissions document released

Guidance, Research assessment, Research news

The REF2014 Guidance on Submissions document was released on Thursday and can be accessed on the HEFCE website here: REF Guidance on Submissions

We have prepared a summary document of the key points that can be accessed on the I drive: I:\CRKT\Public\RDU\REF

At the end of July the REF team will publish the draft panel working methods and criteria documents which will be open to consultation until the autumn. The Research Development Unit will be coordinating the BU response to the consultation – further details will be available once the documentation is released.

links for 2011-07-14

Research news

Interesting links tagged this week in the BU Research delicious pages:

First BU Research Blog Poll Results

BU research

Are journal impact factors a good indicator of quality?

Following the launch of the first BU Research Blog Poll, we received 28 responses to the above question which were split as follows:

Yes – always 2
Yes – but in STEM disciplines only 1
Sometimes 22
No, never 2

The majority of responses indicate that there may be some doubt about the usefulness of impact factors when used as a proxy for journal quality. This is perhaps because there are a number of factors that could affect a journal’s perceived quality that cannot be demonstrated through metrics alone. Also, the use of journal metrics like impact factors are not necessarily perceived as being robust enough yet to give an accurate indication of journal or article quality, hence HEFCE’s decision not to rely solely on metrics in the Research Excellence Framework (REF).

To continue the debate on this, do feel free to post a reply below, or suggest a topic for a future poll by responding to Julie’s original post. For more information about journal impact factors, have a look at the previous blog post on this subject.

In the meantime, why not get involved in the current poll which can be accessed from the top right-hand side of the blog homepage – it will take just seconds to complete and will help shape the support offered to BU academics in the future.