Tagged / rejected proposal

Understanding and benefiting from EU evaluation reports

BU research

This post continues series of blog posts on institutional learning from funder feedback. Today’s topic – understanding and benefiting from EU evaluation reports.

EU’s Horizon Europe, as well as previous framework programmes, stands out by always providing useful funder feedback known as Evaluation Summary Report (ESR). This document is available on EU Funding and Tender opportunities portal for all registered applicants at any time regardless of the outcome of their application.

If rejection decision has been made, the Commission will notify proposal coordinator if their proposal has been rejected because:

  • it is found to be inadmissible or ineligible (before or during the evaluation)
  • it falls short of the relevant thresholds
  • it is too far down the ranked list to qualify for the limited amount of funding available
  • if it fails to obtain ethics clearance, following an ethics review, or
  • it raises security concerns.

After the finalisation of the evaluation, all applicants will receive the ESR (they may also call it Proposal Evaluation Form). The layout of provided feedback may differ depending on particular funding stream you have applied for. However, there are common features applicable to all ESRs containing general information related to the call and your application, abstract, total score, scoring by each criterion and evaluators’ comments indicating strengths and weakness of the proposed project.

If you decide to resubmit your proposal to another call, considering the feedback provided in ESR will be extremely important because funders may expect a substantial change to your application. More about resubmissions you may find in our blog published on Tuesday this week.

Professor Jian Chang has been successful in submitting applications as a supervisor to Marie Skłodowska-Curie Actions, here is what he says about ESR: ‘Evaluation Summary Report is really useful document, especially if you resubmit your application to repeating call. Information provided in this document explains both strengths and weaknesses of your proposal, so you can focus on necessary improvements and save a lot of time for developing successful application.’

BU academics can find real-life example of the EU Evaluation Summary Report on Brightspace. RDS has a practice to analyse examples of feedback from funders during workshops dedicated to specific funding calls as well as supporting academics individually to facilitate improvement and resubmission of rejected applications.

For further support, feel free to get in touch with your Research Facilitator.

Managing rejection – part two

BU research

In yesterday’s post, we looked at the initial moment your application is rejected, and how to digest what’s happened. Today’s post will look at what to do once you’ve reflected on the feedback.

For your own sanity, the first thing to consider is whether it is worth pursuing further, or if time has moved on and it’s time for something new. Whatever, you decide, you know that time and planning is a big factor. Like a research project, you also need to map out your application preparation as a project, to ensure you have sufficient time to craft it into a fundable bid. If you need partners, this needs even more time to find the right ones, develop the relationship, and ensure it’s trusted and beneficial to the research.

Resubmissions

To be blunt, most funders will not accept a resubmission. Funder success rates are low enough without old applications being repeatedly submitted. You also shouldn’t expect a different result from doing the same thing. Funders will expect a substantial change to an application if you’re going to submit the same research as a new bid. If the only changes you make are in response to the reviewer comments then you’re 99% more likely to be rejected again, and possibly before it’s even gone to reviewers. If a funder has a ‘no resubmission’ policy, they will chuck out anything that looks like one (they know the tricks too, and so you can’t fool them by swapping personnel or changing the title).

If a funder does allow resubmissions or has invited one, you’ll need to declare this upfront. If you thoroughly revise the application, take the advice of your peers, and resubmit, you could be successful.

Submit to different funders

You may be tempted to submit to another funder. However, beware! The funder may have a whole different approach, criteria, priorities, and schemes. Remember that you tailored your application originally for a specific funding call or a funders priorities. You will still need to substantially revise your application. If you start moving sections to fit into new ones, you may undermine your message, and your research becomes unclear. We’re back to using the crowbar again.

New idea

If you decide that you want to make a fresh start, do think about the tips given above and in yesterday’s post, and employ these to your next bid.

Alternative funding opportunities can be found on Research Professional. All BU academics have an account.

In RDS, we can help you review your feedback and determine what steps you should take next. We’re building a bank of funder feedback and ensuring that our training and development for research reflects the most common weaknesses identified. More information will be provided in Friday’s post, written by Research Facilitator, Alex Pekalski.

Managing rejection – part one

BU research

This week, we’ll be running a series of blog posts on managing rejection. These will include experience from academics, advice on what to do next, tips of understanding EU evaluation reports, and what support is available for you to take the next steps. Leading on from this, next week, we will run a series of posts on institutional learning from funder feedback, and so make sure you’re glued to the research blog for the next couple of weeks. The first and second posts are about managing rejection.

Unfortunately, rejection is part and parcel of academic life. If you watch the TV series ‘Ozark’, rejection can feel like Darlene Snell offering to get the lemonade. You need to try not to take it personally, no matter how hard that is given the weekends and evenings you may have given up to craft your bid. There may be many factors as to why you didn’t get funded, and hopefully, you’ve been provided with feedback from the funding panel (not all do, sadly). The most frustrating feedback is when you were deemed fundable, but there wasn’t sufficient budget.

When a funder doesn’t provide panel feedback, you should at least receive the reviewers’ comments (note that research councils will send these to you in advance of  the panel so that you have a right to reply). These can be a mixed bag, and so don’t focus on the odd sentence that stands out (there is always one), but look collectively at what the reviewers’ are saying.

All is not lost! A huge amount of work goes into the development of a proposal. It is a great shame to park your idea, when it could be re-worked, and submitted to an alternative funder. Alternatively, it might be time to develop a new project idea. Before you decide, you need to take time to reflect on the failed application.

After you’ve given yourself a couple of days to get over the shock, grab a cuppa and revisit the feedback, together with your application, and the funder guidance. This will give you an opportunity to reflect on what you would do differently. This might involve having the right networks in place; did you really meet the call criteria or did you crowbar your research to fit; did you articulate well enough in the sections of the application; did you use the right research methods, was your research really state of the art? Ultimately, did you leave enough time to plan out the application and submit a high-quality bid?

Once you’ve digested all of this, it’s time to think about what to do next. Tomorrow’s post will explore this in more detail.

Coping with rejection: what to do if your grant application is unsuccessful part 2

Guidance

With only so much grant money in the world, Adam Golberg’s second of a two-part series, looks at how to move forward when it becomes clear that your time courting a potential funder comes to an end. Adam works at the University of Nottingham and runs the Cash for Questions blog – http://socialscienceresearchfunding.co.uk/.

You can read the first part here – Coping with rejection: what to do if your grant application in unsuccessful part 1

In the first part of this series, I argued that it’s important not to misunderstand or misinterpret the reasons for a grant application being unsuccessful. In the comments, Jo VanEvery shared a phrase that she’s heard from a senior figure at one of the Canadian Research Councils – that research funding “is not a test, it’s a contest”. Not getting funded doesn’t necessarily mean that your research isn’t considered to be of high quality. This second entry is about what steps to consider next.

1.  Some words of wisdom

‘Tis a lesson you should heed:  Try, try, try again.
If at first you don’t succeed, Try, try, try again
William Edward Hickson (1803-1870)

The definition of insanity is doing the same thing over and over but expecting different results
Ben Franklin, Albert Einstein, or Narcotics Anonymous

I like these quotes because they’re both correct in their own way. There’s value to Hickson’s exhortation. Success rates are low for most schemes and most funders, so even if you’ve done everything right, the chances are against you. To be successful, you need a degree of resilience to look for another funder or a new project, rather than embarking on a decade-long sulk, muttering plaintively about how “the ESRC doesn’t like” your research whenever the topic of external funding is raised.

However Franklin et al (or al?) also have a point about not learning from the experience, and repeating the same mistakes without learning anything as you drift from application to application. While doing this, you can convince yourself that research funding is a lottery (which it isn’t) and all you have to do is to submit enough applications and eventually your number will come up (which it won’t). This is the kind of approach (on the part of institutions as well as individuals) that’s pushed us close to ‘demand management’ measures with the ESRC. More on learning from the experience in a moment or two.

2.  Can you do the research anyway?

This might seem like an odd question to ask, but it’s always the first one I ask academic colleagues who’ve been unsuccessful with a grant application (yes, this does happen,  even at Nottingham University Business School). The main component of most research projects is staff time. And if you’re fortunate enough to be employed by a research-intensive institution which gives you a generous research time allocation, then this shouldn’t be a problem. Granted, you can’t have that full time research associate you wanted, but could you cut down the project and take on some or all of that work yourself or between the investigators? Could you involve more people – perhaps junior colleagues – to help cover the work? Would others be willing to be involved if they can either co-author or be sole author on some of the outputs? Could it be a PhD project?

Directly incurred research expenses are more of a problem – transcription costs, data costs, travel and expenses – especially if you and your co-investigators don’t have personal research accounts to dip into. But if it turns out that all you need is your expenses paying, then a number of other funding options become viable – some external, but perhaps also some internal.

Of course, doing it anyway isn’t always possible, but it’s worth asking yourself and your team that question. It’s also one that’s well worth asking before you decide to apply for funding.

3.  What can you learn for next time?

It’s not nice not getting your project funded. Part of you probably wants to lock that application away and not think about it again. Move onwards and upwards, and perhaps trying again with another research idea. While resilience is important, it’s just as important to learn whatever lessons there are to learn to give yourself the best possible chance next time.

One lesson you might be able to take from the experience is about planning the application. If you found yourself running out of time, or not getting sufficient input from senior colleagues, not taking full advantage of the support available within your institution, well, that’s a lesson to learn. Give yourself more time, start earlier before the deadline, and don’t make yourself rush it. If you did all this last time, remember that you did, and the difference that it made. If you didn’t, then the fact is that your application was almost certainly not as strong as it could have been. And if your application document is not the strongest possible iteration of your research idea, your chances of getting funded are pretty minimal.

I’d recommend reading through your application and the call guidance notes once again in the light of referees’ comments. Now that you have sufficient distance from the application, you should ‘referee’ it yourself as well. What would you do better next time? Not necessarily individual application-specific aspects, but more general points. Did your application address the priorities of the call specifically enough, or were the crowbar marks far too visible? Did you get the balance right between exposition and background and writing about the current project? Did you pay enough attention to each section? Did you actually answer the questions asked? Do you understand any criticisms that the referees had?

4. Can you reapply?  Should you reapply?

If it’s the ESRC you’re thinking about, then the answer’s no unless you’re invited.  I think we’re still waiting on guidance from the ESRC about what constitutes a resubmission, but if you find yourself thinking about how much you might need to tinker with your unsuccessful project to make it a fresh submission, then the chances are that you’ll be barking up the wrong tree. Worst case scenario is that it’s thrown straight out without review, and best case is probably that you end up with something a little too contrived to stand any serious chance of funding.

Some other research funders do allow resubmissions, but generally you will need to declare it. While you might get lucky with a straight resubmission, my sense is that if it was unsuccessful once it will be unsuccessful again. But if you were to thoroughly revise it, polish it, take advice from anyone willing to give it, and have one more go, well, who knows?

But there’s really no shame in walking away. Onwards and upwards to the next idea. Let this one go for now, and working on something new and fresh and exciting instead. Just remember everything that you learnt along the way. One former colleague once told me that he usually got at least one paper out of an application even it was unsuccessful. I don’t know how true that might be more generally, but you’ve obviously done a literature review and come up with some ideas for future research. Might there be a paper in all that somewhere?

Another option which I hinted at earlier when I mentioned looking for the directly incurred costs only is resubmitting to another funder. My advice on this is simple… don’t resubmit to another funder. Or at least, don’t treat it like a resubmission. Every research funder, every scheme, has different interests and priorities. You wrote an application for one funder, which presumably was tailored to that funder (it was, wasn’t it?). So a few alterations probably won’t be enough.

For one thing, the application form is almost certainly different, and that eight page monstrosity won’t fit into two pages. But cut it down crudely, and if it reads like it’s been cut down crudely, you have no chance. I’ve never worked for a research funding body (unless you count internal schemes where I’ve had a role in managing the process), but I would imagine that if I did, the best way to annoy me (other than using the word ‘impactful‘) would be sending me some other funder’s cast-offs. It’s not quite like romancing a potential new partner and using your old flame’s name by mistake, but you get the picture. Your new funder wants to feel special and loved. They want you to have picked out them – and them alone – for their unique and enlightened approach to funding. Only they can fill the hole in your heart wallet, and satisfy your deep yearning for fulfilment.

And where should you look if your first choice funder does not return your affections? Well, I’m not going to tell you (not without a consultancy fee, anyway). But I’m sure your research funding office will be able to help find you some new prospective partners.

If your research application is unsuccessful then consider running it through our internal peer review scheme (the RPRS) to see about resubmitting the idea to an alternative funding body. Speak to Caroline O’Kane if you’re interested.

Coping with rejection: what to do if your grant application is unsuccessful part 1

Guidance

With only so much grant money in the world, Adam Golberg’s first of a two-part series, looks at how to move forward when it becomes clear that your time courting a potential funder comes to an end. Adam works at the University of Nottingham and runs the Cash for Questions blog – http://socialscienceresearchfunding.co.uk/.

Some application and assessment processes are for limited goods, and some are for unlimited goods, and it’s important to understand the difference.  PhD vivas and driving tests are assessments for unlimited goods – there’s no limit on how many PhDs or driving licenses can be issued.  In principle, everyone could have one if they met the requirements.  You’re not going to fail your driving test because there are better drivers than you.  Other processes are for limited goods – there is (usually) only one job vacancy that you’re all competing for, only so many papers that a top journal accept, and only so much grant money available.

You’d think this was a fairly obvious point to make.  But talking to researchers who have been unsuccessful with a particular application, there’s sometimes more than a hint of hurt in their voices as they discuss it, and talk in terms of their research being rejected, or not being judged good enough.  They end up taking it rather personally.  And given the amount of time and effort that must researchers put into their applications, that’s not surprising.

It reminds me of an unsuccessful job applicant whose opening gambit at a feedback meeting was to ask me why I didn’t think that she was good enough to do the job.  Well, my answer was that I was very confident that she could do the job, it’s just that there was someone more qualified and only one post to fill.  In this case, the unsuccessful applicant was simply unlucky – an exceptional applicant was offered the job, and nothing she could have said or done (short of assassination) would have made much difference.  While I couldn’t give the applicant the job she wanted or make the disappointment go away, I could at least pass on the panel’s unanimous verdict on her appointability.  My impression was that this restored some lost confidence, and did something to salve the hurt and disappointment.  You did the best that you could.  With better luck you’ll get the next one.

Of course, with grant applications, the chances are that you won’t get to speak to the chair of the panel who will explain the decision.  You’ll either get a letter with the decision and something about how oversubscribed the scheme was and how hard the decisions were, which might or might not be true.  Your application might have missed out by a fraction, or been one of the first into the discard pile.

Some funders, like the ESRC, will pass on anonymised referees’ comments, but oddly, this isn’t always constructive and can even damage confidence in the quality of the peer review process.  In my experience, every batch of referees’ comments will contain at least one weird, wrong-headed, careless, or downright bizarre comment, and sometimes several.  Perhaps a claim about the current state of knowledge that’s just plain wrong, a misunderstanding that can only come from not reading the application properly, and/or criticising it on the spurious grounds of not being the project that they would have done.  These apples are fine as far as they go, but they should really taste of oranges.  I like oranges.

Don’t get me wrong – most referees’ reports that I see are careful, conscientious, and insightful, but it’s those misconceived criticisms that unsuccessful applicants will remember.  Even ahead of the valid ones.  And sometimes they will conclude that its those wrong criticisms that are the reason for not getting funded.  Everything else was positive, so that one negative review must be the reason, yes?  Well, maybe not.  It’s also possible that that bizarre comment was discounted by the panel too, and the reason that your project wasn’t funded was simply that the money ran out before they reached your project.  But we don’t know.  I really, really, really want to believe that that’s the case when referees write that a project is “too expensive” without explaining how or why.  I hope the panel read our carefully constructed budget and our detailed justification for resources and treat that comment with the fECing contempt that it deserves.

Fortunately, the ESRC have announced changes to procedures which allow not only a right of reply to referees, but also to communicate the final grade awarded.  This should give a much stronger indication of whether it was a near miss or miles off.  Of course, the news that an application was miles off the required standard may come gifted wrapped with sanctions.   So it’s not all good news.

But this is where we should be heading with feedback.  Funders shouldn’t be shy about saying that the application was a no-hoper, and they should be giving as much detail as possible.  Not so long ago, I was copied into a lovely rejection letter, if there’s any such thing.  It passed on comments, included some platitudes, but also told the applicant what the overall ranking was (very close, but no cigar) and how many applications there were (many more than the team expected).  Now at least one of the comments was surprising, but we know the application was taken seriously and given a thorough review.  And that’s something…

So, in conclusion,  just because your project wasn’t funded doesn’t (necessarily) mean that it wasn’t fundable.  And don’t take it personally.  It’s not personal.  Just the business of research funding.

If your research application is unsuccessful then consider running it through our internal peer review scheme (the RPRS) to see about resubmitting the idea to an alternative funding body. Speak to Caroline O’Kane if you’re interested.

This post is the first in a 2-part series. We’ll published the second part next week. This was originally published on Adam Goldberg’s excellent Cash for Questions blog.