Experiences of a Marie Curie Expert Evaluator

Dr Sara BenettiEnvironmental Sciences Research

Institute, University of Ulster

• Vice-chair for FP7 Marie Curie IEF/IIF/IOF fellowship evaluation (2013)

• Expert evaluator for FP7 Marie Curie IEF/IIF/IOF fellowship evaluation (2011)

• Since 2011: expert evaluator for European Union FP7 programme “New and renewable sources of energy, Energy efficiency & innovation”

IEF: Intra-European Fellowships;IIF: International Incoming Fellowships;IOF: International Outgoing Fellowships;CAR: Career Restart.

Panel:

• Project officer/s (EU person/s);• Chair;• Vice-chairs (normally 1 to 8-15 evaluators);• Expert evaluators (lots and from all fields

within the panel topic!).

Marie Curie fellowships: evaluation process (1)

Candidates submit proposals to specific panel CHE, ECO, ENG, ENV, LIF, MAT, PHY, SOC

Evaluators to declare any conflict of interest and

select proposals (at least 100) to evaluate based on

the abstract

"1" denotes that the proposal is exactly in the expert's field of expertise;"2" denotes that the proposal is in the expert's broader field of expertise; "3" denotes that the proposal is not exactly in the expert's field of expertise s/he could evaluate it if necessary;"No" "No expertise" (by default this option is selected in all proposals).

Marie Curie fellowships: evaluation process (2)

Depending on range of expertise, evaluators are selected and assigned up to 20 proposals to

evaluate in a relatively short period (3-4 weeks)

• Each proposal has 3 evaluators;• Evaluators will not all be expert in the specific field;• They do not have contact with each others;• They are not allowed to contact the applicant/s.• Proposals are evaluated strictly in relation to the evaluation

criteria;• If proposal longer than allowed word count, anything over

the word count will NOT be read or evaluated.

REMOTE EVALUATION

IAR

Marie Curie fellowships: evaluation process (2)

REMOTE EVALUATION: scoring

Marie Curie fellowships: evaluation process (2)

REMOTE EVALUATION:criteria

IEF/IOF

Overall threshold of 70%

Marie Curie fellowships: evaluation process (2)

REMOTE EVALUATION:criteria

Marie Curie fellowships: evaluation process (2)REMOTE EVALUATION:

criteria

Marie Curie fellowships: evaluation process (2)

OUTREACH ACTIVITIES

Marie Curie fellowships: evaluation process (3)

Rapporteur to write CONSENSUS REPORT Draft CR

The rapporteur is one of the 3 evaluators of a proposal. He/she has the responsibility to:• Collate a draft CR taking into account all of the 3

evaluators’ comments.• Lead the consensus meeting (normally in

Brussels).• Write up the final CR and get it approved by

other evaluators and proof-read by his/her vice-chair before final submission.

Marie Curie fellowships: evaluation process (4)

Remotely or in Brussels, consensus meeting among 3 evaluators takes place.

• The 3 evaluators discuss the contents of the 3 IARs and draft CR;• Comments in agreement are normally included in CR;• Conflicting evaluations are discussed in more detail until (if)

agreement among evaluators is reached;• Marks for each criterion are then decided on the basis of the

evaluations.

CONSENSUS MEETING

2nd draft of CR

Marie Curie fellowships: evaluation process (5)

• 2nd draft of CR is checked and approved by all evaluators;• Rapporteur’s vice-chair proof reads, checks for

inconsistencies and match of comments to mark;• If other evaluators do not approve the report, this process

is reiterated until an agreement is reached on the final CR.

If agreement evaluators is reached…

Final CR submitted to EC

Marie Curie fellowships: evaluation process (5b)

• Second (third/fourth!) meeting may be scheduled to allow more time for discussion;

• Fourth evaluator may be selected, he/she will draft an independent IAR and join the following consensus meeting/s;

• A moderator may be brought in (vice-chair; chair; project officer.

• A consensus must be reached!

If agreement among evaluators is not reached…

Several iterations of CR until consensus is reached

Marie Curie fellowships: evaluation process (6)

• Those with an overall mark below threshold are out.• Those with any mark below the threshold for specific

criteria are out.• Funding is allocated from highest scoring proposal

going down, depending on funds (this changes from one year to the next).

All proposals in panel are ranked

Not a rule of thumb but any proposal below 85% is unlikely to get funded!

EU Marie Curie proposals

1. Good science;

2. Well explained and at the right level (evaluators may not be overly familiar with the science but they are scientists and they will have very limited time to read your proposal);

3. Do your homework: check criteria/requirements/etc. and stick to them;

4. Make sure you address every single evaluation criterion;

5. You must ‘prove’ everything! It is not enough that the host institution is ‘well-known’ for…how can you tell?

6. Outreach ‘points’ are easy to get if you make a little effort.

Tips for success:

EU Marie Curie funding: the way forward

1. Name change (politics?);

2. Criteria will be reduced in number and simplified (to avoid repetition as much as possible);

3. It is possible more work will be done remotely (even harder for evaluators to discuss pros and cons of each proposal!);

4. Stick to the tips for success;

5. Level of funding may increase from one year to the next.

Call for Marie Skłodowska-Curie Individual Fellowships (IF)H2020-MSCA-IF-2014 out in March 2014