How to become Expert - Evaluator

in H2020

Marcela Groholova

Research Executive Agency

Brussels

25/2/2015

Bratislava, Slovak Republic

Evaluation - the most important step in the project management cycle

The best proposals selected

=

The best proposals funded

Contents:

•Experts database

•Experts selection

•Conflict of interest

•Confidentiality

•Projects to evaluate

•Remote evaluation

•Central week

•Evaluation reports

•Reimbursement

•Good expert

Experts registration:

Via Participant Portal

https://ec.europa.eu/research/participants/portal/desktop/en/experts/index.html

- Expert creates his/her ECAS login

Registration for new experts

Update your existing profile

Programme Selection Horizon 2020 (including Euratom programme)

COSME (Competitiveness for Small and Medium-sized

Enterprises)

Education, Audiovisual and Culture Executive Agency (EACEA) programmes

CEF (Connecting Europe Facility)

RFCS (Research Fund for Coal and Steel)

Third Health Programme (managed by Chafea)

Personal details – name, nationality, gender (experts statistics)

Contact details – how to contact you and where to send any documents

Education: Qualification, Field, Name of the institution, Country, Year Awarded

Languages:

Writing

Reading

Conversation

Area of expertise:

Specialized field – predefined from the menu

Free keywords

Professional experience:

employer history

Experience in the field

Publications and achievements

How long you worked in your specialist field

Have you ever worked in industrial sector

Have you assisted to EC in evaluation process? (evaluator, reviewer, NCP, etc.)

Do you have any other experience in peer review process

Additional info: upload your most recent CV

Experts selection: •Expertise •Nationality balance •Gender (at least 40% female) •Private – Public balance •At least 25% of newcomers for each call •Max 120 days in 4 consequent years (incl. Review and other services)

•Pre-selection of bigger pool for some actions (bottom-up calls), final selection after call closure

Overview of the experts approval process: Approval of the pool by EC (internal) Valid Bank Account + Legal Entity form for each expert Drafting the contract Signing the contract – electronically by expert and by EC EVALUATION Payment session created Submission of payment documents by expert – electronically,

originals to be kept by expert up to 2 years in case of audit Payment within 30 days Days count in our internal system

Once you are selected for the call – BRIEFING (online or in Brussels)

Conflict of interest: = Reliability of expert! •1st time – when accepting participation and signing contract •2nd time – when list of participants is available (some calls) •3rd time – when full proposal is received in SEP

•Expert can declare CoI at any moment during evaluation!

Note: It is very important to review in detail all the proposals allocated to you so that you are able to promptly identify any

potential conflict of interest and decline the respective task(s).

In case of CoI, please immediately inform your respective PO in EC/agency and/or decline task in SEP

Confidentiality of evaluation: Expert must follow strict confidentiality principles with respect to whole evaluation process. Under no circumstance expert may contact applicants, send any document produced or received during evaluation to the third person or to inform the third person about evaluation result.

Projects to evaluate: • Expertise – Specialist vs Generalist • Number of projects can vary – from 1 (specific top-down

calls) to 20+ (bottom-up calls) • private – public balance for some calls • Number of pages – from 10 to 100 – page limit nearly in

all calls • Complexity of projects – based on the calls

Evaluation steps: 1. REMOTE

2. CENTRAL

3. PANEL MEETING 3-5 experts assigned to each proposal (average)

Remote evaluation: • From home or work place • Can take several weeks (normally 2-3 weeks) • Each expert evaluate all proposals allocated to him/her • Submission of IER (Individual Evaluation Report) electronically • Experts don't know other co-experts evaluating the same

proposals • Evaluation based on own experience in the field and your best

knowledge about the topic • Calibration of marks when more proposals to be evaluated • End of Remote phase:

• All IERs must be submitted in the system • Rapporteurs are allocated to each proposal – preparation for

central week

Central evaluation: • In Brussels – all experts who act as Rapporteurs or remotely

(some cases) • 1st time all experts meet around one table • Day 1 – experts get instructions + their individual schedule for

each consensus meeting (room number, date and timing) • Each proposal can be discussed between 30 minutes and 2 hours

depends on complexity

Consensus must be reached! • If needed, another evaluator may be asked to step in (exceptional

cases – CoI during central week, not possible to reach consensus, etc.)

• Rapporteur - one of the experts • Moderator – either Rapporteur or EC official • End of the meeting – rapporteur drafts CR (consensus report) • All experts must approve CR electronically

Panel meeting: • End of the central week (last day) • All rapporteurs must approve Ranking list • Ranking list automatically created by the system based on

the scores in CR • Ex-aequo cases – Work Programme must be respected • If possible: rapporteurs present briefly most successful

proposals ranked as first in each panel ranking • Free discussion on evaluation process – suggestions,

comments

How to prepare evaluation report:

Example from MSCA call

Drafting evaluation report: - Comments for each criterion and each sub-criterion

(some calls)

OR - Strengths and Weaknesses for each criterion and sub-

criterion (some calls)

H2020 - 3 main criteria: Excellence, Impact and Implementation

Drafting evaluation report: 1. Comment on each element listed underneath the main criteria;

e.g. under "Excellence" for ITN call you should evaluate following sub-criteria:

• Quality and innovative aspects of the research

programme • Quality of the training programme • Quality of the supervision

2. Then assign an overall score for each criterion in line with

your comments.

During Consensus meeting: • All experts should agree on the strengths and weaknesses /

comments. • Experts agree on an overall score for each criterion in line with

the strengths and weaknesses / comments. If requested, for the IER and CR, you may provide overall comments. These should specify the relative importance of the strengths and weaknesses. For example:

‘This proposal is very good overall on this criterion. Its strengths are A, B and C. Its shortcoming is D.’

Score = 4.0

Scoring: • Score each criterion from 0 up 5 points. Decimals or half-points available.

• Some criteria may have a threshold (3 or 4).

• Maximum score:

• 15 for some CSA and collaborative projects • 100 for MSCA (weighted score, calculated automatically)

• Minimum threshold for the overall score of the proposal:

• 10 for CSA and collaborative projects • 70 for MSCA

• The total score calculated automatically.

Scoring:

Please remember to use the full scoring scale

0 – Proposal fails to address or cannot be assessed due to missing or incomplete information

1 – Poor. The criterion is inadequately addressed or there are serious weaknesses

2 – Fair. Proposal broadly addresses the criterion, but there are significant weaknesses

3 – Good. The proposal addresses the criterion well, but number of shortcomings are present

4 – Very good. The proposal addresses the criterion very well, but small number of shortcomings are present

5 – Excellent. Proposal successfully addresses all relevant aspects of the criterion, any shortcomings are minor.

Example of comments in the evaluation report:

Overall comments must be:

• Specific to the relevant criterion – you should not include information e.g. on the quality of the project management under "excellence" when there is a specific section on this under "implementation"

• Clear and substantial

• Your comments should state facts, not opinions, and be precise and final: "We think that…" "This proposal is… "

• The score must be a fair reflection of the comments, balancing strengths and weaknesses.

• Comments should be of adequate length: you should not right just one sentence, but neither should you write a whole booklet.

• Comments judge the proposal, they do not summarise it. you do not need to repeat what the applicants have written in the proposal in your report.

• You are evaluating the quality of the proposal only. So you should not provide in your report advice on improving the proposal.

Good comments explain it

For example,

“This proposal is not innovative in X or

Y, and it does not take Z into account”.

Poor comments merely echo the score

For example,

“The innovative aspects of the research

programme are poor”.

Poor comments are ambiguous

“The resources for the project are

unrealistic”

Good comments are clear

“The resources in WP 4 and 6

are seriously underestimated

given the complexity of the

activity proposed”.

Good comments are precise and final

“The management plan is inadequate. It does not

include clear overall responsibility for the training

activities; it lacks a problem-solving mechanism in the

event of disputes between partners”.

Poor comments are vague, subject to

interpretation

“We think the management plan is probably

inadequate given the duration of the project and the

number of partners”.

Good comments close the question

“Dissemination activities are listed but the proposal lacks a clear

dissemination strategy”.

“The consortium lacks sufficient SME participation.”

“The coordinator does not demonstrate in the proposal an adequate

level of experience in this field.”

Poor comments are inaccurate and provide an opening for a

complaint

“There is no discussion of a dissemination strategy.”

“There is only one SME partner in the consortium.”

When there were actually 2.

“The coordinator is not adequately experienced."

Good comments

include words like…

Because

Percent

Specifically

For example

Poor comments

include words like…

Perhaps

Think

Seems

Assume

Probably

Insufficient, very generic, not evident, unfocused, limited, unclear, no significant impact, overestimated …

For strengths

Extremely relevant, credible, comprehensive, high quality, highly effective, well-formulated, well balanced, clear advances, …

For weaknesses, you might use

Strengths: • The S&T objectives of the research project are well presented and clearly structured. • The partners have complementary expertise. • The joint collaborative research programme is of good quality. • The project is original and the state of the art is adequately presented.

It is essential that your scores are consistent with comments

Extract from an evaluation report from a previous evaluation:

And this project on this criterion was given a score = 4.5 (between very good and excellent)

A score which does not adequately reflect the weaknesses described.

However, Weaknesses: • The methodology for the project is not fully convincing. • The data collection strategy, potential sources of information and data accessibility are unclear. • The proposals fails to sufficiently demonstrate that the consortium has the necessary expertise and capabilities to obtain the necessary information needed for the project.

Weaknesses of the proposal: •The proposal is not specific enough to clearly show its innovative and original aspects. •The contribution of the private sector is not described clearly. •The connection between the basic research topics and the 3 translational projects is not well presented.

This achieved a score = 4.4 (between very good and excellent)

Some more examples of inconsistencies:

Are scores consistent with comments?

Comments of adequate length?

Are all sub-criteria addressed?

Operational capacity confirmed?

No factual errors! (WP7 is not adequately described...) in fact there

where only 5 WPs in the proposal!

No discriminatory comments! (Partner 3 is not capable to fulfil

tasks from WP3 and 4)

Final check before finalizing CR:

Reimbursement: • Online via electronic system, no paper documents to be sent

anymore (originals kept by expert for audit) • End of evaluation experts submit all costs via Participant Portal

• Lump sum of 450 Euro per working day calculated to the

nearest half day • A daily allowance of 92 Euro or 46 Euro if your point of

departure is less that 100 km from Brussels • An accommodation allowance of 100 Euro, if applicable • Full reimbursement of your travel expenses

• Taxi + parking fee is not reimbursed

Good expert: • Timely in delivering reports and during consensus meetings • In case of problem informs ON TIME • Reliable • Flexible • Able and willing to learn and accept our process • Written and spoken English of good quality • Able to judge and evaluate the project, not to copy/paste • If comments are negative – WHY? • If comments are positive – WHY?

We do not look for expert-superman, everybody can learn it...

Good luck