horizon 2020 sme instrument experts briefing part 2 · proposal scoring each evaluation...
TRANSCRIPT
Horizon 2020 SME Instrument Experts Briefing
Part 2
CONTENT
1. The Evaluation Process;
2. Your role as independent experts:
3. The Individual Evaluation Report;
4. Tips & tricks to complete the IER.
1. The Evaluation Process
Overview of the Evaluation Process for the SME Instrument
Receipt of proposals
Individual evaluation
Automated Consensus
Report Finalisation
Remote evaluation
Individual Evaluation Reports
(explanatory comments)
Automated Consensus
Report (arithmetic median/
No negotiation)
Standardised Evaluation Summary
Report
Eligibility check
Allocation of proposals to evaluators
Final ranked list
Informing the
applicants
Admissibility and Eligibility Checks
Admissibility - proposals must be:
Readable, Accessible and Printable;
Complete (all requested forms);
Eligibility:
SME status, country;
Limited number of pages (10 – Phase 1; 30 - Phase 2)
Excess pages are watermarked - ignore them;
Only one application per company allowed for all phases (no concurrent submission or implementation).
If you spot an issue relating to eligibility, please inform EASME
Evaluation time line
Allocation of all proposals right after the cut-off date;
Accept/Decline task within 24hrs- otherwise the task will be reallocated to another expert;
7 calendar days to complete the evaluations;
Inform us as early as possible if you face difficulties to meet this deadline;
Completing the evaluation reports before the deadline is appreciated.
Payment of experts
New fee since January 2017
Fee = 90 € (or 2 working units) per Phase 1 proposal and 135€ (or 3 working units)/Phase 2 proposal;
1 working unit is worth 45€ (0,1*450€) - 1 working day represents 10 working units (450€)
Payment is done separately for Phase 1 & 2 and starts at the end of the evaluation process;
You will receive an email informing you that the payment procedure is starting;
Please submit your request for payment within the given deadline;
If you miss claiming your reimbursement within the designated period, we will be unable to do ad-hoc payments and there will be a long delay in payment;
2. Your role as independent expert
Guiding principles
Independence: evaluating in your personal capacity;
Impartiality: treating all proposals equally;
Objectivity: evaluating each proposal as submitted;
Accuracy: evaluating against the official evaluation criteria;
Consistency: applying same standards of judgment.
Confidentiality
As we are dealing with close-to-market activities, confidentiality is crucial and is the basis for a trusted evaluation.
Do not discuss evaluation matters;
Do not contact partners in the consortium, sub-contractors or any third parties;
Maintain the confidentiality of documents: destroy or delete all confidential documents upon completing your work.
Conflict of interest
It is the responsibility of the evaluator to determine whether a conflict of interest exists.
Before accepting any evaluation:
• Check if you are in a situation of conflict of interest;
• If yes, decline the task assigned and justify the reason;
• Inform the SME Instrument team.
Please note that EASME has the final decision on whether a conflict of interest exists.
Definition of conflict of interest (1/2)
Check Article 2 – Annex 1 - Code of conduct of the experts' contract.
A conflict of interest arises if an expert:
a) was involved in the preparation of the proposal;
b) stands to benefit directly or indirectly if the proposal is accepted;
c) has a close family or personal relationship with any person representing an applicant legal entity;
d) is a director, trustee or partner or is in any way involved in the management of an applicant legal entity;
e) is employed or contracted by one of the applicant legal entities or any named subcontractors;
f) is a member of an Advisory Group set up by the Commission to advise on the preparation of EU or Euratom Horizon 2020 Work Programmes, or Work Programmes in an area related to the call for proposals in question;
g) is a National Contact Point, or is directly working for the Enterprise Europe Network;
h) is a member of a Programme Committee.
Definition of conflict of interest (2/2)
In the following situations, EASME will decide whether a COI exists
Were employed by an applicant or sub-contractor in the last 3 years;
Were involved in a grant agreement, the membership of management structures or a business collaboration with an applicant in the last 3 years;
Are in any other situation that casts doubt on your impartiality or that could reasonably appear to do so
Am I in a conflict of interest situation?
I have been involved as a consultant/advisor/service provider/applicant preparing a proposal.
Yes, if you are evaluating proposals for the SME Instrument whatever the topic. Please note that you may be required to suspend your evaluator activities during the ongoing evaluation.
I have been asked to give a presentation on the programme.
No, there is no conflict of interest if you speak in general about the Programme.
Yes, if you mention the contents/details of a proposal you have evaluated.
Can I be an evaluator and a coach at the same time?
No, in this combination of roles there is a potential conflict of interest.
Am I in a conflict of interest situation?
3. Individual Evaluation Report
Proposal scoring Each evaluation sub-criterion is scored out of 10 points (one
decimal may be used);
Each evaluation sub-criterion question has the same weight, except overall perception that weights 25% of the total score of that criterion;
The individual scores (from 0 to 10) given to each sub-criterion are automatically converted to a scale from 0 to 5 to calculate each of the three Criterion scores per evaluator.
The total maximum score for a proposal is 15 points.
The scale used to obtain the qualitative assessment is the following:
Scores from 0 to 2.99 – generate "Insufficient" Scores from 3 to 4.99 – generate "Insufficient to Fair" Scores from 5 to 6.99 – generate "Fair to Good" Scores from 7 to 8.99 – generate "Good to Very Good" Scores from 9 to 10.0 – generate "Very Good to Excellent"
The final score
Each proposal is assessed by 4 evaluators;
The median score across all 4 evaluators for each criterion
Overall score is the sum of the 3 criteria median scores.
How is an IER structured? Please note that the formulation of several sub-criteria was
modified and the order has changed compared to 2016
PHASE 1 PHASE 2
Operational capacity
Impact Impact
Excellence Excellence
Quality and efficiency of implementation
Quality and efficiency of implementation
Subcontracting
Scope of the proposal Scope of the proposal
Threshold: 13 Threshold: 12
Impact PHASE 1 PHASE 2
Threshold: 4 Threshold: 4
Sub-criteria: 9 Sub-criteria: 9
Main aspects to tackle in the evaluation:
• Compliance with the relevant Work Programme; • The demand/market of the innovation proposed; • The targeted users or user groups; • The market conditions; • The impact on the growth of the applying company; • The commercialisation plan; • The European dimension; • The IPR filing status and ownership, licensing; • The Regulatory and standard requirements. To determine the ranking, the score for the criterion ‘impact’ will be given a weight of 1.5.
Excellence
PHASE 1 PHASE 2
Threshold: 4 Threshold: 3
Sub-criteria: 7 Sub-criteria: 8
Main aspects to tackle in the evaluation: • The new market opportunities for EU/global challenges; • The current stage of development of the innovation (for
Phase 2 TRL6 or above except SMEInst-05 and phasing out PHC-12);
• The comparison with known commercial solutions; • The objectives and the approach/activities to be
developed; • The commercial viability of the innovation proposed; • The risks and opportunities of the market introduction.
Implementation
PHASE 1 PHASE 2
Threshold: 4 Threshold: 3
Sub-criteria: 4 Sub-criteria: 5
Main aspects to tackle in the evaluation: • The resources to develop the activities; • The technical/scientific knowledge/management
experience; • The time frame and the implementation description; • The work packages and major deliverables and
milestones.
For Phase 2 only: subcontracting
Technological Readiness Level TRL – NEW
Phase 2 only
The TRL described in the proposal has to be assessed by replying to the following question:
Is the project proposed containing activities above TRL 8?
TRL 8 corresponds to 'system complete and qualified' (not yet proven in operational environment).
The answer is set to 'No' by default in the Individual Evaluation Report Form (IER). If your assessment reveals a TRL>8, switch the radio button to 'Yes'.
Please note that it is NOT an evaluation criterion. We ask you to note the level of TRL but being above TRL8 will not disqualify the proposal.
Only to be used when a proposal is very clearly submitted in a wrong topic.
If a proposal is partially relevant to the topic, it should be considered within scope.
If a proposal is considered not innovative, not disruptive, not well explained, incomplete, etc., the proposal is still within scope but this opinion should be expressed by means of a lowered score in the relevant (sub) criteria.
The 'out of scope' option
The 'out of scope' option
As long as there is a link between a proposal and the challenges described in the relevant Work Program, it is IN scope. Examples: • A proposal aiming at developing a technology for conversion of
pressurized gas to energy is at TRL3 instead of TRL6. The proposal is IN scope!
• A proposal concerning an innovative textile for the fashion industry is submitted under the Transport topic. The proposal is OUT of scope!
Operational capacity - Phase 2 only
If you believe that an applicant does NOT have the operational capacity to carry out the proposed work, you should choose NO, justify the reason and score the Quality and efficiency of implementation below the threshold (<3).
Operational capacity - Phase 2 only
Examples: • The company does not have technical & financial resources to
manage the project/subcontractors. NO!
• The company will subcontract many tasks but has the resources to manage the core activities. YES!
Suggestions: • Check the company's website
• Check the team involved in the project
Subcontracting - Phase 2 only
• Regulated under Art 13 of the H2020 SME Instrument Phase 2 Model Grant Agreement (see link - page 551);
• Subcontracting is NOT restricted to a limited part of the action;
• It is in the SME Instrument spirit that the applying SME has the capacity to carry out the activity;
• Compliance with best value-for-money is assessed during the evaluation (not for topic SMEInst-05-2016-2017 and phasing out PHC12).
Subcontracting - Phase 2 only
• A table in annex 4-5 to detail each subcontractor and task subcontracted.
Subcontracting - Phase 2 only
For each subcontracted task, there are only two options: • Yes • No or lack of explanation If "no or lack of explanation", experts need to justify and reflect this in the assessment in the Quality & efficiency of implementation criterion (score below the threshold: <3). By default, the task is set to 'yes' even when the proposal does not foresee any subcontracting. If there are no subcontractors in the proposal you shall not change this "yes" set by default.
Health topics
SMEInst-06-2016-2017 - Accelerating market introduction of ICT solutions for Health, Well-Being and Ageing Well (same rules as other topics)
SMEInst-05-2016-2017
a) Supporting innovative SMEs in the healthcare biotechnology sector (No phase 1 applications at all in 2017)
b) Clinical research for the validation of biomarkers and/or diagnostic medical devices (only at the first cut-off date in 2017 and for phase 2 applications - phasing out of the PHC12-2014-2015 topic introduced in the Work Programme 2014-2015)
Specific Rules for SMEInst-05
• TRL indication does not apply • Contribution up to € 5 million • Funding rate at 100% • General model Grant Agreement applies • No evaluation of best value for money of subcontracting
4. Tips and tricks to complete the IER
How to write a quality IER?
Maintain a very high quality standard – do not be too positive;
Exercise critical judgement - assess the credibility or plausibility of very positive growth figures for instance
Reflect shortcomings in a lower score for the relevant criterion;
Provide explanation of shortcomings but do not give recommendations;
Remember if you score above threshold, the proposal has a high chance to get funding so think as an investor, would you put money in this proposal
No grant negotiation phase!
− You evaluate each proposal as submitted, not on its potential if certain changes were to be made
− Maintain a very high quality standard. Any proposal which does not meet this should not receive above-threshold scores
− If you identify shortcomings (other than minor ones and obvious clerical errors), you must reflect those in a lower score for the relevant criterion
− You should explain the shortcomings, but do not make recommendations
− Proposals with significant shortcomings should not receive above-threshold scores
− Any proposal with scores above the thresholds and for which there is sufficient budget will be selected as submitted
How to write a quality IER?
Yes
No
Dedicated and thorough comments for each sub-criterion
Comments repeating the sub-criterion description and/or identical comments for all sub-criteria and/or blank text boxes
Consistency between scores and comments
Good score but negative comments
Coherence between operational capacity/subcontracting and the score of the 'Implementation' criterion
Selecting NO for operational capacity/subcontracting and scoring 'Implementation' above threshold (3)
During each cut-off
Due to the time constraints your availability and accountability is crucial! • Be accurate when you specify your availability; • Accept/decline the tasks assigned within 24 hours; • Let us know if you can take more evaluations; • Stick to the deadlines; • Inform us asap if you cannot cope with the workload; First cut-off Phase 2 on the 18 of January 2017 !
Further sources of information Horizon 2020 Participant Portal
http://ec.europa.eu/research/participants/portal/
EASME website
https://ec.europa.eu/easme/en/horizons-2020-sme-instrument
FAQ:
https://ec.europa.eu/easme/sites/easme-site/files/FAQ_for_experts_SME%20Instrument_02_10_15.pdf
Functional mail box for all your questions and comments
Project Officer responsible for each topic;
THANK YOU FOR YOUR ATTENTION
All images except SME Support logo © DrAfter123 iStockphoto.com