independent observers’ report - european commission · 2013-12-12 · independent observers’...

PART 1 FP7 - HEALTH-2010-single-stage

FP7 - HEALTH-2010-two-stage (1. stage) FP7 - INFLUENZA-2010

PART 2

FP7 - HEALTH-2010-two-stage (2. stage) FP7 - HEALTH-2010-Alternative-Testing

Evaluation November 2009 - March 2010

Independent Observers’ Report

Jan Krzysztof Frackowiak*) Director, Polish Science Contact Agency

Former Deputy Minister of Science – Poland

and

Hans Lehmann**) Former Head of National Contact Points, Life Sciences, Germany.

Former Member of the German Delegation to Life Sciences Programmes, FP3–FP7.

and

Guy Ebinger, MD, PhD (Rapporteur)**) Former Head of Department Neurology Free University Brussels (VUB)

Research Group on Experimental Pharmacology -VUB

*) Active for PART 1 **) Active for PARTs 1 and 2

FP7-Health-2010 Independent Observers’ Report

2

Abbreviations CR Consensus Report EIR Ethical Issues Report EPSS Electronic Proposal Submission System ERR Ethics Review Report ESR Evaluation Summary Report FP(6 or 7) Framework Programme IER Individual Evaluation Report IPR Intellectual Property Rights RIvET Software-support-tool for the remote evaluation of proposals S/T Scientific and or Technological Excellence SME Small and Medium Enterprise(s)


3

Table of Contents 1. Introduction 1.1 Summary of Major Observations 2. Independent Observers: Role and Approach Taken 2.1 Role of the Independent Observers 2.2 Approach taken to perform the task 3. Evaluation Process 3.1 General Design of the Evaluation Process 3.2 Briefing of Moderators 3.3 Selection and Briefing of Experts 4. Evaluation Criteria: Observations and Recommendations 4.1 Single-stage evaluation procedure 4.2 Two-stage evaluation procedure (first stage) 4.3 Two-stage evaluation procedure (second stage) 4.4 Proposal Scoring 5. Evaluation Process: Observations and Recommendations 5.1 Single-stage evaluation procedure 5.2 Two-stage evaluation procedure, first stage 5.3 Two-stage evaluation procedure, second stage 5.4 Single-stage evaluation procedure (Alternative Testing) 6. IT- Evaluation System – RIvET and Remote Evaluation 7. Consensus meetings – General Comments 7.1 Consensus Meetings: should all be readers? 7.2 In which order are the proposals discussed? 7.3 Role of “rapporteurs” 7.4 When are the scores to be disclosed to the evaluators? 7.5 Disclosure of scores to experts having a conflict of interest? 8. Ranking Panel Meetings 9. Ethical issues 10. Organisational Aspects of the Evaluation Procedure 11. Implementation General Conclusions Acknowledgements


4

1. Introduction. This report outlines the actual observation of the evaluation procedures applied under the FP7 Specific Programme “Cooperation”: FP7-HEALTH-2010-single-stage, FP7-HEALTH-2010-two-stage (1. stage), FP7-INFLUENZA-2010 (referred to as PART 1), FP7-HEALTH-2010-two-stage (2. stage) and FP7-HEALTH-2010-Alternative Testing (the latter two referred to as PART 2, see also cover sheet)1. The Independent Observers evaluate the conduct, the correct application of the evaluation criteria and the fairness of all phases of the process. They verify whether the correct procedures have been followed to ensure full transparency. The Independent Observers assess solely the work process and not the outcome of the procedures. They never intervene in the discussion and never express their views on a given proposal. The Independent Observers are expected to provide input and advice for the Euro-pean Commission in order to further ensure transparency and to support the further development of project evaluation and selection procedures in FP7-Health. Such assessment by Independent Observers of the evaluation procedures is a standard for the Theme “Health” as part of the Framework Programmes of the European Commission. For PART 1 three Independent Observers have been active, for PART 2 two (for details see cover sheet). The results will be reported to the Programme Management Committee with representatives from the Member States of the European Union and States Associated to FP7. This report will be published as were the preceding ones2. This report will be presented on March 22nd, (as regards PART 1) and on April 28th, 2010 (as regards PART 2), to the Programme Management Committee with representatives from the Member States of the European Union and States Associated to FP7. 1.1 Summary of Major Observations The observers found the entire process combining both the remote and central evaluations to be of high quality, transparent, rigorous and impartial. They were fully satisfied that the procedures set out in the guidelines were being adhered to.

1 For call documents see http://cordis.europa.eu/fp7/dc/, then follow “Closed calls”, “Health” 2 For reports of the Independent Observers for the first and second call, see http://cordis. europa.eu/fp7/dc/, as before, then follow “Closed calls”, “Health” and “Additional Documents”, respectively see http://cordis.europa.eu/fp7/health/library_en.html "About the evaluation process", all calls.


5

The interviewed evaluators shared this opinion: the general approach combining individual remote reviews of experts with Consensus - and Panel Meetings in Brussels proved to be appropriate in order to reach a transparent and fair decision. The high standard of the expertise of the evaluators and the professionalism and dedication of the EU Commission staff ensured an independent impartial and fair review process oriented towards a very high standard. We are satisfied that the procedure strictly followed the rules as laid down in the work programme and the respective Guides for Applicants2. The European Commission review process is based on scientific excellence and their approach compares favourably with those applied by other prominent international granting agencies. It constitutes a good standard for top-ranking evaluation procedures. The RIvET system used for the on-line remote evaluations facilitated greatly this phase of the review process and was unanimously appreciated by the interviewed evaluators. Also, the observers recommend that the current evaluation system should be retained for the future. Given the already high standard of the evaluation process, our recommendations mostly deal with minor aspects. They are outlined in sections 3 -11. Taking them into account could contribute to more transparency and efficiency in the formulation of the work programme and improve the evaluation procedure as a whole. 2. Independent Observers: Role and Approach Taken 2.1 Role of the Independent Observers The role of the Independent Observers is to give independent advice to the Commission on the conduct of all phases of the evaluation process, on the application of the evaluation criteria used by the evaluators in the sessions. They formulate recommendations and suggestions for further improvements of the evaluation procedures on ways in which procedures could be improved. The role of the Independent Observers is also to ensure a high degree of transparency and to ensure that throughout the entire evaluation and decision process, it fulfils the well-established principles of Excellence, Transparency, Fairness and Impartiality, Confidentiality, Efficiency and Ethics and Security. The Independent Observers were present during all stages of the evaluation process. The observers respected the same confidentiality obligations as evaluators and signed confidentiality agreements and conflict of interest declarations. The observers also entered into informal discussions with the evaluating experts and Commission officials, involved in the evaluation process, in order to observe the process, obtain feedback and develop recommendations for improvements.

2 See call documents, reference on page 4.


6

As outlined in their remit, the Independent Observers did not intervene in the discussions or did they express views on proposals under examination. It is not the role of the Independent Observers to influence the evaluation outcome, unless the outcome of the evaluation would be strongly connected with operational aspects. The involvement of Independent Observers was most welcomed by the evaluators and Commission staff alike and was seen to contribute positively to credibility and quality of the evaluation system, ensuring an independent outside view on a well-established system. 2.2 Approach taken to perform the task The Independent Observers received all required written information of the evaluation procedure well in advance. At the beginning of the consensus and panel sessions the observers had an extensive personal briefing from Commission Staff. The Independent Observers attended experts’ briefing sessions, Consensus Meetings, Panel Meetings and ethical screening. All sessions and meetings were observed in a shared time mode. The observers had extensive opportunities to discuss procedures with the evaluators both individually and in groups at all stages of the evaluation process. In addition, the Commission Staff including scientific officers, moderators, heads of units and the director of the health research directorate, gave ample time for discussions with the Independent Observers. The observers had finally the opportunity to discuss extensively their preliminary observations and recommendations with Mrs. Ruxandra DRAGHIA-AKLI, Director F and Mr Stephane HOGAN (Head of Unit F1). In this meeting, the Independent Observers received an extensive feedback of the implementation of the previous Observer’s recommendations as formulated in the Report on Evaluation FP7-2007 -2009 . 3. Evaluation Process Actually, the proposal evaluation process was considered to be of the highest standard. However, there are some points that could be further improved; the main observations and a number of recommendations are outlined in this Section. When a modification, suggested by some experts or observers, merits further thorough consideration before being formulated as a recommendation, we headed the paragraph as “Suggestion”.


7

3.1 General Design of the Evaluation Process: Figure 1 The evaluation exercise is managed by the Commission Staff responsible for the different areas/topics. Scientific Officers act as moderators of Consensus Meetings. The Panel Meetings are mainly moderated by Heads of Units.

Figure 1: Overview of the Evaluation Process The scheme represents the full single-stage process as whole. The stage 1 of the two-stage process is highlighted. Stage 2 of the two-stage process follows the single-stage evaluation process. 3.1.1 Submission and evaluation procedure in general Two-stage submission procedure and evaluation procedure: For the second time there was a specific call in the Theme Health implemented by the two-stage submission and evaluation procedure for large scale collaborative projects (in areas 1.1 and 2.1). Within the call HEALTH-2010-two-stage, the deadline for the submission of outline proposals (stage 1) was October 29th 2009, with a later submission of 87 full proposals (stage 2) in 2010 . With a budget of 205 million Euro, the two-stage scheme in the programme Health has been significantly extended.


8

The scoring system was adapted for the first as well as for the second stage of this process. All other areas (most of the work programme 2009) following the single-stage submission and evaluation procedure used the standard scoring system as set out in the rules. The reaction of the evaluators to the first-stage of the two-stage procedure was unanimously positive. At first glance the ease of work is indeed obvious, comparing a success rate of 33% of full proposals in the two-stage with a success rate of a single-stage procedure of approximately 18.5% in calls FP7-Health-2007-A and B, for example, with 326 successful out of 1759 eligible proposals. As an example, from the observation and discussion with experts during Consensus Meetings and Panel Meetings (in 2.4.1. cancer) it was clear that the evaluators appreciated greatly the two-stage evaluation procedure and took it seriously. The experts in the Panel Meetings had in particular a close look into those cases, where more than three proposals in one topic passed the threshold according to Consensus Meeting. In other words they considered those cases, where the success rate for the second stage would have been 25% at best. They clearly stated, that they considered this as being against the intention of the two stage procedure - to reduce the amount of unnecessary work for applicants and to increase the success rate for full proposals. Therefore in such cases the scores of the Consensus Meetings were discussed again in the Panel Meeting and finally a few of them changed with the result that the number of proposals was reduced to three in each of the topics. This was discussed and decided unanimously. Finally, it must be reminded that the preparation of outline proposal for stage one of a two-stage call is a big effort by its own and should not be underestimated. Recommendation: The prospects of the two-stage evaluation procedure should be fully explored and the procedures (single-stage versus two-stages) should be assessed at a later phase by Commission Services, preferably by collecting and analysing a feedback of applicants of the call. 3.1.2 Information on the work programme As, in the past it occasionally happened that persons were mentioned as likely collaborators of a consortium without their knowledge. The Commission must assure that all named partners of a consortium are fully informed and willing to collaborate with the coordinator in case the application is successful. Recommendation: When applicants provide a list of names of scientists willing to participate to the consortium, they should provide to the Commission at least a signed letter of intent of these eventual future participants. A number of applications propose to organise studies that are by all means real clinical studies on patients or volunteers. Often the essential information necessary


9

to be able to judge the quality of a clinical study is lacking: number of patients, calculation of power, inclusion and exclusion criteria, safety issues, recruitment procedures, insurance issues, financial rewards, ethical aspects (vulnerable patients, patients unable to give informed consent). Animal studies are also often insufficiently documented and justified. Recommendation clinical studies and animal studies:

• The Guide for Applicants should draw attention to such necessary information and provide a structure to document all these aspects. Provide guidance for the assessment of clinical studies in the guide for applicants.

• Moderators and experts alike should receive extensive briefing in quality criteria for evaluating clinical studies and animal studies.

3.2 Briefing of moderators: Commission Staff act as moderators for the Consensus Meetings and Panel Meetings. Moderators play an important role in the selection of the experts for the evaluation of the proposals. They have to assure an appropriate range of competencies and a balance between academic and industrial expertise. In addition, a fair distribution of geographical origins of the experts and an “acceptable” gender balance. They shall assure a regular rotation of their experts (ideally said to be 25% from call to call). A major responsibility of the moderators is to handle conflicts of Interest (COI) set out in the appointment letter: identifying disqualifying COI and potential COI. The guidelines for moderators state, that the moderator is moderating, not chairing the meeting; he or she is not supposed to influence the opinion of the experts. In the Consensus - and Panel Meetings, the moderators are expected to act as neutral facilitators helping the experts to reach a consensual view on a proposal, regarding to both the calibration of the markings and the corresponding comments. The moderator is not supposed to influence the opinion of the experts. The moderator may never express an opinion on the quality or on other aspects of a proposal, even if asked. The moderators may summarize the scores by the panel during the remote evaluation, note the final consensus’ scores but should refrain to propose scores him- or herself. The moderator’s task therefore is a difficult one and has to be handled with great diplomacy and care. Most moderators were very well acquainted with their role and the interviewed experts appreciated that the structured guidance of moderators is crucial in order to achieve a high quality consensus decision and a convincing consensus report. In preparation of this evaluation there was, as in the past, a moderator briefing session; moderator guidelines are available. All guidelines for moderating the Consensus Meetings and the Panel were extensively explained. Emphasis was also placed on the handling of the conflicts of interest: how to handle potential conflicts of interest, in which case experts with a “potential COI” may be allowed exceptionally to participate in a panel review, etc.


10

It is recommended that the moderator should ask systematically before the discussion of each proposal whether experts - which were not remote evaluators - have any conflict of interest with a proposal to be discussed. Moderators must ensure that experts with a conflict of interests leave the room immediately before the proposal is discussed. Recommendations:

• The observers underline the essential role of a continuous training of moderators: the high quality of Consensus Meetings (and eventually Panels) depends to a great extent on well trained moderators.

• The observers appreciated, that the moderators paid special attention to dealing with conflicts of interest, however this should be further improved.

• We recommend that, as a rule, Panel Meetings for a final ranking should always be moderated by a Head of Unit.

Suggestion:

• In case that a junior moderator is not very familiar with the scientific domain concerned, it could be wise to appoint one of the senior experts as a chairman for the Consensus Meeting to support the moderator.

3.3 Selection and Briefing of Experts 3.3.1 Selection of Experts To evaluate the proposals submitted in response to a call, the Commission selects experts from all over the world and draws up a list of independent experts with the help of the Expert Management Module, where experts have to register before they can be formally invited. This database contains approximately 50 000 people. The lists are drawn up by the Commission services using the following selection criteria:

- An appropriate range of competencies, expertise and experience; - A balance between academic and industrial expertise and users; - A distribution of geographical origins of the experts; - A reasonable gender balance,

The evaluators are selected by the scientific officers (moderators) responsible for the different areas/topics in the call for proposals. Based on our observations of the discussions and deliberations in the meetings, it can be confirmed by the Independent Observers that the scientific level of the experts involved in the evaluation exercise was excellent. We observed that it happened - rarely - that some Consensus Meetings did not have a reasonable gender balance or even had no female experts on board. This is difficult to understand. Recommendation: If a reasonable gender balance could not be achieved, the moderator should give a short explanation or justification to the Head of Unit.


11

3.3.2 Briefing of Experts While the experts received all the necessary information well in advance of their stay in Brussels, the briefing sessions are important to ensure that all experts have an equal level of information and understanding on the whole evaluation process. During the briefing sessions, the key issues of the evaluation exercise, starting from the objectives and contents of the call, to general aspects such as evaluation criteria and conflict of interest issues to detailed information about budgetary aspects were presented and explained. The briefings are standardized and were very well structured and detailed, giving the experts sufficient overview about their assignment, regarding their specific roles, time schedule and documents to be generated during their stay in Brussels. Sufficient time was also given for asking questions. The briefing sessions were conducted by Commission staff and formed a sound basis for a common under-standing of the tasks of the whole evaluation exercise. Their approach taken was remarkably coherent across all units of the directorate. The evaluators reported that the briefing sessions were very helpful. After general briefing for experts from several topics, moderators of Consensus Meetings make for their topic experts more detailed briefing sessions before beginning discussion of proposals. The “topic briefings” often repeat some basic information from the general briefing but encourage much more questions and more intensive discussion; this provides experts with better practical understanding of evaluation and consensus reaching processes. However, a more detailed explanation of such aspects as ‘out of scope’ and the evaluation of the ‘Impact’ criterion would further enhance the process. Suggestion: Some experts advocated to send them the slides of the briefing on beforehand, or even organise a Web Meeting to ask questions before closing the remote evaluation. To be considered eventually. Recommendations:

• Continued importance should also be maintained in the management of conflict of interest; the observers welcomed that in cases of conflict of interest, the respective expert had to leave the meeting room immediately;

• Briefing sessions could be extended to include more information on aspects such as ‘out of scope’ and evaluating the ‘Impact’ criterion.

• Ensure IPR issues are better understood by the evaluators by providing improved information and guidelines for how they should be assessed; it is seen as important that moderators are well prepared to provide the adequate information should questions arise related to IPR issues.

4. Evaluation Criteria and Scoring The evaluation criteria including the threshold marks are part of the work programme and have been implemented into the software-evaluation tool RIvET.


12

They are explained in full detail in the Guide for Applicants3 and in the Work Programme. However, guidance on how to assess clinical studies would be helpful. 4.1 Single-stage evaluation procedure Criterion 1: S/T Quality threshold: 3/5 "Scientific and/or technological excellence (relevance to the topics addressed by the call)"

- Soundness of concept and quality of objectives; - Progress beyond the state-of-the-art; - Quality and effectiveness of the S/T methodology and the associated work

plan. Criterion 2: Implementation - People, Consortium, Management threshold: 3/5 "Quality and efficiency of the implementation and the management"

- Appropriateness of the management structure and procedures; - Quality and relevant experience of the individual participants; - Quality of the consortium as a whole (including complementarity, balance); - Appropriateness of the allocation and justification of the resources to be

committed (budget, staff, equipment). Criterion 3: Impact - Use, dissemination, exploitation threshold: 3/5 "Potential impact on science, technology and application"

- Contribution at the European and/or international level to the expected impacts listed in the work programme under the relevant topic/activity. Note: Refer to the applicable impact(s) as specified in the work programme

- Appropriateness of measures for the dissemination and/or exploitation of project results and the management of intellectual property.

Overall threshold: 10/15 4.2 Two-stage evaluation procedure (first-stage) Criterion 1: S/T Quality threshold: 4/5 "Scientific and/or technological excellence (relevance to the topics addressed by the call)"

- Soundness of concept and quality of objectives; - Progress beyond the state-of-the-art; - Quality and effectiveness of the S/T methodology and the associated work

plan. Criterion 2: “Potential impact” threshold: 3/5

3 a_gfacp_200901_en.pdf resp. a_gfacp_200902_en.pdf, see http://cordis.europa.eu/fp7/dc/, then follow

“closed calls”, then “Health”


13

- Contribution at the European and/or international level to the expected impacts listed in the work programme under the relevant topic/activity; Note: Refer to the applicable impact(s) as specified in the work programme.

Overall threshold: 8/10

4.3 Two-stage evaluation procedure (second stage) Identical with "4.1 Single-stage evaluation procedure" with the following two important modifications4: Criterion 1: S/T Quality threshold: 4/5

Overall threshold: 12/15 4.4 Proposal Scoring With regard to proposal scoring, the definitions of the scores have been modified last year (see guide for applicants, ref. 5): 0 - The proposal fails to address the criterion under examination or cannot be

judged due to missing or incomplete information 1 - Poor. The criterion is addressed in an inadequate manner, or there are serious

inherent weaknesses 2 - Fair. While the proposal broadly addresses the criterion, there are significant

weaknesses. 3 - Good. The proposal addresses the criterion well, although improvements would

be necessary. 4 - Very Good. The proposal addresses the criterion very well, although certain

improvements are still possible. 5 - Excellent. The proposal successfully addresses all relevant aspects of the cri-

terion in question. Any shortcomings are minor. 5. Evaluation Process: Observations and Recommendations 5.1. Single-stage evaluation procedure: Currently, each of the three criteria (see 4.3.1) is scored 0 to 5, hence a maximum score of 15. Half marks can be given. Currently the individual threshold for each cri-terion is 3 and the overall threshold is 10. In case of the single-stage evaluation procedure - the system of 3 criteria in FP7 already has strengthened the importance of scientific excellence and was recognized a real progress as compared to former Framework Programmes.

4 See call documents http://cordis.europa.eu/fp7/dc/, then follow “Closed calls”, “Health”, "Information package" and in particular guide for applicants "a_gfanoe_201001_en.pdf", "a_gfacp_201001_en.pdf", annexes 2 respectively.


14

Therefore, in order to emphasize the importance of scientific excellence in FP7 funded proposals and to harmonize the single-stage procedure with the two-stage procedure, we recommend one common threshold of 4 for S/T quality. Unfortunately, this is currently only adopted for the two-stage procedure. To reflect this focus on research excellence, it was recommended that (in any case) the overall threshold for the single-stage-procedure should be increased from 10 to at least 11 and possibly to 12, as it is the case for the second-stage of the two-stage procedure. (see for example: “Guide for Applicants: Collaborative projects, call identifier FP-Health-2010-two-stage”, cf. reference 4 on page 13 of this document). 5.2. Two-stage evaluation procedure, first stage: There are now only two criteria scored 0 to 5 as before, hence a maximum score of 10. The threshold for S/T has been raised to 4, the overall threshold is thus 8. The threshold of the S/T criterion has been raised to 4 for the new two-stage procedure (for both stages). This is in line with former Independent Observers’ Reports, who repeatedly emphasized the importance of this criterion. Consequently in case of the single-stage procedure the S/T threshold criterion should be raised to 4 too, resulting in one common S/T criterion for the theme Health. Recommendation: Consequently it is advocated that the overall threshold for the single-stage procedure should also be raised from the current 10 to at least 11 and possibly 12, equally to the two-stage procedure. When, in a two-step evaluation a proposal fails the first threshold (S&T quality) in all IER’s, there is no need to further evaluate the remaining two criteria. We witnessed that during some discussion the ‘Impact’ criterion was not always clearly defined in the mind of the experts and not evaluated in the same way by different consensus groups. Experts still reported to us difficulties to correctly judge and score the criterion “potential impact” despite written instructions and briefings. As this criterion is of strategic importance for programme orientated research funding, it is essential that this criterion is not misunderstood neither by applicants nor evaluators. Recommendation: Improve wording and documentation concerning the ‘Impact’ criterion in the guides for applicants and in the briefing of the evaluators. Recommendation: Consider creation of a check-list of questions/sub criteria helpful in impact evaluation. Each question/sub criterion must be marked by an expert; the check-list should serve not only as an auxiliary evaluation tool but would help authors of proposals as well .


15

Recommendation 0.25 incremental steps: In particular in case of ex aequo’s after the Consensus Meetings. A finer final grading with 0.25 incremental steps would be very helpful for the ranking Panel. Also we recommend such 0.25 incremental steps rating to give a more accurate reflection of the consensus groups’ opinion. This will facilitate the final ranking in the panel meetings. Recommendation detailed budget breakdown: At present the evaluators do not dispose of a sufficiently detailed budget breakdown: sometimes limited to 4 budget lines. This renders impossible an in-depth discussion of the relevance of the budget neither does it permit a clear detailed and motivated suggestion to modify the budget. Experts repeatedly requested more detailed break-down information and more detailed justification of the budget, as it was felt to be very difficult to judge the financial aspects of proposals properly without this information. An example of the NIH procedures could be used. If the Commission expects the experts to give their educated opinion on the budget and eventually to propose reasonable changes to the budget, the detailed information must be made available. 5.3. Two-stage evaluation procedure, second stage: As a result of the first stage of the call (FP7-HEALTH-2010-two-stage, see 5.2 above) 87 outline proposals had been selected with the chance to present a full proposal with a deadline in 2010. The reaction of the evaluators to the second stage of the two-stage procedure was unanimously positive too. 5.4. Single-stage evaluation procedure (Alternative Testing): This call is an innovation. The call text as such has been processed and published according to standard Commission procedures (see document a_ct_201003_en.pdf, reference 1 above). However, it has been prepared in coordination with COLIPA, "The European Cosmetic Toiletry and Perfumery Association". Firstly, COLIPA has been involved in the formulation of the text by nominating candidates for the joint expert group, contributing to a draft as part of the work programme to be presented to the Programme Management Committee. Secondly, there is a binding commitment of COLIPA to contribute up to 25 M€ in addition to community funding of the same amount of money, as it reads in the official Call Fiche (already referred to above): "This budget of EUR 25 million only relates to the contribution to be paid by the EC. The cosmetics industry has announced its commitment to provide additional funding equal to the contribution of the EC (see COLIPA website www.colipa.eu )". Thirdly, submission of proposals, evaluation, and selection have been carried out


16

exclusively via the European Commissions submission and evaluation rules. The Independent Observers confirm, that there has been no difference as compared to the single-stage procedure, as reported above (see 5.1) and that they have been impressed by the design and the implementation of the call. Suggestion: At present and with respect to the Health Programme under FP7, there are two concepts for a closer collaboration of academia and industry (IMI, see http://www.imi-europe.org and COLIPA, this call), which - this has to be admitted - are difficult to compare, concerning scope, implementation, duration and amount of financial commitment. It is out of scope of our mandate as Independent Observers with respect to the present call to discuss this in detail. However, we suggest to carefully following these two approaches of collaboration of industry with academia with respect to responsibility, commitment and achievements. 6. IT- Systems EPSS – electronic proposal submission system Recommendation maximum page-limit-system: Frequently the maximum number of pages indicated for four of the five sections of the B-part is not respected. Currently, the evaluators are instructed to ignore pages over the limit, but the evaluators felt strongly that this should be implemented consistently and automatically. We thus advocate a maximum page-limit-system per section which could be applied to the electronic proposal submission system (EPSS) making it impossible to write beyond the page limit which is now standard practice in most electronic grant submission systems. Suggestion:

• On could accept eventually a strictly limited number of pages in a “Part B1” whereas tables and addenda could be submitted in a “Part B2”

RIvET and Remote Evaluation The RIvET software tool gives restricted remote access to proposals to be evaluated by individual evaluators. RIvET has a strong positive impact on the efficiency and quality of the evaluation process. RIvET gives the experts access via Internet to the proposals they are entitled to evaluate and they have to introduce their completed Individual Evaluation Reports (IER) into the RIvET-System. Note that the “Individual Evaluation Report” is confusingly called “Individual Assessment Report” in RIvET. Thereby the experts are better prepared for more in depth discussions when they participate in Consensus Meetings and/or Panels in Brussels. A further advantage of remote evaluation is that the duration of the central Consensus and Panel Meetings can be limited to a maximum of three days, which


17

allows the Commission to have on board the highly qualified experts that would otherwise not have time to participate in longer meetings. However, there is a number of points, where RIvET could be made more efficient and easier to handle. We recommend updating the tool. The specifications for such an update should be carefully discussed and prepared with the support of experienced moderators. We mention a number of such recommendations based on the suggestions of the evaluators: Suggestion with respect to the structure of the Individual Evaluation Reports: A number of evaluators and moderators advocated a more standardized structure of the IER’s. The reports should provide sufficient written information in order to better justify the individual scores. This could be done for example by highlighting systematically first the strongest and then the weakest aspects of the proposal. Recommendation: Include the ethical screening and review documents into RIvET, for more detail please see Section 9. “Ethics’ Issues” Recommendation flexible access to proposals: Actually, the access is restricted to individual “panels”, related to given topics. It should be possible to combine these “panels” in a flexible way, for further use in the example for Panel Meetings. Recommendation access to all proposals of a given topic: When the IER’s in RIvET are finalised, the evaluators should have the opportunity to see all the proposals in their topic Consensus Meeting or Panel. This has two advantages: firstly the experts may study in depth proposals that will be discussed in his/her Consensus Meeting and fruitfully participate in the discussion. Secondly, he/she will be made aware on beforehand of any unexpected conflict of interest and could declare this before travelling to Brussels.

Recommendations–technical: The following technical points should be taken into consideration:

• Some experts reported that they felt distressed by receiving a RIvET message asking if they were sure about a score putting the proposal out of consideration. They felt that this message tended to encourage increasing the scores. We recommend not sending such a message (eventually place it as a general recalling, without interaction with a current score).

• The automatic “time out” of RIvET should be increased and an automatic save function provided before time out. At present, typed text may get lost, as typing itself is not considered to be an “action”.

• The font size for printed output should be adjustable • The “Global Report” should automatically deliver an Excel-table with an

overview suited for Consensus and Panel Meetings.


18

• Uniform and readable printing reports from the system are required: readability of the reports printed for the rapporteur is hampered by the small font size.

7. Consensus Meetings - General Comments It was observed by us and by many experts that the final scores for the Consensus Report were at times different from the mean scores that had been given in the Individual Evaluation Reports. This is a strong indication of the added value of the Consensus Meetings for the evaluation process. The commitment and seriousness of experts was convincing. The evaluators were encouraged by the moderators to make clear and accurate remarks and statements with regard to the different criteria and sub-criteria. We witnessed that all the discussions in Consensus Meetings seen were of high scientific quality and were characterized by mutual respect and by a common commitment to reach consensus. The experts appreciated greatly the complementary scientific expertise of the other evaluators. The quality of the evaluation procedures contributes significantly to the further development of the European Research Area. Every proposal was assessed remotely by 4 to 9 experts (required are 3 experts/proposal as a minimum). Based on long-term experience, it can be confirmed, that 5 to 6 evaluators per proposal seem to be an ideal number to ensure a balanced assessment. Commission Staff act as moderators for consensus groups and panels. Moderators have to act as neutral facilitators; with the role of enabling the group of experts to reach a consensual view on a proposal taking into account both the markings as well as the corresponding comments. Most moderators were very well acquainted with their role and it was seen that the structured guidance of moderators is a crucial, contributing factor to achieve a high quality consensus decision and report. Introductory briefings by moderators were detailed, concrete, well prepared. Moderators used well-prepared power point presentations. Presentations slightly differ one from another, adjusted to differences according to differences among topics. The strict deadline of submission of the complete Individual Evaluation Reports (three working days before the Consensus Meeting) is usually respected. 7.1. Consensus Meetings: should all be readers? In Consensus Meetings, depending on the total number of proposals, only some of the evaluators have read the proposal under discussion. Sometimes (in particular


19

when marks given by “readers” differ considerably) “non-readers” are asked to read and evaluate proposals on the spot, what prolongs Consensus Meetings. Recommendation: Give the evaluators access to all proposals of their respective consensus group/panel before the Consensus Meeting in Brussels. This would improve the discussions during the Consensus Meetings, as evaluators would have the opportunity to read all proposals to be discussed. Another advantage is the fact that experts will detect in advance any conflict of interest with a proposal to be discussed in their Consensus Meeting. 7.2. Consensus meetings: in which order are the proposals discussed? Every proposal was discussed in detail (see below for proposals, who fail with respect to the S/T-criterion) even when it was clear from the remote scores, that it was going to fail. This was done in order to give the best possible comments and feedback to the applicants. Consensus groups discussed the “S/T-Quality” criterion first. In most cases, the discussions were shortened or ended as soon as this criterion was marked below threshold. In such cases, the rapporteurs were advised to provide summarised remarks based on the comments during the meeting but also based of the Individual Assessment Reports. However, there was also some concern that, doing so, too much time was spent to low quality proposals. Therefore a number of experts advised to group the proposals: firstly one could discuss the proposals with all scores above threshold. Especially in the first stage of the two-stage procedure such proposals will quasi automatically go further and there is no need to very lengthy discussion. Secondly, one could eliminate more rapidly the proposals that score below threshold for all criteria, without spending too much time to low quality proposals. Doing so, more time would be available for in-depth discussion and decision making on the proposals that score around the threshold were the most difficult decisions are to be taken. Suggestion: Study the pros and cons of a procedure whereby the proposals are grouped as high quality, low quality and intermediate quality proposals and evaluated in that order. If applicable, the grouping of thematically closely related proposals would be helpful. 7.3 Consensus Meetings: role of “rapporteurs” For each proposal, one of the experts acts as rapporteur introducing the proposal and following the consensus process. The term ‘rapporteur’ is an EC neologism designating the expert responsible for recording the scores and comments as outcome of the discussion using the appropriate form and preparing the Consensus Report. The discussions were in general of a high quality due to high competences


20

of the experts and the skills of the moderators. For more efficient Consensus Meet-ings: it is recommended to make it a rule to let the rapporteur summarize the project first, thereafter ask the experts who have read the proposal (‘readers’) to give their opinions and to then ask if other evaluators may have questions for clarification. Currently in many cases the experts are only informed when they arrive at the Con-sensus meeting, to which proposals they are attributed to act as rapporteurs. In such cases experts often request to know earlier that attribution. Recommendation:

• The experts requested that the Commission should decide who is going to be rapporteur for each of the proposals, when IER’s in RIvET are closed in the week before the meetings in Brussels. That would give the evaluators more time to improve their preparation for the meetings.

• For more efficient Consensus Meetings: it is recommended to make it a rule to let the rapporteur summarize the project first, thereafter ask other evaluators for comments.

7.4. When are the scores disclosed to the evaluators? We remarked different approaches to acquaint experts with marks given by their colleagues:

a) At the very beginning of the Consensus Meeting – all marks given to all proposals are revealed. b) At the beginning of discussion of each proposal - and only the marks of that proposal – are revealed. c) After presenting reports and after preliminary discussion on the proposal’s merits and faults, the experts can see the proposal’s marks only at the beginning of consensus process.

Suggestion: Consider all pros and cons for the disclosure time of the of the marks of the proposals: experts’ positions in the discussion of the proposal could be influenced by in advance knowledge of other experts’ opinion; on the other hand that knowledge usually accelerates the consensus reaching. 7.5. Disclosure of scores to experts having a conflict of interest? At times an expert having declared a conflict of interest is allowed to participate in the Consensus Meeting and will be asked to leave the room only when the incriminated proposal is discussed. There are further cases, where an expert - as a non-reader - realizes during the Consensus Meeting that he or she is expected to participate in one of the proposals (without having been informed by the proposer). In both cases the expert will have the possibility to take note of the scores, as given by the individual experts. We had the opportunity to ask several of the experts for their opinion. It was most interesting, that none of them considered this situation as problematic, provided, the


21

potential conflicts of interests were made public and those, being associated with one of the proposals, left the room during in depth discussion. We don't have a unanimous answer opinion to this. With the highly interconnected scientific work, it will be difficult to find experts with the necessary expertise, which at the same time definitely are not involved in a number of proposals. In addition, the presentation of the tableau of scores (all scores of all experts for all proposals in a Consensus Meeting) proved to be an ideal instrument in order to reach consensus. 8. Panel Meetings The Panel Meetings are necessary in case of one budget line for a number of (quite) different areas and topics - as foreseen by the work programme. This is a delicate process and it is usually moderated by Heads of Units, who oversee the variety of topics. The evaluators, participating in this important meeting are a subset of those of the Consensus Meetings. The outcome of this Panel Meeting is an overall ranked list of proposals, complemented by a list of proposals below threshold and a list of ineligible proposals. Rules for ranking proposals are set out in the call fiche and annex 2 of the work programme. Ranking is done firstly by taking into account the overall score, then the S/T score and then the impact criterion. This difficult task has been handled perfectly by the Commission, which is herewith fully acknowledged. Some experts and some observers felt that there is no need to change the scores, at this late stage, in order to rank two “ex-aequo” proposals: an extensive motivation by the Panel members should be accepted. Recommendations:

• As a rule Heads of Units should chair Panel Meetings. • The use of 0.25 incremental steps for scoring could be helpful in ranking

otherwise ex-aequo proposals 9. Ethical issues The ethics screening is managed by the Directorate F - Health Research. All proposals above threshold and with a high probability to be funded undergo an ethics screening and eventually are forwarded for an ethics review. The categories that imply automatic forwarding for full ethics review include the involvement respectively the use of:

- Children, Vulnerable Adults, Human Intervention, - Human Tissues, Foetal Tissues, human embryonic Stem Cells - Animals, Genetically-modified Animals and organisms, - Non-human Primates


22

- Involvement of Developing Countries - Data Protection and Privacy issues - Likely Dual Use or Improper Use

Each B-part of a proposal, containing the Ethics Section and Table, is read at least by two experts. In case of disagreement between the two experts, a third one will, in addition, give his/her advice. Finally, the issue were discussed with all experts in short Consensus Meetings. There have been experts for an overall time of 2 weeks for this task. The outcome categories are:

- No ethics issues required ("ER"), - "Screening report required ", - "More information required " - "Mandatory".

• Mandatory means: it has to be passed to the ethics review, managed by

Directorate L (Ethics). • More information: in this case the applicants have to supply further information to

Directorate F; in general, with the exception of simple clarifications, it is then passed to the Ethics Directorate as well.

• Screening report: the report gives guidance to the responsible scientific officer for contract negotiations.

This procedure is in force since FP7, when the applicants had to include an Ethics Section and Table in the proposal, providing all details necessary to enable a valid ethics evaluation. The Health Programme Management Committee decided, that in case the information is incomplete, to give the possibility to supply additional information on demand (category "more information"). As mentioned before, the major shortcoming, in this respect, of a number of proposals reside in the insufficient documentation given in case of real clinical studies embedded in more general work packages. We consider the ethical screening, followed, where necessary, by forwarding for full Ethics’ review, as exemplary: the ethics’ issues are handled correctly according to the specifications of the Framework Programme FP7. Furthermore the Independent Observers have been impressed by the expertise of the experts, the quality and critical fairness of the discussions in the Consensus Meetings. These specifications are crucial and the result of intensive discussions in the co-decision procedure of the framework programme and they are closely linked to the scientific content of the respective themes, like Health. Due to this an ethics scrutiny as part of the scientific evaluation and in collaboration with Directorate F, as it is now, is the right way to handle these issues.


23

Recommendations: • The Ethical Issues Reports (EIR), which will be attached to the proposals before

RIvET is closed for this call, but not for the ethic review reports (ERR). We recommend to fully implementing both of them into RIvET.

• The EIRs of the scientific as well as the ethics experts should be handled as IER’s and the final EIR as the Consensus Report, all to be attached to the propo-sals within RIvET.

10. Organisational Aspects of the Evaluation Procedure Evaluation building facilities The facilities for Consensus and Panel Meetings in the Covent Garden building were greatly appreciated. The IT support is now excellent and greatly appreciated by evaluators. With a view to earlier Independent Observers Reports, it is welcomed that the technical conditions in the Commission building "Covent Garden" have been improved, they are now of a high standard. There have been no complaints with respect to access to IT equipment. Private laptops were accepted and a wireless internet access provided, which could be used in the building. However, in the first two weeks of evaluation (end November to December 2009) there was no internet access available in Covent Garden. This was justified with the need of confidentiality for an evaluation related to the housing of an ICT programme. Now the whole evaluation in the programme health as well as ICT rests on the re-mote software tool RIvET. The means to ensure confidentiality is the contract with the signed confidentiality declaration both of evaluators as well as of independent observers. Why this has to be interpreted in a different way during Consensus Meetings in the Covent Garden Building (notwithstanding internet access in the respective hotels of the evaluators for example) is incomprehensible. In order to better follow the discussions during Consensus Meetings discussion, it is - for an independent observer - an advantage to having direct access to RIvET for example, where all information concerning the allocation of projects to evaluators, the abstracts as well as the Individual Evaluation Reports of the remote evaluation and so on are easily and directly available. The Independent Observers very much appreciate the handling of any issues by the Commission Staff. In addition, the scientific officers and administrative staff of the Directorate F provided excellent support to the Independent Observers and the evaluators throughout the entire process. The staff is of excellent quality, very committed to the process and show an exceptionally high level of competence, dedication and working. It is also recognized that the several weeks of an evaluation process present a very high working load to the Commission staff, which was nevertheless continuously highly committed during the entire period.


24

11. Implementation

The Commission Services (Directory F) provided an extensive feedback concerning the implementation of the previous Independent Observers' Recommendations, which will facilitate the work of future Independent Observers teams. With a view to past Independent Observers' Reports we appreciate, that a number of changes - in line with former recommendations – had been implemented. The Categories of Recommendations concerned:

• Evaluation procedure in general • The Criteria and Scoring • Briefings • Evaluators’ recruitment and briefings • Consensus Meetings • Panel Meetings and proposal ranking • Proposals and RIvET • Covent Garden facilities • Wording and structure of the work programme • Feed-back of Implementation

A large number of recommendations are actually implemented or under further scrutiny to study the practical feasibility of their implementation. We recognise that some recommendations may not have been implemented due to the fact that this would require consent over all themes of FP7, not only for Health. General Conclusions The Independent Observers came to the conclusion that of the entire evaluation process of research proposals within the EU FP7 “Cooperation” Theme “Health” is of high standards, rigorous, transparent, fair and impartial. The consensus approach, the high standards and the complementary character of the expertise of the evaluators ensures that the evaluation strategy, implemented by the European Commission, is most efficient and among the highest quality world-wide. All interviewed experts underlined that the European Commission review process is based on rigorous scientific excellence and can be ranked among the best international evaluation processes. Acknowledgements The Independent Observers gratefully acknowledge that their presence and role was fully supported and seen as helpful by the evaluators and the Commission staff.


25

The Independent Observers would like to sincerely thank the Commission staff members and the expert evaluators for their assistance and full cooperation. Our special thanks is addressed to Mrs Gesa HANSEN responsible for Guidelines, Harmonisation and Independent Observers within the Evaluation Team, for her efficient day-to-day support.