training module review

PEER REVIEW STRATEGYWarzaw, May 2013Sofia Kuhn and Ingemar PongratzMuninn: Managing and promoting science

OUTLINE OF TALK

1. Setting up a proposal review strategy

2. General Considerations

3. Planning the call

4. Publication of the call text

5. How to manage an open call

6. Review process

7. Publication of Results

8. Closing the call

WWW.MUNINNSCIENCE.COM

1. SETTING UP A REVIEW STRATEGY

• Plan in advance• Develop an realistic time-frame

• Include ample time to identify the key players and key actions

• Reviewers

• Profile

• Check for conflicts of interest

• links

• Training needs for personnel

• Communication strategy

• Strategy need to be developed together with financing provider

• In general advisor(s) could support this phase

• Identify problems in advance


GENERAL CONSIDERATIONS

• Take into account:• Timing

• Dissemination strategy

• Where to provide information about the call

• How to provide information

• Which is the audience

• How many proposals do you expect

• Yearly (similar) calls

• Develop and publish guidelines

• Evaluate your guidelines

• Inform


PREPARING THE CALL

• What are you going to fund• Research Projects (consumables, salaries etc)

• Career development (courses, training, schools etc)

• Fellowships (PhD student, post-doc, researchers)

• Conferences meetings

• Who is your target• Academia

• Industry

• Translational research


PREPARING THE CALL

• Focus of the call• Open or theme based call

• Applied of frontline research

• Outcomes of the research

• Knowledge or Innovation

• Dissemination and communication• Internal communication (own organization)

• Communicate to possible recipients

• Communicate to scientists/organizations that will not receive funding

• Media, trade organizations, general public


PREPARING THE CALL

• Dissemination/Communication strategy• Timeline

• Communication plan

• Prepare online documentation

• Administrative planning• Several calls

• Use a dedicated solution

• Open and secure intra-section

• Test in advance

• Guidelines to staff (written mandate)

• Adhoc experts


PREPARING THE CALL

• How to handle applications• Webb-based or paper based

• Register (file) all documents

• Inform the applicants

• Develop a web-based application portal• Several calls

• Use a dedicated solution

• Open and secure intra-section

• Test in advance


THE REVIEW PROCESS

• Core principles of the review


Core

PrinciplesIntegrityQuality

Admin

Method

THE REVIEW PROCESS

• The participants• Applicants

• Evaluators

• Programme officers and staff

• Proper communication• Applicants

• Staff

• Review duties• Objective

• Impartial

• Fair


KEY DUTIES

• Avoid/declare conflicts of interest• Perceived

• Public trust

• Real• Identify early

• Staff duty, define clear guidelines

• Inform reviewers

• Search in advance

• Confidentiality of the review• Open

• Blind (double, single)

• Legislation may be important


KEY DUTIES

• Communication of the results• Reviewers/advisors

• Non-disclosure agreement

• Code of Conduct

• Administrative staff

• Explicit mandate to communicate

• Other


INTERVENTION BY APPLICANTS

• Applicant right to to intervene is critical for the integrity of the process• Stress this point

• Critical for the integrity and reputation of the call

• Redress• No interference with the review process

• Factual errors need to be corrected (Advisors)

• New review?

• Multistep process• Allows for more interaction between applicants and reviewers

• Information to all in advance


SUMMARY OF CALL CYCLE


Monitoring

and

Evaluation

Communi-

cation

Preparation

Call publication

Preliminary check

Expert selection

Review and decision

Rebutal

PREPARATORY PHASE

• Mandate• Launch or Re-launch

• Communication and PR

• Programme:• Needs

• Objectives

• Target groups

• Performance monitoring

• Administration and implementation• Work plan and logistics

• Responsibilities

• Budget (Review process)

• Tech Resources


PREPARATORY PHASE

• Staff and Resources• Contact points for applicants, journalists, other

• Eligibility checks, Conflicts of Interests, etc

• KEY ROLE IN THE REVIEW

• Training

• Formal mandate


PREPARATORY PHASE

• Peer Review• Main features of the call

• Full proposal or two stage

• Communication between Reviewers and applicants

• Peer review model

• Remote

• Meetings


PREPARATORY PHASE

• Administrative issues• Support for the panels

• Information flow • Reviewers

• Reviewers-administration

• Monitoring• Administrative

• Advisors

• Manuals• Call manuals, scope information, information about eligibility etc

• Other documents


CALL PREPARATION

• Documentation

• Call for proposals (call text)• Scientific

• Eligibility criteria

• Format

• Review process

• Guidelines to applicants• Mandatory templates, page length etc


PREPARATORY PHASE

• Reference material• Guidelines for applicants

• Guidelines for reviewers

• Code of conduct

• Redress

• FAQ

• Webb-


LAUNCH OF THE PROGRAMME

• Dissemination• Web-based promotion

• Scientific media

• Mailing lists, newsletters

• Meetings

• Opening of the call• Stated time

• Closing• Communicated to all participants

• Receipts of submitted application

• Avoid extensions


PROCESSING

• Validate the status of the application• Readable

• All the submitted material is present

• Pre-screening• Screen for complete proposals

• Submitted in time

• Affiliation

• Ethical concerns

• Inform early

• Acknowledgement• Full Review


EXPERT SELECTION

• Very difficult task

• Expert Identification• Keyword matching

• Remote evaluation and consensus meeting

• Panel Rapporteur

• Monitoring

• Selection criteria• Excellence and experience

• Active in the field

• International / National


REVIEW

• Incentives• Increases participation but mot quality

• Briefing of reviewers• Opportunity to provide formal advice

• Conducted in place

• Provide equal opportunities for all applicants• Message should be the same

• Communication• Timely

• Correct


FINAL DECISION

• Consensus report• Provided by the panel rapporteurs

• Minutes of the meetings (including ranking) should be recorded

• With clear explanations for the applicants

• Funding decision• Taken by the organizations

• Based on the consensus report

• Information to applicants and other stakeholders• Fast

• Clear information on how to appeal

• Redress


COMMUNICATION

• General• Timely

• Effective

• Efficient

• Planned in advance

• Train for leaks


COMMUNICATION

• Applicants• Receipts and validation

• Inform of possible technical problems

• Eligibility

• Review outcome

• Right to answer

• Reviewers• Necessary and useful information

• Complimentary

• Stakeholders and media• Post-results

• guidelines (in advance)


AFTER THE CALL CYCLE

• Evaluate the process • Evaluate the different steps individually

• Identify strengths and weakness

• Possibly by outside experts

• Quantify the problems

• Identify corrective actions

• Evaluate the views of all the participants• Administrative staff

• Funders

• Reviewers

• Researchers


AFTER THE CALL CYCLE

• Identify weakness after every call• Even if no follow up is planned

• Corrective action before new cycle is planned/decided• Specially if weakness is technical or structural

• Update Guidelines

• Publish with care


FUNDING INSTRUMENTS

• Solicited or unsolicited research• Solicited: Clear guidelines (preparatory phase)

• Solicited: Expert panels can be established in advance

• Unsolicited: Experts identified after the applications have been received

• Pre-registration is an option

• Multi / Inter disciplinary research• One topic but several approaches

• Cross-disciplinary• Topics at the intersection of different research areas

• Review panels are more difficult to establish• Expert selection need to be discussed in advance


Dis

c 3

T

FUNDING INSTRUMENTS


T

Disc 1

Disc 2

Disc 3

T

Disc 1

IndividualProjects

MultidisciplinaryProjects

T

Disc 1Disc 2

Disc 3

InterdisciplinaryProjects

Disc 1

CrossdisciplinaryProjects

T

Disc 1Disc 2

Disc 3

TransdisciplinaryProjects

Disc 1

FUNDING INSTRUMENTS

• Career Development• PhD students grants

• Post-doc grants

• Advance Career Grants

• Many proposals

• Conflict of interests are common

• Project Applications• One researcher/research group

• Often unsolicited

• Experts panels need to be set up after applications have been received

• Many panels available in advance (continuous calls)


FUNDING INSTRUMENTS

• Collaborative Research• Can be mono, multi or intradisciplinary

• Involve several partners

• Eligibility criteria

• Networking• No research funding

• Workshops and meetings

• Thematic

• National or international

• Eligibility and commitment


FUNDING INSTRUMENTS

• Center of Excellence• Very Very competitive

• Political

• International review panels

• Infrastructure grants• Replace or Acquire new equipment

• Often the applicants contact funders in advance

• Application can also be endorsed by the University


training module review

Education