machine vision and media processing unit department of electrical engineering and infotech oulu the...

Machine Vision and Media Processing Unit Machine Vision and Media Processing Unit Department of Electrical Engineering and Infotech OuluDepartment of Electrical Engineering and Infotech Oulu

The peer review process and the task of a referee

Olli SilvénUniversity of Oulu


Contents

IntroductionPeer review process

JournalsConferencesResearch programmesThe tasks of a refereeReviewing a research paperPreparing the referee report & recommendationsEvaluating a research proposal

Acting as an editor or program chairperson How to become a referee?Final words


IntroductionA scientific paper is expected to provide a sufficient contribution to the knowledge base of its field

• Number of scientific papers and articles (2000): > 600 000 (ISI)• About 50% in the fields of science and technology

The number of papers and articles submitted for publication is much larger

• refereeing process selects the ones to be published

Examples of acceptance rates after refereeing:• IEEE journals: ~10-20% (large variance)• IEEE conferences: ~10-50%• Workshops: ~30%-90%

Refereeing is also used in selecting research projects to be funded


Introduction

What is a sufficient contribution? • new result, theoretical or experimental• new insight• novel synthesis of ideas• useful survey• useful tutorial

What is not a sufficient contribution• new, novel, useful• badly written• erroneous data

MPI = Minimum Publishable Increment depends on the forum


Peer review process

Peer reviews are carried out by anonymous referees who evaluate the sufficiency of contribution

• novelty, significance, correctness, readability

Refereeing is public service to the scientific community• professional obligation, • carried out on volunteer basis• requires high expertice• helps in improving one’s own expertice• ensures the integrity of science


Peer review process of a journal

editor

associateeditors

author

referees

acceptrejectrevise

submission

reviewsrecommendations

selection of refereeschecking of revised papers

publish

selection of associate editor


Peer review process of a conference

program chairprogram

committeeauthor

referees

accept/reject/accept with revisions

submission

accept/reject/minor revision recommendations

selection of therefereeschecking ofrevisions

extra referees


Peer review process of a workshop

program chairprogram

committeeauthor

extra referees

accept/reject

submit

refereeingchecking ofrevisions


Peer review process of a research programme

steeringcommittee

proposer

referees

accept with partial funding/reject

submission

Notice: not representative of TEKES or EU research programmes


The tasks of a referee

The reviewer grades a paper based on its novelty, significance,correctness, and readability

1. In case of substantial conflicts of interest or if the paper is out of the field of the reviewer, the editor must be informed promptly

2. Both positive and negative findings are summarized in a referee report

confidential part only for the editor/program committee:

information that could reveal the identity of the reviewer or in minor conflicts of interest

non-confidential part for the author/program committee

3. Learn from the other reviews, if they are sent to you after the process


The right attitude: I can learn something!

Humbleness and an open mind needed; 100% self-confidence can be harmful

Early assumptions on the correctness of the paper or the sufficiency of its references should be avoided

• an elegantly written paper may have zero actual contribution• a paper with broken English may contain a major new idea

The papers recommended for acceptance should have novelty and be correct

• If the reviewer can’t check a fact or is unsure, this should be stated in the review report

But don’t waste your time on analysing in detail a paper that is never publishable

• a single crucial error is enough


Reviewing a research paperThe paper to be reviewed is typically accompanied with a review

form• fill the five point scale questions last• it is most important to write an itemized review report

1. Relevance [ ] poor [ ] marginal [ ] fair [ x ] good [ ] excellent

2. Originality [ ] poor [ ] marginal [ ] fair [ x ] good [ ] excellent

3. Background knowledge of the subject and references [ ] poor [ ] marginal [ ] fair [ x ] good [ ] excellent

4. Technical content [ ] poor [ ] marginal [ ] fair [ x ] good [ ] excellent

5. Presentation [ ] poor [ ] marginal [ ] fair [ x ] good [ ] excellent


Reviewing a research paper: analysis

The analysis of a paper can be done by generating explanations to the following eight points (Smith 1990)

1. What is the purpose of the paperIs the problem clearly stated and have the key issues been

pointed out?Is it clear what has been accomplished?

2. Is the paper appropriate for the intended forum?If it is not, what could be a better choice?

3. Is the goal significant = has the work been worth doing? Are the results just trivial variations or extensions of previous

results?Are there any new ideas, or novelties in research methodology?

Citation analysis using electronic libraries are a big help!


Reviewing a research paper: analysis (cont’d)

4. Is the method of approach clear and valid?Is there something fundamentally flawed in the approach? Are the assumptions realistic and does that matter?Is the method new? Can it be generalized to other problems?

Again, electronic libraries are most useful.

5. Is the actual execution of the research correct?Are the mathematics and statistics correct? Check!Have the simulations been described in sufficient detail for

replication?What about the boundary conditions?Do the results make sense?

This part may require considerable effort from the reviewer...


Reviewing a research paper: analysis (cont’d)

5. Are the conclusions correct?Wht are the applications or implications of the results and are

the results analysed to an adequate depth?

6. Is the presentation satisfactory?Is the paper readable? Is it structured according to the

convenstionsof scientific publications?

7. What did you as the reviewer learn?If you didn’t learn anything, then the paper is not publishable(provided that you understood the paper)


Reviewing a research paper: analyzing the references

It is researcher’s professional obligation to cite prior work• the manuscript being reviewed includes claims of novelties;

regularly citing prior research• the reviewer needs to check the validity of the claims• most efficient to carry out the analysis using electronic libraries

At minimum:1. Check what is found using the key words of the article 2. Study the references you don’t know beforehand3. Check which recent papers cite the same references4. Check the references of those recent papers


Reviewing a research paper: reference analysis example


Writing the referee reportNo fixed rules exist, the following ones are according to (Smith 1990)

Most important: make your opinions clear; avoid ”perhaps” and ”maybe”;evaluate the paper, not the author; itemize the contributions

1. State the recommendation and its justification; the five point scale part of the evaluation form is not enough

2. Show with a few summarizing sentences that you have understood the paper. The editor may use this part and compare your summary to those of the other reviewers

3. Evaluate the significance and validity of the research goal4. Evaluate the quality of methodology, techniques, accuracy and

presentation; recommendations for revisions can be written here5. Make a clear recommendation for or against publication with

justifications


Compiling the recommendationsClassification of papers (Smith 1990)

1. Very significant; includes major results (<1% of all papers)2. Interesting work, a good contribution (<10%)3. Minor positive contribution (10-30%)4. Elegant and technically correct, but useless5. Neither elegant nor useful, but not wrong6. Wrong and misleading7. Unreadable, impossible to evaluate

The acceptance level of the journals and conferences vary; 1,2, and perhaps 3(-4)


Research proposalsA research proposal is a request for funding submitted to, e.g,

• Finnish Academy of Sciences• European Commission• Tekes• other funding organization such as a foundation

The key difference to reviewing research papers is thatthe reviewers also evaluate the proposers

Not all organization use peer review as a means for selecting proposals for funding


Evaluating research proposalsThe evaluation criteria vary between funding organizations

Key criteria:

1. Is the research topic significant?2. Are the goals realistic?3. Has the proposer sufficient expertice and facilities to reach the

goals?4. Is the requested funding reasonable?


Sidenote: Evaluation of EU RTD proposals

Often arranged as evaluation meetings lasting for several days

3-4 reviewers evaluate each proposal• about 30 questions to be answered with at least one sentence• in addition a 5 point evaluation of each area• consolidation meetings to analyze differences of opinion• around 2 hours per proposal, each ~30 pages

To be selected, none of the evaluated issues should fall below 4 pointsSelf evaluation possible; evaluation instructions are public

Key criteria to reject proposals:• Lacking significance• Quality of the consortium


Acting as an editor or program chairperson

The editor • maintains correspondence with authors and referees• finds new referees if the ones assigned fail to act in given time• decides on acceptance, rejection or a revision round based on

2-4 review statements. • should distribute all review statements to the referees• receives occasional negative feedback

Review is not a vote! The editor is likely to line himself according to the best justified recommendations

Conference program committees often rely on the numerical evaluations, occasionally resorting to vote


How to become a referee

Writing a publication that is cited is the most certain way to become a referee

Coordination or technical coordination of an EU RTD project is a direct road to proposal evaluations

Refereeing is very rewarding, helps to keep up-to-date andaware of developments in fields adjacent to ones own specialty


Final words

Good referee reports are valuable and free of charge• help in improving the paper• help in improving as a researcher• help in improving as a referee

Refereeing is a learning experience

Scientific progress rests heavily on peer reviews

machine vision and media processing unit department of electrical engineering and infotech oulu the...

Documents