1

How to get the best paper award

Hans-Dieter Burkhard

Humboldt University Berlin

2

… I don’t know

(but it happens)

3

What is a good paper?Would you read it?

Why:• Interesting topics• Surprising results• Nice story

What is the message?What to learn from the paper?

4

What is a good paper?

Examples:

• New efficient quality measure in SE.

• New facts about neural control in animals.

• A first correctness proof.

…

5

What is a good paper?

Who decides:

1. Reviewers of journals/conferences

2. Readers (citations of the paper)

6

What is a good paper?

Typical criteria of reviewers:

Contents

• Contribution

• Originality/novelty

• Significance to theory and practice

• Relevance to call of papers/to journal

7

What is a good paper?

Typical criteria of reviewers:

Technical aspects

• Technically sound

• Readability and organization

• Appropriate length of the paper

• Content style and clarity of writing

• Quality of English

8

Highly ranked papers

• Articles in main journals– (what are the most important journals in your

research field?)

• Contributions in main international conferences– (what are the most important conferences in

your research field?)– What is the acceptance rate

9

Acceptance rateFrom a mail by the program chairs:We have almost completed the decision phase for the 531

papers that we received. We have followed the recommendations in 96% of the cases. There have been 21 papers out of 531 decisions where we changed between reject – poster or poster - accept compared to recommendations we received. In each case, it was done only after reading all reviews, the discussion, and the paper itself …

• Accepted: 119 (22%)• Posters: 133 (25%)• Rejected: 277 (52%)• Transferred to Industrial Track: 2

10

Highly ranked papers need time

• Leading researchers can produce 1-2 high quality papers per year

• Result of work in their group

• Result of discussions with others

11

Other papers

• Other (peer reviewed!) journals• Other (peer reviewed!) international conferences

• National journals• National conferences

• WorkshopsQuality Check: Which kind of proceedings (e.g. by Springer)

There are so many conferences/journals

12

Why workshop papers?

Workshop papers are not highly ranked.

But:

• Don’t start publication at the highest level• Early presentation of your work• Discussion: critics and information by others

(hopefully by reviewers)

13

What about books and book articles?

Large differences in ranking:

Fundamental book (cited by everyone)…

…yet another textbook (among 100 others)

(What are the standard books in your field?)

14

Organization of conference reviews

Program Chairs invite

members of the Program Committee (PC)

15

Organization of conference reviews

Chairs invite Program Committee (PC)

Dear Anton,

The 5th International Conference on Multi-Agent Systems will be held in Leipzig (Germany), the 25-27 September 2007. … CEEMAS conferences (http://www.ceemas.org/) take place every second year in the Central and Eastern European region. The programme committee of the conference series consists of well known researchers from the region and renowned international colleagues. The conference proceedings are planned to be published by Springer Verlag in the LNAI series as before.

In view of your expertise on the field, the Chairs would like to invite you to take part in the Program Committee of the conference. In addition to the task of reviewing a few papers in April/May 2007, your duties as a PC member will be to actively promote the workshop and to solicit paper submissions from colleagues.

Please let us know if you are able, as we hope, to accept this invitation.

Best regards …

16

Sure! What a pleasure to join you, my dear friends.

Thank for the invitation; unfortunately I am totally overloaded right now and so much decline.

I will be honored to join the PC as it was the case for the previous ones. Please count on me.

Thank you for the invite. I will be happy to help out on the PC.

17

Organization: ConfMaster System• Web based system (same as for authors)

– Authors upload their papers– Reviewers can download assigned papers

• Each paper is usually assigned to 3 PC members• PC members usually get 4-10 papers to review

• PC members ask their colleagues, assistants …(look for the list of reviewers in the proceedings …)

18

How papers are assigned to reviewers

• By Chairs only.

• By competencies: – PC members indicate their interests/expertise

by given keywords

• By bidding for papers: – PC members indicate their interests/expertise

based on paper abstracts

19

Format of reviews

• Fixed Attributes– Ranking, e.g. between 1 … 5 (best)

Example:Evaluation of work and contribution : 4Significance to theory and practice : 4Originality novelty : 3Relevance to the call of papers : 3Readability and organization : 4Overall recommendation : 4Reviewer familiarity : 4

The important one

20

Format of reviews

• Additional text fields

-- Comments to the author(s): I was trying to find why it is so important to build coalitions and in what dynamic environments you are trying to find them. Is it for a plant? What are the application fields of your coalition formation? What is an example for a noisy environment?-- Comments to the PC: -- Summary:

21

Remarks of the reviewers

General remarks Generally, I don’t understand what the intention of the

authors is. Why do they call it data mining if, as I understand, it ends up as a specific way of searching? Also, the paper does not mention any related work, most notably from the case-based reasoning area where applications of that kind have been developed more than 10 years ago.

Contributions of the paper… (here you can read what the reviewer has understood)…

What are the ways in which the paper should be improved? Firstly, name the problem precisely. Do you address search

or data mining? If the latter, what is the result of the data mining process? Investigate related work and then describe what really is unique and new.

22

Remarks of the reviewers

The idea of using relative information between objects/landmarks is not new. At least, it has been used in SLAM (Simultaneous Localization and Mapping) for years. See the line of research onRelative Mapping (e.g. [Smith et al., 1990], … ). …In the paper, it does not describe if the observations arecommunicated to ALL robots (in the team). If it is the case, wouldit be better to have a central control where the observation are collected/computed and results send to all robots ?

Detailed comments are as following:"Line point" is mentioned many times in the paper. I do notunderstand. It is true that the robot observes line segments ?

I do not understand the figures 4a. Please explain in more details. The figure 5b is too small to see. Please make it larger and explain in a better way. Please clarify the last paragraph of page3, column 1. …

23

Final decision

• By Chairs only

(mainly based on overall evaluation)

• By discussion between reviewers (try to find consensus)

Overall evaluation1 – strong reject2 – weak reject3 – balanced4 – weak accept5 – strong accept

24

Tight schedule (!) for conferences• Bidding Period: January 22 to February 1

PC Members/Reviewers notified of their allocation: 5 February

Due dates for referee reports: March 2Consensus Discussions: March 2 to 12

Author notification: March 23

(leaves some time for preparing final version)

25

Best paper award

• Candidates: – Best ranked papers– Proposals of PC members

• Decision (vote) by all PC members

Inofficial rule: The next year award is often not given to same topic, country, university,…

Award is subject to “scientific/political” goals …

26

What you can learn by reviews

• What do others think about your work

• References to other interesting work

• Open questions in your paper– Presentation problems– Scientific problems

27

Writing papers You should already have some results …• The good case:

Problems become more clear while writing

• The bad case:You still don’t understand what you are writing

NEEDS TIME!!!• Write it two times (with a delay of 4 weeks or more)

28

What are interesting topics?

• Exploration of new fields (may arise by practical questions)– Main stream? Private research field?

(who reads your paper)

Time

# papers

Ø quality

29

What are interesting topics?

• Improvements of other work:– New insights– Better estimations, better experiments, …– Fewer preconditions …– Corrections …

• Interdisciplinary: Go to other communities

Why should someone read my paper?

30

What a paper should contain

What people (reviewers) read first:• Abstract, introduction, conclusion, references • Figures (readability!), tables

They want to get an idea:

What are the main contributions?

For their later reconsideration (for fixing ranking):

What were the main contributions?

31

What a paper should contain

• Good structure – outline/explanation in the introduction

• Clear (formal) Definitions and Results (!)• If appropriate: Evaluation by experiments

– with discussion!

• Related work – what is new in your paper• Open problems

• Don’t forget to give the credits to other work

32

Credits, Acknowledgements, Citations

• Strict honesty with regard to contributions of partners, competitors and predecessors

33

How to find relevant papers

• Libraries

• References in other papers

• Citation indices: Links to citations of a paper– Citeseer – Google Scholar

34

35

36

37

38

Use of citation indices

• Investigation of more recent work (citing articles are newer than the cited article)

• Relevance of work (count of citations)

• Additionally: Who reads your papers?

39

Technical aspects

• Given formats:– paper size– pages– citation styles

– Works best with LaTeX

• Use the correct terminology …

• Include a nice story

40

I. INTRODUCTION The classic example of negative information was described

in the Sherlock Holmes case “Silver Blaze.” In this case, a house has been broken into. Under such circumstances, one would expect the watch-dog to bark. The curious incident of the non-barking of the dog in the nighttime provides Holmes with the information that the dog must know the burglar, allowing him to solve the case. Applied to mobile robot localization, this means that conclusions can be drawn from expected but actually missing sensor measurements [5].

41

Open problems …

What can it mean to the reader: – You didn’t like to work more on it?

Reviewer answers (hopefully not):– Present the paper if you are finished.

But of course: Be honest with respect to unsolved questions and further plans.– Why is it unsolved?

… will be left for future research …

42

Co-authorship

• Cooperative work can help a lot.

• Authorship: – all cooperating people– order by (inverse) alphabet or better by contribution

(clarify role/contribution of co-authors)

• Co-Authorship by recognized people can help to be accepted – Inclusion of the professor (if she/he agrees)?

43

How to be visible by others

• Be present on the Web – provide papers for down load

• Use appropriate key words

• Discuss with others on the conferences

• Join scientific networks!

44

Write more papers Why?

– want to have larger publication list– institution pays travel expenses

• Consistently question your own findings• Co-authoring

• Reuse of papers … ? cut-and-paste-papers …?

45

If the paper was rejected

Consider it as in sports:

• a step to future success.

• referees are also humans.

• learn from comments (hopefully they explain the reasons).

46

If the paper was accepted

• Final version: Follow the hints of reviewers.– Journals: further review after revision.– Conferences: no time for further review.

• Exception: after conference proceedings

• Technical aspects: – No problem with LaTeX styles.

47

Presentation at conference

• Keep the timePrepare parts (proofs, remarks, explanations, slides, …)

near the end that you can easily– add for remaining time – skip for missing time

• Preparation:

Test the talk with clock and audience (with audience it will take more time than expected)

48

Presentation at conference

Technical aspects • Common Fonts, not less than 24 points• Color

consider physiological aspects

and: projector may change color

• Sufficient size of figures (readable inscriptions!)

• Parts of programs (implementation details) are usually not interesting and not readable

red on blue ?

yellow on white?

Can you read ?

49

Presentation at conference

Get the interest of the audience by …(a lot of papers are presented!)if possible: recall other talks from the session

Discussion: • Source of new questions for future work• If you are lucky: You will be invited for

common work

50

Poster-Presentation at conferences

• Highly competitive• Try to get attention:

– (provocative) thesis– interesting picture

• Stay at your poster for explanation/discussion

• Keep ready: business card and printed papers (if available)

51

Good luck!

52

After discussion• It might be the case that I talked too much about

introduction and conclusion, but not enough about the main part in between. It is like eating: The proof of the meal is to eat. The proof of the paper is to read. But to get hungry, you need a good guess …

• And if you write honestly the introduction and the conclusion, then you will learn a lot what your paper is really like.

The important thing is not to stop questioning.

(A. Einstein)

53

54

Another topic:

RoboCup 2009 …

… will be in Graz (Austria)

i.e. not so far from here

55

Another topic:

By the way:

Not only Soccer Robots, but

Rescue Robots,

Robots for Home&Garden…