Two Issues Concerning Research Conferences

Dave PattersonOctober 2004

1) Big Idea/New Directions Papers

Problem Trying to Solve: Given avalanche of submissions to conferences, Big Idea / New Directions papers rarely accepted at many conferences Program Committee (PC) must reject 100-170

papers, easy to find bullet holes in such papers, they get a few low scores, and PC rejects them

Occasionally “hero” PC member gets 1 thru, but not sustainable (not just call for papers, PC chair)

Numbers making many conferences conservative Given important stature of conferences

today, Big Ideas / New Directions papers even more important now

1) Big Idea/New Directions Papers

Proposal:Try a 3 – 4 year experiment to attract such papers at some conferences

Mechanism:Separate Big Idea / New Directions PC Former PC chairs + people with reputation for

Big Idea / New Directions select ~ 3 papers Dedicate 1 session to such papers

Evaluate value of Big Idea papers to decide if discard/keep/expand this approach: attendance at session, good discussions at conference, lead to subsequent papers, …

1) Big Idea/New Directions Papers

Our conference already accepts them Great! Please share your experience,

but no need to try this idea. But we lose slots for 3 “good” papers?

For 3 years, let’s drop 1 panel session or 1 keynote (often nil in my experience)

Discuss! Suggestions for conferences and/or

people to contact as likely match?

2) Is large scale a threat to conference quality? Background – Number of paper

submissions increasing over time while number of papers presented roughly constant

2) Is large scale a threat to conference quality?

PapersSubmitted

0

50

100

150

200

250

300

350

400

2000 2001 2002 2003 2004

Su

bm

itte

d SIGMOD

STOC

ISCA

PODC

2) Is large scale a threat to conference quality?

AcceptanceRate

0%

5%

10%

15%

20%

25%

30%

35%

40%

45%

50%

2000 2001 2002 2003 2004

Acc

epta

nce

Rat

e

STOC

PODC

ISCA

SIGMOD

2) Is large scale a threat to conference quality?

Impact on Program Committee – Either more papers per member or larger PC

Either affects quality of reviews and of decision? Impact on Authors of Rejected Papers –

Inaccurate reviews decision feels arbitrary, unfair

Resubmit them to related conferences or submit more papers per conference more papers per conference

Impact on Conference,Field – Losing good ideas?

Are papers ranked 31-60 really that bad? (Its research)

Researchers don’t try riskier ideas since less chance?

2) Is large scale a threat to conference quality? Proposal – Create an ad hoc committee

to write report (& talk in Feb) on whether / when a problem including: Why are numbers increasing? (More people

in field, resubmiting, multiple papers/conf) What size is a problem? (150? 250? 350?) What have some conferences tried, and did

it work or not? (best/worst practices) New good ideas to help to cope with issue

2) Is large scale a threat to conference quality? Examples of Best/Worst Practice

STOC – Went to dual tracks PODC – 5 minute Brief Announcements (1

page in proceedings) + 25-min papers 75 talks in 3 days, 50% - 50%, single track

SIGMOD – lower bound to accept rate, so more papers submittedmore presented

SIGGRAPH – add 5-minute recent result papers with later deadline, quick review

DAC? – Hierarchical Program Committees (divide by topic, allocate by track)

2) Is large scale a threat to conference quality? Examples of New Ideas

No downside to submitting, so create downside by publishing authors’ batting averages to

Limit number of papers one can submit to a single conference (as often a limit for PC members)

Since now many related conferences, if some submit papers to each, go to a common paper deadline to force authors to decide target

Journals cope with surge by having editors reject papers outright without further review, for otherwise surge would break review system: give PC chair same authority smaller PC

2) Is large scale a threat to conference quality?

Request – If you think it’s a good idea, suggest names of people who might be willing to volunteer for ad hoc committee, possibly a chair? Important to get people from diverse

communities to collect practices, ideas