Towards a Model of Computer Systems Research

Tom Anderson University of Washington

2

P2P vs. Systems Research

P2P

No centralized controlEmergent behaviorHeavy tailed distributionsIncentives matterRandomness helps

Systems Research

No centralized controlEmergent behaviorHeavy tailed distributions?Incentives matter?Randomness hurts?

This talk:

•Explain systems research using tools from P2P systems research

•Suggest some mechanisms to better align author and conference incentives

3

Mean Score + StdDev NSDI 08

4

Mean Score + StdDev OSDI 06

5

Mean Score + StdDev SOSP 07

6

Randomness is Fundamental?

Little consensus as to what constitutes merit− Importance of problem?− Creativity of solution?− Completeness of evaluation?− Effectiveness of presentation?− All of the above?

Large #’s of submissions makes consistency hard to achieve− Small PC, huge workload, burnout, lack of

attention to detail− Large PC, lower workload, less consistency

7

SIGCOMM 06 Experiment

Manage randomness explicitly− Large PC, split between “light” and “heavy”− Light + heavy PC: bin into accept, marginal, reject

• With as few reviews as possible• Add reviews for papers with high variance• Add reviews for papers at the margin

Program committee meeting (just heavy PC)− Pre-accept half the papers− Pre-select 2x to discuss− Each paper under discussion read by at least 5 from heavy

PC− Result: success disaster

• Little basis for discriminating between papers at the boundary

8

Two Models of Distribution of Merit

9

Citation Distribution for SOSP

10

Incentives for Marginal Effort

With unit merit and no noise:− Impulse function at accept threshold

With unit merit and noise, single conference:− Gaussian function at accept threshold

With unit merit, high noise, and multiple conferences:− Peak incentive well below accept threshold− Repeated attempts without improving paper

We’d like effort to reflect the underlying merit of the idea

− Good ideas are pursued, even after publication− Mediocre ideas are published, and the author quickly

moves on

11

A Modest Suggestion

Reward, like merit, should be a continuous function

Publish rank and error bars for every paper accepted at a conference− Computed automatically from individual PC ranking− Post-hoc (benefit from perspectives of all reviewers)

After some time has elapsed, re-rank− Encourage continued effort on good ideas− Like test in time, but applied to all published papers

12

Afternoon Discussion Topics Double-blind vs. single-blind reviews Should authors disclose previous reviews of the

same paper? Are author-rebuttals useful? When should ``open reviews'' be used? Should we review the reviewers? CS-wide citation reporting and indexing Travel reduction Decoupling publication from presentation How do we quantify the merit of a conference? Do PCs tend to favor PC-authored papers? How random are PC decisions? How big is the rejected-paper tumbleweed?

13

Afternoon Discussion Topics Is there a correlation between PC size and conference

impact? Does overlapping membership between PCs decrease

diversity? Is there a correlation between number of papers accepted

and quality? Do overall scores predict what gets accepted? What do authors like and dislike about reviews? How to handle suspected author misbehavior How to handle suspected reviewer misbehavior When, why, and how to shepherd Reviews of review-management software Proposals for new or improved review-management features