a preliminary analysis of aaai-99 submissions devika subramanian rice university
Post on 26-Dec-2015
220 Views
Preview:
TRANSCRIPT
A preliminary analysis of AAAI-99 submissions
Devika SubramanianRice University
Distribution of 400 submitted papers by area
Distribution by area (submitted papers)
17%
13%
13%
13%
11%
8%
6%
5%
4%
4% 4% 2%multiagents/agents
KR
reasoning/unc
learning
planning
vision/robotics
search
interfaces
NL
IR/IE
neural
misc
Trends in submissions
Multi-agent/agents, KR, reasoning/UAI, planning, learning are the largest constituencies.
Robotics papers are coming into AAAI; however these are mobile robot papers.
Neural computation papers are not a large part of the submissions.
Overall acceptance statistics Before SPC meeting:
– Accepts (105)– Undecided (50)– Rejects (255)
After SPC meeting– Accepts (104)– Conditional accepts (5)– Rejects (290)
15 papers accepted with 1 A/2M or R. 3 papers rejected with 2 A/1 strong R. Every other accepted paper had two As. 32 accepted papers had 3 As! (just under 30%)
Distribution of 109 accepted papers by area
Distribution by area (accepted papers)6%
7%
8%
7%
10%
6%12%
7%
7%
13%
10%
7%multiagents/agents
KR
reasoning/unc
learning
planning
vision/robotics
search
interfaces
NL
IR/IE
neural
misc
Trends in accepted papers
Search, planning, neural computation, and information extraction/information retrieval have higher than average acceptance rates.
Data on information extraction/information retrieval and neural computation unreliable because of small sample size.
Other trends in papers
AAAI still attracts the best work in planning, constraint satisfaction, search, multi-agent systems and KR.
Other trends in papers
The best work in machine learning, uncertainty, KDD, neural computation, natural language and Web agents (not including multi-agent systems) is not being submitted to the conference.
The work in mobile robotics submitted to the conference is not competitive with the work represented at the top vision/robotics conferences..
Any new ideas in the papers?
Proverb: a system that solves NY Times crosswords. A tour-de-force integration of ideas in AI.
Hybrid approaches to collaborative filtering. New extensions to Graphplan and Satplan. Active learning: analysis and implementation. Integrating the fields of constraint satisfaction and
classical planning.
Remarks gathered from SPC
Number of NLP papers submitted to AAAI is increasing; they are of much better quality than in years past. Still not the very best papers (which tend to go to ACL), but solid work at the intersection of statistics/machine learning and NLP.
Neural computation papers should be reviewed on a special track as they were this year (with two special SPC members Giles and Sun) even though the actual number of submissions from that community was small this year.
More remarks from the SPC Need to develop mechanisms to get the best work in
machine learning, UAI, agents, KDD, robotics and neural computation to be submitted to conference.
Many were concerned about the fact that most accepted papers were incremental advances.– signs of a maturing field? Or self-selection among
submissions because of reputation as archival conference? Mechanisms to detect and encourage revolutionary
work among the submissions not working well.– reflects fundamental split in community on what a
significant result is, and about the extent of evaluation needed to “prove” that an idea works.
Some SPC statistics
SPC Papers Accepts %Acceptkautzh 20 9 45kollerd 20 8 40weldd 20 7 35thruns 18 6 33.33hirschbergj 18 6 33.33sunr 15 5 33.33gilesl 16 5 31.25kaelblingl 16 5 31.25morgensternl 14 4 28.57spectorl 18 5 27.78steinl 19 5 26.32mcallesterd 20 5 25crawfordj 20 5 25forbusk 19 4 21.05boutilierc 19 4 21.05piatetskyshapirog 15 3 20porterb 17 3 17.65nayakp 19 3 15.79tambem 19 3 15.79konoligek 19 3 15.79greinerr 20 3 15velosom 18 1 5.56
Acceptance rates vary widely among SPC.Partially explains variation in acceptance rates across sub-areas.Mean=26.37stdev=10.22
0
5
10
15
20
25
30
Number of PC
0 to10
10 to19
20 to29
30 to39
40 to49
50 to59
60 to69
70 to100
Acceptance % ranges
PC statistics for AAAI-99
110 PC members
SPC statistics for AAAI-98SPC Total Accepts %acceptsfinint 13 8 61.54porterb 18 10 55.56knoblockc 15 8 53.33moorej 11 5 45.45etheringtond 18 8 44.44crawfordj 20 8 40freudere 20 8 40hendlerj 17 6 35.29epsteins 20 7 35tambem 19 6 31.58gratchj 16 5 31.25myersk 16 5 31.25greinerr 13 4 30.77boutilierc 17 5 29.41konoligek 15 4 26.67horvitze 19 5 26.32suttonr 19 5 26.32biswasg 16 4 25forbusk 16 4 25borgidaa 17 4 23.53bonissonep 13 3 23.08morgensternl 18 4 22.22lochbaumk 15 3 20pereiraf 20 4 20simmonsr 18 3 16.67subramaniand 18 3 16.67goldszmidtm 13 2 15.38hirshh 20 3 15
Acceptance ratesvary widely.
Mean=30.95stdev=12.33
0
10
20
30
40
50
60
Number of PC
0 to 9 10 to19
20 to29
30 to39
40 to49
50 to59
60 to69
70 to79
89 to100
%accepts
PC statistics for AAAI-98
204 PC members
Issues to consider for AAAI-2000 and beyond
Size of SPC and PC to get consistent reviewing standards. Ways of attracting “work-in-progress” to AAAI and to set
good evaluation guidelines for them. Reconsider need for an SPC meeting. Paper assignment to reviewers benefits from a manual
component. Consider providing electronic access to paper abstracts and to assignment software so chairs can teleconference and do reviewer assignment.
Consider accepting few papers for plenary presentation (say 20) and have all papers presented at poster sessions.
How to exist and cooperate/compete with speciality conferences.
top related