A Comparison of On-line Computer ScienceCitation Databases

Vaclav Petricek, Ingemar J. Cox, Hui Han, Isaac G. Councill, C. Lee Giles

v.petricek@cs.ucl.ac.ukhttp://www.cs.ucl.ac.uk/staff/V.Petricek

2

Motivation

Autonomous databases have advantages compared to manually constructed

- Easier maintenance- Lower cost

Is it really an equivalent solution that is just cheaper?

Does the automated acquisition introduce any bias?

3

Talk Overview

Datasets Acquisition bias and models CS Citation Distribution Conclusions Future Work

4

Datasets - DBLP

DBLP was operated by Micheal Ley since 1994 [8]. It currently contains over 550,000 computer science references from around 368,000 authors.

Each entry is manually inserted by a group of volunteers and occasionally hired students. The entries are obtained from conference proceeding and journals.

5

Datasets - CiteSeer

CiteSeer was created by Steve Lawrence and C. Lee Giles in 1997. It currently contains over 716,797 documents.

In contrast, each entry in CiteSeer is automatically entered from an analysis of documents found on the Web.

6

Datasets – Publication year

CiteSeer DBLP

Declining CiteSeer maintenance

Increased DBLP funding

7

Author bias

CiteSeer papers have higher average number of authors Both databases show growing team sizes

8

Author bias

Crossover for low number of authors

CiteSeer has higher proportion of multiauthor papers than DBLP

(for number of authors <4)

9

Author bias

“Papers with higher number of authors are more likely to be included in CiteSeer”

Hypothesis

Crawler suffers from acquisition bias due to - Submission- Crawling

10

Models - CiteSeer

CiteSeer Submission model

Probability of a document being submitted grows with number of authors

- Publication submitted with probability β- Probabilities independent for coauthors

citeseers(i) = (1-(1- β )i) * all(i)

11

Models - CiteSeer

CiteSeer crawler model- Probability of crawling a document grows with number of its

online copies- Probability of a document being online grows with number

of authors- Probabilities independent between authors- Publication published online with probability δ- Publication found by crawler with probability γ

citeseerc(i) = (1-(1- γδ)i) * all(i)

Both models result in equivalent type of bias

12

Coverage

Can we estimate the coverage of dblp? Can we estimate the coverage of CiteSeer? Can we estimate the coverage of CS

literature?

We need a model of DBLP acquisition method

13

Models - DBLP

DBLP model- Publication included in DBLP with probability α- α is a parameter reflecting DBLP “coverage” of CS

literature

dblp(i) = α * all(i)

14

Coverage

citeseer(i) = (1-(1- β )^i) * all(i)

dblp(i) = α * all(i)

r(i) = dblp(i) / citeseer(i)

r(i) = α / (1-(1- β )^i)

15

Results

• r(i) = α / (1-(1- β )^i)

Alpha ~ 0.3

DBLP covers approx 30%

of CS literature

CiteSeer covers approx 40%

CS literature ~ 2M publications

Citation distribution

17

Citation distribution

Studied before Follow a power-law Redner, Laherrere et al, Lehmann and

others Mostly physics community

We use a subset of CiteSeer and DBLP papers that have citation information

18

Citation distribution

Power law Sparse data for

high number of citations

19

Citation distribution

Exponential binning Data aggregated in

exponentially increasing ‘bins’

Equivalent to constant bins on a logarithmic scale

Easier interpolation

20

Citation distribution

Distribution of citations more uneven in CS than in Physics Significant differences between DBLP and CiteSeer

slope

# citations Lehmann DBLP CiteSeer

< 50 -1.29 -1.876 -1.504

> 50 -2.32 -3.509 -3.074

21

Citation distribution

CiteSeer contains fewer low cited papers than DBLP

No model yet Lawrence

- “Online or invisible?”

22

Conclusions - authors

CiteSeer and DBLP have very different acquisition methods

Significant bias against papers with low number of authors (less than 4) in CiteSeer.

Single author papers appear to be disadvantaged with regard to the CiteSeer acquisition method.

two probabilistic models for paper acquisition in CiteSeer resulting in the same type of bias

- Crawler model- Submission model

23

Conclusions - coverage

Simple model of DBLP coverage predicts coverage of approx 30% of the entire Computer Science literature.

This gives us CiteSeer coverage of approx 40%

and total number of CS papers around 2M

24

Conclusions - citations

CiteSeer and DBLP citation distributions are different

Both indicate that highly cited papers in Computer Science receive a larger citation share than in Physics.

CiteSeer contains fewer low cited papers

25

Future Work

Repeat experiments on most recent CiteSeer data

Other methods to estimate Computer science literature size and trends

- Overlap of CiteSeer and DBLP

Bias introduced by bibliography parsing Collaborative network analysis Connection to internet surveys?

Thank you