interference in text categorization experiments giorgio di nunzio university of padua peter bruza...
TRANSCRIPT
Interference in text categorization experiments
Giorgio di Nunzio
University of Padua
Peter Bruza and Laurianne Sitbon
Queensland University of Technology
EU Marie Curie FP7 project “QONTEXT”
© Jerome Busemeyer
The Experimental Conditions
• Condition 1: Make an action decision without
reporting any categorization
• Condition 2: Make an action decision after
categorizing a face
Categorization—Decision Experiment
© Jerome Busemeyer
© Jerome Busemeyer
Decision making under uncertainty => incompatible perspectives (subspaces)
Incompatible subspaces => violation of the law of total probability
The port of Philadelphia was closed when a Cypriot oil tanker “Seapride” ran aground after hitting a 200-foot tower supporting power lines across the river; a Coast Guard spokesman said. He said there was no oil spill but the ship is lodged on rocks opposite the Hope Creek nuclear power plant in New Jersey. He said the port would be closed until today when they hoped to refloat the ship on the high tide. After delivering oil to a refinery in Paulsboro; New Jersey; the ship apparently lost its steering and hit the power transmission line carrying power from the nuclear plant to the state of Delaware.
Is it about “crude oil”, “shipping”, “shipping” BUT NOT “crude oil”?
PHILADELPHIA PORT CLOSED BY TANKER CRASH
Are topical subspaces incompatible in some cases?
… we conducted an experiment to find out…
Design: One vs. two step topical classification
d: doc
S: “shipping”
Ŝ: not “shipping”
C: “crude”
Ĉ: not “crude”
Materials
Documents drawn from the Reuters-21578 collection
Manually classified set of Reuters newsfeeds (1988) by a group of experts (72 categories)
Subjects (crowdsourcing)
HIT: “Human Intelligence task” on Mechanical Turk (Amazon)
The higher the number of HITS, the higher the expertise – “masters” have “demonstrated excellence”
Each document categorized by ~10 subjects (workers)
Quality check used to remove unreliable observations
Results
Differences in P(c2) – law of total probability is being violatedOriginal Reuters classification seems to fit a 2 stage decision model. (In ML, categorization decisions are assumed to be in isolation i.e., a one stage model)
c1 = crudec2 = shipping
"Closer examination of the results, however, shows that the kind of errors made are quite different. Human errors stem mainly from inconsistent application of categories, especially the categories with the vaguest definitions, and from failing to specify all the categories when several should have been assigned to a story.”(Hayes and Weinstein 1990)
Did each expert had in mind a specific order of the 72 categories? Maybe the same order in which they were given (alphabetical order? subject area?). If this is the case, they were actually performing a sort of n-step classification (n<=6).
Concluding remarks
No theoretical new insights
Some evidence for incompatiblity in topical subspaces (a la (Busemeyer, Wang & Lambert-Mogiliansky, 2009), but more extensive studies needed
Is it really incompatibility, or just two probability spaces?
… not particularly surprising
“Kolmogorov realized that different sample spaces are needed for different experiments, but his theory does not provide a coherent principle for relating these separate experiments. This is exactly what quantum probability theory is designed to do” (Busemeyer & Bruza 2012).
Impact on Bayesian classifiers (machine learning)
We know the difficult categories by the performance of automated classifiers
Compute the interference term for such categories
Augment the Bayesian classifier with the interference term
Further work
(a) Incompatible subspaces in the Linda problem(b) Incompatible subspaces in document relevance(c) Incompatibility between dimensions of relevance (topicality
vs sentiment)
We are trying to come up with models of users – by the decsions they make about information