Unsupervised Transfer Classification

Application to Text Categorization

Tianbao Yang, Rong Jin, Anil Jain, Yang Zhou, Wei Tong

Michigan State University

Overview

Introduction Related Work Unsupervised Transfer Classification

Problem Definition Approach & Analysis

Experiments Conclusions

Introduction

Classification: supervised learning semi-supervised learning

What if No label information is available? impossible but not with

some additional information

supervised

semi-supervised

unsupervised classification

Introduction

Unsupervised transfer classification (UTC) a collection of training examples and their

assignments to auxiliary classes to build a classification model for a target

class….

auxiliary class 1

auxiliary class K

target class

No Labeled training examples

prior

conditional probabilities

Introduction: Motivated Examples

Image Annotationsky

1

sun

0

1 1

0 1

water

0

0

1

0 0 1

grass?

?

?

?

Social Tagging

phone

verizon

apple

1

google

0 0

1 1 0

0

0

1

0

1

1

?

?

?

?

How to predict an annotation word/social tag that does not appear in the training data ?

?//// / / ?

auxiliary classes

auxiliary classes

target classestarget classes

Related Work

Transfer Learning transfer knowledge from source domain to

target domain similarity: transfer label information for

auxiliary classes to target class difference: assume NO label information for

target class

Multi-Label Learning, Maximum Entropy Model

Unsupervised Transfer Classification Data

for auxiliary class

target class

target class label

target classification model

Goal

Prior probability

conditional probabilities

Class Information

Examples

Auxiliary Classes

assignments to auxiliary classes

Maximum Entropy Model (MaxEnt)

Favor uniform

distribution

Favor uniform

distribution

Feature statistics computed

from conditional model

Feature statistics computed

from training data: the jth feature function

Generalized MaxEnt

With a large probability

Equality constraints

Inequality constraints

Generalized MaxEnt

Generalized MaxEnt

is unknown for target class is unknown for target class

How to extend generalized MaxEnt to unsupervised transfer classification ?

Estimating feature statistics of target class from those of the auxiliary classes

Unsupervised Transfer Classification

~~

Unsupervised Transfer Classification Build up Relation between Auxiliary

Classes and Target Class

Independence Assumption

Unsupervised Transfer Classification Estimating feature statistics for the

target class by regression

Feature Statistics for

Auxiliary Classes

Feature Statistics for

Auxiliary Classes

Feature Statistics for Target Class

Feature Statistics for Target Class

Class Informati

on

Class Informati

on

Unsupervised Transfer Classification Dual problem

: function of U; definition can be found in paper

Consistency Result

With a large probability

The optimal dual solution using the label information for the target class

The dual solution obtained by the proposed approach

Experiments

Text categorization Data sets: multi-labeled data

Protocol: leave one-class out as the target class

Metric: AUC (Area under ROC curve)

Experiments: Baselines

cModel train a classifier for each auxiliary class linearly combine them for the target class cLabel predict the assignment of the target class for training

examples by linearly combining the labels of auxiliary classes

train a classifier using the predicted labels for target class

GME-avg use generalized maxent model compute the feature statistics for the target class by

linearly combining those for the auxiliary classes

Proposed Approach: GME-Reg

Experiment (I)

Estimate class information from training data

Estimate class information from training data

Compare to the classifier of the target class learned by supervised learning

Experiment (I)

1500 2500

Experiment (II)

Obtain class information from external sources

Datasets: bibtex and delicious bibsonomy www.bibsonomy.org/tagsbibtex ACM DL www.portal.acm.orgbibtex deli.cio.us www.delicious.com/tag

delicious

Experiment (II)

Comparison with Supervised Classification

650

1000~1200

Conclusions

A new problem: unsupervised transfer classification

A statistical framework for unsupervised transfer classification based on generalized maximum entropy robust estimate feature statistics for target class provable performance by consistency analysis

Future Work relax independence assumption better estimation of feature statistics for target

class

Thanks

Questions ?