leveraging sequence classification by taxonomy-based ...€¦ · taxonomy-based multitask learning...

Leveraging Sequence Classification by

Taxonomy-based Multitask Learning

Christian Widmer,1 Jose Leiva,1,2 Yasemin Altun,2 Gunnar Ratsch1

1 Friedrich Miescher Laboratory of the Max Planck Society, Tubingen, Germany2 Max Planck Institute for Biological Cybernetics, Tubingen, Germany

RECOMB 2010

August 15th, 2010

Christian Widmer (FML, Tubingen) Taxonomy-based Multitask Learning 1 / 22

Talk Outline fml

I Introduction of the setting

I Methods: 3 approaches to Hierarchical Mulitask Learning

I Experiments (Toy, Splicing)

I Outlook

I Summary

Introduction

Why Multitask learning? fml

I Model quality limited by insufficient training data

⇒ Idea: Use information from related domains/tasks

I Related Distributions: Different but similar

I In Computational Biology domains correspond to organismsI Organisms share evolutionary historyI Many biological processes are conserved

Introduction

Why Hierarchies? fmlHierarchical structure arises naturally from the Tree of Life

I Taxonomy used to define relationship between tasks

I Closer tasks benefit more from each other

Introduction

What problems could benefit from this? fmlI Prediction of Sequence Signals in Gene Finding (mGene [4])

I Numerous potential applications for Multitask LearningI Focus on splice-sites (Donor)

Introduction

What problems could benefit from this? fmlI Prediction of Sequence Signals in Gene Finding (mGene [4])

I Numerous potential applications for Multitask LearningI Focus on splice-sites (Donor)

Methods

Formal Problem Definition fml

Multitask Learning

I Consider M tasks Ti , where i ∈ {1, ...,M}I We are given data Di = {(x1, y1), ..., (xNi

, yNi)} for each task

I We want to train M predictors f1, ..., fM , each taking intoaccount all available information

I For that we would like to utilize a given taxonomy T , thatrelates the tasks at hand

⇒ We need algorithms to efficiently exploit T for transfer learning

Methods

Multitask Learning

Methods

Multitask Learning

Methods

Multitask Learning

Methods

Multitask Learning

Methods

Two ways of leveraging a given taxonomy T fml

Methods

Domain Adaptation by Regularization fmlIdea: Enforce model similarity via Regularization Term

I Regular SVM

minw,b

2‖w‖2 + C

∑(x,y)∈D

` (〈Φ(x),w〉+ b, y)

I DA-SVM

minw,b

2‖w−wpar‖2 + C

∑(x,y)∈D

` (〈Φ(x),w〉+ b, y) ,

where ` is the hinge loss, `(z , y) = max{1− yz , 0}.Christian Widmer (FML, Tubingen) Taxonomy-based Multitask Learning 8 / 22

Methods

Domain Adaptation by Regularization fmlIdea: Enforce model similarity via Regularization Term

I Regular SVM

minw,b

2‖w‖2 + C

∑(x,y)∈D

` (〈Φ(x),w〉+ b, y)

I DA-SVM

minw,b

2‖w−wpar‖2 + C

∑(x,y)∈D

` (〈Φ(x),w〉+ b, y) ,

where ` is the hinge loss, `(z , y) = max{1− yz , 0}.Christian Widmer (FML, Tubingen) Taxonomy-based Multitask Learning 8 / 22

Methods

Hierarchical Top-Down Approach fmlIdea: Exploit taxonomy G algorithmically

I Initialization: w0 trained on union of task dataI Top-Down for each node t:

I Train on Di =⋃

j4i Dj

I Regularize wi against parent predictor wpar : ‖wi−wpar‖2

I Use leaf predictors for classification

(a) Top-level training (b) Inner training (c) Taxon training

Methods

I Train on Di =⋃

j4i Dj

Methods

I Train on Di =⋃

j4i Dj

Methods

Pairwise Approach fml

minw1,...,wM

M∑t=1

M∑s=1

γt,s‖wt −ws‖2 +M∑t=1

∑(x,y)∈Dt

` (〈x,wt〉 , y) .

where ` is the hinge loss, `(z , y) = max{1− yz , 0}.I Train all classifiers wi at the same time

I Loss is evaluated independently on datasets Di

I Similarity is enforced via pairwise regularization term

I Task closeness controlled by γt,s

⇒ efficient solution possible via decompositionEvgeniou et al. [2]

Methods

Multitask Kernel Approach fml

maxα−1

n∑i=1

n∑j=1

αiαjyiyj k(xi , xj) +n∑

s.t. 0 ≤ αi ≤ C ∀i ∈ [1, n]

αTy = 0,

wherek((xi , s), (xj , t)) = ktask(s, t)︸︷︷︸

·k(xi , xj)

I Easily implemented by altering existing kernel functions (WDK)I Reuse existing kernel algorithms (SVM)

Daume III [1], Vert et al. [3]

Methods

Method Summary fmlI Multitask Learning Methods

I Top-DownI Pairwise RegularizationI Multitask Kernel

I Additional Baselines

Plain Union

Experiments

Toy Data Generation fmlI Sample sequences for M tasks

I Evolve generative model (PSSM) according to given structure

I We want to controlI How hard the problems areI How different the problems areI → use relative entropy DKL(P||Q) =

∑i P(i)log P(i)

Experiments

Results Toy fml

8 tasks 16 tasks 32 tasks 64 tasks0.62

plainunionpairwisemultikerneltop-down

I MTL methods outperform baselinesI High mutation rate leads to decreased performance in deep

hierarchiesChristian Widmer (FML, Tubingen) Taxonomy-based Multitask Learning 14 / 22

Experiments

Application to splice-site recognition fmlI Formulate as binary classification problem

I Utilize 15 organisms related by taxonomy

Experiments

Results Splicing Data fml

I MTL methods outperform baselines

I Best performer is Top-Down

I Potential increase depends on the particular organism

Experiments

Outlook fml

I Techniques for estimation of edge lengths

I Apply to Promoter prediction: ARTS [5] ⇒ ARTS.MTL

I Extension to structured output learning

Summary

Summary fml

I Presented three MTL methodsI Two extensions of existing MTL methodsI One novel method

I Demonstrated performance on two datasetsI Toy Data: RobustnessI Splicing Data: Outperform baselines

I Implementations available in Shogun [6]

I Relevant to many application in Comp-Bio

Summary

Summary fml

Summary

Summary fml

Summary

Summary fml

Summary

Acknowledgments fmlI Jose Leiva 1,2

I Yasemin Altun 2

I Soren Sonnenburg 2

I Gabrielle Schweikert 1,2

I Bernhard Scholkopf 2

I Klaus Robert Muller 3

I Gunnar Ratsch 1

1 Friedrich Miescher Laboratory of the Max Planck Society

2 Max Planck Institute for Biological Cybernetics

3 Berlin Institute of Technology

Thank you for your attention.

Summary

Acknowledgments fmlI Jose Leiva 1,2

I Yasemin Altun 2

I Soren Sonnenburg 2

I Gabrielle Schweikert 1,2

I Bernhard Scholkopf 2

I Klaus Robert Muller 3

I Gunnar Ratsch 1

1 Friedrich Miescher Laboratory of the Max Planck Society

2 Max Planck Institute for Biological Cybernetics

3 Berlin Institute of Technology

Thank you for your attention.

Summary

References I

H. Daume.

Frustratingly easy domain adaptation.

In ACL. The Association for Computer Linguistics, 2007.

T. Evgeniou, C.A. Micchelli, and M. Pontil.

Learning multiple tasks with kernel methods.

Journal of Machine Learning Research, 6:615–637, 2005.

L. Jacob and J.P. Vert.

Efficient peptide-MHC-I binding prediction for alleles with few knownbinders.

Bioinformatics, 24(3):358, 2008.

Summary

References II

Gabriele Schweikert, Alexander Zien, Georg Zeller, Jonas Behr, ChristophDieterich, Cheng Soon Ong, Petra Philips, Fabio De Bona, Lisa Hartmann,Anja Bohlen, Nina Kruger, Soren Sonnenburg, and Gunnar Ratsch.

mGene: accurate SVM-based gene finding with an application to nematodegenomes.

Genome research, 19(11):2133–43, November 2009.

S. Sonnenburg, A. Zien, and G. Ratsch.

ARTS: accurate recognition of transcription starts in human.

Bioinformatics, 22(14):e472, 2006.

Summary

References III

Soren Sonnenburg, Gunnar Ratsch, Sebastian Henschel, Christian Widmer,Jonas Behr, Alexander Zien, Fabio de Bona, Alexander Binder, ChristianGehl, and Vojtech Franc.

The shogun machine learning toolbox.

Journal of Machine Learning Research, 2010.

accepted.

leveraging sequence classification by taxonomy-based ...€¦ · taxonomy-based multitask learning...

Documents