barking up the wrong treelength kevin liu, serita nelesen, sindhu raghavan, c. randal linder, and...
TRANSCRIPT
Barking Up the Wrong Treelength
Kevin Liu, Serita Nelesen, Sindhu Raghavan, C. Randal Linder, and Tandy Warnow
IEEE TCCB 2009
Minimizing Treelength
Generalized Input: set S of sequences and a function f(s, s') for
the edit distance between sequences s and s' Output: A tree T, leaf-labelled by set S, with
additional sequences labelling the internal nodes of T, so as to minimize treelength (total edit distance on the edges of the tree)
Fixed Tree variant
POY
POY (from the American Museum of Natural History, Ward Wheeler and colleagues) is the main software for this.
Minimizing treelength is also known as “Direct Optimization”
POY has passionate adherents who believe in treelength
POY also has been heavily criticized
POY
Input: set S of sequences (unaligned), gap-open cost, gap-extend cost, and transition/transversion ratio
Default settings for gap-open and gap-extend in POY are “simple” (gap-open cost is 0)
POY can also be used to score a fixed input tree under the desired treelength definition.
Ogden and Rosenberg 2007
Ogden and Rosenberg study compared POY 3.0 to MP(ClustalW) Model conditions – mostly 16 taxa (some 64 taxon trees),
K2P substitution model, short gaps (expected length 4) Optimization Problem – Multiple edit distances, all on simple
gap penalties (gap-open cost is 0) Performance metrics
Tree errors Alignment errors No mention of treelength
Result: MP(ClustalW) much more accurate than POY
O&R concluded that Treelength is BAD!
O&R simulation study showed that POY alignments worse than ClustalW more than 99% of the time, and POY trees less accurate than ClustalW on average.
“Therefore, traditional multiple sequence alignment approaches appear to vastly outperform direct optimization-like approaches in terms of alignment accuracy, at least for the data sets and parameter settings that have been examined thus far.” Ogden and Rosenberg 2007
Treelength is BAD!
“Although our data represents a fairly simple case, for data sets similar to these the traditional two-step approach will almost always give a more accurate alignment and will most likely recover equally or more accurate phylogenetic relationships than direct optimization as implemented in POY.” Ogden and Rosenberg 2007
Our question
Does minimizing treelength work poorly in general,
or
Is it minimizing treelength under simple gap penalties that works poorly?
Gap penalties
Simple: a gap of length k costs kC Affine: a gap of length k costs Copen+kCextend
Other types of penalties are possible
“Treelength not so bad!”(paraphrasing Liu et al 2009)
Liu et al. 2009 show Treelength can be a good criterion, if based
upon affine gap penalty We developed POY*: a version of POY which
uses: a particular affine gap penalty, and a particular starting tree
Our Study 2008
Our study compares POY 4.0 to multiple methods Model conditions – 25 and 100 taxa, GTR+Gamma
for the substitution model, short and long gaps Optimization Problem – Multiple edit distances,
based upon both simple and affine gap penalties Results
Tree error Alignment error Treelength
Gap cost functions we studied
Simple1 – all mismatches and indels cost 1 Simple2 – indels cost 2, transversions cost 2 and
transitions cost 1 Affine – gap of length k costs 4 + k, transversions cost
2, and transitions cost 1
Simulation Study Overview
Model trees Birth-death Deviation from ultrametricity
Sequence evolution Estimation of trees and alignments Statistics
Simulation Study Overview
Model trees Sequence evolution
GTR model of evolution from Tree of Life project Gamma-distributed rates across sites Gap model
Estimation of trees and alignments Statistics
Simulation Study Overview
Model trees Sequence evolution Estimation of trees and alignments
POY POY* - POY with particular starting tree (Probtree,
using a particular Affine gap penalty Several two-phase methods (best alignments
followed by MP and ML) PS (POY-score) on various trees
Statistics
Simulation Study Overview
Model trees Sequence evolution Estimation of trees and alignments Statistics
1. Alignment error
2. Tree error
3. Treelength under each gap cost function
Simulation Study Model Conditions
4 model conditions 80 replicate datasets apiece Different numbers of taxa allow us to explore
taxonomic sampling effects
Results – Alignment
Errors Simple vs. affine
penalties Note: story
changes for affine penalties, especially on long gap event distribution
Alignment Error: ClustalW vs. POY*
POY* better than ClustalW over 50% in (b), and 90% of time under (a)
Compare with Ogden and Rosenberg, who find ClustalW better than POY 99.9% of time
Results – Alignment
Errors
PS is POY used to estimate alignments on various trees
Note: PS produces worse alignments than ClustalW if simple gap cost functions are used, even if applied to the true tree
Tree error
POY and POY* both use the same gap penalty (affine)
Results shown on 100 taxon short gap simulated datasets (results for other models similar)
Tree Error
POY and POY* both use the same gap penalty (affine)
Results shown on 100 taxon short gap simulated datasets (results for other models similar)
Tree error
POY and POY* both use the same gap penalty (affine)
Results shown on 100 taxon short gap simulated datasets (results for other models similar)
How well does POY solve its optimization problem?
We examine the treelength found by POY for various model conditions
We let treelength be defined by simple1, simple2, or affine
We compare treelengths found by POY to treelengths achievable in each model condition (as produced by scoring the true tree and other trees)
Results - Treelengths
POY search finds short trees for simple gap penalties, but not for affine
Can we propose a better POY search for affine penalties?
POY*
How well does POY solve its optimization problem?
Simple gap penalties: excellent performance Affine gap penalties: poor performance
But POY* optimizes both well.
The difference is just the starting tree.
Is it a good idea to optimize treelength?
Simple gap penalties: NO! Worse trees and worse alignments.
Affine gap penalties: Let’s see.
Insights
Simple gap penalties were a main cause behind Ogden and Rosenberg's findings
Unable to obtain accurate POY alignments and trees under a simple treelength criterion
Using affine penalties, POY*: Obtains alignments that are more accurate than ClustalW 90% of long gap
datasets, 75% of medium, 55% of short Has tree accuracy that is comparable to the best two-phase method (ML
on good alignments) But poorer alignments than the best alignment methods (e.g., Probtree)
Conclusions Distinguish between the optimization problem,
and the heuristic methods used for those problems
The treelength optimization criteria chosen has a significant impact on the tree and alignment error Simple alignment and trees aren't competitive
relative to two-phase methods, and improving simple criteria treelengths doesn't get better trees
Affine criteria story is still open Can we find shorter trees than two-phase trees? How accurate are such shorter trees?