Hindawi Publishing CorporationApplied Computational Intelligence and Soft ComputingVolume 2013 Article ID 794350 9 pageshttpdxdoiorg1011552013794350

Research ArticleOn the Variability of Neural Network Classification Measures inthe Protein Secondary Structure Prediction Problem

Eric Sakk and Ayanna Alexander

Department of Computer Science Morgan State University Baltimore MD 21251 USA

Correspondence should be addressed to Eric Sakk ericsakkmorganedu

Received 27 April 2012 Revised 19 November 2012 Accepted 27 November 2012

Academic Editor Cheng-Jian Lin

Copyright copy 2013 E Sakk and A Alexander This is an open access article distributed under the Creative Commons AttributionLicense which permits unrestricted use distribution and reproduction in any medium provided the original work is properlycited

We revisit the protein secondary structure prediction problem using linear and backpropagation neural network architecturescommonly applied in the literature In this context neural network mappings are constructed between protein training setsequences and their assigned structure classes in order to analyze the class membership of test data and associated measures ofsignificance We present numerical results demonstrating that classifier performance measures can vary significantly dependingupon the classifier architecture and the structure class encoding technique Furthermore an analytic formulation is introduced inorder to substantiate the observed numerical data Finally we analyze and discuss the ability of the neural network to accuratelymodel fundamental attributes of protein secondary structure

1 Introduction

The protein secondary structure prediction problem can bephrased as a supervised pattern recognition problem [1ndash5] forwhich training data is readily available from reliable databasessuch as the Protein Data Bank (PDB) or CB513 [6] Basedupon training examples subsequences derived from primarysequences are encoded based upon a discrete set of classesFor instance three class encodings are commonly applied inthe literature in order to numerically represent the secondarystructure set (alpha helix (119867) beta sheet (119864) coil (119862)) [7ndash11]By applying a pattern recognition approach subsequences ofunknown classification can then be tested to determine thestructure class to which they belong Phrased in this waybackpropagation neural networks [7 12ndash14] and variationson the neural network theme [8 10 11 15ndash18] have beenapplied to the secondary structure prediction problem withvaried success Furthermore many tools currently applyinghybridmethodologies such as PredictProtein [19 20] JPRED[8 17 21] SCRATCH [22 23] and PSIPRED [24 25] relyon the neural network paradigm as part of their predictionscheme

One of the main reasons for applying the neural networkapproach in the first place is that they tend be good universal

approximators [26ndash30] and theoretically have the potentialto create secondary structure models In other words aftera given network architecture has been chosen and presentedwith a robust set of examples the optimal parametersassociated with the trained network in principle define anexplicit function that can map a given protein sequence toits associated secondary structure If the structure predictedby the network function is generally correct and consis-tent for an arbitrary input sequence not contained in thetraining set one must be left to conclude that the neuralnetwork has accurately modeled some fundamental set ofattributes that define the properties of protein secondarystructure Under these circumstances one should then beable to extract information from the trained neural net-work model parameters thus leading to a solution to thesecondary structure prediction problem as well as a para-metric understanding of the underlying basis for secondarystructure

The purpose of this work is to revisit the application ofneural networks to the protein secondary structure predic-tion problem In this setting we consider the commonlyencountered case where three structure classes (alpha helix(119867) beta sheet (119864) and coil (119862)) are used to classify agiven protein subsequence Given the same set of input

2 Applied Computational Intelligence and Soft Computing

training sequences we demonstrate that for the backprop-agation neural network architecture classification resultsand associated confidence measures can vary when twoequally valid encoding schemes are employed to numericallyrepresent the three structure classes (ie the ldquotarget encodingschemerdquo) Such a result goes against the intuition that thephysical nature of the secondary structure property shouldbe independent of the target encoding scheme chosen

The contribution of this work is not to demonstrateimprovements over existing techniques The hybrid tech-niques outlined above have been demonstrated to outperformneural networks when used alone Instead we focus ourattention on the ability of the neural network model-basedapproach to accurately characterize fundamental attributesof protein secondary structure given that certain modelspresented within this work are demonstrated to yield variableresults Specifically in this work we present

(1) numerical results demonstrating how secondarystructure classification results can vary as function ofclassifier architecture and parameter choices

(2) an analytic formulation in order to explain underwhat circumstances classification variability can arise

(3) an outline of specific challenges associated withthe neural network model-based approach outlinedabove

The conclusions reported here are relevant because theybring into discussion a body of literature that has purportedto offer a viable path to the solution of the secondary struc-ture prediction problem Section 3 describes the methodsapplied in this work examine the total number that retainedtheir classification using In particular this section providesdetails concerning the encoding of the protein sequencedata (Section 31) the encoding of the structure classes(Section 32) as well as the neural network architectures(Sections 33-34) and the classifier performance measures(Section 35) applied in this work Section 4 then presentsresults from numerical secondary structure classificationexperiments Section 5 presents an analytic formulationfor the linear network and the backpropagation networkdescribed in Section 3 in order to explain the numericalresults given in Section 4

2 Notation for the SupervisedClassification Problem

In the supervised classification problem [1 2] it is assumedthat a training set consists of119873 training pairs

(1198831 1198841) (1198832 1198842) (119883

119873 119884119873) (1)

where 119883119894isin 119877119899 are 119899-dimensional input column vectors and

119884119894isin 119877119901 are 119901-dimensional output column vectors The goal

of the supervised classifier approach is to ensure that thedesired response to a given input vector 119883

119894of dimension 119899

from the training set is the 119901-dimensional output vector 119884119894

Furthermore when the training data can be partitioned into

119872 distinct classes a set E equiv 1198901 119890

119895 119890

119872 of target 119901-

dimensional column vectors 119890119895isin 119877119901 are chosen to encode

(ie mathematically represent) each class for 119895 = 1 119872Under these circumstances each output training vector 119884

119894

is derived from the set E Based upon this discussion wesummarize the use of following symbols

(i) 119899 is the dimension of a classifier input vector(ii) 119901 is the dimension of a classifier output vector(iii) 119872 is the number of discrete classes for the classifica-

tion problem(iv) 119873 is the number of training pairs for the supervised

classification problem

3 Methods

In order to apply the neural network paradigm two numeri-cal issues must be addressed First since the input data comesin the form of an amino acid sequence Section 31 discusses asimple encoding scheme for converting the amino acid alpha-bet into a usable numerical form Second for this work oursecondary structure target alphabet consists of elements fromthe set 119864119867 119862 = Beta Sheet AlphaHelix Coil Hencean encoding schememust also be chosen for representing theneural network classifier output Section 32 discusses twoapproaches to encoding the output in fine detail because itis critical to the main point of this paper Specifically wechoose two different target vector encoding schemes thatcan be related by a simple mathematical relationship Suchan approach will allow us to compare classifier performancemeasures based upon the target vector encoding in addi-tion it will facilitate the analytic formulation presented inSection 5 Finally Sections 33ndash35 review the neural networkarchitectures and the specific classifier performancemeasuresemployed in this work Section 6 then concludes with somefinal observations regarding the neural network model-based approach to the protein secondary structure predictionproblem

31 Encoding of Protein Sequence Input Data For the numer-ical experiments the training set was constructed usingone hundred protein sequences randomly chosen from theCB513 database [6] available through the JPRED secondarystructure prediction engine [21] Furthermore we employa moving window of length 17 to each protein sequencewhere in order to avoid protein terminal effects the firstand last 50 amino acids are omitted from the analysis Thesecondary structure classification of the central residue isthen assigned to each window of 17 amino acids For theone hundred sequences analyzed a total of 12000 windowsof length 17 were extracted The window size value of 17was chosen based upon the assumption that the eight closestneighboring residues will have the greatest influence on thesecondary structure conformation of the central residueThisassumption is consistent with similar approaches reported inthe literature [7 12ndash14]

To encode the input amino acid sequences of length 17 weemploy sparse orthogonal encoding [31] whichmaps symbols

Applied Computational Intelligence and Soft Computing 3

from a given sequence alphabet onto a set of orthogonalvectors Specifically for an alphabet containing 119870 symbols aunique119870-dimensional unit vector is assigned to each symbolfurthermore the 119896th unit vector is one at the 119896th position andis zero at all other positions Hence if all training sequencesand unknown test sequences are of uniform length 119871 anencoded input vector will be of dimension 119899 where 119899 = 119871119870In our case 119870 = 20 and 119871 = 17 hence the dimension of anygiven input vector is 119899 = 340

The above input vector encoding technique is commonlyapplied in the bioinformatics and secondary structure pre-diction literature [7 15] While many different and superiorapproaches to this phase of the machine learning problemhave been suggested [3ndash5] we have chosen orthogonalencoding because of its simplicity and the fact that the resultsof this work do not depend upon the input encoding schemeInstead our work specifically focuses on potential neuralnetwork classifier variabilities induced by choice of the targetvector encoding scheme

32 Target Vector Encoding Analytically characterizing theinvariance of classifier performancemeasures clearly involvesfirst establishing a relationship between different sets of targetvectors E and E As a means of making the invarianceformulation presented in this paper more tractable weassume that two alternative sets of target vectors can berelated via an affine transformation involving a translation119879 a rigid rotation 119877 where 119877 is an orthogonal matrix and ascale factor 119886

Γ equiv 119886119877 (Γ minus 119879) (2)

where

Γ equiv [11989011198902 119890119872] Γ equiv [119890

11198902 119890119872]

119879 equiv [11990511199052 119905119872]

(3)

is a matrix of translation column vectors applied to eachtarget vector Many target vector choices regularly applied inthe literature can be related via the transformation in (2) Forinstance two equally valid and commonly applied encodingschemes for the three class problem are orthogonal encoding[31] where

119867 997888rarr [

[

1

0

0

]

]

119864 997888rarr [

[

0

1

0

]

]

119862 997888rarr [

[

0

0

1

]

]

(4)

and

119867 997888rarr

[[[[[[

[

minus1

2

radic3

2

0

]]]]]]

]

119864 997888rarr

[[[[[[

[

minus1

2

minusradic3

2

0

]]]]]]

]

119862 997888rarr [

[

1

0

0

]

]

(5)

where class encodings are chosen on the vertices of a trianglein a two-dimensional plane [14] It turns out that (4) and (5)can be phrased in terms of (2) [32] hence the numerical

results presented in this work will apply this set of encodingsMore precisely the secondary structure classification associ-ated with a given input vector is encoded using (4) and (5)(hence 119901 = 3) The set of target vectorsE is derived from (4)and the set of target vectors E is derived from (5) Both thelinear and the backpropagation networks are tested first bytraining using E and then comparing classifier performancewith their counterparts trained using E In all numericalexperiments MATLAB has been used for simulating andtesting these networks

33 The Linear Network When the supervised classifiermodel in (1) assumes an affine relationship between theinput and output data sets (as in the case of multiple linearregression) matrices of the form

119883 equiv [1198831 119883

119873] (6)

119884 equiv [1198841 119884

119873] (7)

are generally introduced Specifically the linear networkseeks to determine a 119901 times 119899 matrix 120573 of coefficients and aconstant 119901-dimensional column vector 119888 such that the 119894thoutput vector 119884

119894in the training set can be approximated by

120573119883119894+ 119888 (8)

Given this model we can form a weight matrix of unknowncoefficients

119882 equiv [120573 119888] (9)that ideally will map each input training vector into thecorresponding output training vector If the bottom rowof theinput data matrix 119883 is appended with a row of ones leadingto the (119899 + 1) times 119873matrix

119860 equiv [1198831 119883

119873

1 1] (10)

in matrix form the goal is then to find a 119901 times 119899 + 1 weightmatrix119882 that minimizes the sum squared-error

119864linear

equiv

119873

sum119894=1

(119882119860119894minus 119884119894)119879

(119882119860119894minus 119884119894) (11)

over the set of data pairs

119860119894equiv [119883119894

1] (12)

by satisfying the first derivative condition 120597119864linear120597119882 = 0The least squares solution to this problem is found via thepseudoinverse 119860dagger [33] where

119882 = 119884119860dagger (13)

Once the optimal set of weights has been computed thenetwork response to an unknown 119899 times 1 input vector 119909 can bedetermined by defining the (119899 + 1) times 1 vector

119911 equiv [119909 1]119879 (14)

and calculating119891 (119911) = 119882119911 (15)

where 119891(119911) is a 119901 times 1 column vector

4 Applied Computational Intelligence and Soft Computing

34The Backpropagation Network Given an input vector 119909 isin119877119899 the model 119891 119877119899 rarr 119877119901 for a backpropagation neuralnetwork with a single hidden layer consisting of 119898 nodes isdescribed as

119891 (119909) =

[[[[[

[

1198911(119909)

1198912(119909)

119891119901(119909)

]]]]]

]

= 120572120590 (119882119909 minus 120591) (16)

where 120572 isin 119877119901times119898 119882 isin 119877119898times119899 and 120591 isin 119877119898 define the setof network weights W = 119882 120591 120572 and 120590 119877119898 rarr 119877119901 isa ldquosigmoidalrdquo function that is bounded and monotonicallyincreasing To perform supervised training in a mannersimilar to the linear networkW is determined byminimizingthe objective function

119864119887119901=

119873

sum119894=1

(119891 (119883119894) minus 119884119894)119879

(119891 (119883119894) minus 119884119894) (17)

given the training data defined in (1) Since 119891(119883119894) is no

longer linear numerical techniques such as the gradientdescent algorithm and variations thereof are relied upon tocompute the setW that satisfies the first derivative condition120597119864119887119901120597W = 0Consider the following definitions

120575119865119896equiv 119891 (119883

119896) minus 119884119896=[[

[

1198911(119883119896) minus (119884

119896)1

1198912(119883119896) minus (119884

119896)2

119891119901(119883119896) minus (119884

119896)119901

]]

]

119878119896equiv

[[[[[[[[[[[[[[

[

120590(

119899

sum119895=1

1198821119895(119883119896)119895minus 1205911)

120590(

119899

sum119895=1

1198822119895(119883119896)119895minus 1205912)

120590(

119899

sum119895=1

119882119898119895(119883119896)119895minus 120591119898)

]]]]]]]]]]]]]]

]

1198781015840

119896equiv

[[[[[[[[[[[[[[

[

1205901015840(

119899

sum119895=1

1198821119895(119883119896)119895minus 1205911)

1205901015840(

119899

sum119895=1

1198822119895(119883119896)119895minus 1205912)

1205901015840(

119899

sum119895=1

119882119898119895(119883119896)119895minus 120591119898)

]]]]]]]]]]]]]]

]

(18)

where 1205901015840(119911) = 119889120590119889119911 The first derivative conditions forthe network weights prescribed by (16) and (17) can then bewritten in matrix form as follows

120597119864119887119901

120597120572=

119873

sum119896=1

120575119865119896119878119879

119896= 0 (19)

(where ldquo119879rdquo denotes the matrix transpose)

120597119864119887119901

120597120591= minus

119873

sum119896=1

diag (1198781015840119896) 120572119879120575119865119879

119896= 0

120597119864119887119901

120597119882=

119873

sum119896=1

diag (1198781015840119896) 120572119879120575119865119879

119896119883119879

119896= 0

(20)

where diag(1198781015840119896) is a square diagonal matrix such that the

diagonal entries consist of components from the vector 1198781015840119896

35 ClassificationMeasures After a given classifier is trainedwhen presented with an input vector 119909 isin 119877119899 of unknownclassification it will respond with an output 119891(119909) isin 119877119901 Theassociated class membership 119862(119909) is then often determinedby applying a minimum distance criterion

119862 (119909) equiv argmin119894

119889 (119891 (119909) 119890119894) (21)

where the target vector 119890119894that is closest to 119891(119909) implies the

class Furthermore when characterizing the performance of apattern classifier one often presents a set 119878 of test vectors andanalyzes the associated output In addition to determiningthe class membership it is also possible to rank the distancebetween a specific target vector and the classifier responseto 119878 In this case a similar distance criterion can be appliedin order to rank an input vector 119909 isin 119878 with respect to class119869 isin 1 119872

120588119869(119909) = 119889 (119891 (119909) 119890

119869) (22)

For the purposes of this work (15) facilitates the determina-tion of the classmembership and rankingwith respect to class119869 for the linear network Similarly assuming a set of weightsW = 119882 120591 120572 for a trained backpropagation network (21)and (22) would be applied using (16)

It is well established that in the case of a normallydistributed data the classification measures presented aboveminimize the probability of a classification error and aredirectly related to the statistical significance of a classificationdecision [1] Given the neural network as the supervisedclassification technique and two distinct choices for the set oftarget vectorsE and E we demonstrate in certain instancesthat the classification and ranking results do not remaininvariant such that 119862(119909) = 119862(119909) and 120588

119869(119909) = 120588

119869(119909) for any

input vector 119909

4 Noninvariance of SecondaryStructure Predictions

In this section we numerically demonstrate that when differ-ent target vector encodings are applied neural network clas-sifier measures outlined above in certain cases are observedto vary widely For each neural network architecture underconsideration an analytic formulation is then presented inSection 5 in order to explain the observed numerical data

As mentioned in Section 32 numerical experiments areperformed first by training using E and then comparing

Applied Computational Intelligence and Soft Computing 5

Table 1 Ranking results for the linear network where 119894 representsthe test vector index and 120588

1(119909) and 120588

1(119909) represent the distance with

respect to the helix class vectors 1198901and 1198901 Out of 1800 vectors tested

vectors referred to in this table were ranked from 1 to 20

119894E 1205881(119909) 119894E 120588

1(119909)

1205 00780 1205 0117042 00867 42 013001031 00976 1031 014641773 01113 1773 01670598 01238 598 018571761 01267 1761 01900862 01354 862 020311073 01409 1073 02114277 01459 277 02188115 01540 115 023091505 01821 1505 02731392 01839 392 027591421 01904 1421 02856147 02001 147 03001990 02044 990 030661457 02127 1457 031911288 02150 1288 03225352 02160 352 032391232 02198 1232 03297280 02311 280 03466

classifier performance with their counterparts trained usingE Multiple cross-validation trials are required in order toprevent potential dependency of the evaluated accuracy onthe particular training or test sets chosen [7 15] In this workwe apply a hold-n-out strategy similar to that of [14] using85 of the 12000 encoded sequences as training data (ie119873 = 10200) and 15 as test data to validate the classificationresults Recognition rates for both the linear and backpropa-gation rates using either set of target vector encodings wereapproximately 65 which is typical of this genre of classifiersthat have applied similar encoding methodologies [7 12ndash14] Although these aggregate values remain consistent using(21) and (22) we now present data demonstrating that whileclass membership and ranking remain invariant for the linearnetwork these measures of performance vary considerablyfor the backpropagation network which was trained with119898 = 17 seventeen hidden nodes and amean squared trainingerror less than 02 Ranking results from a representative testfor the linear and backpropagation networks are presented forthe top 20 ranked vectors in Tables 1 and 3 Classmembershipdata are presented in Tables 2 and 4 Observe that forthe linear network indices for the top 20 ranked vectorsremain invariant indicating ranking invariance in additionno change in class membership is observed On the otherhand Tables 3 and 4 clearly indicate a lack of consistencywhen considering the ranking and class membership of testvectors A particularly troubling observation is that very fewvectors ranked in the top 20 with respect to 119890

1were ranked in

the top 20 with respect to 1198901 Furthermore Table 4 indicates

Table 2 Class membership results for the linear network For eachclass the total number of vectors classified using E is analyzed toexamine the total number that retained their classification using E

Class E E change119867 202 202 0119864 621 621 0119862 977 977 0

that the class membership of a substantial number of testvectors changed when an alternative set of target vectors wasemployed The data also indicates that the greatest changein class membership took place for alpha helical sequencesthus implying that there is substantial disagreement overthe modeling of this secondary structure element by thebackpropagation network due to a simple transformation ofthe target vectors

5 Analysis

The results in Section 4 clearly show that while the patternrecognition results for the linear network remain invariantunder a change in target vectors those for the backpropa-gation network do not In this section we present analyticresults in order to clearly explain and understand why thesetwo techniques lead to different conclusions

51 Invariance Formulation Let us begin by considering twodefinitions

Definition 1 Given two sets of target vectors E and E theclass membership is invariant under a transformation of thetarget vectors if for any input vector 119909

119889(119891 (119909) 119890119894) gt 119889(119891 (119909) 119890

119895)997904rArr119889 (119891 (119909) 119890

119894) gt 119889(119891 (119909) 119890

119895)

(23)

where 119891(119909) is the output of the classifier with target vectors119890119895119872

119895=1

Definition 2 Given two sets of target vectors E and E theranking with respect to a specific class 119869 is invariant under atransformation of the target vectors if for any input vectors1199091and 119909

2

119889(119891(1199091) 119890119869) gt 119889(119891(119909

2) 119890119869)997904rArr119889(119891 (119909

1) 119890119869) gt 119889(119891 (119909

2) 119890119869)

(24)

Based upon these definitions the following has beenestablished [32]

Proposition 3 Given two sets of target vectorsE and E if theranking is invariant then the class membership of an arbitraryinput vector 119909 will remain invariant

In the analysis presented the strategy for characterizingneural network performance depends upon the data from theprevious section For the linear network since both ranking

6 Applied Computational Intelligence and Soft Computing

Table 3 Ranking results for the backpropagation network where 119894represents the test vector index and 120588

1(119909) and 120588

1(119909) represent the

distance with respect to the helix class vectors 1198901and 1198901 Out of 1800

vectors tested vectors referred to in this table were ranked from 1 to20

119894E 1205881(119909) 119894E 120588

1(119909)

817 00107 926 00101887 00231 1604 00130264 00405 887 002091183 00711 1145 00214684 00727 461 00232623 00874 583 00329911 00891 1086 003391382 00917 1382 004781610 00939 413 00489551 01060 225 006081042 01150 438 00609924 01322 911 00613727 01339 207 00774438 01356 559 00885577 01363 481 00945896 01500 1548 00947175 01513 962 009681138 01549 85 01012583 01581 195 01111559 01655 9 01167

Table 4 Class membership results for the backpropagationnetwork For each class the total number of vectors classified usingE is analyzed to examine the total number that retained theirclassification using E

Class E E change119867 225 142 369119864 581 476 181119862 994 878 117

and classification were observed to remain invariant it ismore sensible to characterize the invariance of this networkusing Definition 2 Then based upon Proposition 3 classmembership invariance naturally follows On the other handto explain the noninvariance of both class membership andranking observed in the backpropagation network the anal-ysis is facilitated by considering Definition 1 The noninvari-ance of ranking then naturally follows from Proposition 3

511 Invariance Analysis for the Linear Network When thetarget vectors are subjected to the transformation defined in(2) the network output can be expressed as

119891 (119911) = 119911 = 119860dagger119911 = 119886119877 (119884 minus 119879

1015840)119860dagger119911 (25)

where 1198791015840 is derived from 119879 such that the translation vector119905119894associated with 119890

119894isin E is appropriately aligned with the

correct target vector in matrix 119884 In other words when theoutput data matrix in (7) is of the form

119884 = [1198901198941

1198901198942

119890119894119873

] (26)

then

1198791015840= [1199051198941

1199051198942

119905119894119873

] (27)

where 119894119895isin 1 119872 for 119895 = 1 119873 Given this network

the following result is applicable [32]

Proposition 4 If

(i) the number of training observations 119873 exceeds thevector dimension 119899

(ii) the rows of the matrix 119860 are linearly independent(iii) E and E are related according to (2)(iv) for some 119905 isin 119877119901 119905

119894= 119905 for all 119894 = 1 119872 in (2)

then the ranking and hence the classmembership for the linearnetwork will remain invariant

In other words if the columns of the matrix 1198791015840 in (25)are all equal then using (15) and (25) will result in (23)being satisfiedThe above result is applicable to the presentednumerical data with 119899 = 340 and119873 = 10200 hence rankingand class membership invariances are corroborated by thedata in Tables 1 and 2

512 Invariance Analysis for the Backpropagation NetworkIn this section we seek to characterize the noninvarianceobserved in class membership using the backpropagationnetwork If the class membership varies due to a changein target vectors then this variation should be quantifiableby characterizing the boundary separating two respectiveclasses The decision boundary between class 119894 and class 119895 isdefined by points 119909 such that

1003817100381710038171003817119891 (119909) minus 11989011989410038171003817100381710038172

=10038171003817100381710038171003817119891 (119909) minus 119890

119895

10038171003817100381710038171003817

2

(28)

where 119894 119895 isin 1 119872 119894 = 119895 and 119891(119909) is the classifier outputUnder these circumstances if an ℓ

2norm is applied in (21)

(119891 (119909) minus 119890119894)119879

(119891 (119909) minus 119890119894) = (119891 (119909) minus 119890

119895)119879

(119891 (119909) minus 119890119895)

(29)

The solution set to this equation consists of all 119909 such that119891(119909) is equidistant from 119890

119894and 119890119895where for the purposes of

this section119891(119909) is defined by (16) Expanding terms on bothsides of this equation leads to the condition

(119890119894minus 119890119895)119879

119891 (119909) = minus1

2(119890119879

119895119890119895minus 119890119879

119894119890119894) (30)

If the classmembership of a representative vector is to remaininvariant under a change of target vectors this same set ofpoints must also satisfy

( 119890119894minus 119890119895)119879

119891 (119909) = minus1

2( 119890119879

119895119890119895minus 119890119879

119894119890119894) (31)

Applied Computational Intelligence and Soft Computing 7

Assuming that two networks have been trained using twodifferent sets of target vectors E and E the set of weights119882 120591 120572 and 120591 determines the network output in (16)Without loss of generality we consider the case where alltarget vectors are normalized to a value of one such that1198901198942= 1 and 119890

1198942= 1 for 119894 = 1 119872 In this case the

conditions in (30) and (31) become

(119890119894minus 119890119895)119879

120572120590 (119882119909 minus 120591) = 0 (32)

(119890119894minus 119890119895)119879

120590 (119909 minus 120591) = 0 (33)

We first consider a special case where the target vectorsare related according to Γ = 119886119877Γ with 119879 = 0 in (2) Underthese circumstances if the choice

W equiv 120591 = 119882 120591 119886119877120572 (34)

is made it should be clear that since119877119879 = 119877minus1 and 119894= 119886119877119884

119894

119864119887119901=

119873

sum119894=1

(119891 (119883119894) minus 119894)119879

(119891 (119883119894) minus 119894)

= 1198862

119873

sum119894=1

(119877119891 (119883119894) minus 119877119884

119894)119879

(119877119891 (119883119894) minus 119877119884

119894)

= 1198862

119873

sum119894=1

(119891 (119883119894) minus 119884119894)119879

119877119879119877 (119891 (119883

119894) minus 119884119894)

= 1198862119864119887119901

(35)

is minimized by the condition 120597119864119887119901120597W = 0 Another way tosee this is to observe that (19) (20) remain invariant for thischoice of target vectors and network weights Hence we havethe following

Proposition 5 For a specific choice of 119886 and 119877 if 119879 = 0 in (2)and

W = 120591 = 119882 120591 119886119877120572 (36)

then the classmembership for the backpropagation networkwillremain invariant

Proof Simply consider (32) and (33) and choose any 119909satisfying

(119890119894minus 119890119895)119879

120572120590 (119882119909 minus 120591) = 0 (37)

It then immediately follows that

(119890119894minus 119890119895)119879

120590 (119909 minus 120591)

= (119890119894minus 119890119895)119879

119877119879119877120572120590 (119882119909 minus 120591)

= 1198862(119890119894minus 119890119895)119879

120572120590 (119882119909 minus 120591)

= 0

(38)

Therefore if 119909 satisfies (32) then it also satisfies (33) andhence is a point on the decision boundary for both net-works

Intuitively a scaled rigid rotation of the target vectorsshould not affect the decision boundary However when themore general transformation of (2) is applied with 119879 = 0we now demonstrate that due to the nonlinearity of 120590 in(16) no simple relationship exists such that (32) and (33)can simultaneously be satisfied by the same set of pointsWe first investigate the possibility of establishing an analyticrelationship between the set of weights 120591 and 119882 120591 120572for both networks In other words we seek ideally invertiblefunctions 119866

120572 119866120591 and 119866

119882

= 119866120572(119882 120591 120572)

120591 = 119866120591(119882 120591 120572)

= 119866119882(119882 120591 120572)

(39)

such that the setW can be transformed into W If this can bedone then an analytic procedure similar to that presented inthe proof of Proposition 5 can be established in order to relate(32) to (33) for the general case Since (19) (20) define the setW it is reasonable to rephrase these equations in terms of theobjective function 119864119887119901

120597119864119887119901

120597=

119873

sum119896=1

(119891 (119883119896) minus 119896) 119878119879

119896= 0

120597119864119887119901

120597120591= minus

119873

sum119896=1

diag (1198781015840119896) 119879(119891 (119883

119896) minus 119896)119879

= 0

120597119864119887119901

120597=

119873

sum119896=1

diag (1198781015840119896) 119879(119891 (119883

119896) minus 119896)119879

119883119879

119896= 0

(40)

where 119891(119883119896) = 120590(119883

119896minus 120591) and

119896= 119886119877(119884

119896minus 1199051015840) such that

1199051015840 is the translation vector associated with the target vectorreferred to by 119884

119896 From these equations it should be clear

that no simple analytic relationship exists that will transformW into W A numerical algorithm such as gradient descentwill assuming a localminimumactually exists arrive at somesolution for both W and W We must therefore be contentwith the assumed existence of some set of functions definedby (39) Again let us consider any point 119909 on the decisionboundary such that

(119890119894minus 119890119895)119879

120572120590 (119882119909 minus 120591) = 0 (41)

Such a point must also simultaneously satisfy

(119890119894minus 119890119895)119879

120590 (119909 minus 120591)

= ((119886119877119890119894minus 119905119894) minus (119886119877119890

119895minus 119905119895))119879

119866120572(119882 120591 120572)

8 Applied Computational Intelligence and Soft Computing

times 120590 (119866119882(119882 120591 120572) 119909 minus 119866

120591(119882 120591 120572))

= 119886((119890119894minus 119890119895) minus (119905

119894minus 119905119895))119879

119877119879119866120572(119882 120591 120572)

times 120590 (119866119882(119882 120591 120572) 119909 minus 119866

120591(119882 120591 120572))

= 0

(42)

At first glance a choice such as 119866120572(119882 120591 120572) = 119886119877120572 and

119905119894= 119905119895for 119894 119895 = 1 119872 (as in Proposition 4) appears

to bring us close to a solution However the term involving120590(119866119882(119882 120591 120572)119909 minus 119866

120591(119882 120591 120572)) is problematic Although a

choice such as 119866119882(119882 120591 120572) = 119882 and 119866

120591(119882 120591 120572) = 120591 would

yield the solution to (42) it should be clear that these valueswould not satisfy (40)

Another way to analyze this problem is to first set 119905119894= 119905119895

for 119894 119895 = 1 119872 Then for any 119909 on the decision boundaryfrom (41) and (42) equate the terms

120572120590 (119882119909 minus 120591) = 119896119877119879119866120572(119882 120591 120572)

times 120590 (119866119882(119882 120591 120572) 119909 minus 119866

120591(119882 120591 120572))

(43)

where 119896 is some constant and 119866120572 119866120591 and 119866

119882satisfy (40)

Given an arbitrary training set defined by (1) it is highlyunlikely that this constraint can be satisfied One remotescenario might be that the terms 119866

119882(119882 120591 120572)119909 minus 119866

120591(119882 120591 120572)

and119882119909 minus 120591 are always small In this case given a sigmoidalfunction 120590(119911) that is linear near 119911 = 0 a linearized versionof (43) could be solved using techniques described in [32]However this again is an unlikely set of events given anarbitrary training set Therefore given the transformation of(2) we are left to conclude that class membership invarianceand hence ranking invariance are in general not achievableusing the backpropagation neural network

52 Discussion Intuitively given a reasonable target encod-ing scheme one would desire that properties related to pro-tein secondary structure would be independent of the targetvectors chosen However we have presented numerical dataand a theoretical foundation demonstrating that secondarystructure classification and confidence measures can varydepending on the type of neural network architecture andtarget vector encoding scheme employed Specifically linearnetwork classification has been demonstrated to remaininvariant under a change in the target structure encodingscheme while the backpropagation network has not As119873 increases for the methodology applied in this workrecognition rates remain consistent with those reported inthe literature however we have observed that adding moretraining data does not improve the invariance of classificationmeasures for the backpropagation network This conclusionis corroborated by the analytic formulation presented above

6 Conclusions

As pointed out in the introduction one major purpose of theneural network is to create a stable and reliable model that

maps input training data to an output classification with thehope of extracting informative parameters When methodssimilar to those in the literature are applied [7 12ndash14] wehave demonstrated that classifier performance measures canvary considerably Under these circumstances parametersderived from a trained network for analytically describingprotein secondary structure may not comprise a reliableset for the model-based approach Furthermore classifiervariabilitywould imply that a stable parametricmodel has notbeen derived It is in some sense paradoxical that the neuralnetwork has been applied for structure classification and yetassociated parameters have not been applied for describingprotein secondary structure The neural network approachto deriving a solution to the protein secondary structureprediction problem therefore requires deeper exploration

Acknowledgments

This publication was made possible by Grant NumberG12RR017581 from the National Center for ResearchResources (NCRR) a component of the National Institutesof Health (NIH) The authors would also like to thank thereviewers for their helpful comments

References

[1] C M Bishop Pattern Recognition and Machine LearningSpringer 2007

[2] M T Hagan H B Demuth and M H Beale Neural NetworkDesign PWS Publishing 1996

[3] M N Nguyen and J C Rajapakse ldquoMulti-class support vectormachines for protein secondary structure predictionrdquo GenomeInformatics vol 14 pp 218ndash227 2003

[4] H J Hu Y Pan R Harrison and P C Tai ldquoImprovedprotein secondary structure prediction using support vectormachine with a new encoding scheme and an advanced tertiaryclassifierrdquo IEEE Transactions onNanobioscience vol 3 no 4 pp265ndash271 2004

[5] W Zhong G Altun X Tian R Harrison P C Tai and YPan ldquoParallel protein secondary structure prediction schemesusing Pthread and OpenMP over hyper-threading technologyrdquoJournal of Supercomputing vol 41 no 1 pp 1ndash16 2007

[6] J A Cuff and G J Barton ldquoApplication of enhanced multiplesequence alignment profiles to improve protein secondarystructure predictionrdquo Proteins vol 40 pp 502ndash511 2000

[7] J M Chandonia and M Karplus ldquoNeural networks for sec-ondary structure and structural class predictionsrdquo ProteinScience vol 4 no 2 pp 275ndash285 1995

[8] J A Cuff and G J Barton ldquoEvaluation and improvement ofmultiple sequence methods for protein secondary structurepredictionrdquo Proteins vol 34 pp 508ndash519 1999

[9] G E Crooks and S E Brenner ldquoProtein secondary structureentropy correlations andpredictionrdquoBioinformatics vol 20 no10 pp 1603ndash1611 2004

[10] L H Wang J Liu and H B Zhou ldquoA comparison of twomachine learning methods for protein secondary structurepredictionrdquo in Proceedings of 2004 International Conference onMachine Learning and Cybernetics pp 2730ndash2735 chn August2004

Applied Computational Intelligence and Soft Computing 9

[11] G Z Zhang D S Huang Y P Zhu and Y X Li ldquoImprovingprotein secondary structure prediction by using the residueconformational classesrdquo Pattern Recognition Letters vol 26 no15 pp 2346ndash2352 2005

[12] NQian and T J Sejnowski ldquoPredicting the secondary structureof globular proteins using neural network modelsrdquo Journal ofMolecular Biology vol 202 no 4 pp 865ndash884 1988

[13] L HowardHolley andMKarplus ldquoProtein secondary structureprediction with a neural networkrdquo Proceedings of the NationalAcademy of Sciences of the United States of America vol 86 no1 pp 152ndash156 1989

[14] J M Chandonia and M Karplus ldquoThe importance of largerdata sets for protein secondary structure prediction with neuralnetworksrdquo Protein Science vol 5 no 4 pp 768ndash774 1996

[15] B Rost and C Sander ldquoImproved prediction of proteinsecondary structure by use of sequence profiles and neuralnetworksrdquoProceedings of theNational Academy of Sciences of theUnited States of America vol 90 no 16 pp 7558ndash7562 1993

[16] B Rost and C Sander ldquoPrediction of protein secondary struc-ture at better than 70 accuracyrdquo Journal of Molecular Biologyvol 232 no 2 pp 584ndash599 1993

[17] J A Cuff M E Clamp A S Siddiqui M Finlay and GJ Barton ldquoJPred a consensus secondary structure predictionserverrdquo Bioinformatics vol 14 no 10 pp 892ndash893 1998

[18] S Hua and Z Sun ldquoA novel method of protein secondary struc-ture prediction with high segment overlap measure supportvector machine approachrdquo Journal of Molecular Biology vol308 no 2 pp 397ndash407 2001

[19] B Rost ldquoPHD predicting one-dimensional protein structureby profile-based neural networksrdquoMethods in Enzymology vol266 pp 525ndash539 1996

[20] B Rost G Yachdav and J Liu ldquoThe PredictProtein serverrdquoNucleic Acids Research vol 32 pp W321ndashW326 2004

[21] C Cole J D Barber and G J Barton ldquoThe Jpred 3 secondarystructure prediction serverrdquo Nucleic Acids Research vol 36 ppW197ndashW201 2008

[22] G Pollastri D Przybylski B Rost and P Baldi ldquoImprovingthe prediction of protein secondary structure in three and eightclasses using recurrent neural networks and profilesrdquo Proteinsvol 47 no 2 pp 228ndash235 2002

[23] J Cheng A Z Randall M J Sweredoski and P BaldildquoSCRATCH a protein structure and structural feature predic-tion serverrdquoNucleic Acids Research vol 33 no 2 ppW72ndashW762005

[24] D T Jones ldquoProtein secondary structure prediction basedon position-specific scoring matricesrdquo Journal of MolecularBiology vol 292 no 2 pp 195ndash202 1999

[25] K Bryson L J McGuffin R L Marsden J J Ward J SSodhi and D T Jones ldquoProtein structure prediction servers atUniversity College Londonrdquo Nucleic Acids Research vol 33 no2 pp W36ndashW38 2005

[26] G Cybenko ldquoApproximation by superpositions of a sigmoidalfunctionrdquo Mathematics of Control Signals and Systems vol 2no 4 pp 303ndash314 1989

[27] J Moody and C J Darken ldquoFast learning in networks of locallytuned processing unitsrdquo Neural Computation vol 1 pp 281ndash294 1989

[28] T Poggio and F Girosi ldquoNetworks for approximation andlearningrdquo Proceedings of the IEEE vol 78 no 9 pp 1481ndash14971990

[29] D F Specht ldquoProbabilistic neural networksrdquo Neural Networksvol 3 no 1 pp 109ndash118 1990

[30] P Andras ldquoOrthogonal RBF neural network approximationrdquoNeural Processing Letters vol 9 no 2 pp 141ndash151 1999

[31] P Baldi and S Brunak Bioinformatics The Machine LearningApproach MIT Press 1998

[32] E Sakk D J Schneider C R Myers and SW Cartinhour ldquoOnthe selection of target vectors for a class of supervised patternrecognizersrdquo IEEE Transactions on Neural Networks vol 20 no5 pp 745ndash757 2009

[33] G H Golub and C F Van Loan Matrix Computations JohnsHopkins University Press 1989

Submit your manuscripts athttpwwwhindawicom

Computer Games Technology

International Journal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Distributed Sensor Networks

International Journal of

Advances in

FuzzySystems

Hindawi Publishing Corporationhttpwwwhindawicom

Volume 2014

International Journal of

ReconfigurableComputing

Hindawi Publishing Corporation httpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Applied Computational Intelligence and Soft Computing

thinspAdvancesthinspinthinsp

Artificial Intelligence

HindawithinspPublishingthinspCorporationhttpwwwhindawicom Volumethinsp2014

Advances inSoftware EngineeringHindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Electrical and Computer Engineering

Journal of

Journal of

Computer Networks and Communications

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporation

httpwwwhindawicom Volume 2014

Advances in

Multimedia

International Journal of

Biomedical Imaging

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

ArtificialNeural Systems

Advances in

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

RoboticsJournal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Computational Intelligence and Neuroscience

Industrial EngineeringJournal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Modelling amp Simulation in EngineeringHindawi Publishing Corporation httpwwwhindawicom Volume 2014

The Scientific World JournalHindawi Publishing Corporation httpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Human-ComputerInteraction

Advances in

Computer EngineeringAdvances in

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

2 Applied Computational Intelligence and Soft Computing

training sequences we demonstrate that for the backprop-agation neural network architecture classification resultsand associated confidence measures can vary when twoequally valid encoding schemes are employed to numericallyrepresent the three structure classes (ie the ldquotarget encodingschemerdquo) Such a result goes against the intuition that thephysical nature of the secondary structure property shouldbe independent of the target encoding scheme chosen

The contribution of this work is not to demonstrateimprovements over existing techniques The hybrid tech-niques outlined above have been demonstrated to outperformneural networks when used alone Instead we focus ourattention on the ability of the neural network model-basedapproach to accurately characterize fundamental attributesof protein secondary structure given that certain modelspresented within this work are demonstrated to yield variableresults Specifically in this work we present

(1) numerical results demonstrating how secondarystructure classification results can vary as function ofclassifier architecture and parameter choices

(2) an analytic formulation in order to explain underwhat circumstances classification variability can arise

(3) an outline of specific challenges associated withthe neural network model-based approach outlinedabove

The conclusions reported here are relevant because theybring into discussion a body of literature that has purportedto offer a viable path to the solution of the secondary struc-ture prediction problem Section 3 describes the methodsapplied in this work examine the total number that retainedtheir classification using In particular this section providesdetails concerning the encoding of the protein sequencedata (Section 31) the encoding of the structure classes(Section 32) as well as the neural network architectures(Sections 33-34) and the classifier performance measures(Section 35) applied in this work Section 4 then presentsresults from numerical secondary structure classificationexperiments Section 5 presents an analytic formulationfor the linear network and the backpropagation networkdescribed in Section 3 in order to explain the numericalresults given in Section 4

2 Notation for the SupervisedClassification Problem

In the supervised classification problem [1 2] it is assumedthat a training set consists of119873 training pairs

(1198831 1198841) (1198832 1198842) (119883

119873 119884119873) (1)

where 119883119894isin 119877119899 are 119899-dimensional input column vectors and

119884119894isin 119877119901 are 119901-dimensional output column vectors The goal

of the supervised classifier approach is to ensure that thedesired response to a given input vector 119883

119894of dimension 119899

from the training set is the 119901-dimensional output vector 119884119894

Furthermore when the training data can be partitioned into

119872 distinct classes a set E equiv 1198901 119890

119895 119890

119872 of target 119901-

dimensional column vectors 119890119895isin 119877119901 are chosen to encode

(ie mathematically represent) each class for 119895 = 1 119872Under these circumstances each output training vector 119884

119894

is derived from the set E Based upon this discussion wesummarize the use of following symbols

(i) 119899 is the dimension of a classifier input vector(ii) 119901 is the dimension of a classifier output vector(iii) 119872 is the number of discrete classes for the classifica-

tion problem(iv) 119873 is the number of training pairs for the supervised

classification problem

3 Methods

In order to apply the neural network paradigm two numeri-cal issues must be addressed First since the input data comesin the form of an amino acid sequence Section 31 discusses asimple encoding scheme for converting the amino acid alpha-bet into a usable numerical form Second for this work oursecondary structure target alphabet consists of elements fromthe set 119864119867 119862 = Beta Sheet AlphaHelix Coil Hencean encoding schememust also be chosen for representing theneural network classifier output Section 32 discusses twoapproaches to encoding the output in fine detail because itis critical to the main point of this paper Specifically wechoose two different target vector encoding schemes thatcan be related by a simple mathematical relationship Suchan approach will allow us to compare classifier performancemeasures based upon the target vector encoding in addi-tion it will facilitate the analytic formulation presented inSection 5 Finally Sections 33ndash35 review the neural networkarchitectures and the specific classifier performancemeasuresemployed in this work Section 6 then concludes with somefinal observations regarding the neural network model-based approach to the protein secondary structure predictionproblem

31 Encoding of Protein Sequence Input Data For the numer-ical experiments the training set was constructed usingone hundred protein sequences randomly chosen from theCB513 database [6] available through the JPRED secondarystructure prediction engine [21] Furthermore we employa moving window of length 17 to each protein sequencewhere in order to avoid protein terminal effects the firstand last 50 amino acids are omitted from the analysis Thesecondary structure classification of the central residue isthen assigned to each window of 17 amino acids For theone hundred sequences analyzed a total of 12000 windowsof length 17 were extracted The window size value of 17was chosen based upon the assumption that the eight closestneighboring residues will have the greatest influence on thesecondary structure conformation of the central residueThisassumption is consistent with similar approaches reported inthe literature [7 12ndash14]

To encode the input amino acid sequences of length 17 weemploy sparse orthogonal encoding [31] whichmaps symbols

Applied Computational Intelligence and Soft Computing 3

from a given sequence alphabet onto a set of orthogonalvectors Specifically for an alphabet containing 119870 symbols aunique119870-dimensional unit vector is assigned to each symbolfurthermore the 119896th unit vector is one at the 119896th position andis zero at all other positions Hence if all training sequencesand unknown test sequences are of uniform length 119871 anencoded input vector will be of dimension 119899 where 119899 = 119871119870In our case 119870 = 20 and 119871 = 17 hence the dimension of anygiven input vector is 119899 = 340

The above input vector encoding technique is commonlyapplied in the bioinformatics and secondary structure pre-diction literature [7 15] While many different and superiorapproaches to this phase of the machine learning problemhave been suggested [3ndash5] we have chosen orthogonalencoding because of its simplicity and the fact that the resultsof this work do not depend upon the input encoding schemeInstead our work specifically focuses on potential neuralnetwork classifier variabilities induced by choice of the targetvector encoding scheme

32 Target Vector Encoding Analytically characterizing theinvariance of classifier performancemeasures clearly involvesfirst establishing a relationship between different sets of targetvectors E and E As a means of making the invarianceformulation presented in this paper more tractable weassume that two alternative sets of target vectors can berelated via an affine transformation involving a translation119879 a rigid rotation 119877 where 119877 is an orthogonal matrix and ascale factor 119886

Γ equiv 119886119877 (Γ minus 119879) (2)

where

Γ equiv [11989011198902 119890119872] Γ equiv [119890

11198902 119890119872]

119879 equiv [11990511199052 119905119872]

(3)

is a matrix of translation column vectors applied to eachtarget vector Many target vector choices regularly applied inthe literature can be related via the transformation in (2) Forinstance two equally valid and commonly applied encodingschemes for the three class problem are orthogonal encoding[31] where

119867 997888rarr [

[

1

0

0

]

]

119864 997888rarr [

[

0

1

0

]

]

119862 997888rarr [

[

0

0

1

]

]

(4)

and

119867 997888rarr

[[[[[[

[

minus1

2

radic3

2

0

]]]]]]

]

119864 997888rarr

[[[[[[

[

minus1

2

minusradic3

2

0

]]]]]]

]

119862 997888rarr [

[

1

0

0

]

]

(5)

where class encodings are chosen on the vertices of a trianglein a two-dimensional plane [14] It turns out that (4) and (5)can be phrased in terms of (2) [32] hence the numerical

results presented in this work will apply this set of encodingsMore precisely the secondary structure classification associ-ated with a given input vector is encoded using (4) and (5)(hence 119901 = 3) The set of target vectorsE is derived from (4)and the set of target vectors E is derived from (5) Both thelinear and the backpropagation networks are tested first bytraining using E and then comparing classifier performancewith their counterparts trained using E In all numericalexperiments MATLAB has been used for simulating andtesting these networks

33 The Linear Network When the supervised classifiermodel in (1) assumes an affine relationship between theinput and output data sets (as in the case of multiple linearregression) matrices of the form

119883 equiv [1198831 119883

119873] (6)

119884 equiv [1198841 119884

119873] (7)

are generally introduced Specifically the linear networkseeks to determine a 119901 times 119899 matrix 120573 of coefficients and aconstant 119901-dimensional column vector 119888 such that the 119894thoutput vector 119884

119894in the training set can be approximated by

120573119883119894+ 119888 (8)

Given this model we can form a weight matrix of unknowncoefficients

119882 equiv [120573 119888] (9)that ideally will map each input training vector into thecorresponding output training vector If the bottom rowof theinput data matrix 119883 is appended with a row of ones leadingto the (119899 + 1) times 119873matrix

119860 equiv [1198831 119883

119873

1 1] (10)

in matrix form the goal is then to find a 119901 times 119899 + 1 weightmatrix119882 that minimizes the sum squared-error

119864linear

equiv

119873

sum119894=1

(119882119860119894minus 119884119894)119879

(119882119860119894minus 119884119894) (11)

over the set of data pairs

119860119894equiv [119883119894

1] (12)

by satisfying the first derivative condition 120597119864linear120597119882 = 0The least squares solution to this problem is found via thepseudoinverse 119860dagger [33] where

119882 = 119884119860dagger (13)

Once the optimal set of weights has been computed thenetwork response to an unknown 119899 times 1 input vector 119909 can bedetermined by defining the (119899 + 1) times 1 vector

119911 equiv [119909 1]119879 (14)

and calculating119891 (119911) = 119882119911 (15)

where 119891(119911) is a 119901 times 1 column vector

4 Applied Computational Intelligence and Soft Computing

34The Backpropagation Network Given an input vector 119909 isin119877119899 the model 119891 119877119899 rarr 119877119901 for a backpropagation neuralnetwork with a single hidden layer consisting of 119898 nodes isdescribed as

119891 (119909) =

[[[[[

[

1198911(119909)

1198912(119909)

119891119901(119909)

]]]]]

]

= 120572120590 (119882119909 minus 120591) (16)

where 120572 isin 119877119901times119898 119882 isin 119877119898times119899 and 120591 isin 119877119898 define the setof network weights W = 119882 120591 120572 and 120590 119877119898 rarr 119877119901 isa ldquosigmoidalrdquo function that is bounded and monotonicallyincreasing To perform supervised training in a mannersimilar to the linear networkW is determined byminimizingthe objective function

119864119887119901=

119873

sum119894=1

(119891 (119883119894) minus 119884119894)119879

(119891 (119883119894) minus 119884119894) (17)

given the training data defined in (1) Since 119891(119883119894) is no

longer linear numerical techniques such as the gradientdescent algorithm and variations thereof are relied upon tocompute the setW that satisfies the first derivative condition120597119864119887119901120597W = 0Consider the following definitions

120575119865119896equiv 119891 (119883

119896) minus 119884119896=[[

[

1198911(119883119896) minus (119884

119896)1

1198912(119883119896) minus (119884

119896)2

119891119901(119883119896) minus (119884

119896)119901

]]

]

119878119896equiv

[[[[[[[[[[[[[[

[

120590(

119899

sum119895=1

1198821119895(119883119896)119895minus 1205911)

120590(

119899

sum119895=1

1198822119895(119883119896)119895minus 1205912)

120590(

119899

sum119895=1

119882119898119895(119883119896)119895minus 120591119898)

]]]]]]]]]]]]]]

]

1198781015840

119896equiv

[[[[[[[[[[[[[[

[

1205901015840(

119899

sum119895=1

1198821119895(119883119896)119895minus 1205911)

1205901015840(

119899

sum119895=1

1198822119895(119883119896)119895minus 1205912)

1205901015840(

119899

sum119895=1

119882119898119895(119883119896)119895minus 120591119898)

]]]]]]]]]]]]]]

]

(18)

where 1205901015840(119911) = 119889120590119889119911 The first derivative conditions forthe network weights prescribed by (16) and (17) can then bewritten in matrix form as follows

120597119864119887119901

120597120572=

119873

sum119896=1

120575119865119896119878119879

119896= 0 (19)

(where ldquo119879rdquo denotes the matrix transpose)

120597119864119887119901

120597120591= minus

119873

sum119896=1

diag (1198781015840119896) 120572119879120575119865119879

119896= 0

120597119864119887119901

120597119882=

119873

sum119896=1

diag (1198781015840119896) 120572119879120575119865119879

119896119883119879

119896= 0

(20)

where diag(1198781015840119896) is a square diagonal matrix such that the

diagonal entries consist of components from the vector 1198781015840119896

35 ClassificationMeasures After a given classifier is trainedwhen presented with an input vector 119909 isin 119877119899 of unknownclassification it will respond with an output 119891(119909) isin 119877119901 Theassociated class membership 119862(119909) is then often determinedby applying a minimum distance criterion

119862 (119909) equiv argmin119894

119889 (119891 (119909) 119890119894) (21)

where the target vector 119890119894that is closest to 119891(119909) implies the

class Furthermore when characterizing the performance of apattern classifier one often presents a set 119878 of test vectors andanalyzes the associated output In addition to determiningthe class membership it is also possible to rank the distancebetween a specific target vector and the classifier responseto 119878 In this case a similar distance criterion can be appliedin order to rank an input vector 119909 isin 119878 with respect to class119869 isin 1 119872

120588119869(119909) = 119889 (119891 (119909) 119890

119869) (22)

For the purposes of this work (15) facilitates the determina-tion of the classmembership and rankingwith respect to class119869 for the linear network Similarly assuming a set of weightsW = 119882 120591 120572 for a trained backpropagation network (21)and (22) would be applied using (16)

It is well established that in the case of a normallydistributed data the classification measures presented aboveminimize the probability of a classification error and aredirectly related to the statistical significance of a classificationdecision [1] Given the neural network as the supervisedclassification technique and two distinct choices for the set oftarget vectorsE and E we demonstrate in certain instancesthat the classification and ranking results do not remaininvariant such that 119862(119909) = 119862(119909) and 120588

119869(119909) = 120588

119869(119909) for any

input vector 119909

4 Noninvariance of SecondaryStructure Predictions

In this section we numerically demonstrate that when differ-ent target vector encodings are applied neural network clas-sifier measures outlined above in certain cases are observedto vary widely For each neural network architecture underconsideration an analytic formulation is then presented inSection 5 in order to explain the observed numerical data

As mentioned in Section 32 numerical experiments areperformed first by training using E and then comparing

Applied Computational Intelligence and Soft Computing 5

Table 1 Ranking results for the linear network where 119894 representsthe test vector index and 120588

1(119909) and 120588

1(119909) represent the distance with

respect to the helix class vectors 1198901and 1198901 Out of 1800 vectors tested

vectors referred to in this table were ranked from 1 to 20

119894E 1205881(119909) 119894E 120588

1(119909)

1205 00780 1205 0117042 00867 42 013001031 00976 1031 014641773 01113 1773 01670598 01238 598 018571761 01267 1761 01900862 01354 862 020311073 01409 1073 02114277 01459 277 02188115 01540 115 023091505 01821 1505 02731392 01839 392 027591421 01904 1421 02856147 02001 147 03001990 02044 990 030661457 02127 1457 031911288 02150 1288 03225352 02160 352 032391232 02198 1232 03297280 02311 280 03466

classifier performance with their counterparts trained usingE Multiple cross-validation trials are required in order toprevent potential dependency of the evaluated accuracy onthe particular training or test sets chosen [7 15] In this workwe apply a hold-n-out strategy similar to that of [14] using85 of the 12000 encoded sequences as training data (ie119873 = 10200) and 15 as test data to validate the classificationresults Recognition rates for both the linear and backpropa-gation rates using either set of target vector encodings wereapproximately 65 which is typical of this genre of classifiersthat have applied similar encoding methodologies [7 12ndash14] Although these aggregate values remain consistent using(21) and (22) we now present data demonstrating that whileclass membership and ranking remain invariant for the linearnetwork these measures of performance vary considerablyfor the backpropagation network which was trained with119898 = 17 seventeen hidden nodes and amean squared trainingerror less than 02 Ranking results from a representative testfor the linear and backpropagation networks are presented forthe top 20 ranked vectors in Tables 1 and 3 Classmembershipdata are presented in Tables 2 and 4 Observe that forthe linear network indices for the top 20 ranked vectorsremain invariant indicating ranking invariance in additionno change in class membership is observed On the otherhand Tables 3 and 4 clearly indicate a lack of consistencywhen considering the ranking and class membership of testvectors A particularly troubling observation is that very fewvectors ranked in the top 20 with respect to 119890

1were ranked in

the top 20 with respect to 1198901 Furthermore Table 4 indicates

Table 2 Class membership results for the linear network For eachclass the total number of vectors classified using E is analyzed toexamine the total number that retained their classification using E

Class E E change119867 202 202 0119864 621 621 0119862 977 977 0

that the class membership of a substantial number of testvectors changed when an alternative set of target vectors wasemployed The data also indicates that the greatest changein class membership took place for alpha helical sequencesthus implying that there is substantial disagreement overthe modeling of this secondary structure element by thebackpropagation network due to a simple transformation ofthe target vectors

5 Analysis

The results in Section 4 clearly show that while the patternrecognition results for the linear network remain invariantunder a change in target vectors those for the backpropa-gation network do not In this section we present analyticresults in order to clearly explain and understand why thesetwo techniques lead to different conclusions

51 Invariance Formulation Let us begin by considering twodefinitions

Definition 1 Given two sets of target vectors E and E theclass membership is invariant under a transformation of thetarget vectors if for any input vector 119909

119889(119891 (119909) 119890119894) gt 119889(119891 (119909) 119890

119895)997904rArr119889 (119891 (119909) 119890

119894) gt 119889(119891 (119909) 119890

119895)

(23)

where 119891(119909) is the output of the classifier with target vectors119890119895119872

119895=1

Definition 2 Given two sets of target vectors E and E theranking with respect to a specific class 119869 is invariant under atransformation of the target vectors if for any input vectors1199091and 119909

2

119889(119891(1199091) 119890119869) gt 119889(119891(119909

2) 119890119869)997904rArr119889(119891 (119909

1) 119890119869) gt 119889(119891 (119909

2) 119890119869)

(24)

Based upon these definitions the following has beenestablished [32]

Proposition 3 Given two sets of target vectorsE and E if theranking is invariant then the class membership of an arbitraryinput vector 119909 will remain invariant

In the analysis presented the strategy for characterizingneural network performance depends upon the data from theprevious section For the linear network since both ranking

6 Applied Computational Intelligence and Soft Computing

Table 3 Ranking results for the backpropagation network where 119894represents the test vector index and 120588

1(119909) and 120588

1(119909) represent the

distance with respect to the helix class vectors 1198901and 1198901 Out of 1800

vectors tested vectors referred to in this table were ranked from 1 to20

119894E 1205881(119909) 119894E 120588

1(119909)

817 00107 926 00101887 00231 1604 00130264 00405 887 002091183 00711 1145 00214684 00727 461 00232623 00874 583 00329911 00891 1086 003391382 00917 1382 004781610 00939 413 00489551 01060 225 006081042 01150 438 00609924 01322 911 00613727 01339 207 00774438 01356 559 00885577 01363 481 00945896 01500 1548 00947175 01513 962 009681138 01549 85 01012583 01581 195 01111559 01655 9 01167

Table 4 Class membership results for the backpropagationnetwork For each class the total number of vectors classified usingE is analyzed to examine the total number that retained theirclassification using E

Class E E change119867 225 142 369119864 581 476 181119862 994 878 117

and classification were observed to remain invariant it ismore sensible to characterize the invariance of this networkusing Definition 2 Then based upon Proposition 3 classmembership invariance naturally follows On the other handto explain the noninvariance of both class membership andranking observed in the backpropagation network the anal-ysis is facilitated by considering Definition 1 The noninvari-ance of ranking then naturally follows from Proposition 3

511 Invariance Analysis for the Linear Network When thetarget vectors are subjected to the transformation defined in(2) the network output can be expressed as

119891 (119911) = 119911 = 119860dagger119911 = 119886119877 (119884 minus 119879

1015840)119860dagger119911 (25)

where 1198791015840 is derived from 119879 such that the translation vector119905119894associated with 119890

119894isin E is appropriately aligned with the

correct target vector in matrix 119884 In other words when theoutput data matrix in (7) is of the form

119884 = [1198901198941

1198901198942

119890119894119873

] (26)

then

1198791015840= [1199051198941

1199051198942

119905119894119873

] (27)

where 119894119895isin 1 119872 for 119895 = 1 119873 Given this network

the following result is applicable [32]

Proposition 4 If

(i) the number of training observations 119873 exceeds thevector dimension 119899

(ii) the rows of the matrix 119860 are linearly independent(iii) E and E are related according to (2)(iv) for some 119905 isin 119877119901 119905

119894= 119905 for all 119894 = 1 119872 in (2)

then the ranking and hence the classmembership for the linearnetwork will remain invariant

In other words if the columns of the matrix 1198791015840 in (25)are all equal then using (15) and (25) will result in (23)being satisfiedThe above result is applicable to the presentednumerical data with 119899 = 340 and119873 = 10200 hence rankingand class membership invariances are corroborated by thedata in Tables 1 and 2

512 Invariance Analysis for the Backpropagation NetworkIn this section we seek to characterize the noninvarianceobserved in class membership using the backpropagationnetwork If the class membership varies due to a changein target vectors then this variation should be quantifiableby characterizing the boundary separating two respectiveclasses The decision boundary between class 119894 and class 119895 isdefined by points 119909 such that

1003817100381710038171003817119891 (119909) minus 11989011989410038171003817100381710038172

=10038171003817100381710038171003817119891 (119909) minus 119890

119895

10038171003817100381710038171003817

2

(28)

where 119894 119895 isin 1 119872 119894 = 119895 and 119891(119909) is the classifier outputUnder these circumstances if an ℓ

2norm is applied in (21)

(119891 (119909) minus 119890119894)119879

(119891 (119909) minus 119890119894) = (119891 (119909) minus 119890

119895)119879

(119891 (119909) minus 119890119895)

(29)

The solution set to this equation consists of all 119909 such that119891(119909) is equidistant from 119890

119894and 119890119895where for the purposes of

this section119891(119909) is defined by (16) Expanding terms on bothsides of this equation leads to the condition

(119890119894minus 119890119895)119879

119891 (119909) = minus1

2(119890119879

119895119890119895minus 119890119879

119894119890119894) (30)

If the classmembership of a representative vector is to remaininvariant under a change of target vectors this same set ofpoints must also satisfy

( 119890119894minus 119890119895)119879

119891 (119909) = minus1

2( 119890119879

119895119890119895minus 119890119879

119894119890119894) (31)

Applied Computational Intelligence and Soft Computing 7

Assuming that two networks have been trained using twodifferent sets of target vectors E and E the set of weights119882 120591 120572 and 120591 determines the network output in (16)Without loss of generality we consider the case where alltarget vectors are normalized to a value of one such that1198901198942= 1 and 119890

1198942= 1 for 119894 = 1 119872 In this case the

conditions in (30) and (31) become

(119890119894minus 119890119895)119879

120572120590 (119882119909 minus 120591) = 0 (32)

(119890119894minus 119890119895)119879

120590 (119909 minus 120591) = 0 (33)

We first consider a special case where the target vectorsare related according to Γ = 119886119877Γ with 119879 = 0 in (2) Underthese circumstances if the choice

W equiv 120591 = 119882 120591 119886119877120572 (34)

is made it should be clear that since119877119879 = 119877minus1 and 119894= 119886119877119884

119894

119864119887119901=

119873

sum119894=1

(119891 (119883119894) minus 119894)119879

(119891 (119883119894) minus 119894)

= 1198862

119873

sum119894=1

(119877119891 (119883119894) minus 119877119884

119894)119879

(119877119891 (119883119894) minus 119877119884

119894)

= 1198862

119873

sum119894=1

(119891 (119883119894) minus 119884119894)119879

119877119879119877 (119891 (119883

119894) minus 119884119894)

= 1198862119864119887119901

(35)

is minimized by the condition 120597119864119887119901120597W = 0 Another way tosee this is to observe that (19) (20) remain invariant for thischoice of target vectors and network weights Hence we havethe following

Proposition 5 For a specific choice of 119886 and 119877 if 119879 = 0 in (2)and

W = 120591 = 119882 120591 119886119877120572 (36)

then the classmembership for the backpropagation networkwillremain invariant

Proof Simply consider (32) and (33) and choose any 119909satisfying

(119890119894minus 119890119895)119879

120572120590 (119882119909 minus 120591) = 0 (37)

It then immediately follows that

(119890119894minus 119890119895)119879

120590 (119909 minus 120591)

= (119890119894minus 119890119895)119879

119877119879119877120572120590 (119882119909 minus 120591)

= 1198862(119890119894minus 119890119895)119879

120572120590 (119882119909 minus 120591)

= 0

(38)

Therefore if 119909 satisfies (32) then it also satisfies (33) andhence is a point on the decision boundary for both net-works

Intuitively a scaled rigid rotation of the target vectorsshould not affect the decision boundary However when themore general transformation of (2) is applied with 119879 = 0we now demonstrate that due to the nonlinearity of 120590 in(16) no simple relationship exists such that (32) and (33)can simultaneously be satisfied by the same set of pointsWe first investigate the possibility of establishing an analyticrelationship between the set of weights 120591 and 119882 120591 120572for both networks In other words we seek ideally invertiblefunctions 119866

120572 119866120591 and 119866

119882

= 119866120572(119882 120591 120572)

120591 = 119866120591(119882 120591 120572)

= 119866119882(119882 120591 120572)

(39)

such that the setW can be transformed into W If this can bedone then an analytic procedure similar to that presented inthe proof of Proposition 5 can be established in order to relate(32) to (33) for the general case Since (19) (20) define the setW it is reasonable to rephrase these equations in terms of theobjective function 119864119887119901

120597119864119887119901

120597=

119873

sum119896=1

(119891 (119883119896) minus 119896) 119878119879

119896= 0

120597119864119887119901

120597120591= minus

119873

sum119896=1

diag (1198781015840119896) 119879(119891 (119883

119896) minus 119896)119879

= 0

120597119864119887119901

120597=

119873

sum119896=1

diag (1198781015840119896) 119879(119891 (119883

119896) minus 119896)119879

119883119879

119896= 0

(40)

where 119891(119883119896) = 120590(119883

119896minus 120591) and

119896= 119886119877(119884

119896minus 1199051015840) such that

1199051015840 is the translation vector associated with the target vectorreferred to by 119884

119896 From these equations it should be clear

that no simple analytic relationship exists that will transformW into W A numerical algorithm such as gradient descentwill assuming a localminimumactually exists arrive at somesolution for both W and W We must therefore be contentwith the assumed existence of some set of functions definedby (39) Again let us consider any point 119909 on the decisionboundary such that

(119890119894minus 119890119895)119879

120572120590 (119882119909 minus 120591) = 0 (41)

Such a point must also simultaneously satisfy

(119890119894minus 119890119895)119879

120590 (119909 minus 120591)

= ((119886119877119890119894minus 119905119894) minus (119886119877119890

119895minus 119905119895))119879

119866120572(119882 120591 120572)

8 Applied Computational Intelligence and Soft Computing

times 120590 (119866119882(119882 120591 120572) 119909 minus 119866

120591(119882 120591 120572))

= 119886((119890119894minus 119890119895) minus (119905

119894minus 119905119895))119879

119877119879119866120572(119882 120591 120572)

times 120590 (119866119882(119882 120591 120572) 119909 minus 119866

120591(119882 120591 120572))

= 0

(42)

At first glance a choice such as 119866120572(119882 120591 120572) = 119886119877120572 and

119905119894= 119905119895for 119894 119895 = 1 119872 (as in Proposition 4) appears

to bring us close to a solution However the term involving120590(119866119882(119882 120591 120572)119909 minus 119866

120591(119882 120591 120572)) is problematic Although a

choice such as 119866119882(119882 120591 120572) = 119882 and 119866

120591(119882 120591 120572) = 120591 would

yield the solution to (42) it should be clear that these valueswould not satisfy (40)

Another way to analyze this problem is to first set 119905119894= 119905119895

for 119894 119895 = 1 119872 Then for any 119909 on the decision boundaryfrom (41) and (42) equate the terms

120572120590 (119882119909 minus 120591) = 119896119877119879119866120572(119882 120591 120572)

times 120590 (119866119882(119882 120591 120572) 119909 minus 119866

120591(119882 120591 120572))

(43)

where 119896 is some constant and 119866120572 119866120591 and 119866

119882satisfy (40)

Given an arbitrary training set defined by (1) it is highlyunlikely that this constraint can be satisfied One remotescenario might be that the terms 119866

119882(119882 120591 120572)119909 minus 119866

120591(119882 120591 120572)

and119882119909 minus 120591 are always small In this case given a sigmoidalfunction 120590(119911) that is linear near 119911 = 0 a linearized versionof (43) could be solved using techniques described in [32]However this again is an unlikely set of events given anarbitrary training set Therefore given the transformation of(2) we are left to conclude that class membership invarianceand hence ranking invariance are in general not achievableusing the backpropagation neural network

52 Discussion Intuitively given a reasonable target encod-ing scheme one would desire that properties related to pro-tein secondary structure would be independent of the targetvectors chosen However we have presented numerical dataand a theoretical foundation demonstrating that secondarystructure classification and confidence measures can varydepending on the type of neural network architecture andtarget vector encoding scheme employed Specifically linearnetwork classification has been demonstrated to remaininvariant under a change in the target structure encodingscheme while the backpropagation network has not As119873 increases for the methodology applied in this workrecognition rates remain consistent with those reported inthe literature however we have observed that adding moretraining data does not improve the invariance of classificationmeasures for the backpropagation network This conclusionis corroborated by the analytic formulation presented above

6 Conclusions

As pointed out in the introduction one major purpose of theneural network is to create a stable and reliable model that

maps input training data to an output classification with thehope of extracting informative parameters When methodssimilar to those in the literature are applied [7 12ndash14] wehave demonstrated that classifier performance measures canvary considerably Under these circumstances parametersderived from a trained network for analytically describingprotein secondary structure may not comprise a reliableset for the model-based approach Furthermore classifiervariabilitywould imply that a stable parametricmodel has notbeen derived It is in some sense paradoxical that the neuralnetwork has been applied for structure classification and yetassociated parameters have not been applied for describingprotein secondary structure The neural network approachto deriving a solution to the protein secondary structureprediction problem therefore requires deeper exploration

Acknowledgments

This publication was made possible by Grant NumberG12RR017581 from the National Center for ResearchResources (NCRR) a component of the National Institutesof Health (NIH) The authors would also like to thank thereviewers for their helpful comments

References

[1] C M Bishop Pattern Recognition and Machine LearningSpringer 2007

[2] M T Hagan H B Demuth and M H Beale Neural NetworkDesign PWS Publishing 1996

[3] M N Nguyen and J C Rajapakse ldquoMulti-class support vectormachines for protein secondary structure predictionrdquo GenomeInformatics vol 14 pp 218ndash227 2003

[4] H J Hu Y Pan R Harrison and P C Tai ldquoImprovedprotein secondary structure prediction using support vectormachine with a new encoding scheme and an advanced tertiaryclassifierrdquo IEEE Transactions onNanobioscience vol 3 no 4 pp265ndash271 2004

[5] W Zhong G Altun X Tian R Harrison P C Tai and YPan ldquoParallel protein secondary structure prediction schemesusing Pthread and OpenMP over hyper-threading technologyrdquoJournal of Supercomputing vol 41 no 1 pp 1ndash16 2007

[6] J A Cuff and G J Barton ldquoApplication of enhanced multiplesequence alignment profiles to improve protein secondarystructure predictionrdquo Proteins vol 40 pp 502ndash511 2000

[7] J M Chandonia and M Karplus ldquoNeural networks for sec-ondary structure and structural class predictionsrdquo ProteinScience vol 4 no 2 pp 275ndash285 1995

[8] J A Cuff and G J Barton ldquoEvaluation and improvement ofmultiple sequence methods for protein secondary structurepredictionrdquo Proteins vol 34 pp 508ndash519 1999

[9] G E Crooks and S E Brenner ldquoProtein secondary structureentropy correlations andpredictionrdquoBioinformatics vol 20 no10 pp 1603ndash1611 2004

[10] L H Wang J Liu and H B Zhou ldquoA comparison of twomachine learning methods for protein secondary structurepredictionrdquo in Proceedings of 2004 International Conference onMachine Learning and Cybernetics pp 2730ndash2735 chn August2004

Applied Computational Intelligence and Soft Computing 9

[11] G Z Zhang D S Huang Y P Zhu and Y X Li ldquoImprovingprotein secondary structure prediction by using the residueconformational classesrdquo Pattern Recognition Letters vol 26 no15 pp 2346ndash2352 2005

[12] NQian and T J Sejnowski ldquoPredicting the secondary structureof globular proteins using neural network modelsrdquo Journal ofMolecular Biology vol 202 no 4 pp 865ndash884 1988

[13] L HowardHolley andMKarplus ldquoProtein secondary structureprediction with a neural networkrdquo Proceedings of the NationalAcademy of Sciences of the United States of America vol 86 no1 pp 152ndash156 1989

[14] J M Chandonia and M Karplus ldquoThe importance of largerdata sets for protein secondary structure prediction with neuralnetworksrdquo Protein Science vol 5 no 4 pp 768ndash774 1996

[15] B Rost and C Sander ldquoImproved prediction of proteinsecondary structure by use of sequence profiles and neuralnetworksrdquoProceedings of theNational Academy of Sciences of theUnited States of America vol 90 no 16 pp 7558ndash7562 1993

[16] B Rost and C Sander ldquoPrediction of protein secondary struc-ture at better than 70 accuracyrdquo Journal of Molecular Biologyvol 232 no 2 pp 584ndash599 1993

[17] J A Cuff M E Clamp A S Siddiqui M Finlay and GJ Barton ldquoJPred a consensus secondary structure predictionserverrdquo Bioinformatics vol 14 no 10 pp 892ndash893 1998

[18] S Hua and Z Sun ldquoA novel method of protein secondary struc-ture prediction with high segment overlap measure supportvector machine approachrdquo Journal of Molecular Biology vol308 no 2 pp 397ndash407 2001

[19] B Rost ldquoPHD predicting one-dimensional protein structureby profile-based neural networksrdquoMethods in Enzymology vol266 pp 525ndash539 1996

[20] B Rost G Yachdav and J Liu ldquoThe PredictProtein serverrdquoNucleic Acids Research vol 32 pp W321ndashW326 2004

[21] C Cole J D Barber and G J Barton ldquoThe Jpred 3 secondarystructure prediction serverrdquo Nucleic Acids Research vol 36 ppW197ndashW201 2008

[22] G Pollastri D Przybylski B Rost and P Baldi ldquoImprovingthe prediction of protein secondary structure in three and eightclasses using recurrent neural networks and profilesrdquo Proteinsvol 47 no 2 pp 228ndash235 2002

[23] J Cheng A Z Randall M J Sweredoski and P BaldildquoSCRATCH a protein structure and structural feature predic-tion serverrdquoNucleic Acids Research vol 33 no 2 ppW72ndashW762005

[24] D T Jones ldquoProtein secondary structure prediction basedon position-specific scoring matricesrdquo Journal of MolecularBiology vol 292 no 2 pp 195ndash202 1999

[25] K Bryson L J McGuffin R L Marsden J J Ward J SSodhi and D T Jones ldquoProtein structure prediction servers atUniversity College Londonrdquo Nucleic Acids Research vol 33 no2 pp W36ndashW38 2005

[26] G Cybenko ldquoApproximation by superpositions of a sigmoidalfunctionrdquo Mathematics of Control Signals and Systems vol 2no 4 pp 303ndash314 1989

[27] J Moody and C J Darken ldquoFast learning in networks of locallytuned processing unitsrdquo Neural Computation vol 1 pp 281ndash294 1989

[28] T Poggio and F Girosi ldquoNetworks for approximation andlearningrdquo Proceedings of the IEEE vol 78 no 9 pp 1481ndash14971990

[29] D F Specht ldquoProbabilistic neural networksrdquo Neural Networksvol 3 no 1 pp 109ndash118 1990

[30] P Andras ldquoOrthogonal RBF neural network approximationrdquoNeural Processing Letters vol 9 no 2 pp 141ndash151 1999

[31] P Baldi and S Brunak Bioinformatics The Machine LearningApproach MIT Press 1998

[32] E Sakk D J Schneider C R Myers and SW Cartinhour ldquoOnthe selection of target vectors for a class of supervised patternrecognizersrdquo IEEE Transactions on Neural Networks vol 20 no5 pp 745ndash757 2009

[33] G H Golub and C F Van Loan Matrix Computations JohnsHopkins University Press 1989

Submit your manuscripts athttpwwwhindawicom

Computer Games Technology

International Journal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Distributed Sensor Networks

International Journal of

Advances in

FuzzySystems

Hindawi Publishing Corporationhttpwwwhindawicom

Volume 2014

International Journal of

ReconfigurableComputing

Hindawi Publishing Corporation httpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Applied Computational Intelligence and Soft Computing

thinspAdvancesthinspinthinsp

Artificial Intelligence

HindawithinspPublishingthinspCorporationhttpwwwhindawicom Volumethinsp2014

Advances inSoftware EngineeringHindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Electrical and Computer Engineering

Journal of

Journal of

Computer Networks and Communications

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporation

httpwwwhindawicom Volume 2014

Advances in

Multimedia

International Journal of

Biomedical Imaging

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

ArtificialNeural Systems

Advances in

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

RoboticsJournal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Computational Intelligence and Neuroscience

Industrial EngineeringJournal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Modelling amp Simulation in EngineeringHindawi Publishing Corporation httpwwwhindawicom Volume 2014

The Scientific World JournalHindawi Publishing Corporation httpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Human-ComputerInteraction

Advances in

Computer EngineeringAdvances in

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Applied Computational Intelligence and Soft Computing 3

from a given sequence alphabet onto a set of orthogonalvectors Specifically for an alphabet containing 119870 symbols aunique119870-dimensional unit vector is assigned to each symbolfurthermore the 119896th unit vector is one at the 119896th position andis zero at all other positions Hence if all training sequencesand unknown test sequences are of uniform length 119871 anencoded input vector will be of dimension 119899 where 119899 = 119871119870In our case 119870 = 20 and 119871 = 17 hence the dimension of anygiven input vector is 119899 = 340

The above input vector encoding technique is commonlyapplied in the bioinformatics and secondary structure pre-diction literature [7 15] While many different and superiorapproaches to this phase of the machine learning problemhave been suggested [3ndash5] we have chosen orthogonalencoding because of its simplicity and the fact that the resultsof this work do not depend upon the input encoding schemeInstead our work specifically focuses on potential neuralnetwork classifier variabilities induced by choice of the targetvector encoding scheme

32 Target Vector Encoding Analytically characterizing theinvariance of classifier performancemeasures clearly involvesfirst establishing a relationship between different sets of targetvectors E and E As a means of making the invarianceformulation presented in this paper more tractable weassume that two alternative sets of target vectors can berelated via an affine transformation involving a translation119879 a rigid rotation 119877 where 119877 is an orthogonal matrix and ascale factor 119886

Γ equiv 119886119877 (Γ minus 119879) (2)

where

Γ equiv [11989011198902 119890119872] Γ equiv [119890

11198902 119890119872]

119879 equiv [11990511199052 119905119872]

(3)

is a matrix of translation column vectors applied to eachtarget vector Many target vector choices regularly applied inthe literature can be related via the transformation in (2) Forinstance two equally valid and commonly applied encodingschemes for the three class problem are orthogonal encoding[31] where

119867 997888rarr [

[

1

0

0

]

]

119864 997888rarr [

[

0

1

0

]

]

119862 997888rarr [

[

0

0

1

]

]

(4)

and

119867 997888rarr

[[[[[[

[

minus1

2

radic3

2

0

]]]]]]

]

119864 997888rarr

[[[[[[

[

minus1

2

minusradic3

2

0

]]]]]]

]

119862 997888rarr [

[

1

0

0

]

]

(5)

where class encodings are chosen on the vertices of a trianglein a two-dimensional plane [14] It turns out that (4) and (5)can be phrased in terms of (2) [32] hence the numerical

results presented in this work will apply this set of encodingsMore precisely the secondary structure classification associ-ated with a given input vector is encoded using (4) and (5)(hence 119901 = 3) The set of target vectorsE is derived from (4)and the set of target vectors E is derived from (5) Both thelinear and the backpropagation networks are tested first bytraining using E and then comparing classifier performancewith their counterparts trained using E In all numericalexperiments MATLAB has been used for simulating andtesting these networks

33 The Linear Network When the supervised classifiermodel in (1) assumes an affine relationship between theinput and output data sets (as in the case of multiple linearregression) matrices of the form

119883 equiv [1198831 119883

119873] (6)

119884 equiv [1198841 119884

119873] (7)

are generally introduced Specifically the linear networkseeks to determine a 119901 times 119899 matrix 120573 of coefficients and aconstant 119901-dimensional column vector 119888 such that the 119894thoutput vector 119884

119894in the training set can be approximated by

120573119883119894+ 119888 (8)

Given this model we can form a weight matrix of unknowncoefficients

119882 equiv [120573 119888] (9)that ideally will map each input training vector into thecorresponding output training vector If the bottom rowof theinput data matrix 119883 is appended with a row of ones leadingto the (119899 + 1) times 119873matrix

119860 equiv [1198831 119883

119873

1 1] (10)

in matrix form the goal is then to find a 119901 times 119899 + 1 weightmatrix119882 that minimizes the sum squared-error

119864linear

equiv

119873

sum119894=1

(119882119860119894minus 119884119894)119879

(119882119860119894minus 119884119894) (11)

over the set of data pairs

119860119894equiv [119883119894

1] (12)

by satisfying the first derivative condition 120597119864linear120597119882 = 0The least squares solution to this problem is found via thepseudoinverse 119860dagger [33] where

119882 = 119884119860dagger (13)

Once the optimal set of weights has been computed thenetwork response to an unknown 119899 times 1 input vector 119909 can bedetermined by defining the (119899 + 1) times 1 vector

119911 equiv [119909 1]119879 (14)

and calculating119891 (119911) = 119882119911 (15)

where 119891(119911) is a 119901 times 1 column vector

4 Applied Computational Intelligence and Soft Computing

34The Backpropagation Network Given an input vector 119909 isin119877119899 the model 119891 119877119899 rarr 119877119901 for a backpropagation neuralnetwork with a single hidden layer consisting of 119898 nodes isdescribed as

119891 (119909) =

[[[[[

[

1198911(119909)

1198912(119909)

119891119901(119909)

]]]]]

]

= 120572120590 (119882119909 minus 120591) (16)

where 120572 isin 119877119901times119898 119882 isin 119877119898times119899 and 120591 isin 119877119898 define the setof network weights W = 119882 120591 120572 and 120590 119877119898 rarr 119877119901 isa ldquosigmoidalrdquo function that is bounded and monotonicallyincreasing To perform supervised training in a mannersimilar to the linear networkW is determined byminimizingthe objective function

119864119887119901=

119873

sum119894=1

(119891 (119883119894) minus 119884119894)119879

(119891 (119883119894) minus 119884119894) (17)

given the training data defined in (1) Since 119891(119883119894) is no

longer linear numerical techniques such as the gradientdescent algorithm and variations thereof are relied upon tocompute the setW that satisfies the first derivative condition120597119864119887119901120597W = 0Consider the following definitions

120575119865119896equiv 119891 (119883

119896) minus 119884119896=[[

[

1198911(119883119896) minus (119884

119896)1

1198912(119883119896) minus (119884

119896)2

119891119901(119883119896) minus (119884

119896)119901

]]

]

119878119896equiv

[[[[[[[[[[[[[[

[

120590(

119899

sum119895=1

1198821119895(119883119896)119895minus 1205911)

120590(

119899

sum119895=1

1198822119895(119883119896)119895minus 1205912)

120590(

119899

sum119895=1

119882119898119895(119883119896)119895minus 120591119898)

]]]]]]]]]]]]]]

]

1198781015840

119896equiv

[[[[[[[[[[[[[[

[

1205901015840(

119899

sum119895=1

1198821119895(119883119896)119895minus 1205911)

1205901015840(

119899

sum119895=1

1198822119895(119883119896)119895minus 1205912)

1205901015840(

119899

sum119895=1

119882119898119895(119883119896)119895minus 120591119898)

]]]]]]]]]]]]]]

]

(18)

where 1205901015840(119911) = 119889120590119889119911 The first derivative conditions forthe network weights prescribed by (16) and (17) can then bewritten in matrix form as follows

120597119864119887119901

120597120572=

119873

sum119896=1

120575119865119896119878119879

119896= 0 (19)

(where ldquo119879rdquo denotes the matrix transpose)

120597119864119887119901

120597120591= minus

119873

sum119896=1

diag (1198781015840119896) 120572119879120575119865119879

119896= 0

120597119864119887119901

120597119882=

119873

sum119896=1

diag (1198781015840119896) 120572119879120575119865119879

119896119883119879

119896= 0

(20)

where diag(1198781015840119896) is a square diagonal matrix such that the

diagonal entries consist of components from the vector 1198781015840119896

35 ClassificationMeasures After a given classifier is trainedwhen presented with an input vector 119909 isin 119877119899 of unknownclassification it will respond with an output 119891(119909) isin 119877119901 Theassociated class membership 119862(119909) is then often determinedby applying a minimum distance criterion

119862 (119909) equiv argmin119894

119889 (119891 (119909) 119890119894) (21)

where the target vector 119890119894that is closest to 119891(119909) implies the

class Furthermore when characterizing the performance of apattern classifier one often presents a set 119878 of test vectors andanalyzes the associated output In addition to determiningthe class membership it is also possible to rank the distancebetween a specific target vector and the classifier responseto 119878 In this case a similar distance criterion can be appliedin order to rank an input vector 119909 isin 119878 with respect to class119869 isin 1 119872

120588119869(119909) = 119889 (119891 (119909) 119890

119869) (22)

For the purposes of this work (15) facilitates the determina-tion of the classmembership and rankingwith respect to class119869 for the linear network Similarly assuming a set of weightsW = 119882 120591 120572 for a trained backpropagation network (21)and (22) would be applied using (16)

It is well established that in the case of a normallydistributed data the classification measures presented aboveminimize the probability of a classification error and aredirectly related to the statistical significance of a classificationdecision [1] Given the neural network as the supervisedclassification technique and two distinct choices for the set oftarget vectorsE and E we demonstrate in certain instancesthat the classification and ranking results do not remaininvariant such that 119862(119909) = 119862(119909) and 120588

119869(119909) = 120588

119869(119909) for any

input vector 119909

4 Noninvariance of SecondaryStructure Predictions

In this section we numerically demonstrate that when differ-ent target vector encodings are applied neural network clas-sifier measures outlined above in certain cases are observedto vary widely For each neural network architecture underconsideration an analytic formulation is then presented inSection 5 in order to explain the observed numerical data

As mentioned in Section 32 numerical experiments areperformed first by training using E and then comparing

Applied Computational Intelligence and Soft Computing 5

Table 1 Ranking results for the linear network where 119894 representsthe test vector index and 120588

1(119909) and 120588

1(119909) represent the distance with

respect to the helix class vectors 1198901and 1198901 Out of 1800 vectors tested

vectors referred to in this table were ranked from 1 to 20

119894E 1205881(119909) 119894E 120588

1(119909)

1205 00780 1205 0117042 00867 42 013001031 00976 1031 014641773 01113 1773 01670598 01238 598 018571761 01267 1761 01900862 01354 862 020311073 01409 1073 02114277 01459 277 02188115 01540 115 023091505 01821 1505 02731392 01839 392 027591421 01904 1421 02856147 02001 147 03001990 02044 990 030661457 02127 1457 031911288 02150 1288 03225352 02160 352 032391232 02198 1232 03297280 02311 280 03466

classifier performance with their counterparts trained usingE Multiple cross-validation trials are required in order toprevent potential dependency of the evaluated accuracy onthe particular training or test sets chosen [7 15] In this workwe apply a hold-n-out strategy similar to that of [14] using85 of the 12000 encoded sequences as training data (ie119873 = 10200) and 15 as test data to validate the classificationresults Recognition rates for both the linear and backpropa-gation rates using either set of target vector encodings wereapproximately 65 which is typical of this genre of classifiersthat have applied similar encoding methodologies [7 12ndash14] Although these aggregate values remain consistent using(21) and (22) we now present data demonstrating that whileclass membership and ranking remain invariant for the linearnetwork these measures of performance vary considerablyfor the backpropagation network which was trained with119898 = 17 seventeen hidden nodes and amean squared trainingerror less than 02 Ranking results from a representative testfor the linear and backpropagation networks are presented forthe top 20 ranked vectors in Tables 1 and 3 Classmembershipdata are presented in Tables 2 and 4 Observe that forthe linear network indices for the top 20 ranked vectorsremain invariant indicating ranking invariance in additionno change in class membership is observed On the otherhand Tables 3 and 4 clearly indicate a lack of consistencywhen considering the ranking and class membership of testvectors A particularly troubling observation is that very fewvectors ranked in the top 20 with respect to 119890

1were ranked in

the top 20 with respect to 1198901 Furthermore Table 4 indicates

Table 2 Class membership results for the linear network For eachclass the total number of vectors classified using E is analyzed toexamine the total number that retained their classification using E

Class E E change119867 202 202 0119864 621 621 0119862 977 977 0

that the class membership of a substantial number of testvectors changed when an alternative set of target vectors wasemployed The data also indicates that the greatest changein class membership took place for alpha helical sequencesthus implying that there is substantial disagreement overthe modeling of this secondary structure element by thebackpropagation network due to a simple transformation ofthe target vectors

5 Analysis

The results in Section 4 clearly show that while the patternrecognition results for the linear network remain invariantunder a change in target vectors those for the backpropa-gation network do not In this section we present analyticresults in order to clearly explain and understand why thesetwo techniques lead to different conclusions

51 Invariance Formulation Let us begin by considering twodefinitions

Definition 1 Given two sets of target vectors E and E theclass membership is invariant under a transformation of thetarget vectors if for any input vector 119909

119889(119891 (119909) 119890119894) gt 119889(119891 (119909) 119890

119895)997904rArr119889 (119891 (119909) 119890

119894) gt 119889(119891 (119909) 119890

119895)

(23)

where 119891(119909) is the output of the classifier with target vectors119890119895119872

119895=1

Definition 2 Given two sets of target vectors E and E theranking with respect to a specific class 119869 is invariant under atransformation of the target vectors if for any input vectors1199091and 119909

2

119889(119891(1199091) 119890119869) gt 119889(119891(119909

2) 119890119869)997904rArr119889(119891 (119909

1) 119890119869) gt 119889(119891 (119909

2) 119890119869)

(24)

Based upon these definitions the following has beenestablished [32]

Proposition 3 Given two sets of target vectorsE and E if theranking is invariant then the class membership of an arbitraryinput vector 119909 will remain invariant

In the analysis presented the strategy for characterizingneural network performance depends upon the data from theprevious section For the linear network since both ranking

6 Applied Computational Intelligence and Soft Computing

Table 3 Ranking results for the backpropagation network where 119894represents the test vector index and 120588

1(119909) and 120588

1(119909) represent the

distance with respect to the helix class vectors 1198901and 1198901 Out of 1800

vectors tested vectors referred to in this table were ranked from 1 to20

119894E 1205881(119909) 119894E 120588

1(119909)

817 00107 926 00101887 00231 1604 00130264 00405 887 002091183 00711 1145 00214684 00727 461 00232623 00874 583 00329911 00891 1086 003391382 00917 1382 004781610 00939 413 00489551 01060 225 006081042 01150 438 00609924 01322 911 00613727 01339 207 00774438 01356 559 00885577 01363 481 00945896 01500 1548 00947175 01513 962 009681138 01549 85 01012583 01581 195 01111559 01655 9 01167

Table 4 Class membership results for the backpropagationnetwork For each class the total number of vectors classified usingE is analyzed to examine the total number that retained theirclassification using E

Class E E change119867 225 142 369119864 581 476 181119862 994 878 117

and classification were observed to remain invariant it ismore sensible to characterize the invariance of this networkusing Definition 2 Then based upon Proposition 3 classmembership invariance naturally follows On the other handto explain the noninvariance of both class membership andranking observed in the backpropagation network the anal-ysis is facilitated by considering Definition 1 The noninvari-ance of ranking then naturally follows from Proposition 3

511 Invariance Analysis for the Linear Network When thetarget vectors are subjected to the transformation defined in(2) the network output can be expressed as

119891 (119911) = 119911 = 119860dagger119911 = 119886119877 (119884 minus 119879

1015840)119860dagger119911 (25)

where 1198791015840 is derived from 119879 such that the translation vector119905119894associated with 119890

119894isin E is appropriately aligned with the

correct target vector in matrix 119884 In other words when theoutput data matrix in (7) is of the form

119884 = [1198901198941

1198901198942

119890119894119873

] (26)

then

1198791015840= [1199051198941

1199051198942

119905119894119873

] (27)

where 119894119895isin 1 119872 for 119895 = 1 119873 Given this network

the following result is applicable [32]

Proposition 4 If

(i) the number of training observations 119873 exceeds thevector dimension 119899

(ii) the rows of the matrix 119860 are linearly independent(iii) E and E are related according to (2)(iv) for some 119905 isin 119877119901 119905

119894= 119905 for all 119894 = 1 119872 in (2)

then the ranking and hence the classmembership for the linearnetwork will remain invariant

In other words if the columns of the matrix 1198791015840 in (25)are all equal then using (15) and (25) will result in (23)being satisfiedThe above result is applicable to the presentednumerical data with 119899 = 340 and119873 = 10200 hence rankingand class membership invariances are corroborated by thedata in Tables 1 and 2

512 Invariance Analysis for the Backpropagation NetworkIn this section we seek to characterize the noninvarianceobserved in class membership using the backpropagationnetwork If the class membership varies due to a changein target vectors then this variation should be quantifiableby characterizing the boundary separating two respectiveclasses The decision boundary between class 119894 and class 119895 isdefined by points 119909 such that

1003817100381710038171003817119891 (119909) minus 11989011989410038171003817100381710038172

=10038171003817100381710038171003817119891 (119909) minus 119890

119895

10038171003817100381710038171003817

2

(28)

where 119894 119895 isin 1 119872 119894 = 119895 and 119891(119909) is the classifier outputUnder these circumstances if an ℓ

2norm is applied in (21)

(119891 (119909) minus 119890119894)119879

(119891 (119909) minus 119890119894) = (119891 (119909) minus 119890

119895)119879

(119891 (119909) minus 119890119895)

(29)

The solution set to this equation consists of all 119909 such that119891(119909) is equidistant from 119890

119894and 119890119895where for the purposes of

this section119891(119909) is defined by (16) Expanding terms on bothsides of this equation leads to the condition

(119890119894minus 119890119895)119879

119891 (119909) = minus1

2(119890119879

119895119890119895minus 119890119879

119894119890119894) (30)

If the classmembership of a representative vector is to remaininvariant under a change of target vectors this same set ofpoints must also satisfy

( 119890119894minus 119890119895)119879

119891 (119909) = minus1

2( 119890119879

119895119890119895minus 119890119879

119894119890119894) (31)

Applied Computational Intelligence and Soft Computing 7

Assuming that two networks have been trained using twodifferent sets of target vectors E and E the set of weights119882 120591 120572 and 120591 determines the network output in (16)Without loss of generality we consider the case where alltarget vectors are normalized to a value of one such that1198901198942= 1 and 119890

1198942= 1 for 119894 = 1 119872 In this case the

conditions in (30) and (31) become

(119890119894minus 119890119895)119879

120572120590 (119882119909 minus 120591) = 0 (32)

(119890119894minus 119890119895)119879

120590 (119909 minus 120591) = 0 (33)

We first consider a special case where the target vectorsare related according to Γ = 119886119877Γ with 119879 = 0 in (2) Underthese circumstances if the choice

W equiv 120591 = 119882 120591 119886119877120572 (34)

is made it should be clear that since119877119879 = 119877minus1 and 119894= 119886119877119884

119894

119864119887119901=

119873

sum119894=1

(119891 (119883119894) minus 119894)119879

(119891 (119883119894) minus 119894)

= 1198862

119873

sum119894=1

(119877119891 (119883119894) minus 119877119884

119894)119879

(119877119891 (119883119894) minus 119877119884

119894)

= 1198862

119873

sum119894=1

(119891 (119883119894) minus 119884119894)119879

119877119879119877 (119891 (119883

119894) minus 119884119894)

= 1198862119864119887119901

(35)

is minimized by the condition 120597119864119887119901120597W = 0 Another way tosee this is to observe that (19) (20) remain invariant for thischoice of target vectors and network weights Hence we havethe following

Proposition 5 For a specific choice of 119886 and 119877 if 119879 = 0 in (2)and

W = 120591 = 119882 120591 119886119877120572 (36)

then the classmembership for the backpropagation networkwillremain invariant

Proof Simply consider (32) and (33) and choose any 119909satisfying

(119890119894minus 119890119895)119879

120572120590 (119882119909 minus 120591) = 0 (37)

It then immediately follows that

(119890119894minus 119890119895)119879

120590 (119909 minus 120591)

= (119890119894minus 119890119895)119879

119877119879119877120572120590 (119882119909 minus 120591)

= 1198862(119890119894minus 119890119895)119879

120572120590 (119882119909 minus 120591)

= 0

(38)

Therefore if 119909 satisfies (32) then it also satisfies (33) andhence is a point on the decision boundary for both net-works

Intuitively a scaled rigid rotation of the target vectorsshould not affect the decision boundary However when themore general transformation of (2) is applied with 119879 = 0we now demonstrate that due to the nonlinearity of 120590 in(16) no simple relationship exists such that (32) and (33)can simultaneously be satisfied by the same set of pointsWe first investigate the possibility of establishing an analyticrelationship between the set of weights 120591 and 119882 120591 120572for both networks In other words we seek ideally invertiblefunctions 119866

120572 119866120591 and 119866

119882

= 119866120572(119882 120591 120572)

120591 = 119866120591(119882 120591 120572)

= 119866119882(119882 120591 120572)

(39)

such that the setW can be transformed into W If this can bedone then an analytic procedure similar to that presented inthe proof of Proposition 5 can be established in order to relate(32) to (33) for the general case Since (19) (20) define the setW it is reasonable to rephrase these equations in terms of theobjective function 119864119887119901

120597119864119887119901

120597=

119873

sum119896=1

(119891 (119883119896) minus 119896) 119878119879

119896= 0

120597119864119887119901

120597120591= minus

119873

sum119896=1

diag (1198781015840119896) 119879(119891 (119883

119896) minus 119896)119879

= 0

120597119864119887119901

120597=

119873

sum119896=1

diag (1198781015840119896) 119879(119891 (119883

119896) minus 119896)119879

119883119879

119896= 0

(40)

where 119891(119883119896) = 120590(119883

119896minus 120591) and

119896= 119886119877(119884

119896minus 1199051015840) such that

1199051015840 is the translation vector associated with the target vectorreferred to by 119884

119896 From these equations it should be clear

that no simple analytic relationship exists that will transformW into W A numerical algorithm such as gradient descentwill assuming a localminimumactually exists arrive at somesolution for both W and W We must therefore be contentwith the assumed existence of some set of functions definedby (39) Again let us consider any point 119909 on the decisionboundary such that

(119890119894minus 119890119895)119879

120572120590 (119882119909 minus 120591) = 0 (41)

Such a point must also simultaneously satisfy

(119890119894minus 119890119895)119879

120590 (119909 minus 120591)

= ((119886119877119890119894minus 119905119894) minus (119886119877119890

119895minus 119905119895))119879

119866120572(119882 120591 120572)

8 Applied Computational Intelligence and Soft Computing

times 120590 (119866119882(119882 120591 120572) 119909 minus 119866

120591(119882 120591 120572))

= 119886((119890119894minus 119890119895) minus (119905

119894minus 119905119895))119879

119877119879119866120572(119882 120591 120572)

times 120590 (119866119882(119882 120591 120572) 119909 minus 119866

120591(119882 120591 120572))

= 0

(42)

At first glance a choice such as 119866120572(119882 120591 120572) = 119886119877120572 and

119905119894= 119905119895for 119894 119895 = 1 119872 (as in Proposition 4) appears

to bring us close to a solution However the term involving120590(119866119882(119882 120591 120572)119909 minus 119866

120591(119882 120591 120572)) is problematic Although a

choice such as 119866119882(119882 120591 120572) = 119882 and 119866

120591(119882 120591 120572) = 120591 would

yield the solution to (42) it should be clear that these valueswould not satisfy (40)

Another way to analyze this problem is to first set 119905119894= 119905119895

for 119894 119895 = 1 119872 Then for any 119909 on the decision boundaryfrom (41) and (42) equate the terms

120572120590 (119882119909 minus 120591) = 119896119877119879119866120572(119882 120591 120572)

times 120590 (119866119882(119882 120591 120572) 119909 minus 119866

120591(119882 120591 120572))

(43)

where 119896 is some constant and 119866120572 119866120591 and 119866

119882satisfy (40)

Given an arbitrary training set defined by (1) it is highlyunlikely that this constraint can be satisfied One remotescenario might be that the terms 119866

119882(119882 120591 120572)119909 minus 119866

120591(119882 120591 120572)

and119882119909 minus 120591 are always small In this case given a sigmoidalfunction 120590(119911) that is linear near 119911 = 0 a linearized versionof (43) could be solved using techniques described in [32]However this again is an unlikely set of events given anarbitrary training set Therefore given the transformation of(2) we are left to conclude that class membership invarianceand hence ranking invariance are in general not achievableusing the backpropagation neural network

52 Discussion Intuitively given a reasonable target encod-ing scheme one would desire that properties related to pro-tein secondary structure would be independent of the targetvectors chosen However we have presented numerical dataand a theoretical foundation demonstrating that secondarystructure classification and confidence measures can varydepending on the type of neural network architecture andtarget vector encoding scheme employed Specifically linearnetwork classification has been demonstrated to remaininvariant under a change in the target structure encodingscheme while the backpropagation network has not As119873 increases for the methodology applied in this workrecognition rates remain consistent with those reported inthe literature however we have observed that adding moretraining data does not improve the invariance of classificationmeasures for the backpropagation network This conclusionis corroborated by the analytic formulation presented above

6 Conclusions

As pointed out in the introduction one major purpose of theneural network is to create a stable and reliable model that

maps input training data to an output classification with thehope of extracting informative parameters When methodssimilar to those in the literature are applied [7 12ndash14] wehave demonstrated that classifier performance measures canvary considerably Under these circumstances parametersderived from a trained network for analytically describingprotein secondary structure may not comprise a reliableset for the model-based approach Furthermore classifiervariabilitywould imply that a stable parametricmodel has notbeen derived It is in some sense paradoxical that the neuralnetwork has been applied for structure classification and yetassociated parameters have not been applied for describingprotein secondary structure The neural network approachto deriving a solution to the protein secondary structureprediction problem therefore requires deeper exploration

Acknowledgments

This publication was made possible by Grant NumberG12RR017581 from the National Center for ResearchResources (NCRR) a component of the National Institutesof Health (NIH) The authors would also like to thank thereviewers for their helpful comments

References

[1] C M Bishop Pattern Recognition and Machine LearningSpringer 2007

[2] M T Hagan H B Demuth and M H Beale Neural NetworkDesign PWS Publishing 1996

[3] M N Nguyen and J C Rajapakse ldquoMulti-class support vectormachines for protein secondary structure predictionrdquo GenomeInformatics vol 14 pp 218ndash227 2003

[4] H J Hu Y Pan R Harrison and P C Tai ldquoImprovedprotein secondary structure prediction using support vectormachine with a new encoding scheme and an advanced tertiaryclassifierrdquo IEEE Transactions onNanobioscience vol 3 no 4 pp265ndash271 2004

[5] W Zhong G Altun X Tian R Harrison P C Tai and YPan ldquoParallel protein secondary structure prediction schemesusing Pthread and OpenMP over hyper-threading technologyrdquoJournal of Supercomputing vol 41 no 1 pp 1ndash16 2007

[6] J A Cuff and G J Barton ldquoApplication of enhanced multiplesequence alignment profiles to improve protein secondarystructure predictionrdquo Proteins vol 40 pp 502ndash511 2000

[7] J M Chandonia and M Karplus ldquoNeural networks for sec-ondary structure and structural class predictionsrdquo ProteinScience vol 4 no 2 pp 275ndash285 1995

[8] J A Cuff and G J Barton ldquoEvaluation and improvement ofmultiple sequence methods for protein secondary structurepredictionrdquo Proteins vol 34 pp 508ndash519 1999

[9] G E Crooks and S E Brenner ldquoProtein secondary structureentropy correlations andpredictionrdquoBioinformatics vol 20 no10 pp 1603ndash1611 2004

[10] L H Wang J Liu and H B Zhou ldquoA comparison of twomachine learning methods for protein secondary structurepredictionrdquo in Proceedings of 2004 International Conference onMachine Learning and Cybernetics pp 2730ndash2735 chn August2004

Applied Computational Intelligence and Soft Computing 9

[11] G Z Zhang D S Huang Y P Zhu and Y X Li ldquoImprovingprotein secondary structure prediction by using the residueconformational classesrdquo Pattern Recognition Letters vol 26 no15 pp 2346ndash2352 2005

[12] NQian and T J Sejnowski ldquoPredicting the secondary structureof globular proteins using neural network modelsrdquo Journal ofMolecular Biology vol 202 no 4 pp 865ndash884 1988

[13] L HowardHolley andMKarplus ldquoProtein secondary structureprediction with a neural networkrdquo Proceedings of the NationalAcademy of Sciences of the United States of America vol 86 no1 pp 152ndash156 1989

[14] J M Chandonia and M Karplus ldquoThe importance of largerdata sets for protein secondary structure prediction with neuralnetworksrdquo Protein Science vol 5 no 4 pp 768ndash774 1996

[15] B Rost and C Sander ldquoImproved prediction of proteinsecondary structure by use of sequence profiles and neuralnetworksrdquoProceedings of theNational Academy of Sciences of theUnited States of America vol 90 no 16 pp 7558ndash7562 1993

[16] B Rost and C Sander ldquoPrediction of protein secondary struc-ture at better than 70 accuracyrdquo Journal of Molecular Biologyvol 232 no 2 pp 584ndash599 1993

[17] J A Cuff M E Clamp A S Siddiqui M Finlay and GJ Barton ldquoJPred a consensus secondary structure predictionserverrdquo Bioinformatics vol 14 no 10 pp 892ndash893 1998

[18] S Hua and Z Sun ldquoA novel method of protein secondary struc-ture prediction with high segment overlap measure supportvector machine approachrdquo Journal of Molecular Biology vol308 no 2 pp 397ndash407 2001

[19] B Rost ldquoPHD predicting one-dimensional protein structureby profile-based neural networksrdquoMethods in Enzymology vol266 pp 525ndash539 1996

[20] B Rost G Yachdav and J Liu ldquoThe PredictProtein serverrdquoNucleic Acids Research vol 32 pp W321ndashW326 2004

[21] C Cole J D Barber and G J Barton ldquoThe Jpred 3 secondarystructure prediction serverrdquo Nucleic Acids Research vol 36 ppW197ndashW201 2008

[22] G Pollastri D Przybylski B Rost and P Baldi ldquoImprovingthe prediction of protein secondary structure in three and eightclasses using recurrent neural networks and profilesrdquo Proteinsvol 47 no 2 pp 228ndash235 2002

[23] J Cheng A Z Randall M J Sweredoski and P BaldildquoSCRATCH a protein structure and structural feature predic-tion serverrdquoNucleic Acids Research vol 33 no 2 ppW72ndashW762005

[24] D T Jones ldquoProtein secondary structure prediction basedon position-specific scoring matricesrdquo Journal of MolecularBiology vol 292 no 2 pp 195ndash202 1999

[25] K Bryson L J McGuffin R L Marsden J J Ward J SSodhi and D T Jones ldquoProtein structure prediction servers atUniversity College Londonrdquo Nucleic Acids Research vol 33 no2 pp W36ndashW38 2005

[26] G Cybenko ldquoApproximation by superpositions of a sigmoidalfunctionrdquo Mathematics of Control Signals and Systems vol 2no 4 pp 303ndash314 1989

[27] J Moody and C J Darken ldquoFast learning in networks of locallytuned processing unitsrdquo Neural Computation vol 1 pp 281ndash294 1989

[28] T Poggio and F Girosi ldquoNetworks for approximation andlearningrdquo Proceedings of the IEEE vol 78 no 9 pp 1481ndash14971990

[29] D F Specht ldquoProbabilistic neural networksrdquo Neural Networksvol 3 no 1 pp 109ndash118 1990

[30] P Andras ldquoOrthogonal RBF neural network approximationrdquoNeural Processing Letters vol 9 no 2 pp 141ndash151 1999

[31] P Baldi and S Brunak Bioinformatics The Machine LearningApproach MIT Press 1998

[32] E Sakk D J Schneider C R Myers and SW Cartinhour ldquoOnthe selection of target vectors for a class of supervised patternrecognizersrdquo IEEE Transactions on Neural Networks vol 20 no5 pp 745ndash757 2009

[33] G H Golub and C F Van Loan Matrix Computations JohnsHopkins University Press 1989

Submit your manuscripts athttpwwwhindawicom

Computer Games Technology

International Journal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Distributed Sensor Networks

International Journal of

Advances in

FuzzySystems

Hindawi Publishing Corporationhttpwwwhindawicom

Volume 2014

International Journal of

ReconfigurableComputing

Hindawi Publishing Corporation httpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Applied Computational Intelligence and Soft Computing

thinspAdvancesthinspinthinsp

Artificial Intelligence

HindawithinspPublishingthinspCorporationhttpwwwhindawicom Volumethinsp2014

Advances inSoftware EngineeringHindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Electrical and Computer Engineering

Journal of

Journal of

Computer Networks and Communications

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporation

httpwwwhindawicom Volume 2014

Advances in

Multimedia

International Journal of

Biomedical Imaging

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

ArtificialNeural Systems

Advances in

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

RoboticsJournal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Computational Intelligence and Neuroscience

Industrial EngineeringJournal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Modelling amp Simulation in EngineeringHindawi Publishing Corporation httpwwwhindawicom Volume 2014

The Scientific World JournalHindawi Publishing Corporation httpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Human-ComputerInteraction

Advances in

Computer EngineeringAdvances in

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

4 Applied Computational Intelligence and Soft Computing

34The Backpropagation Network Given an input vector 119909 isin119877119899 the model 119891 119877119899 rarr 119877119901 for a backpropagation neuralnetwork with a single hidden layer consisting of 119898 nodes isdescribed as

119891 (119909) =

[[[[[

[

1198911(119909)

1198912(119909)

119891119901(119909)

]]]]]

]

= 120572120590 (119882119909 minus 120591) (16)

where 120572 isin 119877119901times119898 119882 isin 119877119898times119899 and 120591 isin 119877119898 define the setof network weights W = 119882 120591 120572 and 120590 119877119898 rarr 119877119901 isa ldquosigmoidalrdquo function that is bounded and monotonicallyincreasing To perform supervised training in a mannersimilar to the linear networkW is determined byminimizingthe objective function

119864119887119901=

119873

sum119894=1

(119891 (119883119894) minus 119884119894)119879

(119891 (119883119894) minus 119884119894) (17)

given the training data defined in (1) Since 119891(119883119894) is no

longer linear numerical techniques such as the gradientdescent algorithm and variations thereof are relied upon tocompute the setW that satisfies the first derivative condition120597119864119887119901120597W = 0Consider the following definitions

120575119865119896equiv 119891 (119883

119896) minus 119884119896=[[

[

1198911(119883119896) minus (119884

119896)1

1198912(119883119896) minus (119884

119896)2

119891119901(119883119896) minus (119884

119896)119901

]]

]

119878119896equiv

[[[[[[[[[[[[[[

[

120590(

119899

sum119895=1

1198821119895(119883119896)119895minus 1205911)

120590(

119899

sum119895=1

1198822119895(119883119896)119895minus 1205912)

120590(

119899

sum119895=1

119882119898119895(119883119896)119895minus 120591119898)

]]]]]]]]]]]]]]

]

1198781015840

119896equiv

[[[[[[[[[[[[[[

[

1205901015840(

119899

sum119895=1

1198821119895(119883119896)119895minus 1205911)

1205901015840(

119899

sum119895=1

1198822119895(119883119896)119895minus 1205912)

1205901015840(

119899

sum119895=1

119882119898119895(119883119896)119895minus 120591119898)

]]]]]]]]]]]]]]

]

(18)

where 1205901015840(119911) = 119889120590119889119911 The first derivative conditions forthe network weights prescribed by (16) and (17) can then bewritten in matrix form as follows

120597119864119887119901

120597120572=

119873

sum119896=1

120575119865119896119878119879

119896= 0 (19)

(where ldquo119879rdquo denotes the matrix transpose)

120597119864119887119901

120597120591= minus

119873

sum119896=1

diag (1198781015840119896) 120572119879120575119865119879

119896= 0

120597119864119887119901

120597119882=

119873

sum119896=1

diag (1198781015840119896) 120572119879120575119865119879

119896119883119879

119896= 0

(20)

where diag(1198781015840119896) is a square diagonal matrix such that the

diagonal entries consist of components from the vector 1198781015840119896

35 ClassificationMeasures After a given classifier is trainedwhen presented with an input vector 119909 isin 119877119899 of unknownclassification it will respond with an output 119891(119909) isin 119877119901 Theassociated class membership 119862(119909) is then often determinedby applying a minimum distance criterion

119862 (119909) equiv argmin119894

119889 (119891 (119909) 119890119894) (21)

where the target vector 119890119894that is closest to 119891(119909) implies the

class Furthermore when characterizing the performance of apattern classifier one often presents a set 119878 of test vectors andanalyzes the associated output In addition to determiningthe class membership it is also possible to rank the distancebetween a specific target vector and the classifier responseto 119878 In this case a similar distance criterion can be appliedin order to rank an input vector 119909 isin 119878 with respect to class119869 isin 1 119872

120588119869(119909) = 119889 (119891 (119909) 119890

119869) (22)

For the purposes of this work (15) facilitates the determina-tion of the classmembership and rankingwith respect to class119869 for the linear network Similarly assuming a set of weightsW = 119882 120591 120572 for a trained backpropagation network (21)and (22) would be applied using (16)

It is well established that in the case of a normallydistributed data the classification measures presented aboveminimize the probability of a classification error and aredirectly related to the statistical significance of a classificationdecision [1] Given the neural network as the supervisedclassification technique and two distinct choices for the set oftarget vectorsE and E we demonstrate in certain instancesthat the classification and ranking results do not remaininvariant such that 119862(119909) = 119862(119909) and 120588

119869(119909) = 120588

119869(119909) for any

input vector 119909

4 Noninvariance of SecondaryStructure Predictions

In this section we numerically demonstrate that when differ-ent target vector encodings are applied neural network clas-sifier measures outlined above in certain cases are observedto vary widely For each neural network architecture underconsideration an analytic formulation is then presented inSection 5 in order to explain the observed numerical data

As mentioned in Section 32 numerical experiments areperformed first by training using E and then comparing

Applied Computational Intelligence and Soft Computing 5

Table 1 Ranking results for the linear network where 119894 representsthe test vector index and 120588

1(119909) and 120588

1(119909) represent the distance with

respect to the helix class vectors 1198901and 1198901 Out of 1800 vectors tested

vectors referred to in this table were ranked from 1 to 20

119894E 1205881(119909) 119894E 120588

1(119909)

1205 00780 1205 0117042 00867 42 013001031 00976 1031 014641773 01113 1773 01670598 01238 598 018571761 01267 1761 01900862 01354 862 020311073 01409 1073 02114277 01459 277 02188115 01540 115 023091505 01821 1505 02731392 01839 392 027591421 01904 1421 02856147 02001 147 03001990 02044 990 030661457 02127 1457 031911288 02150 1288 03225352 02160 352 032391232 02198 1232 03297280 02311 280 03466

classifier performance with their counterparts trained usingE Multiple cross-validation trials are required in order toprevent potential dependency of the evaluated accuracy onthe particular training or test sets chosen [7 15] In this workwe apply a hold-n-out strategy similar to that of [14] using85 of the 12000 encoded sequences as training data (ie119873 = 10200) and 15 as test data to validate the classificationresults Recognition rates for both the linear and backpropa-gation rates using either set of target vector encodings wereapproximately 65 which is typical of this genre of classifiersthat have applied similar encoding methodologies [7 12ndash14] Although these aggregate values remain consistent using(21) and (22) we now present data demonstrating that whileclass membership and ranking remain invariant for the linearnetwork these measures of performance vary considerablyfor the backpropagation network which was trained with119898 = 17 seventeen hidden nodes and amean squared trainingerror less than 02 Ranking results from a representative testfor the linear and backpropagation networks are presented forthe top 20 ranked vectors in Tables 1 and 3 Classmembershipdata are presented in Tables 2 and 4 Observe that forthe linear network indices for the top 20 ranked vectorsremain invariant indicating ranking invariance in additionno change in class membership is observed On the otherhand Tables 3 and 4 clearly indicate a lack of consistencywhen considering the ranking and class membership of testvectors A particularly troubling observation is that very fewvectors ranked in the top 20 with respect to 119890

1were ranked in

the top 20 with respect to 1198901 Furthermore Table 4 indicates

Table 2 Class membership results for the linear network For eachclass the total number of vectors classified using E is analyzed toexamine the total number that retained their classification using E

Class E E change119867 202 202 0119864 621 621 0119862 977 977 0

that the class membership of a substantial number of testvectors changed when an alternative set of target vectors wasemployed The data also indicates that the greatest changein class membership took place for alpha helical sequencesthus implying that there is substantial disagreement overthe modeling of this secondary structure element by thebackpropagation network due to a simple transformation ofthe target vectors

5 Analysis

The results in Section 4 clearly show that while the patternrecognition results for the linear network remain invariantunder a change in target vectors those for the backpropa-gation network do not In this section we present analyticresults in order to clearly explain and understand why thesetwo techniques lead to different conclusions

51 Invariance Formulation Let us begin by considering twodefinitions

Definition 1 Given two sets of target vectors E and E theclass membership is invariant under a transformation of thetarget vectors if for any input vector 119909

119889(119891 (119909) 119890119894) gt 119889(119891 (119909) 119890

119895)997904rArr119889 (119891 (119909) 119890

119894) gt 119889(119891 (119909) 119890

119895)

(23)

where 119891(119909) is the output of the classifier with target vectors119890119895119872

119895=1

Definition 2 Given two sets of target vectors E and E theranking with respect to a specific class 119869 is invariant under atransformation of the target vectors if for any input vectors1199091and 119909

2

119889(119891(1199091) 119890119869) gt 119889(119891(119909

2) 119890119869)997904rArr119889(119891 (119909

1) 119890119869) gt 119889(119891 (119909

2) 119890119869)

(24)

Based upon these definitions the following has beenestablished [32]

Proposition 3 Given two sets of target vectorsE and E if theranking is invariant then the class membership of an arbitraryinput vector 119909 will remain invariant

In the analysis presented the strategy for characterizingneural network performance depends upon the data from theprevious section For the linear network since both ranking

6 Applied Computational Intelligence and Soft Computing

Table 3 Ranking results for the backpropagation network where 119894represents the test vector index and 120588

1(119909) and 120588

1(119909) represent the

distance with respect to the helix class vectors 1198901and 1198901 Out of 1800

vectors tested vectors referred to in this table were ranked from 1 to20

119894E 1205881(119909) 119894E 120588

1(119909)

817 00107 926 00101887 00231 1604 00130264 00405 887 002091183 00711 1145 00214684 00727 461 00232623 00874 583 00329911 00891 1086 003391382 00917 1382 004781610 00939 413 00489551 01060 225 006081042 01150 438 00609924 01322 911 00613727 01339 207 00774438 01356 559 00885577 01363 481 00945896 01500 1548 00947175 01513 962 009681138 01549 85 01012583 01581 195 01111559 01655 9 01167

Table 4 Class membership results for the backpropagationnetwork For each class the total number of vectors classified usingE is analyzed to examine the total number that retained theirclassification using E

Class E E change119867 225 142 369119864 581 476 181119862 994 878 117

and classification were observed to remain invariant it ismore sensible to characterize the invariance of this networkusing Definition 2 Then based upon Proposition 3 classmembership invariance naturally follows On the other handto explain the noninvariance of both class membership andranking observed in the backpropagation network the anal-ysis is facilitated by considering Definition 1 The noninvari-ance of ranking then naturally follows from Proposition 3

511 Invariance Analysis for the Linear Network When thetarget vectors are subjected to the transformation defined in(2) the network output can be expressed as

119891 (119911) = 119911 = 119860dagger119911 = 119886119877 (119884 minus 119879

1015840)119860dagger119911 (25)

where 1198791015840 is derived from 119879 such that the translation vector119905119894associated with 119890

119894isin E is appropriately aligned with the

correct target vector in matrix 119884 In other words when theoutput data matrix in (7) is of the form

119884 = [1198901198941

1198901198942

119890119894119873

] (26)

then

1198791015840= [1199051198941

1199051198942

119905119894119873

] (27)

where 119894119895isin 1 119872 for 119895 = 1 119873 Given this network

the following result is applicable [32]

Proposition 4 If

(i) the number of training observations 119873 exceeds thevector dimension 119899

(ii) the rows of the matrix 119860 are linearly independent(iii) E and E are related according to (2)(iv) for some 119905 isin 119877119901 119905

119894= 119905 for all 119894 = 1 119872 in (2)

then the ranking and hence the classmembership for the linearnetwork will remain invariant

In other words if the columns of the matrix 1198791015840 in (25)are all equal then using (15) and (25) will result in (23)being satisfiedThe above result is applicable to the presentednumerical data with 119899 = 340 and119873 = 10200 hence rankingand class membership invariances are corroborated by thedata in Tables 1 and 2

512 Invariance Analysis for the Backpropagation NetworkIn this section we seek to characterize the noninvarianceobserved in class membership using the backpropagationnetwork If the class membership varies due to a changein target vectors then this variation should be quantifiableby characterizing the boundary separating two respectiveclasses The decision boundary between class 119894 and class 119895 isdefined by points 119909 such that

1003817100381710038171003817119891 (119909) minus 11989011989410038171003817100381710038172

=10038171003817100381710038171003817119891 (119909) minus 119890

119895

10038171003817100381710038171003817

2

(28)

where 119894 119895 isin 1 119872 119894 = 119895 and 119891(119909) is the classifier outputUnder these circumstances if an ℓ

2norm is applied in (21)

(119891 (119909) minus 119890119894)119879

(119891 (119909) minus 119890119894) = (119891 (119909) minus 119890

119895)119879

(119891 (119909) minus 119890119895)

(29)

The solution set to this equation consists of all 119909 such that119891(119909) is equidistant from 119890

119894and 119890119895where for the purposes of

this section119891(119909) is defined by (16) Expanding terms on bothsides of this equation leads to the condition

(119890119894minus 119890119895)119879

119891 (119909) = minus1

2(119890119879

119895119890119895minus 119890119879

119894119890119894) (30)

If the classmembership of a representative vector is to remaininvariant under a change of target vectors this same set ofpoints must also satisfy

( 119890119894minus 119890119895)119879

119891 (119909) = minus1

2( 119890119879

119895119890119895minus 119890119879

119894119890119894) (31)

Applied Computational Intelligence and Soft Computing 7

Assuming that two networks have been trained using twodifferent sets of target vectors E and E the set of weights119882 120591 120572 and 120591 determines the network output in (16)Without loss of generality we consider the case where alltarget vectors are normalized to a value of one such that1198901198942= 1 and 119890

1198942= 1 for 119894 = 1 119872 In this case the

conditions in (30) and (31) become

(119890119894minus 119890119895)119879

120572120590 (119882119909 minus 120591) = 0 (32)

(119890119894minus 119890119895)119879

120590 (119909 minus 120591) = 0 (33)

We first consider a special case where the target vectorsare related according to Γ = 119886119877Γ with 119879 = 0 in (2) Underthese circumstances if the choice

W equiv 120591 = 119882 120591 119886119877120572 (34)

is made it should be clear that since119877119879 = 119877minus1 and 119894= 119886119877119884

119894

119864119887119901=

119873

sum119894=1

(119891 (119883119894) minus 119894)119879

(119891 (119883119894) minus 119894)

= 1198862

119873

sum119894=1

(119877119891 (119883119894) minus 119877119884

119894)119879

(119877119891 (119883119894) minus 119877119884

119894)

= 1198862

119873

sum119894=1

(119891 (119883119894) minus 119884119894)119879

119877119879119877 (119891 (119883

119894) minus 119884119894)

= 1198862119864119887119901

(35)

is minimized by the condition 120597119864119887119901120597W = 0 Another way tosee this is to observe that (19) (20) remain invariant for thischoice of target vectors and network weights Hence we havethe following

Proposition 5 For a specific choice of 119886 and 119877 if 119879 = 0 in (2)and

W = 120591 = 119882 120591 119886119877120572 (36)

then the classmembership for the backpropagation networkwillremain invariant

Proof Simply consider (32) and (33) and choose any 119909satisfying

(119890119894minus 119890119895)119879

120572120590 (119882119909 minus 120591) = 0 (37)

It then immediately follows that

(119890119894minus 119890119895)119879

120590 (119909 minus 120591)

= (119890119894minus 119890119895)119879

119877119879119877120572120590 (119882119909 minus 120591)

= 1198862(119890119894minus 119890119895)119879

120572120590 (119882119909 minus 120591)

= 0

(38)

Therefore if 119909 satisfies (32) then it also satisfies (33) andhence is a point on the decision boundary for both net-works

Intuitively a scaled rigid rotation of the target vectorsshould not affect the decision boundary However when themore general transformation of (2) is applied with 119879 = 0we now demonstrate that due to the nonlinearity of 120590 in(16) no simple relationship exists such that (32) and (33)can simultaneously be satisfied by the same set of pointsWe first investigate the possibility of establishing an analyticrelationship between the set of weights 120591 and 119882 120591 120572for both networks In other words we seek ideally invertiblefunctions 119866

120572 119866120591 and 119866

119882

= 119866120572(119882 120591 120572)

120591 = 119866120591(119882 120591 120572)

= 119866119882(119882 120591 120572)

(39)

such that the setW can be transformed into W If this can bedone then an analytic procedure similar to that presented inthe proof of Proposition 5 can be established in order to relate(32) to (33) for the general case Since (19) (20) define the setW it is reasonable to rephrase these equations in terms of theobjective function 119864119887119901

120597119864119887119901

120597=

119873

sum119896=1

(119891 (119883119896) minus 119896) 119878119879

119896= 0

120597119864119887119901

120597120591= minus

119873

sum119896=1

diag (1198781015840119896) 119879(119891 (119883

119896) minus 119896)119879

= 0

120597119864119887119901

120597=

119873

sum119896=1

diag (1198781015840119896) 119879(119891 (119883

119896) minus 119896)119879

119883119879

119896= 0

(40)

where 119891(119883119896) = 120590(119883

119896minus 120591) and

119896= 119886119877(119884

119896minus 1199051015840) such that

1199051015840 is the translation vector associated with the target vectorreferred to by 119884

119896 From these equations it should be clear

that no simple analytic relationship exists that will transformW into W A numerical algorithm such as gradient descentwill assuming a localminimumactually exists arrive at somesolution for both W and W We must therefore be contentwith the assumed existence of some set of functions definedby (39) Again let us consider any point 119909 on the decisionboundary such that

(119890119894minus 119890119895)119879

120572120590 (119882119909 minus 120591) = 0 (41)

Such a point must also simultaneously satisfy

(119890119894minus 119890119895)119879

120590 (119909 minus 120591)

= ((119886119877119890119894minus 119905119894) minus (119886119877119890

119895minus 119905119895))119879

119866120572(119882 120591 120572)

8 Applied Computational Intelligence and Soft Computing

times 120590 (119866119882(119882 120591 120572) 119909 minus 119866

120591(119882 120591 120572))

= 119886((119890119894minus 119890119895) minus (119905

119894minus 119905119895))119879

119877119879119866120572(119882 120591 120572)

times 120590 (119866119882(119882 120591 120572) 119909 minus 119866

120591(119882 120591 120572))

= 0

(42)

At first glance a choice such as 119866120572(119882 120591 120572) = 119886119877120572 and

119905119894= 119905119895for 119894 119895 = 1 119872 (as in Proposition 4) appears

to bring us close to a solution However the term involving120590(119866119882(119882 120591 120572)119909 minus 119866

120591(119882 120591 120572)) is problematic Although a

choice such as 119866119882(119882 120591 120572) = 119882 and 119866

120591(119882 120591 120572) = 120591 would

yield the solution to (42) it should be clear that these valueswould not satisfy (40)

Another way to analyze this problem is to first set 119905119894= 119905119895

for 119894 119895 = 1 119872 Then for any 119909 on the decision boundaryfrom (41) and (42) equate the terms

120572120590 (119882119909 minus 120591) = 119896119877119879119866120572(119882 120591 120572)

times 120590 (119866119882(119882 120591 120572) 119909 minus 119866

120591(119882 120591 120572))

(43)

where 119896 is some constant and 119866120572 119866120591 and 119866

119882satisfy (40)

Given an arbitrary training set defined by (1) it is highlyunlikely that this constraint can be satisfied One remotescenario might be that the terms 119866

119882(119882 120591 120572)119909 minus 119866

120591(119882 120591 120572)

and119882119909 minus 120591 are always small In this case given a sigmoidalfunction 120590(119911) that is linear near 119911 = 0 a linearized versionof (43) could be solved using techniques described in [32]However this again is an unlikely set of events given anarbitrary training set Therefore given the transformation of(2) we are left to conclude that class membership invarianceand hence ranking invariance are in general not achievableusing the backpropagation neural network

52 Discussion Intuitively given a reasonable target encod-ing scheme one would desire that properties related to pro-tein secondary structure would be independent of the targetvectors chosen However we have presented numerical dataand a theoretical foundation demonstrating that secondarystructure classification and confidence measures can varydepending on the type of neural network architecture andtarget vector encoding scheme employed Specifically linearnetwork classification has been demonstrated to remaininvariant under a change in the target structure encodingscheme while the backpropagation network has not As119873 increases for the methodology applied in this workrecognition rates remain consistent with those reported inthe literature however we have observed that adding moretraining data does not improve the invariance of classificationmeasures for the backpropagation network This conclusionis corroborated by the analytic formulation presented above

6 Conclusions

As pointed out in the introduction one major purpose of theneural network is to create a stable and reliable model that

maps input training data to an output classification with thehope of extracting informative parameters When methodssimilar to those in the literature are applied [7 12ndash14] wehave demonstrated that classifier performance measures canvary considerably Under these circumstances parametersderived from a trained network for analytically describingprotein secondary structure may not comprise a reliableset for the model-based approach Furthermore classifiervariabilitywould imply that a stable parametricmodel has notbeen derived It is in some sense paradoxical that the neuralnetwork has been applied for structure classification and yetassociated parameters have not been applied for describingprotein secondary structure The neural network approachto deriving a solution to the protein secondary structureprediction problem therefore requires deeper exploration

Acknowledgments

This publication was made possible by Grant NumberG12RR017581 from the National Center for ResearchResources (NCRR) a component of the National Institutesof Health (NIH) The authors would also like to thank thereviewers for their helpful comments

References

[1] C M Bishop Pattern Recognition and Machine LearningSpringer 2007

[2] M T Hagan H B Demuth and M H Beale Neural NetworkDesign PWS Publishing 1996

[3] M N Nguyen and J C Rajapakse ldquoMulti-class support vectormachines for protein secondary structure predictionrdquo GenomeInformatics vol 14 pp 218ndash227 2003

[4] H J Hu Y Pan R Harrison and P C Tai ldquoImprovedprotein secondary structure prediction using support vectormachine with a new encoding scheme and an advanced tertiaryclassifierrdquo IEEE Transactions onNanobioscience vol 3 no 4 pp265ndash271 2004

[5] W Zhong G Altun X Tian R Harrison P C Tai and YPan ldquoParallel protein secondary structure prediction schemesusing Pthread and OpenMP over hyper-threading technologyrdquoJournal of Supercomputing vol 41 no 1 pp 1ndash16 2007

[6] J A Cuff and G J Barton ldquoApplication of enhanced multiplesequence alignment profiles to improve protein secondarystructure predictionrdquo Proteins vol 40 pp 502ndash511 2000

[7] J M Chandonia and M Karplus ldquoNeural networks for sec-ondary structure and structural class predictionsrdquo ProteinScience vol 4 no 2 pp 275ndash285 1995

[8] J A Cuff and G J Barton ldquoEvaluation and improvement ofmultiple sequence methods for protein secondary structurepredictionrdquo Proteins vol 34 pp 508ndash519 1999

[9] G E Crooks and S E Brenner ldquoProtein secondary structureentropy correlations andpredictionrdquoBioinformatics vol 20 no10 pp 1603ndash1611 2004

[10] L H Wang J Liu and H B Zhou ldquoA comparison of twomachine learning methods for protein secondary structurepredictionrdquo in Proceedings of 2004 International Conference onMachine Learning and Cybernetics pp 2730ndash2735 chn August2004

Applied Computational Intelligence and Soft Computing 9

[11] G Z Zhang D S Huang Y P Zhu and Y X Li ldquoImprovingprotein secondary structure prediction by using the residueconformational classesrdquo Pattern Recognition Letters vol 26 no15 pp 2346ndash2352 2005

[12] NQian and T J Sejnowski ldquoPredicting the secondary structureof globular proteins using neural network modelsrdquo Journal ofMolecular Biology vol 202 no 4 pp 865ndash884 1988

[13] L HowardHolley andMKarplus ldquoProtein secondary structureprediction with a neural networkrdquo Proceedings of the NationalAcademy of Sciences of the United States of America vol 86 no1 pp 152ndash156 1989

[14] J M Chandonia and M Karplus ldquoThe importance of largerdata sets for protein secondary structure prediction with neuralnetworksrdquo Protein Science vol 5 no 4 pp 768ndash774 1996

[15] B Rost and C Sander ldquoImproved prediction of proteinsecondary structure by use of sequence profiles and neuralnetworksrdquoProceedings of theNational Academy of Sciences of theUnited States of America vol 90 no 16 pp 7558ndash7562 1993

[16] B Rost and C Sander ldquoPrediction of protein secondary struc-ture at better than 70 accuracyrdquo Journal of Molecular Biologyvol 232 no 2 pp 584ndash599 1993

[17] J A Cuff M E Clamp A S Siddiqui M Finlay and GJ Barton ldquoJPred a consensus secondary structure predictionserverrdquo Bioinformatics vol 14 no 10 pp 892ndash893 1998

[18] S Hua and Z Sun ldquoA novel method of protein secondary struc-ture prediction with high segment overlap measure supportvector machine approachrdquo Journal of Molecular Biology vol308 no 2 pp 397ndash407 2001

[19] B Rost ldquoPHD predicting one-dimensional protein structureby profile-based neural networksrdquoMethods in Enzymology vol266 pp 525ndash539 1996

[20] B Rost G Yachdav and J Liu ldquoThe PredictProtein serverrdquoNucleic Acids Research vol 32 pp W321ndashW326 2004

[21] C Cole J D Barber and G J Barton ldquoThe Jpred 3 secondarystructure prediction serverrdquo Nucleic Acids Research vol 36 ppW197ndashW201 2008

[22] G Pollastri D Przybylski B Rost and P Baldi ldquoImprovingthe prediction of protein secondary structure in three and eightclasses using recurrent neural networks and profilesrdquo Proteinsvol 47 no 2 pp 228ndash235 2002

[23] J Cheng A Z Randall M J Sweredoski and P BaldildquoSCRATCH a protein structure and structural feature predic-tion serverrdquoNucleic Acids Research vol 33 no 2 ppW72ndashW762005

[24] D T Jones ldquoProtein secondary structure prediction basedon position-specific scoring matricesrdquo Journal of MolecularBiology vol 292 no 2 pp 195ndash202 1999

[25] K Bryson L J McGuffin R L Marsden J J Ward J SSodhi and D T Jones ldquoProtein structure prediction servers atUniversity College Londonrdquo Nucleic Acids Research vol 33 no2 pp W36ndashW38 2005

[26] G Cybenko ldquoApproximation by superpositions of a sigmoidalfunctionrdquo Mathematics of Control Signals and Systems vol 2no 4 pp 303ndash314 1989

[27] J Moody and C J Darken ldquoFast learning in networks of locallytuned processing unitsrdquo Neural Computation vol 1 pp 281ndash294 1989

[28] T Poggio and F Girosi ldquoNetworks for approximation andlearningrdquo Proceedings of the IEEE vol 78 no 9 pp 1481ndash14971990

[29] D F Specht ldquoProbabilistic neural networksrdquo Neural Networksvol 3 no 1 pp 109ndash118 1990

[30] P Andras ldquoOrthogonal RBF neural network approximationrdquoNeural Processing Letters vol 9 no 2 pp 141ndash151 1999

[31] P Baldi and S Brunak Bioinformatics The Machine LearningApproach MIT Press 1998

[32] E Sakk D J Schneider C R Myers and SW Cartinhour ldquoOnthe selection of target vectors for a class of supervised patternrecognizersrdquo IEEE Transactions on Neural Networks vol 20 no5 pp 745ndash757 2009

[33] G H Golub and C F Van Loan Matrix Computations JohnsHopkins University Press 1989

Submit your manuscripts athttpwwwhindawicom

Computer Games Technology

International Journal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Distributed Sensor Networks

International Journal of

Advances in

FuzzySystems

Hindawi Publishing Corporationhttpwwwhindawicom

Volume 2014

International Journal of

ReconfigurableComputing

Hindawi Publishing Corporation httpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Applied Computational Intelligence and Soft Computing

thinspAdvancesthinspinthinsp

Artificial Intelligence

HindawithinspPublishingthinspCorporationhttpwwwhindawicom Volumethinsp2014

Advances inSoftware EngineeringHindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Electrical and Computer Engineering

Journal of

Journal of

Computer Networks and Communications

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporation

httpwwwhindawicom Volume 2014

Advances in

Multimedia

International Journal of

Biomedical Imaging

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

ArtificialNeural Systems

Advances in

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

RoboticsJournal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Computational Intelligence and Neuroscience

Industrial EngineeringJournal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Modelling amp Simulation in EngineeringHindawi Publishing Corporation httpwwwhindawicom Volume 2014

The Scientific World JournalHindawi Publishing Corporation httpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Human-ComputerInteraction

Advances in

Computer EngineeringAdvances in

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Applied Computational Intelligence and Soft Computing 5

Table 1 Ranking results for the linear network where 119894 representsthe test vector index and 120588

1(119909) and 120588

1(119909) represent the distance with

respect to the helix class vectors 1198901and 1198901 Out of 1800 vectors tested

vectors referred to in this table were ranked from 1 to 20

119894E 1205881(119909) 119894E 120588

1(119909)

1205 00780 1205 0117042 00867 42 013001031 00976 1031 014641773 01113 1773 01670598 01238 598 018571761 01267 1761 01900862 01354 862 020311073 01409 1073 02114277 01459 277 02188115 01540 115 023091505 01821 1505 02731392 01839 392 027591421 01904 1421 02856147 02001 147 03001990 02044 990 030661457 02127 1457 031911288 02150 1288 03225352 02160 352 032391232 02198 1232 03297280 02311 280 03466

classifier performance with their counterparts trained usingE Multiple cross-validation trials are required in order toprevent potential dependency of the evaluated accuracy onthe particular training or test sets chosen [7 15] In this workwe apply a hold-n-out strategy similar to that of [14] using85 of the 12000 encoded sequences as training data (ie119873 = 10200) and 15 as test data to validate the classificationresults Recognition rates for both the linear and backpropa-gation rates using either set of target vector encodings wereapproximately 65 which is typical of this genre of classifiersthat have applied similar encoding methodologies [7 12ndash14] Although these aggregate values remain consistent using(21) and (22) we now present data demonstrating that whileclass membership and ranking remain invariant for the linearnetwork these measures of performance vary considerablyfor the backpropagation network which was trained with119898 = 17 seventeen hidden nodes and amean squared trainingerror less than 02 Ranking results from a representative testfor the linear and backpropagation networks are presented forthe top 20 ranked vectors in Tables 1 and 3 Classmembershipdata are presented in Tables 2 and 4 Observe that forthe linear network indices for the top 20 ranked vectorsremain invariant indicating ranking invariance in additionno change in class membership is observed On the otherhand Tables 3 and 4 clearly indicate a lack of consistencywhen considering the ranking and class membership of testvectors A particularly troubling observation is that very fewvectors ranked in the top 20 with respect to 119890

1were ranked in

the top 20 with respect to 1198901 Furthermore Table 4 indicates

Table 2 Class membership results for the linear network For eachclass the total number of vectors classified using E is analyzed toexamine the total number that retained their classification using E

Class E E change119867 202 202 0119864 621 621 0119862 977 977 0

that the class membership of a substantial number of testvectors changed when an alternative set of target vectors wasemployed The data also indicates that the greatest changein class membership took place for alpha helical sequencesthus implying that there is substantial disagreement overthe modeling of this secondary structure element by thebackpropagation network due to a simple transformation ofthe target vectors

5 Analysis

The results in Section 4 clearly show that while the patternrecognition results for the linear network remain invariantunder a change in target vectors those for the backpropa-gation network do not In this section we present analyticresults in order to clearly explain and understand why thesetwo techniques lead to different conclusions

51 Invariance Formulation Let us begin by considering twodefinitions

Definition 1 Given two sets of target vectors E and E theclass membership is invariant under a transformation of thetarget vectors if for any input vector 119909

119889(119891 (119909) 119890119894) gt 119889(119891 (119909) 119890

119895)997904rArr119889 (119891 (119909) 119890

119894) gt 119889(119891 (119909) 119890

119895)

(23)

where 119891(119909) is the output of the classifier with target vectors119890119895119872

119895=1

Definition 2 Given two sets of target vectors E and E theranking with respect to a specific class 119869 is invariant under atransformation of the target vectors if for any input vectors1199091and 119909

2

119889(119891(1199091) 119890119869) gt 119889(119891(119909

2) 119890119869)997904rArr119889(119891 (119909

1) 119890119869) gt 119889(119891 (119909

2) 119890119869)

(24)

Based upon these definitions the following has beenestablished [32]

Proposition 3 Given two sets of target vectorsE and E if theranking is invariant then the class membership of an arbitraryinput vector 119909 will remain invariant

In the analysis presented the strategy for characterizingneural network performance depends upon the data from theprevious section For the linear network since both ranking

6 Applied Computational Intelligence and Soft Computing

Table 3 Ranking results for the backpropagation network where 119894represents the test vector index and 120588

1(119909) and 120588

1(119909) represent the

distance with respect to the helix class vectors 1198901and 1198901 Out of 1800

vectors tested vectors referred to in this table were ranked from 1 to20

119894E 1205881(119909) 119894E 120588

1(119909)

817 00107 926 00101887 00231 1604 00130264 00405 887 002091183 00711 1145 00214684 00727 461 00232623 00874 583 00329911 00891 1086 003391382 00917 1382 004781610 00939 413 00489551 01060 225 006081042 01150 438 00609924 01322 911 00613727 01339 207 00774438 01356 559 00885577 01363 481 00945896 01500 1548 00947175 01513 962 009681138 01549 85 01012583 01581 195 01111559 01655 9 01167

Table 4 Class membership results for the backpropagationnetwork For each class the total number of vectors classified usingE is analyzed to examine the total number that retained theirclassification using E

Class E E change119867 225 142 369119864 581 476 181119862 994 878 117

and classification were observed to remain invariant it ismore sensible to characterize the invariance of this networkusing Definition 2 Then based upon Proposition 3 classmembership invariance naturally follows On the other handto explain the noninvariance of both class membership andranking observed in the backpropagation network the anal-ysis is facilitated by considering Definition 1 The noninvari-ance of ranking then naturally follows from Proposition 3

511 Invariance Analysis for the Linear Network When thetarget vectors are subjected to the transformation defined in(2) the network output can be expressed as

119891 (119911) = 119911 = 119860dagger119911 = 119886119877 (119884 minus 119879

1015840)119860dagger119911 (25)

where 1198791015840 is derived from 119879 such that the translation vector119905119894associated with 119890

119894isin E is appropriately aligned with the

correct target vector in matrix 119884 In other words when theoutput data matrix in (7) is of the form

119884 = [1198901198941

1198901198942

119890119894119873

] (26)

then

1198791015840= [1199051198941

1199051198942

119905119894119873

] (27)

where 119894119895isin 1 119872 for 119895 = 1 119873 Given this network

the following result is applicable [32]

Proposition 4 If

(i) the number of training observations 119873 exceeds thevector dimension 119899

(ii) the rows of the matrix 119860 are linearly independent(iii) E and E are related according to (2)(iv) for some 119905 isin 119877119901 119905

119894= 119905 for all 119894 = 1 119872 in (2)

then the ranking and hence the classmembership for the linearnetwork will remain invariant

In other words if the columns of the matrix 1198791015840 in (25)are all equal then using (15) and (25) will result in (23)being satisfiedThe above result is applicable to the presentednumerical data with 119899 = 340 and119873 = 10200 hence rankingand class membership invariances are corroborated by thedata in Tables 1 and 2

512 Invariance Analysis for the Backpropagation NetworkIn this section we seek to characterize the noninvarianceobserved in class membership using the backpropagationnetwork If the class membership varies due to a changein target vectors then this variation should be quantifiableby characterizing the boundary separating two respectiveclasses The decision boundary between class 119894 and class 119895 isdefined by points 119909 such that

1003817100381710038171003817119891 (119909) minus 11989011989410038171003817100381710038172

=10038171003817100381710038171003817119891 (119909) minus 119890

119895

10038171003817100381710038171003817

2

(28)

where 119894 119895 isin 1 119872 119894 = 119895 and 119891(119909) is the classifier outputUnder these circumstances if an ℓ

2norm is applied in (21)

(119891 (119909) minus 119890119894)119879

(119891 (119909) minus 119890119894) = (119891 (119909) minus 119890

119895)119879

(119891 (119909) minus 119890119895)

(29)

The solution set to this equation consists of all 119909 such that119891(119909) is equidistant from 119890

119894and 119890119895where for the purposes of

this section119891(119909) is defined by (16) Expanding terms on bothsides of this equation leads to the condition

(119890119894minus 119890119895)119879

119891 (119909) = minus1

2(119890119879

119895119890119895minus 119890119879

119894119890119894) (30)

If the classmembership of a representative vector is to remaininvariant under a change of target vectors this same set ofpoints must also satisfy

( 119890119894minus 119890119895)119879

119891 (119909) = minus1

2( 119890119879

119895119890119895minus 119890119879

119894119890119894) (31)

Applied Computational Intelligence and Soft Computing 7

Assuming that two networks have been trained using twodifferent sets of target vectors E and E the set of weights119882 120591 120572 and 120591 determines the network output in (16)Without loss of generality we consider the case where alltarget vectors are normalized to a value of one such that1198901198942= 1 and 119890

1198942= 1 for 119894 = 1 119872 In this case the

conditions in (30) and (31) become

(119890119894minus 119890119895)119879

120572120590 (119882119909 minus 120591) = 0 (32)

(119890119894minus 119890119895)119879

120590 (119909 minus 120591) = 0 (33)

We first consider a special case where the target vectorsare related according to Γ = 119886119877Γ with 119879 = 0 in (2) Underthese circumstances if the choice

W equiv 120591 = 119882 120591 119886119877120572 (34)

is made it should be clear that since119877119879 = 119877minus1 and 119894= 119886119877119884

119894

119864119887119901=

119873

sum119894=1

(119891 (119883119894) minus 119894)119879

(119891 (119883119894) minus 119894)

= 1198862

119873

sum119894=1

(119877119891 (119883119894) minus 119877119884

119894)119879

(119877119891 (119883119894) minus 119877119884

119894)

= 1198862

119873

sum119894=1

(119891 (119883119894) minus 119884119894)119879

119877119879119877 (119891 (119883

119894) minus 119884119894)

= 1198862119864119887119901

(35)

is minimized by the condition 120597119864119887119901120597W = 0 Another way tosee this is to observe that (19) (20) remain invariant for thischoice of target vectors and network weights Hence we havethe following

Proposition 5 For a specific choice of 119886 and 119877 if 119879 = 0 in (2)and

W = 120591 = 119882 120591 119886119877120572 (36)

then the classmembership for the backpropagation networkwillremain invariant

Proof Simply consider (32) and (33) and choose any 119909satisfying

(119890119894minus 119890119895)119879

120572120590 (119882119909 minus 120591) = 0 (37)

It then immediately follows that

(119890119894minus 119890119895)119879

120590 (119909 minus 120591)

= (119890119894minus 119890119895)119879

119877119879119877120572120590 (119882119909 minus 120591)

= 1198862(119890119894minus 119890119895)119879

120572120590 (119882119909 minus 120591)

= 0

(38)

Therefore if 119909 satisfies (32) then it also satisfies (33) andhence is a point on the decision boundary for both net-works

Intuitively a scaled rigid rotation of the target vectorsshould not affect the decision boundary However when themore general transformation of (2) is applied with 119879 = 0we now demonstrate that due to the nonlinearity of 120590 in(16) no simple relationship exists such that (32) and (33)can simultaneously be satisfied by the same set of pointsWe first investigate the possibility of establishing an analyticrelationship between the set of weights 120591 and 119882 120591 120572for both networks In other words we seek ideally invertiblefunctions 119866

120572 119866120591 and 119866

119882

= 119866120572(119882 120591 120572)

120591 = 119866120591(119882 120591 120572)

= 119866119882(119882 120591 120572)

(39)

such that the setW can be transformed into W If this can bedone then an analytic procedure similar to that presented inthe proof of Proposition 5 can be established in order to relate(32) to (33) for the general case Since (19) (20) define the setW it is reasonable to rephrase these equations in terms of theobjective function 119864119887119901

120597119864119887119901

120597=

119873

sum119896=1

(119891 (119883119896) minus 119896) 119878119879

119896= 0

120597119864119887119901

120597120591= minus

119873

sum119896=1

diag (1198781015840119896) 119879(119891 (119883

119896) minus 119896)119879

= 0

120597119864119887119901

120597=

119873

sum119896=1

diag (1198781015840119896) 119879(119891 (119883

119896) minus 119896)119879

119883119879

119896= 0

(40)

where 119891(119883119896) = 120590(119883

119896minus 120591) and

119896= 119886119877(119884

119896minus 1199051015840) such that

1199051015840 is the translation vector associated with the target vectorreferred to by 119884

119896 From these equations it should be clear

that no simple analytic relationship exists that will transformW into W A numerical algorithm such as gradient descentwill assuming a localminimumactually exists arrive at somesolution for both W and W We must therefore be contentwith the assumed existence of some set of functions definedby (39) Again let us consider any point 119909 on the decisionboundary such that

(119890119894minus 119890119895)119879

120572120590 (119882119909 minus 120591) = 0 (41)

Such a point must also simultaneously satisfy

(119890119894minus 119890119895)119879

120590 (119909 minus 120591)

= ((119886119877119890119894minus 119905119894) minus (119886119877119890

119895minus 119905119895))119879

119866120572(119882 120591 120572)

8 Applied Computational Intelligence and Soft Computing

times 120590 (119866119882(119882 120591 120572) 119909 minus 119866

120591(119882 120591 120572))

= 119886((119890119894minus 119890119895) minus (119905

119894minus 119905119895))119879

119877119879119866120572(119882 120591 120572)

times 120590 (119866119882(119882 120591 120572) 119909 minus 119866

120591(119882 120591 120572))

= 0

(42)

At first glance a choice such as 119866120572(119882 120591 120572) = 119886119877120572 and

119905119894= 119905119895for 119894 119895 = 1 119872 (as in Proposition 4) appears

to bring us close to a solution However the term involving120590(119866119882(119882 120591 120572)119909 minus 119866

120591(119882 120591 120572)) is problematic Although a

choice such as 119866119882(119882 120591 120572) = 119882 and 119866

120591(119882 120591 120572) = 120591 would

yield the solution to (42) it should be clear that these valueswould not satisfy (40)

Another way to analyze this problem is to first set 119905119894= 119905119895

for 119894 119895 = 1 119872 Then for any 119909 on the decision boundaryfrom (41) and (42) equate the terms

120572120590 (119882119909 minus 120591) = 119896119877119879119866120572(119882 120591 120572)

times 120590 (119866119882(119882 120591 120572) 119909 minus 119866

120591(119882 120591 120572))

(43)

where 119896 is some constant and 119866120572 119866120591 and 119866

119882satisfy (40)

Given an arbitrary training set defined by (1) it is highlyunlikely that this constraint can be satisfied One remotescenario might be that the terms 119866

119882(119882 120591 120572)119909 minus 119866

120591(119882 120591 120572)

and119882119909 minus 120591 are always small In this case given a sigmoidalfunction 120590(119911) that is linear near 119911 = 0 a linearized versionof (43) could be solved using techniques described in [32]However this again is an unlikely set of events given anarbitrary training set Therefore given the transformation of(2) we are left to conclude that class membership invarianceand hence ranking invariance are in general not achievableusing the backpropagation neural network

52 Discussion Intuitively given a reasonable target encod-ing scheme one would desire that properties related to pro-tein secondary structure would be independent of the targetvectors chosen However we have presented numerical dataand a theoretical foundation demonstrating that secondarystructure classification and confidence measures can varydepending on the type of neural network architecture andtarget vector encoding scheme employed Specifically linearnetwork classification has been demonstrated to remaininvariant under a change in the target structure encodingscheme while the backpropagation network has not As119873 increases for the methodology applied in this workrecognition rates remain consistent with those reported inthe literature however we have observed that adding moretraining data does not improve the invariance of classificationmeasures for the backpropagation network This conclusionis corroborated by the analytic formulation presented above

6 Conclusions

As pointed out in the introduction one major purpose of theneural network is to create a stable and reliable model that

maps input training data to an output classification with thehope of extracting informative parameters When methodssimilar to those in the literature are applied [7 12ndash14] wehave demonstrated that classifier performance measures canvary considerably Under these circumstances parametersderived from a trained network for analytically describingprotein secondary structure may not comprise a reliableset for the model-based approach Furthermore classifiervariabilitywould imply that a stable parametricmodel has notbeen derived It is in some sense paradoxical that the neuralnetwork has been applied for structure classification and yetassociated parameters have not been applied for describingprotein secondary structure The neural network approachto deriving a solution to the protein secondary structureprediction problem therefore requires deeper exploration

Acknowledgments

This publication was made possible by Grant NumberG12RR017581 from the National Center for ResearchResources (NCRR) a component of the National Institutesof Health (NIH) The authors would also like to thank thereviewers for their helpful comments

References

[1] C M Bishop Pattern Recognition and Machine LearningSpringer 2007

[2] M T Hagan H B Demuth and M H Beale Neural NetworkDesign PWS Publishing 1996

[3] M N Nguyen and J C Rajapakse ldquoMulti-class support vectormachines for protein secondary structure predictionrdquo GenomeInformatics vol 14 pp 218ndash227 2003

[4] H J Hu Y Pan R Harrison and P C Tai ldquoImprovedprotein secondary structure prediction using support vectormachine with a new encoding scheme and an advanced tertiaryclassifierrdquo IEEE Transactions onNanobioscience vol 3 no 4 pp265ndash271 2004

[5] W Zhong G Altun X Tian R Harrison P C Tai and YPan ldquoParallel protein secondary structure prediction schemesusing Pthread and OpenMP over hyper-threading technologyrdquoJournal of Supercomputing vol 41 no 1 pp 1ndash16 2007

[6] J A Cuff and G J Barton ldquoApplication of enhanced multiplesequence alignment profiles to improve protein secondarystructure predictionrdquo Proteins vol 40 pp 502ndash511 2000

[7] J M Chandonia and M Karplus ldquoNeural networks for sec-ondary structure and structural class predictionsrdquo ProteinScience vol 4 no 2 pp 275ndash285 1995

[8] J A Cuff and G J Barton ldquoEvaluation and improvement ofmultiple sequence methods for protein secondary structurepredictionrdquo Proteins vol 34 pp 508ndash519 1999

[9] G E Crooks and S E Brenner ldquoProtein secondary structureentropy correlations andpredictionrdquoBioinformatics vol 20 no10 pp 1603ndash1611 2004

[10] L H Wang J Liu and H B Zhou ldquoA comparison of twomachine learning methods for protein secondary structurepredictionrdquo in Proceedings of 2004 International Conference onMachine Learning and Cybernetics pp 2730ndash2735 chn August2004

Applied Computational Intelligence and Soft Computing 9

[11] G Z Zhang D S Huang Y P Zhu and Y X Li ldquoImprovingprotein secondary structure prediction by using the residueconformational classesrdquo Pattern Recognition Letters vol 26 no15 pp 2346ndash2352 2005

[12] NQian and T J Sejnowski ldquoPredicting the secondary structureof globular proteins using neural network modelsrdquo Journal ofMolecular Biology vol 202 no 4 pp 865ndash884 1988

[13] L HowardHolley andMKarplus ldquoProtein secondary structureprediction with a neural networkrdquo Proceedings of the NationalAcademy of Sciences of the United States of America vol 86 no1 pp 152ndash156 1989

[14] J M Chandonia and M Karplus ldquoThe importance of largerdata sets for protein secondary structure prediction with neuralnetworksrdquo Protein Science vol 5 no 4 pp 768ndash774 1996

[15] B Rost and C Sander ldquoImproved prediction of proteinsecondary structure by use of sequence profiles and neuralnetworksrdquoProceedings of theNational Academy of Sciences of theUnited States of America vol 90 no 16 pp 7558ndash7562 1993

[16] B Rost and C Sander ldquoPrediction of protein secondary struc-ture at better than 70 accuracyrdquo Journal of Molecular Biologyvol 232 no 2 pp 584ndash599 1993

[17] J A Cuff M E Clamp A S Siddiqui M Finlay and GJ Barton ldquoJPred a consensus secondary structure predictionserverrdquo Bioinformatics vol 14 no 10 pp 892ndash893 1998

[18] S Hua and Z Sun ldquoA novel method of protein secondary struc-ture prediction with high segment overlap measure supportvector machine approachrdquo Journal of Molecular Biology vol308 no 2 pp 397ndash407 2001

[19] B Rost ldquoPHD predicting one-dimensional protein structureby profile-based neural networksrdquoMethods in Enzymology vol266 pp 525ndash539 1996

[20] B Rost G Yachdav and J Liu ldquoThe PredictProtein serverrdquoNucleic Acids Research vol 32 pp W321ndashW326 2004

[21] C Cole J D Barber and G J Barton ldquoThe Jpred 3 secondarystructure prediction serverrdquo Nucleic Acids Research vol 36 ppW197ndashW201 2008

[22] G Pollastri D Przybylski B Rost and P Baldi ldquoImprovingthe prediction of protein secondary structure in three and eightclasses using recurrent neural networks and profilesrdquo Proteinsvol 47 no 2 pp 228ndash235 2002

[23] J Cheng A Z Randall M J Sweredoski and P BaldildquoSCRATCH a protein structure and structural feature predic-tion serverrdquoNucleic Acids Research vol 33 no 2 ppW72ndashW762005

[24] D T Jones ldquoProtein secondary structure prediction basedon position-specific scoring matricesrdquo Journal of MolecularBiology vol 292 no 2 pp 195ndash202 1999

[25] K Bryson L J McGuffin R L Marsden J J Ward J SSodhi and D T Jones ldquoProtein structure prediction servers atUniversity College Londonrdquo Nucleic Acids Research vol 33 no2 pp W36ndashW38 2005

[26] G Cybenko ldquoApproximation by superpositions of a sigmoidalfunctionrdquo Mathematics of Control Signals and Systems vol 2no 4 pp 303ndash314 1989

[27] J Moody and C J Darken ldquoFast learning in networks of locallytuned processing unitsrdquo Neural Computation vol 1 pp 281ndash294 1989

[28] T Poggio and F Girosi ldquoNetworks for approximation andlearningrdquo Proceedings of the IEEE vol 78 no 9 pp 1481ndash14971990

[29] D F Specht ldquoProbabilistic neural networksrdquo Neural Networksvol 3 no 1 pp 109ndash118 1990

[30] P Andras ldquoOrthogonal RBF neural network approximationrdquoNeural Processing Letters vol 9 no 2 pp 141ndash151 1999

[31] P Baldi and S Brunak Bioinformatics The Machine LearningApproach MIT Press 1998

[32] E Sakk D J Schneider C R Myers and SW Cartinhour ldquoOnthe selection of target vectors for a class of supervised patternrecognizersrdquo IEEE Transactions on Neural Networks vol 20 no5 pp 745ndash757 2009

[33] G H Golub and C F Van Loan Matrix Computations JohnsHopkins University Press 1989

Submit your manuscripts athttpwwwhindawicom

Computer Games Technology

International Journal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Distributed Sensor Networks

International Journal of

Advances in

FuzzySystems

Hindawi Publishing Corporationhttpwwwhindawicom

Volume 2014

International Journal of

ReconfigurableComputing

Hindawi Publishing Corporation httpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Applied Computational Intelligence and Soft Computing

thinspAdvancesthinspinthinsp

Artificial Intelligence

HindawithinspPublishingthinspCorporationhttpwwwhindawicom Volumethinsp2014

Advances inSoftware EngineeringHindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Electrical and Computer Engineering

Journal of

Journal of

Computer Networks and Communications

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporation

httpwwwhindawicom Volume 2014

Advances in

Multimedia

International Journal of

Biomedical Imaging

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

ArtificialNeural Systems

Advances in

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

RoboticsJournal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Computational Intelligence and Neuroscience

Industrial EngineeringJournal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Modelling amp Simulation in EngineeringHindawi Publishing Corporation httpwwwhindawicom Volume 2014

The Scientific World JournalHindawi Publishing Corporation httpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Human-ComputerInteraction

Advances in

Computer EngineeringAdvances in

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

6 Applied Computational Intelligence and Soft Computing

Table 3 Ranking results for the backpropagation network where 119894represents the test vector index and 120588

1(119909) and 120588

1(119909) represent the

distance with respect to the helix class vectors 1198901and 1198901 Out of 1800

vectors tested vectors referred to in this table were ranked from 1 to20

119894E 1205881(119909) 119894E 120588

1(119909)

817 00107 926 00101887 00231 1604 00130264 00405 887 002091183 00711 1145 00214684 00727 461 00232623 00874 583 00329911 00891 1086 003391382 00917 1382 004781610 00939 413 00489551 01060 225 006081042 01150 438 00609924 01322 911 00613727 01339 207 00774438 01356 559 00885577 01363 481 00945896 01500 1548 00947175 01513 962 009681138 01549 85 01012583 01581 195 01111559 01655 9 01167

Table 4 Class membership results for the backpropagationnetwork For each class the total number of vectors classified usingE is analyzed to examine the total number that retained theirclassification using E

Class E E change119867 225 142 369119864 581 476 181119862 994 878 117

and classification were observed to remain invariant it ismore sensible to characterize the invariance of this networkusing Definition 2 Then based upon Proposition 3 classmembership invariance naturally follows On the other handto explain the noninvariance of both class membership andranking observed in the backpropagation network the anal-ysis is facilitated by considering Definition 1 The noninvari-ance of ranking then naturally follows from Proposition 3

511 Invariance Analysis for the Linear Network When thetarget vectors are subjected to the transformation defined in(2) the network output can be expressed as

119891 (119911) = 119911 = 119860dagger119911 = 119886119877 (119884 minus 119879

1015840)119860dagger119911 (25)

where 1198791015840 is derived from 119879 such that the translation vector119905119894associated with 119890

119894isin E is appropriately aligned with the

correct target vector in matrix 119884 In other words when theoutput data matrix in (7) is of the form

119884 = [1198901198941

1198901198942

119890119894119873

] (26)

then

1198791015840= [1199051198941

1199051198942

119905119894119873

] (27)

where 119894119895isin 1 119872 for 119895 = 1 119873 Given this network

the following result is applicable [32]

Proposition 4 If

(i) the number of training observations 119873 exceeds thevector dimension 119899

(ii) the rows of the matrix 119860 are linearly independent(iii) E and E are related according to (2)(iv) for some 119905 isin 119877119901 119905

119894= 119905 for all 119894 = 1 119872 in (2)

then the ranking and hence the classmembership for the linearnetwork will remain invariant

In other words if the columns of the matrix 1198791015840 in (25)are all equal then using (15) and (25) will result in (23)being satisfiedThe above result is applicable to the presentednumerical data with 119899 = 340 and119873 = 10200 hence rankingand class membership invariances are corroborated by thedata in Tables 1 and 2

512 Invariance Analysis for the Backpropagation NetworkIn this section we seek to characterize the noninvarianceobserved in class membership using the backpropagationnetwork If the class membership varies due to a changein target vectors then this variation should be quantifiableby characterizing the boundary separating two respectiveclasses The decision boundary between class 119894 and class 119895 isdefined by points 119909 such that

1003817100381710038171003817119891 (119909) minus 11989011989410038171003817100381710038172

=10038171003817100381710038171003817119891 (119909) minus 119890

119895

10038171003817100381710038171003817

2

(28)

where 119894 119895 isin 1 119872 119894 = 119895 and 119891(119909) is the classifier outputUnder these circumstances if an ℓ

2norm is applied in (21)

(119891 (119909) minus 119890119894)119879

(119891 (119909) minus 119890119894) = (119891 (119909) minus 119890

119895)119879

(119891 (119909) minus 119890119895)

(29)

The solution set to this equation consists of all 119909 such that119891(119909) is equidistant from 119890

119894and 119890119895where for the purposes of

this section119891(119909) is defined by (16) Expanding terms on bothsides of this equation leads to the condition

(119890119894minus 119890119895)119879

119891 (119909) = minus1

2(119890119879

119895119890119895minus 119890119879

119894119890119894) (30)

If the classmembership of a representative vector is to remaininvariant under a change of target vectors this same set ofpoints must also satisfy

( 119890119894minus 119890119895)119879

119891 (119909) = minus1

2( 119890119879

119895119890119895minus 119890119879

119894119890119894) (31)

Applied Computational Intelligence and Soft Computing 7

Assuming that two networks have been trained using twodifferent sets of target vectors E and E the set of weights119882 120591 120572 and 120591 determines the network output in (16)Without loss of generality we consider the case where alltarget vectors are normalized to a value of one such that1198901198942= 1 and 119890

1198942= 1 for 119894 = 1 119872 In this case the

conditions in (30) and (31) become

(119890119894minus 119890119895)119879

120572120590 (119882119909 minus 120591) = 0 (32)

(119890119894minus 119890119895)119879

120590 (119909 minus 120591) = 0 (33)

We first consider a special case where the target vectorsare related according to Γ = 119886119877Γ with 119879 = 0 in (2) Underthese circumstances if the choice

W equiv 120591 = 119882 120591 119886119877120572 (34)

is made it should be clear that since119877119879 = 119877minus1 and 119894= 119886119877119884

119894

119864119887119901=

119873

sum119894=1

(119891 (119883119894) minus 119894)119879

(119891 (119883119894) minus 119894)

= 1198862

119873

sum119894=1

(119877119891 (119883119894) minus 119877119884

119894)119879

(119877119891 (119883119894) minus 119877119884

119894)

= 1198862

119873

sum119894=1

(119891 (119883119894) minus 119884119894)119879

119877119879119877 (119891 (119883

119894) minus 119884119894)

= 1198862119864119887119901

(35)

is minimized by the condition 120597119864119887119901120597W = 0 Another way tosee this is to observe that (19) (20) remain invariant for thischoice of target vectors and network weights Hence we havethe following

Proposition 5 For a specific choice of 119886 and 119877 if 119879 = 0 in (2)and

W = 120591 = 119882 120591 119886119877120572 (36)

then the classmembership for the backpropagation networkwillremain invariant

Proof Simply consider (32) and (33) and choose any 119909satisfying

(119890119894minus 119890119895)119879

120572120590 (119882119909 minus 120591) = 0 (37)

It then immediately follows that

(119890119894minus 119890119895)119879

120590 (119909 minus 120591)

= (119890119894minus 119890119895)119879

119877119879119877120572120590 (119882119909 minus 120591)

= 1198862(119890119894minus 119890119895)119879

120572120590 (119882119909 minus 120591)

= 0

(38)

Therefore if 119909 satisfies (32) then it also satisfies (33) andhence is a point on the decision boundary for both net-works

Intuitively a scaled rigid rotation of the target vectorsshould not affect the decision boundary However when themore general transformation of (2) is applied with 119879 = 0we now demonstrate that due to the nonlinearity of 120590 in(16) no simple relationship exists such that (32) and (33)can simultaneously be satisfied by the same set of pointsWe first investigate the possibility of establishing an analyticrelationship between the set of weights 120591 and 119882 120591 120572for both networks In other words we seek ideally invertiblefunctions 119866

120572 119866120591 and 119866

119882

= 119866120572(119882 120591 120572)

120591 = 119866120591(119882 120591 120572)

= 119866119882(119882 120591 120572)

(39)

such that the setW can be transformed into W If this can bedone then an analytic procedure similar to that presented inthe proof of Proposition 5 can be established in order to relate(32) to (33) for the general case Since (19) (20) define the setW it is reasonable to rephrase these equations in terms of theobjective function 119864119887119901

120597119864119887119901

120597=

119873

sum119896=1

(119891 (119883119896) minus 119896) 119878119879

119896= 0

120597119864119887119901

120597120591= minus

119873

sum119896=1

diag (1198781015840119896) 119879(119891 (119883

119896) minus 119896)119879

= 0

120597119864119887119901

120597=

119873

sum119896=1

diag (1198781015840119896) 119879(119891 (119883

119896) minus 119896)119879

119883119879

119896= 0

(40)

where 119891(119883119896) = 120590(119883

119896minus 120591) and

119896= 119886119877(119884

119896minus 1199051015840) such that

1199051015840 is the translation vector associated with the target vectorreferred to by 119884

119896 From these equations it should be clear

that no simple analytic relationship exists that will transformW into W A numerical algorithm such as gradient descentwill assuming a localminimumactually exists arrive at somesolution for both W and W We must therefore be contentwith the assumed existence of some set of functions definedby (39) Again let us consider any point 119909 on the decisionboundary such that

(119890119894minus 119890119895)119879

120572120590 (119882119909 minus 120591) = 0 (41)

Such a point must also simultaneously satisfy

(119890119894minus 119890119895)119879

120590 (119909 minus 120591)

= ((119886119877119890119894minus 119905119894) minus (119886119877119890

119895minus 119905119895))119879

119866120572(119882 120591 120572)

8 Applied Computational Intelligence and Soft Computing

times 120590 (119866119882(119882 120591 120572) 119909 minus 119866

120591(119882 120591 120572))

= 119886((119890119894minus 119890119895) minus (119905

119894minus 119905119895))119879

119877119879119866120572(119882 120591 120572)

times 120590 (119866119882(119882 120591 120572) 119909 minus 119866

120591(119882 120591 120572))

= 0

(42)

At first glance a choice such as 119866120572(119882 120591 120572) = 119886119877120572 and

119905119894= 119905119895for 119894 119895 = 1 119872 (as in Proposition 4) appears

to bring us close to a solution However the term involving120590(119866119882(119882 120591 120572)119909 minus 119866

120591(119882 120591 120572)) is problematic Although a

choice such as 119866119882(119882 120591 120572) = 119882 and 119866

120591(119882 120591 120572) = 120591 would

yield the solution to (42) it should be clear that these valueswould not satisfy (40)

Another way to analyze this problem is to first set 119905119894= 119905119895

for 119894 119895 = 1 119872 Then for any 119909 on the decision boundaryfrom (41) and (42) equate the terms

120572120590 (119882119909 minus 120591) = 119896119877119879119866120572(119882 120591 120572)

times 120590 (119866119882(119882 120591 120572) 119909 minus 119866

120591(119882 120591 120572))

(43)

where 119896 is some constant and 119866120572 119866120591 and 119866

119882satisfy (40)

Given an arbitrary training set defined by (1) it is highlyunlikely that this constraint can be satisfied One remotescenario might be that the terms 119866

119882(119882 120591 120572)119909 minus 119866

120591(119882 120591 120572)

and119882119909 minus 120591 are always small In this case given a sigmoidalfunction 120590(119911) that is linear near 119911 = 0 a linearized versionof (43) could be solved using techniques described in [32]However this again is an unlikely set of events given anarbitrary training set Therefore given the transformation of(2) we are left to conclude that class membership invarianceand hence ranking invariance are in general not achievableusing the backpropagation neural network

52 Discussion Intuitively given a reasonable target encod-ing scheme one would desire that properties related to pro-tein secondary structure would be independent of the targetvectors chosen However we have presented numerical dataand a theoretical foundation demonstrating that secondarystructure classification and confidence measures can varydepending on the type of neural network architecture andtarget vector encoding scheme employed Specifically linearnetwork classification has been demonstrated to remaininvariant under a change in the target structure encodingscheme while the backpropagation network has not As119873 increases for the methodology applied in this workrecognition rates remain consistent with those reported inthe literature however we have observed that adding moretraining data does not improve the invariance of classificationmeasures for the backpropagation network This conclusionis corroborated by the analytic formulation presented above

6 Conclusions

As pointed out in the introduction one major purpose of theneural network is to create a stable and reliable model that

maps input training data to an output classification with thehope of extracting informative parameters When methodssimilar to those in the literature are applied [7 12ndash14] wehave demonstrated that classifier performance measures canvary considerably Under these circumstances parametersderived from a trained network for analytically describingprotein secondary structure may not comprise a reliableset for the model-based approach Furthermore classifiervariabilitywould imply that a stable parametricmodel has notbeen derived It is in some sense paradoxical that the neuralnetwork has been applied for structure classification and yetassociated parameters have not been applied for describingprotein secondary structure The neural network approachto deriving a solution to the protein secondary structureprediction problem therefore requires deeper exploration

Acknowledgments

This publication was made possible by Grant NumberG12RR017581 from the National Center for ResearchResources (NCRR) a component of the National Institutesof Health (NIH) The authors would also like to thank thereviewers for their helpful comments

References

[1] C M Bishop Pattern Recognition and Machine LearningSpringer 2007

[2] M T Hagan H B Demuth and M H Beale Neural NetworkDesign PWS Publishing 1996

[3] M N Nguyen and J C Rajapakse ldquoMulti-class support vectormachines for protein secondary structure predictionrdquo GenomeInformatics vol 14 pp 218ndash227 2003

[4] H J Hu Y Pan R Harrison and P C Tai ldquoImprovedprotein secondary structure prediction using support vectormachine with a new encoding scheme and an advanced tertiaryclassifierrdquo IEEE Transactions onNanobioscience vol 3 no 4 pp265ndash271 2004

[5] W Zhong G Altun X Tian R Harrison P C Tai and YPan ldquoParallel protein secondary structure prediction schemesusing Pthread and OpenMP over hyper-threading technologyrdquoJournal of Supercomputing vol 41 no 1 pp 1ndash16 2007

[6] J A Cuff and G J Barton ldquoApplication of enhanced multiplesequence alignment profiles to improve protein secondarystructure predictionrdquo Proteins vol 40 pp 502ndash511 2000

[7] J M Chandonia and M Karplus ldquoNeural networks for sec-ondary structure and structural class predictionsrdquo ProteinScience vol 4 no 2 pp 275ndash285 1995

[8] J A Cuff and G J Barton ldquoEvaluation and improvement ofmultiple sequence methods for protein secondary structurepredictionrdquo Proteins vol 34 pp 508ndash519 1999

[9] G E Crooks and S E Brenner ldquoProtein secondary structureentropy correlations andpredictionrdquoBioinformatics vol 20 no10 pp 1603ndash1611 2004

[10] L H Wang J Liu and H B Zhou ldquoA comparison of twomachine learning methods for protein secondary structurepredictionrdquo in Proceedings of 2004 International Conference onMachine Learning and Cybernetics pp 2730ndash2735 chn August2004

Applied Computational Intelligence and Soft Computing 9

[11] G Z Zhang D S Huang Y P Zhu and Y X Li ldquoImprovingprotein secondary structure prediction by using the residueconformational classesrdquo Pattern Recognition Letters vol 26 no15 pp 2346ndash2352 2005

[12] NQian and T J Sejnowski ldquoPredicting the secondary structureof globular proteins using neural network modelsrdquo Journal ofMolecular Biology vol 202 no 4 pp 865ndash884 1988

[13] L HowardHolley andMKarplus ldquoProtein secondary structureprediction with a neural networkrdquo Proceedings of the NationalAcademy of Sciences of the United States of America vol 86 no1 pp 152ndash156 1989

[14] J M Chandonia and M Karplus ldquoThe importance of largerdata sets for protein secondary structure prediction with neuralnetworksrdquo Protein Science vol 5 no 4 pp 768ndash774 1996

[15] B Rost and C Sander ldquoImproved prediction of proteinsecondary structure by use of sequence profiles and neuralnetworksrdquoProceedings of theNational Academy of Sciences of theUnited States of America vol 90 no 16 pp 7558ndash7562 1993

[16] B Rost and C Sander ldquoPrediction of protein secondary struc-ture at better than 70 accuracyrdquo Journal of Molecular Biologyvol 232 no 2 pp 584ndash599 1993

[17] J A Cuff M E Clamp A S Siddiqui M Finlay and GJ Barton ldquoJPred a consensus secondary structure predictionserverrdquo Bioinformatics vol 14 no 10 pp 892ndash893 1998

[18] S Hua and Z Sun ldquoA novel method of protein secondary struc-ture prediction with high segment overlap measure supportvector machine approachrdquo Journal of Molecular Biology vol308 no 2 pp 397ndash407 2001

[19] B Rost ldquoPHD predicting one-dimensional protein structureby profile-based neural networksrdquoMethods in Enzymology vol266 pp 525ndash539 1996

[20] B Rost G Yachdav and J Liu ldquoThe PredictProtein serverrdquoNucleic Acids Research vol 32 pp W321ndashW326 2004

[21] C Cole J D Barber and G J Barton ldquoThe Jpred 3 secondarystructure prediction serverrdquo Nucleic Acids Research vol 36 ppW197ndashW201 2008

[22] G Pollastri D Przybylski B Rost and P Baldi ldquoImprovingthe prediction of protein secondary structure in three and eightclasses using recurrent neural networks and profilesrdquo Proteinsvol 47 no 2 pp 228ndash235 2002

[23] J Cheng A Z Randall M J Sweredoski and P BaldildquoSCRATCH a protein structure and structural feature predic-tion serverrdquoNucleic Acids Research vol 33 no 2 ppW72ndashW762005

[24] D T Jones ldquoProtein secondary structure prediction basedon position-specific scoring matricesrdquo Journal of MolecularBiology vol 292 no 2 pp 195ndash202 1999

[25] K Bryson L J McGuffin R L Marsden J J Ward J SSodhi and D T Jones ldquoProtein structure prediction servers atUniversity College Londonrdquo Nucleic Acids Research vol 33 no2 pp W36ndashW38 2005

[26] G Cybenko ldquoApproximation by superpositions of a sigmoidalfunctionrdquo Mathematics of Control Signals and Systems vol 2no 4 pp 303ndash314 1989

[27] J Moody and C J Darken ldquoFast learning in networks of locallytuned processing unitsrdquo Neural Computation vol 1 pp 281ndash294 1989

[28] T Poggio and F Girosi ldquoNetworks for approximation andlearningrdquo Proceedings of the IEEE vol 78 no 9 pp 1481ndash14971990

[29] D F Specht ldquoProbabilistic neural networksrdquo Neural Networksvol 3 no 1 pp 109ndash118 1990

[30] P Andras ldquoOrthogonal RBF neural network approximationrdquoNeural Processing Letters vol 9 no 2 pp 141ndash151 1999

[31] P Baldi and S Brunak Bioinformatics The Machine LearningApproach MIT Press 1998

[32] E Sakk D J Schneider C R Myers and SW Cartinhour ldquoOnthe selection of target vectors for a class of supervised patternrecognizersrdquo IEEE Transactions on Neural Networks vol 20 no5 pp 745ndash757 2009

[33] G H Golub and C F Van Loan Matrix Computations JohnsHopkins University Press 1989

Submit your manuscripts athttpwwwhindawicom

Computer Games Technology

International Journal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Distributed Sensor Networks

International Journal of

Advances in

FuzzySystems

Hindawi Publishing Corporationhttpwwwhindawicom

Volume 2014

International Journal of

ReconfigurableComputing

Hindawi Publishing Corporation httpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Applied Computational Intelligence and Soft Computing

thinspAdvancesthinspinthinsp

Artificial Intelligence

HindawithinspPublishingthinspCorporationhttpwwwhindawicom Volumethinsp2014

Advances inSoftware EngineeringHindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Electrical and Computer Engineering

Journal of

Journal of

Computer Networks and Communications

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporation

httpwwwhindawicom Volume 2014

Advances in

Multimedia

International Journal of

Biomedical Imaging

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

ArtificialNeural Systems

Advances in

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

RoboticsJournal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Computational Intelligence and Neuroscience

Industrial EngineeringJournal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Modelling amp Simulation in EngineeringHindawi Publishing Corporation httpwwwhindawicom Volume 2014

The Scientific World JournalHindawi Publishing Corporation httpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Human-ComputerInteraction

Advances in

Computer EngineeringAdvances in

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Applied Computational Intelligence and Soft Computing 7

Assuming that two networks have been trained using twodifferent sets of target vectors E and E the set of weights119882 120591 120572 and 120591 determines the network output in (16)Without loss of generality we consider the case where alltarget vectors are normalized to a value of one such that1198901198942= 1 and 119890

1198942= 1 for 119894 = 1 119872 In this case the

conditions in (30) and (31) become

(119890119894minus 119890119895)119879

120572120590 (119882119909 minus 120591) = 0 (32)

(119890119894minus 119890119895)119879

120590 (119909 minus 120591) = 0 (33)

We first consider a special case where the target vectorsare related according to Γ = 119886119877Γ with 119879 = 0 in (2) Underthese circumstances if the choice

W equiv 120591 = 119882 120591 119886119877120572 (34)

is made it should be clear that since119877119879 = 119877minus1 and 119894= 119886119877119884

119894

119864119887119901=

119873

sum119894=1

(119891 (119883119894) minus 119894)119879

(119891 (119883119894) minus 119894)

= 1198862

119873

sum119894=1

(119877119891 (119883119894) minus 119877119884

119894)119879

(119877119891 (119883119894) minus 119877119884

119894)

= 1198862

119873

sum119894=1

(119891 (119883119894) minus 119884119894)119879

119877119879119877 (119891 (119883

119894) minus 119884119894)

= 1198862119864119887119901

(35)

is minimized by the condition 120597119864119887119901120597W = 0 Another way tosee this is to observe that (19) (20) remain invariant for thischoice of target vectors and network weights Hence we havethe following

Proposition 5 For a specific choice of 119886 and 119877 if 119879 = 0 in (2)and

W = 120591 = 119882 120591 119886119877120572 (36)

then the classmembership for the backpropagation networkwillremain invariant

Proof Simply consider (32) and (33) and choose any 119909satisfying

(119890119894minus 119890119895)119879

120572120590 (119882119909 minus 120591) = 0 (37)

It then immediately follows that

(119890119894minus 119890119895)119879

120590 (119909 minus 120591)

= (119890119894minus 119890119895)119879

119877119879119877120572120590 (119882119909 minus 120591)

= 1198862(119890119894minus 119890119895)119879

120572120590 (119882119909 minus 120591)

= 0

(38)

Therefore if 119909 satisfies (32) then it also satisfies (33) andhence is a point on the decision boundary for both net-works

Intuitively a scaled rigid rotation of the target vectorsshould not affect the decision boundary However when themore general transformation of (2) is applied with 119879 = 0we now demonstrate that due to the nonlinearity of 120590 in(16) no simple relationship exists such that (32) and (33)can simultaneously be satisfied by the same set of pointsWe first investigate the possibility of establishing an analyticrelationship between the set of weights 120591 and 119882 120591 120572for both networks In other words we seek ideally invertiblefunctions 119866

120572 119866120591 and 119866

119882

= 119866120572(119882 120591 120572)

120591 = 119866120591(119882 120591 120572)

= 119866119882(119882 120591 120572)

(39)

such that the setW can be transformed into W If this can bedone then an analytic procedure similar to that presented inthe proof of Proposition 5 can be established in order to relate(32) to (33) for the general case Since (19) (20) define the setW it is reasonable to rephrase these equations in terms of theobjective function 119864119887119901

120597119864119887119901

120597=

119873

sum119896=1

(119891 (119883119896) minus 119896) 119878119879

119896= 0

120597119864119887119901

120597120591= minus

119873

sum119896=1

diag (1198781015840119896) 119879(119891 (119883

119896) minus 119896)119879

= 0

120597119864119887119901

120597=

119873

sum119896=1

diag (1198781015840119896) 119879(119891 (119883

119896) minus 119896)119879

119883119879

119896= 0

(40)

where 119891(119883119896) = 120590(119883

119896minus 120591) and

119896= 119886119877(119884

119896minus 1199051015840) such that

1199051015840 is the translation vector associated with the target vectorreferred to by 119884

119896 From these equations it should be clear

that no simple analytic relationship exists that will transformW into W A numerical algorithm such as gradient descentwill assuming a localminimumactually exists arrive at somesolution for both W and W We must therefore be contentwith the assumed existence of some set of functions definedby (39) Again let us consider any point 119909 on the decisionboundary such that

(119890119894minus 119890119895)119879

120572120590 (119882119909 minus 120591) = 0 (41)

Such a point must also simultaneously satisfy

(119890119894minus 119890119895)119879

120590 (119909 minus 120591)

= ((119886119877119890119894minus 119905119894) minus (119886119877119890

119895minus 119905119895))119879

119866120572(119882 120591 120572)

8 Applied Computational Intelligence and Soft Computing

times 120590 (119866119882(119882 120591 120572) 119909 minus 119866

120591(119882 120591 120572))

= 119886((119890119894minus 119890119895) minus (119905

119894minus 119905119895))119879

119877119879119866120572(119882 120591 120572)

times 120590 (119866119882(119882 120591 120572) 119909 minus 119866

120591(119882 120591 120572))

= 0

(42)

At first glance a choice such as 119866120572(119882 120591 120572) = 119886119877120572 and

119905119894= 119905119895for 119894 119895 = 1 119872 (as in Proposition 4) appears

to bring us close to a solution However the term involving120590(119866119882(119882 120591 120572)119909 minus 119866

120591(119882 120591 120572)) is problematic Although a

choice such as 119866119882(119882 120591 120572) = 119882 and 119866

120591(119882 120591 120572) = 120591 would

yield the solution to (42) it should be clear that these valueswould not satisfy (40)

Another way to analyze this problem is to first set 119905119894= 119905119895

for 119894 119895 = 1 119872 Then for any 119909 on the decision boundaryfrom (41) and (42) equate the terms

120572120590 (119882119909 minus 120591) = 119896119877119879119866120572(119882 120591 120572)

times 120590 (119866119882(119882 120591 120572) 119909 minus 119866

120591(119882 120591 120572))

(43)

where 119896 is some constant and 119866120572 119866120591 and 119866

119882satisfy (40)

Given an arbitrary training set defined by (1) it is highlyunlikely that this constraint can be satisfied One remotescenario might be that the terms 119866

119882(119882 120591 120572)119909 minus 119866

120591(119882 120591 120572)

and119882119909 minus 120591 are always small In this case given a sigmoidalfunction 120590(119911) that is linear near 119911 = 0 a linearized versionof (43) could be solved using techniques described in [32]However this again is an unlikely set of events given anarbitrary training set Therefore given the transformation of(2) we are left to conclude that class membership invarianceand hence ranking invariance are in general not achievableusing the backpropagation neural network

52 Discussion Intuitively given a reasonable target encod-ing scheme one would desire that properties related to pro-tein secondary structure would be independent of the targetvectors chosen However we have presented numerical dataand a theoretical foundation demonstrating that secondarystructure classification and confidence measures can varydepending on the type of neural network architecture andtarget vector encoding scheme employed Specifically linearnetwork classification has been demonstrated to remaininvariant under a change in the target structure encodingscheme while the backpropagation network has not As119873 increases for the methodology applied in this workrecognition rates remain consistent with those reported inthe literature however we have observed that adding moretraining data does not improve the invariance of classificationmeasures for the backpropagation network This conclusionis corroborated by the analytic formulation presented above

6 Conclusions

As pointed out in the introduction one major purpose of theneural network is to create a stable and reliable model that

maps input training data to an output classification with thehope of extracting informative parameters When methodssimilar to those in the literature are applied [7 12ndash14] wehave demonstrated that classifier performance measures canvary considerably Under these circumstances parametersderived from a trained network for analytically describingprotein secondary structure may not comprise a reliableset for the model-based approach Furthermore classifiervariabilitywould imply that a stable parametricmodel has notbeen derived It is in some sense paradoxical that the neuralnetwork has been applied for structure classification and yetassociated parameters have not been applied for describingprotein secondary structure The neural network approachto deriving a solution to the protein secondary structureprediction problem therefore requires deeper exploration

Acknowledgments

This publication was made possible by Grant NumberG12RR017581 from the National Center for ResearchResources (NCRR) a component of the National Institutesof Health (NIH) The authors would also like to thank thereviewers for their helpful comments

References

[1] C M Bishop Pattern Recognition and Machine LearningSpringer 2007

[2] M T Hagan H B Demuth and M H Beale Neural NetworkDesign PWS Publishing 1996

[3] M N Nguyen and J C Rajapakse ldquoMulti-class support vectormachines for protein secondary structure predictionrdquo GenomeInformatics vol 14 pp 218ndash227 2003

[4] H J Hu Y Pan R Harrison and P C Tai ldquoImprovedprotein secondary structure prediction using support vectormachine with a new encoding scheme and an advanced tertiaryclassifierrdquo IEEE Transactions onNanobioscience vol 3 no 4 pp265ndash271 2004

[5] W Zhong G Altun X Tian R Harrison P C Tai and YPan ldquoParallel protein secondary structure prediction schemesusing Pthread and OpenMP over hyper-threading technologyrdquoJournal of Supercomputing vol 41 no 1 pp 1ndash16 2007

[6] J A Cuff and G J Barton ldquoApplication of enhanced multiplesequence alignment profiles to improve protein secondarystructure predictionrdquo Proteins vol 40 pp 502ndash511 2000

[7] J M Chandonia and M Karplus ldquoNeural networks for sec-ondary structure and structural class predictionsrdquo ProteinScience vol 4 no 2 pp 275ndash285 1995

[8] J A Cuff and G J Barton ldquoEvaluation and improvement ofmultiple sequence methods for protein secondary structurepredictionrdquo Proteins vol 34 pp 508ndash519 1999

[9] G E Crooks and S E Brenner ldquoProtein secondary structureentropy correlations andpredictionrdquoBioinformatics vol 20 no10 pp 1603ndash1611 2004

[10] L H Wang J Liu and H B Zhou ldquoA comparison of twomachine learning methods for protein secondary structurepredictionrdquo in Proceedings of 2004 International Conference onMachine Learning and Cybernetics pp 2730ndash2735 chn August2004

Applied Computational Intelligence and Soft Computing 9

[11] G Z Zhang D S Huang Y P Zhu and Y X Li ldquoImprovingprotein secondary structure prediction by using the residueconformational classesrdquo Pattern Recognition Letters vol 26 no15 pp 2346ndash2352 2005

[12] NQian and T J Sejnowski ldquoPredicting the secondary structureof globular proteins using neural network modelsrdquo Journal ofMolecular Biology vol 202 no 4 pp 865ndash884 1988

[13] L HowardHolley andMKarplus ldquoProtein secondary structureprediction with a neural networkrdquo Proceedings of the NationalAcademy of Sciences of the United States of America vol 86 no1 pp 152ndash156 1989

[14] J M Chandonia and M Karplus ldquoThe importance of largerdata sets for protein secondary structure prediction with neuralnetworksrdquo Protein Science vol 5 no 4 pp 768ndash774 1996

[15] B Rost and C Sander ldquoImproved prediction of proteinsecondary structure by use of sequence profiles and neuralnetworksrdquoProceedings of theNational Academy of Sciences of theUnited States of America vol 90 no 16 pp 7558ndash7562 1993

[16] B Rost and C Sander ldquoPrediction of protein secondary struc-ture at better than 70 accuracyrdquo Journal of Molecular Biologyvol 232 no 2 pp 584ndash599 1993

[17] J A Cuff M E Clamp A S Siddiqui M Finlay and GJ Barton ldquoJPred a consensus secondary structure predictionserverrdquo Bioinformatics vol 14 no 10 pp 892ndash893 1998

[18] S Hua and Z Sun ldquoA novel method of protein secondary struc-ture prediction with high segment overlap measure supportvector machine approachrdquo Journal of Molecular Biology vol308 no 2 pp 397ndash407 2001

[19] B Rost ldquoPHD predicting one-dimensional protein structureby profile-based neural networksrdquoMethods in Enzymology vol266 pp 525ndash539 1996

[20] B Rost G Yachdav and J Liu ldquoThe PredictProtein serverrdquoNucleic Acids Research vol 32 pp W321ndashW326 2004

[21] C Cole J D Barber and G J Barton ldquoThe Jpred 3 secondarystructure prediction serverrdquo Nucleic Acids Research vol 36 ppW197ndashW201 2008

[22] G Pollastri D Przybylski B Rost and P Baldi ldquoImprovingthe prediction of protein secondary structure in three and eightclasses using recurrent neural networks and profilesrdquo Proteinsvol 47 no 2 pp 228ndash235 2002

[23] J Cheng A Z Randall M J Sweredoski and P BaldildquoSCRATCH a protein structure and structural feature predic-tion serverrdquoNucleic Acids Research vol 33 no 2 ppW72ndashW762005

[24] D T Jones ldquoProtein secondary structure prediction basedon position-specific scoring matricesrdquo Journal of MolecularBiology vol 292 no 2 pp 195ndash202 1999

[25] K Bryson L J McGuffin R L Marsden J J Ward J SSodhi and D T Jones ldquoProtein structure prediction servers atUniversity College Londonrdquo Nucleic Acids Research vol 33 no2 pp W36ndashW38 2005

[26] G Cybenko ldquoApproximation by superpositions of a sigmoidalfunctionrdquo Mathematics of Control Signals and Systems vol 2no 4 pp 303ndash314 1989

[27] J Moody and C J Darken ldquoFast learning in networks of locallytuned processing unitsrdquo Neural Computation vol 1 pp 281ndash294 1989

[28] T Poggio and F Girosi ldquoNetworks for approximation andlearningrdquo Proceedings of the IEEE vol 78 no 9 pp 1481ndash14971990

[29] D F Specht ldquoProbabilistic neural networksrdquo Neural Networksvol 3 no 1 pp 109ndash118 1990

[30] P Andras ldquoOrthogonal RBF neural network approximationrdquoNeural Processing Letters vol 9 no 2 pp 141ndash151 1999

[31] P Baldi and S Brunak Bioinformatics The Machine LearningApproach MIT Press 1998

[32] E Sakk D J Schneider C R Myers and SW Cartinhour ldquoOnthe selection of target vectors for a class of supervised patternrecognizersrdquo IEEE Transactions on Neural Networks vol 20 no5 pp 745ndash757 2009

[33] G H Golub and C F Van Loan Matrix Computations JohnsHopkins University Press 1989

Submit your manuscripts athttpwwwhindawicom

Computer Games Technology

International Journal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Distributed Sensor Networks

International Journal of

Advances in

FuzzySystems

Hindawi Publishing Corporationhttpwwwhindawicom

Volume 2014

International Journal of

ReconfigurableComputing

Hindawi Publishing Corporation httpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Applied Computational Intelligence and Soft Computing

thinspAdvancesthinspinthinsp

Artificial Intelligence

HindawithinspPublishingthinspCorporationhttpwwwhindawicom Volumethinsp2014

Advances inSoftware EngineeringHindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Electrical and Computer Engineering

Journal of

Journal of

Computer Networks and Communications

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporation

httpwwwhindawicom Volume 2014

Advances in

Multimedia

International Journal of

Biomedical Imaging

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

ArtificialNeural Systems

Advances in

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

RoboticsJournal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Computational Intelligence and Neuroscience

Industrial EngineeringJournal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Modelling amp Simulation in EngineeringHindawi Publishing Corporation httpwwwhindawicom Volume 2014

The Scientific World JournalHindawi Publishing Corporation httpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Human-ComputerInteraction

Advances in

Computer EngineeringAdvances in

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

8 Applied Computational Intelligence and Soft Computing

times 120590 (119866119882(119882 120591 120572) 119909 minus 119866

120591(119882 120591 120572))

= 119886((119890119894minus 119890119895) minus (119905

119894minus 119905119895))119879

119877119879119866120572(119882 120591 120572)

times 120590 (119866119882(119882 120591 120572) 119909 minus 119866

120591(119882 120591 120572))

= 0

(42)

At first glance a choice such as 119866120572(119882 120591 120572) = 119886119877120572 and

119905119894= 119905119895for 119894 119895 = 1 119872 (as in Proposition 4) appears

to bring us close to a solution However the term involving120590(119866119882(119882 120591 120572)119909 minus 119866

120591(119882 120591 120572)) is problematic Although a

choice such as 119866119882(119882 120591 120572) = 119882 and 119866

120591(119882 120591 120572) = 120591 would

yield the solution to (42) it should be clear that these valueswould not satisfy (40)

Another way to analyze this problem is to first set 119905119894= 119905119895

for 119894 119895 = 1 119872 Then for any 119909 on the decision boundaryfrom (41) and (42) equate the terms

120572120590 (119882119909 minus 120591) = 119896119877119879119866120572(119882 120591 120572)

times 120590 (119866119882(119882 120591 120572) 119909 minus 119866

120591(119882 120591 120572))

(43)

where 119896 is some constant and 119866120572 119866120591 and 119866

119882satisfy (40)

Given an arbitrary training set defined by (1) it is highlyunlikely that this constraint can be satisfied One remotescenario might be that the terms 119866

119882(119882 120591 120572)119909 minus 119866

120591(119882 120591 120572)

and119882119909 minus 120591 are always small In this case given a sigmoidalfunction 120590(119911) that is linear near 119911 = 0 a linearized versionof (43) could be solved using techniques described in [32]However this again is an unlikely set of events given anarbitrary training set Therefore given the transformation of(2) we are left to conclude that class membership invarianceand hence ranking invariance are in general not achievableusing the backpropagation neural network

52 Discussion Intuitively given a reasonable target encod-ing scheme one would desire that properties related to pro-tein secondary structure would be independent of the targetvectors chosen However we have presented numerical dataand a theoretical foundation demonstrating that secondarystructure classification and confidence measures can varydepending on the type of neural network architecture andtarget vector encoding scheme employed Specifically linearnetwork classification has been demonstrated to remaininvariant under a change in the target structure encodingscheme while the backpropagation network has not As119873 increases for the methodology applied in this workrecognition rates remain consistent with those reported inthe literature however we have observed that adding moretraining data does not improve the invariance of classificationmeasures for the backpropagation network This conclusionis corroborated by the analytic formulation presented above

6 Conclusions

As pointed out in the introduction one major purpose of theneural network is to create a stable and reliable model that

maps input training data to an output classification with thehope of extracting informative parameters When methodssimilar to those in the literature are applied [7 12ndash14] wehave demonstrated that classifier performance measures canvary considerably Under these circumstances parametersderived from a trained network for analytically describingprotein secondary structure may not comprise a reliableset for the model-based approach Furthermore classifiervariabilitywould imply that a stable parametricmodel has notbeen derived It is in some sense paradoxical that the neuralnetwork has been applied for structure classification and yetassociated parameters have not been applied for describingprotein secondary structure The neural network approachto deriving a solution to the protein secondary structureprediction problem therefore requires deeper exploration

Acknowledgments

This publication was made possible by Grant NumberG12RR017581 from the National Center for ResearchResources (NCRR) a component of the National Institutesof Health (NIH) The authors would also like to thank thereviewers for their helpful comments

References

[1] C M Bishop Pattern Recognition and Machine LearningSpringer 2007

[2] M T Hagan H B Demuth and M H Beale Neural NetworkDesign PWS Publishing 1996

[3] M N Nguyen and J C Rajapakse ldquoMulti-class support vectormachines for protein secondary structure predictionrdquo GenomeInformatics vol 14 pp 218ndash227 2003

[4] H J Hu Y Pan R Harrison and P C Tai ldquoImprovedprotein secondary structure prediction using support vectormachine with a new encoding scheme and an advanced tertiaryclassifierrdquo IEEE Transactions onNanobioscience vol 3 no 4 pp265ndash271 2004

[5] W Zhong G Altun X Tian R Harrison P C Tai and YPan ldquoParallel protein secondary structure prediction schemesusing Pthread and OpenMP over hyper-threading technologyrdquoJournal of Supercomputing vol 41 no 1 pp 1ndash16 2007

[6] J A Cuff and G J Barton ldquoApplication of enhanced multiplesequence alignment profiles to improve protein secondarystructure predictionrdquo Proteins vol 40 pp 502ndash511 2000

[7] J M Chandonia and M Karplus ldquoNeural networks for sec-ondary structure and structural class predictionsrdquo ProteinScience vol 4 no 2 pp 275ndash285 1995

[8] J A Cuff and G J Barton ldquoEvaluation and improvement ofmultiple sequence methods for protein secondary structurepredictionrdquo Proteins vol 34 pp 508ndash519 1999

[9] G E Crooks and S E Brenner ldquoProtein secondary structureentropy correlations andpredictionrdquoBioinformatics vol 20 no10 pp 1603ndash1611 2004

[10] L H Wang J Liu and H B Zhou ldquoA comparison of twomachine learning methods for protein secondary structurepredictionrdquo in Proceedings of 2004 International Conference onMachine Learning and Cybernetics pp 2730ndash2735 chn August2004

Applied Computational Intelligence and Soft Computing 9

[11] G Z Zhang D S Huang Y P Zhu and Y X Li ldquoImprovingprotein secondary structure prediction by using the residueconformational classesrdquo Pattern Recognition Letters vol 26 no15 pp 2346ndash2352 2005

[12] NQian and T J Sejnowski ldquoPredicting the secondary structureof globular proteins using neural network modelsrdquo Journal ofMolecular Biology vol 202 no 4 pp 865ndash884 1988

[13] L HowardHolley andMKarplus ldquoProtein secondary structureprediction with a neural networkrdquo Proceedings of the NationalAcademy of Sciences of the United States of America vol 86 no1 pp 152ndash156 1989

[14] J M Chandonia and M Karplus ldquoThe importance of largerdata sets for protein secondary structure prediction with neuralnetworksrdquo Protein Science vol 5 no 4 pp 768ndash774 1996

[15] B Rost and C Sander ldquoImproved prediction of proteinsecondary structure by use of sequence profiles and neuralnetworksrdquoProceedings of theNational Academy of Sciences of theUnited States of America vol 90 no 16 pp 7558ndash7562 1993

[16] B Rost and C Sander ldquoPrediction of protein secondary struc-ture at better than 70 accuracyrdquo Journal of Molecular Biologyvol 232 no 2 pp 584ndash599 1993

[17] J A Cuff M E Clamp A S Siddiqui M Finlay and GJ Barton ldquoJPred a consensus secondary structure predictionserverrdquo Bioinformatics vol 14 no 10 pp 892ndash893 1998

[18] S Hua and Z Sun ldquoA novel method of protein secondary struc-ture prediction with high segment overlap measure supportvector machine approachrdquo Journal of Molecular Biology vol308 no 2 pp 397ndash407 2001

[19] B Rost ldquoPHD predicting one-dimensional protein structureby profile-based neural networksrdquoMethods in Enzymology vol266 pp 525ndash539 1996

[20] B Rost G Yachdav and J Liu ldquoThe PredictProtein serverrdquoNucleic Acids Research vol 32 pp W321ndashW326 2004

[21] C Cole J D Barber and G J Barton ldquoThe Jpred 3 secondarystructure prediction serverrdquo Nucleic Acids Research vol 36 ppW197ndashW201 2008

[22] G Pollastri D Przybylski B Rost and P Baldi ldquoImprovingthe prediction of protein secondary structure in three and eightclasses using recurrent neural networks and profilesrdquo Proteinsvol 47 no 2 pp 228ndash235 2002

[23] J Cheng A Z Randall M J Sweredoski and P BaldildquoSCRATCH a protein structure and structural feature predic-tion serverrdquoNucleic Acids Research vol 33 no 2 ppW72ndashW762005

[24] D T Jones ldquoProtein secondary structure prediction basedon position-specific scoring matricesrdquo Journal of MolecularBiology vol 292 no 2 pp 195ndash202 1999

[25] K Bryson L J McGuffin R L Marsden J J Ward J SSodhi and D T Jones ldquoProtein structure prediction servers atUniversity College Londonrdquo Nucleic Acids Research vol 33 no2 pp W36ndashW38 2005

[26] G Cybenko ldquoApproximation by superpositions of a sigmoidalfunctionrdquo Mathematics of Control Signals and Systems vol 2no 4 pp 303ndash314 1989

[27] J Moody and C J Darken ldquoFast learning in networks of locallytuned processing unitsrdquo Neural Computation vol 1 pp 281ndash294 1989

[28] T Poggio and F Girosi ldquoNetworks for approximation andlearningrdquo Proceedings of the IEEE vol 78 no 9 pp 1481ndash14971990

[29] D F Specht ldquoProbabilistic neural networksrdquo Neural Networksvol 3 no 1 pp 109ndash118 1990

[30] P Andras ldquoOrthogonal RBF neural network approximationrdquoNeural Processing Letters vol 9 no 2 pp 141ndash151 1999

[31] P Baldi and S Brunak Bioinformatics The Machine LearningApproach MIT Press 1998

[32] E Sakk D J Schneider C R Myers and SW Cartinhour ldquoOnthe selection of target vectors for a class of supervised patternrecognizersrdquo IEEE Transactions on Neural Networks vol 20 no5 pp 745ndash757 2009

[33] G H Golub and C F Van Loan Matrix Computations JohnsHopkins University Press 1989

Submit your manuscripts athttpwwwhindawicom

Computer Games Technology

International Journal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Distributed Sensor Networks

International Journal of

Advances in

FuzzySystems

Hindawi Publishing Corporationhttpwwwhindawicom

Volume 2014

International Journal of

ReconfigurableComputing

Hindawi Publishing Corporation httpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Applied Computational Intelligence and Soft Computing

thinspAdvancesthinspinthinsp

Artificial Intelligence

HindawithinspPublishingthinspCorporationhttpwwwhindawicom Volumethinsp2014

Advances inSoftware EngineeringHindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Electrical and Computer Engineering

Journal of

Journal of

Computer Networks and Communications

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporation

httpwwwhindawicom Volume 2014

Advances in

Multimedia

International Journal of

Biomedical Imaging

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

ArtificialNeural Systems

Advances in

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

RoboticsJournal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Computational Intelligence and Neuroscience

Industrial EngineeringJournal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Modelling amp Simulation in EngineeringHindawi Publishing Corporation httpwwwhindawicom Volume 2014

The Scientific World JournalHindawi Publishing Corporation httpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Human-ComputerInteraction

Advances in

Computer EngineeringAdvances in

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Applied Computational Intelligence and Soft Computing 9

[11] G Z Zhang D S Huang Y P Zhu and Y X Li ldquoImprovingprotein secondary structure prediction by using the residueconformational classesrdquo Pattern Recognition Letters vol 26 no15 pp 2346ndash2352 2005

[12] NQian and T J Sejnowski ldquoPredicting the secondary structureof globular proteins using neural network modelsrdquo Journal ofMolecular Biology vol 202 no 4 pp 865ndash884 1988

[13] L HowardHolley andMKarplus ldquoProtein secondary structureprediction with a neural networkrdquo Proceedings of the NationalAcademy of Sciences of the United States of America vol 86 no1 pp 152ndash156 1989

[14] J M Chandonia and M Karplus ldquoThe importance of largerdata sets for protein secondary structure prediction with neuralnetworksrdquo Protein Science vol 5 no 4 pp 768ndash774 1996

[15] B Rost and C Sander ldquoImproved prediction of proteinsecondary structure by use of sequence profiles and neuralnetworksrdquoProceedings of theNational Academy of Sciences of theUnited States of America vol 90 no 16 pp 7558ndash7562 1993

[16] B Rost and C Sander ldquoPrediction of protein secondary struc-ture at better than 70 accuracyrdquo Journal of Molecular Biologyvol 232 no 2 pp 584ndash599 1993

[17] J A Cuff M E Clamp A S Siddiqui M Finlay and GJ Barton ldquoJPred a consensus secondary structure predictionserverrdquo Bioinformatics vol 14 no 10 pp 892ndash893 1998

[18] S Hua and Z Sun ldquoA novel method of protein secondary struc-ture prediction with high segment overlap measure supportvector machine approachrdquo Journal of Molecular Biology vol308 no 2 pp 397ndash407 2001

[19] B Rost ldquoPHD predicting one-dimensional protein structureby profile-based neural networksrdquoMethods in Enzymology vol266 pp 525ndash539 1996

[20] B Rost G Yachdav and J Liu ldquoThe PredictProtein serverrdquoNucleic Acids Research vol 32 pp W321ndashW326 2004

[21] C Cole J D Barber and G J Barton ldquoThe Jpred 3 secondarystructure prediction serverrdquo Nucleic Acids Research vol 36 ppW197ndashW201 2008

[22] G Pollastri D Przybylski B Rost and P Baldi ldquoImprovingthe prediction of protein secondary structure in three and eightclasses using recurrent neural networks and profilesrdquo Proteinsvol 47 no 2 pp 228ndash235 2002

[23] J Cheng A Z Randall M J Sweredoski and P BaldildquoSCRATCH a protein structure and structural feature predic-tion serverrdquoNucleic Acids Research vol 33 no 2 ppW72ndashW762005

[24] D T Jones ldquoProtein secondary structure prediction basedon position-specific scoring matricesrdquo Journal of MolecularBiology vol 292 no 2 pp 195ndash202 1999

[25] K Bryson L J McGuffin R L Marsden J J Ward J SSodhi and D T Jones ldquoProtein structure prediction servers atUniversity College Londonrdquo Nucleic Acids Research vol 33 no2 pp W36ndashW38 2005

[26] G Cybenko ldquoApproximation by superpositions of a sigmoidalfunctionrdquo Mathematics of Control Signals and Systems vol 2no 4 pp 303ndash314 1989

[27] J Moody and C J Darken ldquoFast learning in networks of locallytuned processing unitsrdquo Neural Computation vol 1 pp 281ndash294 1989

[28] T Poggio and F Girosi ldquoNetworks for approximation andlearningrdquo Proceedings of the IEEE vol 78 no 9 pp 1481ndash14971990

[29] D F Specht ldquoProbabilistic neural networksrdquo Neural Networksvol 3 no 1 pp 109ndash118 1990

[30] P Andras ldquoOrthogonal RBF neural network approximationrdquoNeural Processing Letters vol 9 no 2 pp 141ndash151 1999

[31] P Baldi and S Brunak Bioinformatics The Machine LearningApproach MIT Press 1998

[32] E Sakk D J Schneider C R Myers and SW Cartinhour ldquoOnthe selection of target vectors for a class of supervised patternrecognizersrdquo IEEE Transactions on Neural Networks vol 20 no5 pp 745ndash757 2009

[33] G H Golub and C F Van Loan Matrix Computations JohnsHopkins University Press 1989

Submit your manuscripts athttpwwwhindawicom

Computer Games Technology

International Journal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Distributed Sensor Networks

International Journal of

Advances in

FuzzySystems

Hindawi Publishing Corporationhttpwwwhindawicom

Volume 2014

International Journal of

ReconfigurableComputing

Hindawi Publishing Corporation httpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Applied Computational Intelligence and Soft Computing

thinspAdvancesthinspinthinsp

Artificial Intelligence

HindawithinspPublishingthinspCorporationhttpwwwhindawicom Volumethinsp2014

Advances inSoftware EngineeringHindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Electrical and Computer Engineering

Journal of

Journal of

Computer Networks and Communications

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporation

httpwwwhindawicom Volume 2014

Advances in

Multimedia

International Journal of

Biomedical Imaging

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

ArtificialNeural Systems

Advances in

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

RoboticsJournal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Computational Intelligence and Neuroscience

Industrial EngineeringJournal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Modelling amp Simulation in EngineeringHindawi Publishing Corporation httpwwwhindawicom Volume 2014

The Scientific World JournalHindawi Publishing Corporation httpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Human-ComputerInteraction

Advances in

Computer EngineeringAdvances in

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Submit your manuscripts athttpwwwhindawicom

Computer Games Technology

International Journal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Distributed Sensor Networks

International Journal of

Advances in

FuzzySystems

Hindawi Publishing Corporationhttpwwwhindawicom

Volume 2014

International Journal of

ReconfigurableComputing

Hindawi Publishing Corporation httpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Applied Computational Intelligence and Soft Computing

thinspAdvancesthinspinthinsp

Artificial Intelligence

HindawithinspPublishingthinspCorporationhttpwwwhindawicom Volumethinsp2014

Advances inSoftware EngineeringHindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Electrical and Computer Engineering

Journal of

Journal of

Computer Networks and Communications

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporation

httpwwwhindawicom Volume 2014

Advances in

Multimedia

International Journal of

Biomedical Imaging

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

ArtificialNeural Systems

Advances in

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

RoboticsJournal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Computational Intelligence and Neuroscience

Industrial EngineeringJournal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Modelling amp Simulation in EngineeringHindawi Publishing Corporation httpwwwhindawicom Volume 2014

The Scientific World JournalHindawi Publishing Corporation httpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Human-ComputerInteraction

Advances in

Computer EngineeringAdvances in

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014